Lua for Game Development — Chapter 6: AI Systems — BT, FSM, Utility AI, GOAP, Navigation, Hybrid AI

Leeting Yan

Game AI is one of Lua’s strongest domains.
Lua scripts drive NPCs in:

  • World of Warcraft
  • Don’t Starve
  • CryEngine titles
  • Roblox (Luau)
  • Cocos2d-x Lua games
  • Thousands of mobile/indie games

Game studios rely heavily on Lua due to:

  • Ease of scripting
  • Data-driven behavior
  • Fast iteration
  • Easy debugging and hot reload
  • Coroutine-based timelines

This chapter covers all major AI paradigms used in modern games:

  1. FSM (Finite State Machines)
  2. Behavior Trees (BT)
  3. Utility AI (Score-based decision AI)
  4. GOAP (Goal-Oriented Action Planning)
  5. Navigation patterns
  6. Blackboard memory systems
  7. Hybrid AI architecture for bosses
  8. Performance tuning for large AI crowds

1. Finite State Machines (FSM)

FSMs are simple, predictable, and great for:

  • Basic enemies
  • Simple bosses
  • UI states
  • Simple scripted sequences

1.1 FSM Implementation (Complete)

local FSM = {}
FSM.__index = FSM

function FSM.new(initial)
  return setmetatable({state = initial, states = {}}, FSM)
end

function FSM:add(name, fn)
  self.states[name] = fn
end

function FSM:update(dt, entity)
  local fn = self.states[self.state]
  if fn then
    self.state = fn(self, dt, entity) or self.state
  end
end

return FSM

1.2 Example: Simple Enemy

local fsm = FSM.new("idle")

fsm:add("idle", function(self, dt, e)
  if distance_to_player(e) < 50 then
    return "chase"
  end
end)

fsm:add("chase", function(self, dt, e)
  move_towards_player(e, dt)
  if distance_to_player(e) < 10 then
    return "attack"
  end
end)

fsm:add("attack", function(self, dt, e)
  do_attack(e)
  return "idle"
end)

FSMs are simple but scale poorly for complex behavior.

2. Behavior Trees (BT)

BTs are the AAA industry’s most common AI approach.

Used in:

  • Halo
  • God of War
  • Overwatch
  • Assassin’s Creed
  • WoW mobs
  • Roblox NPCs

BTs are modular, debug-friendly, and scale better than FSMs.

2.1 Node Types

local BT = {}

function BT.action(fn)
  return { type="action", run=fn }
end

function BT.sequence(children)
  return { type="sequence", children=children }
end

function BT.selector(children)
  return { type="selector", children=children }
end

return BT

2.2 BT Runner

local function run(node, entity)
  if node.type == "action" then
    return node.run(entity)
  end

  if node.type == "sequence" then
    for _, child in ipairs(node.children) do
      if run(child, entity) ~= "success" then
        return "fail"
      end
    end
    return "success"
  end

  if node.type == "selector" then
    for _, child in ipairs(node.children) do
      if run(child, entity) == "success" then
        return "success"
      end
    end
    return "fail"
  end
end

2.3 Example: Enemy Behavior

local tree = BT.selector({
  BT.action(function(e)
    if e.hp < 20 then flee(e); return "success" end
    return "fail"
  end),

  BT.sequence({
    BT.action(function(e)
      if see_player(e) then return "success" end
      return "fail"
    end),
    BT.action(function(e)
      chase(e)
      return "success"
    end)
  }),

  BT.action(function(e)
    patrol(e)
    return "success"
  end)
})

run(tree, enemy)

BTs provide clean, readable logic.

3. Utility AI (Score-Based Decision Making)

Utility AI chooses the “best” action using real-time scores.

Used in:

  • The Sims
  • Total War series
  • Shadow of Mordor
  • Many RTS/strategy games

3.1 Action Score Definition

local util_ai = {}

function util_ai.choose(entity, actions)
  local best_score, best_action = -1, nil
  for _, a in ipairs(actions) do
    local score = a.score(entity)
    if score > best_score then
      best_score = score
      best_action = a
    end
  end
  return best_action
end

return util_ai

3.2 Example: Actions

local actions = {
  {
    name = "heal",
    score = function(e) return 1 - (e.hp / e.max_hp) end,
    run = function(e) heal(e) end
  },
  {
    name = "attack",
    score = function(e) return distance_to_player(e) < 10 and 0.9 or 0.1 end,
    run = function(e) attack(e) end
  },
  {
    name = "patrol",
    score = function(e) return 0.2 end,
    run = function(e) patrol(e) end
  },
}

local chosen = util_ai.choose(enemy, actions)
chosen.run(enemy)

Utility AI = flexible, scalable, easy to tune.

4. GOAP (Goal-Oriented Action Planning)

Used in:

  • FEAR
  • The Sims
  • RimWorld
  • Some Ubisoft titles

GOAP is like a mini-AI planner inside the game.

4.1 Actions with Preconditions & Effects

local Action = {}
Action.__index = Action

function Action.new(cfg)
  return setmetatable({
    name = cfg.name,
    cost = cfg.cost,
    pre = cfg.pre,
    effect = cfg.effect,
  }, Action)
end

return Action

4.2 Planner (Simplified)

local function satisfies(state, goal)
  for k, v in pairs(goal) do
    if state[k] ~= v then return false end
  end
  return true
end

local function apply(state, effect)
  for k, v in pairs(effect) do
    state[k] = v
  end
end

local function plan(state, actions, goal, depth)
  if satisfies(state, goal) then return {} end
  if depth <= 0 then return nil end

  for _, action in ipairs(actions) do
    local ok = true
    for k, v in pairs(action.pre) do
      if state[k] ~= v then ok = false break end
    end

    if ok then
      local new_state = {}
      for k, v in pairs(state) do new_state[k] = v end
      apply(new_state, action.effect)

      local sub = plan(new_state, actions, goal, depth - 1)
      if sub then
        table.insert(sub, 1, action)
        return sub
      end
    end
  end
end

4.3 Example GOAP Setup

local state = {has_weapon=false, enemy_visible=true}
local goal = {enemy_dead=true}

local actions = {
  Action.new({
    name="pickup",
    pre={has_weapon=false},
    effect={has_weapon=true},
  }),
  Action.new({
    name="shoot",
    pre={has_weapon=true, enemy_visible=true},
    effect={enemy_dead=true},
  })
}

local plan_seq = plan(state, actions, goal, 4)

for _, act in ipairs(plan_seq) do
  print("Action:", act.name)
end

GOAP produces dynamic plans like a small RPG brain.

5. Navigation Patterns

Lua typically handles navigation via:

  • Grid-based A*
  • Navigation meshes (navmesh)
  • Steering behaviors
  • Formation movement
  • Simple chaser/pursuer logic

Below is a basic example.

5.1 A* Pathfinding (Simplified Grid)

This is a compact version suitable for Lua-based engines.

local function astar(start, goal, grid)
  local open = {[start]=true}
  local came = {}
  local cost = {[start]=0}

  while next(open) do
    local current = nil
    for k in pairs(open) do current = k break end
    open[current] = nil

    if current == goal then break end

    for _, n in ipairs(grid.neighbors(current)) do
      local new_cost = cost[current] + 1
      if not cost[n] or new_cost < cost[n] then
        cost[n] = new_cost
        came[n] = current
        open[n] = true
      end
    end
  end

  local path = {}
  local cur = goal
  while cur do
    table.insert(path, 1, cur)
    cur = came[cur]
  end

  return path
end

Works well for grid-based games.

6. Blackboard Architecture (Shared AI Memory)

Used in:

  • Behavior Trees
  • Utility AI
  • GOAP planners
  • Squad/group AI
  • Boss multi-phase AI

6.1 Blackboard Module

local Blackboard = {}
Blackboard.__index = Blackboard

function Blackboard.new()
  return setmetatable({data = {}}, Blackboard)
end

function Blackboard:set(key, value)
  self.data[key] = value
end

function Blackboard:get(key)
  return self.data[key]
end

return Blackboard

6.2 Example

bb = Blackboard.new()
bb:set("player_pos", {x=10,y=20})
bb:set("alert", false)

AI systems read/write to the shared blackboard.

7. Hybrid AI: Combining FSM + BT + Utility + Timers

Real games usually combine systems:

  • FSM controls high-level phases
  • BT controls behavior flow
  • Utility AI chooses tactical actions
  • Coroutines run timed actions
  • Blackboard stores shared memory

7.1 Example Hybrid Boss AI

boss.fsm = FSM.new("phase1")

boss.fsm:add("phase1", function(self, dt, e)
  if e.hp < 50 then return "phase2" end
  run(tree_phase1, e)
end)

boss.fsm:add("phase2", function(self, dt, e)
  run(tree_phase2, e)

  local action = util_ai.choose(e, boss_actions)
  action.run(e)
end)

This scales to multi-phase bosses with complex patterns.

8. Performance Tuning for 100+ AI Agents

Lua can handle large crowds if optimized:

8.1 Always prefer local variables

Local lookup is 10–30% faster.

8.2 Avoid creating tables in loops

Reuse tables in AI calculations.

8.3 Keep BT nodes static

Don’t recreate nodes every frame.

8.4 For large crowds, update in slices

Update 1/4 of NPCs per frame.

8.5 Avoid heavy math inside Lua

Delegate vector math/physics to C-level engine code.

9. Complete AI Example (BT + Utility + Blackboard + Timers)

local function enemy_ai(e, dt)
  if not e.blackboard:get("initialized") then
    e.blackboard:set("initialized", true)
  end

  -- update timers
  if e.cooldown > 0 then e.cooldown = e.cooldown - dt end

  -- high-level decisions
  local action = util_ai.choose(e, actions)
  action.run(e)

  -- BT logic for moving/attacking
  run(tree, e)
end

This structure is used in professional action RPGs.

10. Summary of Chapter 6

You now understand:

  • FSM systems for baseline AI
  • Behavior Trees for scalable modular logic
  • Utility AI for tactical scoring
  • GOAP for high-level planning
  • Navigation (A*, steering)
  • Blackboard memory
  • Hybrid AI models (industry standard)
  • Performance patterns for large AI groups

Lua is exceptionally strong for building flexible, scalable, game-ready AI systems.