Lua for Game Development — Chapter 14: Combat AI, Behavior Trees, Utility AI & Enemy Design

Leeting Yan

November 26, 2025

Combat AI determines the challenge, pacing, and tension of any action game.

Modern games use:

State Machines (simple AI)
Behavior Trees (modular, readable)
Utility AI (dynamic decisions)
Blackboards (shared memory)
Squad AI (formations, flanking, roles)
Boss AI (scripted phases + BT nodes)

Lua’s flexibility makes it ideal for building AI layers.

This chapter builds:

Enemy archetypes
Finite State Machine (FSM) combat AI
Behavior Tree engine
Utility-based decision system
Aggro & threat systems
Group tactics (flank, assist, swarm)
Boss scripting (phases, triggers, timelines)

1. Enemy Archetypes

Define enemies in data:

enemies/
goblin.lua
skeleton.lua
archer.lua
boss_dragon.lua

1.1 Example Enemy Definition

return {
  name = "Goblin",
  hp = 40,
  speed = 60,
  behaviors = {"bt_goblin"},
  abilities = {"slash"},
  aggro_range = 120,
  attack_range = 25,
}

2. Finite State Machine (FSM) for Simple AI

FSM is lightweight and good for:

weak melee enemies
simple ranged foes
environmental hazards

2.1 FSM Template

local FSM = {}
FSM.__index = FSM

function FSM.new(initial)
  return setmetatable({state=initial, states={}}, FSM)
end

function FSM:add(name, obj)
  self.states[name] = obj
end

function FSM:update(entity, dt)
  local s = self.states[self.state]
  if s.update then
    local new = s.update(entity, dt)
    if new then self.state = new end
  end
end

return FSM

2.2 Example Goblin FSM

fsm:add("idle", {
  update = function(e, dt)
    if distance(e, player) < e.data.aggro_range then
      return "chase"
    end
  end
})

fsm:add("chase", {
  update = function(e, dt)
    e:move_toward(player.x, player.y, e.data.speed * dt)
    if distance(e, player) < e.data.attack_range then
      return "attack"
    end
  end
})

fsm:add("attack", {
  update = function(e, dt)
    e:attack(player)
    return "chase"
  end
})

3. Behavior Trees (BT)

Behavior Trees scale better than FSMs.

BTs are used by:

Unreal Engine
Unity
Godot
Hades
Dead Cells
Halo / Gears / Doom AI

BTs are modular and descriptive.

3.1 Behavior Tree Node Structure

Nodes have:

status: success, failure, running
tick: main logic

3.2 BT Base Node

local Node = {}
Node.__index = Node

function Node.new()
  return setmetatable({}, Node)
end

function Node:tick(entity, dt)
  error("tick() not implemented")
end

return Node

4. Composite Nodes (Selectors, Sequences)

4.1 Selector (OR)

Try children until one succeeds.

local Selector = {}
Selector.__index = Selector

function Selector.new(children)
  return setmetatable({children=children}, Selector)
end

function Selector:tick(entity, dt)
  for _, c in ipairs(self.children) do
    if c:tick(entity, dt) then return true end
  end
  return false
end

return Selector

4.2 Sequence (AND)

All children must succeed.

local Sequence = {}
Sequence.__index = Sequence

function Sequence.new(children)
  return setmetatable({children=children}, Sequence)
end

function Sequence:tick(entity, dt)
  for _, c in ipairs(self.children) do
    if not c:tick(entity, dt) then return false end
  end
  return true
end

return Sequence

5. Leaf Nodes (Actions & Conditions)

Condition Example

local IsPlayerNear = {}
function IsPlayerNear:tick(e, dt)
  return distance(e, player) < e.data.aggro_range
end

Action Example

local MoveToPlayer = {}
function MoveToPlayer:tick(e, dt)
  e:move_toward(player.x, player.y, e.data.speed * dt)
  return true
end

6. Goblin BT Example

local goblin_bt = Selector.new({
  Sequence.new({
    IsPlayerNear,
    MoveToPlayer,
  }),
  IdleNode,
})

Readable and modular.

7. Utility AI (Scoring-Based Decisions)

Utility AI enables:

dynamic decisions
reacting to player behavior
selecting best actions based on score

Examples:

“Should enemy dodge?”
“Should heal or run?”
“Should call allies?”

7.1 Utility Score Table

function utility_attack(e)
  return (e.hp > 20) and 0.8 or 0.4
end

function utility_run(e)
  return (e.hp < 20) and 0.9 or 0.1
end

function utility_call_allies(e)
  return (#nearby_allies < 2) and 0.7 or 0.2
end

7.2 Decision Function

function choose_action(e)
  local scores = {
    attack = utility_attack(e),
    run = utility_run(e),
    call = utility_call_allies(e)
  }

  local best, best_v = nil, -1
  for k,v in pairs(scores) do
    if v > best_v then
      best, best_v = k, v
    end
  end
  return best
end

8. Aggro & Threat System

Enemies focus on targets based on:

damage dealt
proximity
taunts
healing support

Used in MMORPGs, ARPGs, and group combat.

8.1 Threat Table

enemy.threat = {}  -- {player_id = threat_value}

8.2 Adding Threat

function enemy:add_threat(source, amount)
  self.threat[source] = (self.threat[source] or 0) + amount
end

Damage and healing add threat:

event:on("damage_done", function(src, tgt, dmg)
  if tgt.is_enemy then
    tgt:add_threat(src, dmg)
  end
end)

8.3 Choosing Target

function enemy:target()
  local best, v = nil, -1
  for p, t in pairs(self.threat) do
    if t > v then best, v = p, t end
  end
  return best
end

9. Squad & Group Tactics

Enemies coordinate:

flanking
rushing
retreating and regrouping
surrounding player
focusing fire

9.1 Shared Blackboard

SquadBB = {
  target_pos = nil,
  formation = "line",
  rally_point = {x=300, y=200},
}

9.2 Flanking Behavior

function flank(entity)
  local lp = SquadBB.target_pos
  entity:move_toward(lp.x+50, lp.y-20, entity.data.speed)
end

9.3 Squad Leader Logic

function squad_leader:update(dt)
  -- update common goals
  SquadBB.target_pos = {x=player.x, y=player.y}

  if player.hp < player.max_hp * 0.5 then
    SquadBB.formation = "aggressive"
  else
    SquadBB.formation = "spread"
  end
end

10. Boss AI Scripting (Phases + BT + Timeline)

Bosses combine:

behavior tree core
scripted phases (HP thresholds)
timeline for attacks
summons & environment changes
enraged states

10.1 Boss Phases

boss.phases = {
  {hp=0.7, script="phase_2"},
  {hp=0.4, script="phase_3"},
  {hp=0.1, script="enrage"},
}

Triggered automatically:

function boss:update(dt)
  local ratio = self.hp / self.max_hp
  for _, p in ipairs(self.phases) do
    if not p.triggered and ratio < p.hp then
      p.triggered = true
      require("boss_scripts."..p.script).run(self)
    end
  end

  self.behavior:tick(self, dt)
end

10.2 Example Boss Script (Timeline)

phase_2.lua:

return {
  run = function(boss)
    timeline:add(function()
      ui:flash_red()
      camera:shake(10, 1)
      wait(0.3)

      boss:roar()
      spawn_enemies("minions", 3)

      boss:use_ability("fire_breath")
      wait(2)

      boss:use_ability("slam")
    end)
  end
}

This is AAA-level boss scripting with simple Lua.

11. Putting It All Together — Full Enemy AI Pipeline

function enemy:update(dt)
  self:check_aggro()
  self:choose_target()
  self.behavior:tick(self, dt)
  self:move_and_attack(dt)
end

12. Summary of Chapter 14

You now understand how to build:

Enemy archetypes
FSM AI for simple enemies
Behavior Tree engine for complex AI
Utility AI scoring decisions
Aggro & threat systems
Squad/group tactics
Boss AI with phases + timeline
Modular, data-driven combat scripting

Lua is an exceptional scripting language for AI behavior orchestration.