Lua

Lua for Game Development — Chapter 14: Combat AI, Behavior Trees, Utility AI & Enemy Design

A complete and production-ready system for combat AI, behavior trees, enemy archetypes, utility scores, aggro management, squad tactics and boss scripting using Lua.

Leeting Yan 5 min read 929 words

Combat AI determines the challenge, pacing, and tension of any action game.

Modern games use:

  • State Machines (simple AI)
  • Behavior Trees (modular, readable)
  • Utility AI (dynamic decisions)
  • Blackboards (shared memory)
  • Squad AI (formations, flanking, roles)
  • Boss AI (scripted phases + BT nodes)

Lua’s flexibility makes it ideal for building AI layers.

This chapter builds:

  1. Enemy archetypes
  2. Finite State Machine (FSM) combat AI
  3. Behavior Tree engine
  4. Utility-based decision system
  5. Aggro & threat systems
  6. Group tactics (flank, assist, swarm)
  7. Boss scripting (phases, triggers, timelines)

1. Enemy Archetypes

Define enemies in data:

enemies/
goblin.lua
skeleton.lua
archer.lua
boss_dragon.lua

1.1 Example Enemy Definition

return {
  name = "Goblin",
  hp = 40,
  speed = 60,
  behaviors = {"bt_goblin"},
  abilities = {"slash"},
  aggro_range = 120,
  attack_range = 25,
}

2. Finite State Machine (FSM) for Simple AI

FSM is lightweight and good for:

  • weak melee enemies
  • simple ranged foes
  • environmental hazards

2.1 FSM Template

local FSM = {}
FSM.__index = FSM

function FSM.new(initial)
  return setmetatable({state=initial, states={}}, FSM)
end

function FSM:add(name, obj)
  self.states[name] = obj
end

function FSM:update(entity, dt)
  local s = self.states[self.state]
  if s.update then
    local new = s.update(entity, dt)
    if new then self.state = new end
  end
end

return FSM

2.2 Example Goblin FSM

fsm:add("idle", {
  update = function(e, dt)
    if distance(e, player) < e.data.aggro_range then
      return "chase"
    end
  end
})

fsm:add("chase", {
  update = function(e, dt)
    e:move_toward(player.x, player.y, e.data.speed * dt)
    if distance(e, player) < e.data.attack_range then
      return "attack"
    end
  end
})

fsm:add("attack", {
  update = function(e, dt)
    e:attack(player)
    return "chase"
  end
})

3. Behavior Trees (BT)

Behavior Trees scale better than FSMs.

BTs are used by:

  • Unreal Engine
  • Unity
  • Godot
  • Hades
  • Dead Cells
  • Halo / Gears / Doom AI

BTs are modular and descriptive.

3.1 Behavior Tree Node Structure

Nodes have:

  • status: success, failure, running
  • tick: main logic

3.2 BT Base Node

local Node = {}
Node.__index = Node

function Node.new()
  return setmetatable({}, Node)
end

function Node:tick(entity, dt)
  error("tick() not implemented")
end

return Node

4. Composite Nodes (Selectors, Sequences)

4.1 Selector (OR)

Try children until one succeeds.

local Selector = {}
Selector.__index = Selector

function Selector.new(children)
  return setmetatable({children=children}, Selector)
end

function Selector:tick(entity, dt)
  for _, c in ipairs(self.children) do
    if c:tick(entity, dt) then return true end
  end
  return false
end

return Selector

4.2 Sequence (AND)

All children must succeed.

local Sequence = {}
Sequence.__index = Sequence

function Sequence.new(children)
  return setmetatable({children=children}, Sequence)
end

function Sequence:tick(entity, dt)
  for _, c in ipairs(self.children) do
    if not c:tick(entity, dt) then return false end
  end
  return true
end

return Sequence

5. Leaf Nodes (Actions & Conditions)

Condition Example

local IsPlayerNear = {}
function IsPlayerNear:tick(e, dt)
  return distance(e, player) < e.data.aggro_range
end

Action Example

local MoveToPlayer = {}
function MoveToPlayer:tick(e, dt)
  e:move_toward(player.x, player.y, e.data.speed * dt)
  return true
end

6. Goblin BT Example

local goblin_bt = Selector.new({
  Sequence.new({
    IsPlayerNear,
    MoveToPlayer,
  }),
  IdleNode,
})

Readable and modular.

7. Utility AI (Scoring-Based Decisions)

Utility AI enables:

  • dynamic decisions
  • reacting to player behavior
  • selecting best actions based on score

Examples:

  • “Should enemy dodge?”
  • “Should heal or run?”
  • “Should call allies?”

7.1 Utility Score Table

function utility_attack(e)
  return (e.hp > 20) and 0.8 or 0.4
end

function utility_run(e)
  return (e.hp < 20) and 0.9 or 0.1
end

function utility_call_allies(e)
  return (#nearby_allies < 2) and 0.7 or 0.2
end

7.2 Decision Function

function choose_action(e)
  local scores = {
    attack = utility_attack(e),
    run = utility_run(e),
    call = utility_call_allies(e)
  }

  local best, best_v = nil, -1
  for k,v in pairs(scores) do
    if v > best_v then
      best, best_v = k, v
    end
  end
  return best
end

8. Aggro & Threat System

Enemies focus on targets based on:

  • damage dealt
  • proximity
  • taunts
  • healing support

Used in MMORPGs, ARPGs, and group combat.

8.1 Threat Table

enemy.threat = {}  -- {player_id = threat_value}

8.2 Adding Threat

function enemy:add_threat(source, amount)
  self.threat[source] = (self.threat[source] or 0) + amount
end

Damage and healing add threat:

event:on("damage_done", function(src, tgt, dmg)
  if tgt.is_enemy then
    tgt:add_threat(src, dmg)
  end
end)

8.3 Choosing Target

function enemy:target()
  local best, v = nil, -1
  for p, t in pairs(self.threat) do
    if t > v then best, v = p, t end
  end
  return best
end

9. Squad & Group Tactics

Enemies coordinate:

  • flanking
  • rushing
  • retreating and regrouping
  • surrounding player
  • focusing fire

9.1 Shared Blackboard

SquadBB = {
  target_pos = nil,
  formation = "line",
  rally_point = {x=300, y=200},
}

9.2 Flanking Behavior

function flank(entity)
  local lp = SquadBB.target_pos
  entity:move_toward(lp.x+50, lp.y-20, entity.data.speed)
end

9.3 Squad Leader Logic

function squad_leader:update(dt)
  -- update common goals
  SquadBB.target_pos = {x=player.x, y=player.y}

  if player.hp < player.max_hp * 0.5 then
    SquadBB.formation = "aggressive"
  else
    SquadBB.formation = "spread"
  end
end

10. Boss AI Scripting (Phases + BT + Timeline)

Bosses combine:

  • behavior tree core
  • scripted phases (HP thresholds)
  • timeline for attacks
  • summons & environment changes
  • enraged states

10.1 Boss Phases

boss.phases = {
  {hp=0.7, script="phase_2"},
  {hp=0.4, script="phase_3"},
  {hp=0.1, script="enrage"},
}

Triggered automatically:

function boss:update(dt)
  local ratio = self.hp / self.max_hp
  for _, p in ipairs(self.phases) do
    if not p.triggered and ratio < p.hp then
      p.triggered = true
      require("boss_scripts."..p.script).run(self)
    end
  end

  self.behavior:tick(self, dt)
end

10.2 Example Boss Script (Timeline)

phase_2.lua:

return {
  run = function(boss)
    timeline:add(function()
      ui:flash_red()
      camera:shake(10, 1)
      wait(0.3)

      boss:roar()
      spawn_enemies("minions", 3)

      boss:use_ability("fire_breath")
      wait(2)

      boss:use_ability("slam")
    end)
  end
}

This is AAA-level boss scripting with simple Lua.

11. Putting It All Together — Full Enemy AI Pipeline

function enemy:update(dt)
  self:check_aggro()
  self:choose_target()
  self.behavior:tick(self, dt)
  self:move_and_attack(dt)
end

12. Summary of Chapter 14

You now understand how to build:

  • Enemy archetypes
  • FSM AI for simple enemies
  • Behavior Tree engine for complex AI
  • Utility AI scoring decisions
  • Aggro & threat systems
  • Squad/group tactics
  • Boss AI with phases + timeline
  • Modular, data-driven combat scripting

Lua is an exceptional scripting language for AI behavior orchestration.

Keep Reading

Follow the engineering thread

Get the next practical Birdor note, or browse the archive for related systems, tooling, and architecture work.

Join newsletter Browse articles