Hierarchical Command Learning for Formation-Based Tactical Combat

This project explores whether reinforcement learning can act as a high-level commander in formation-based tactical combat inspired by games such as Total War and Mount & Blade. Instead of learning individual movement and combat, the system delegates those behaviours to deterministic controllers and learns persistent commands for groups of Squads. A four-level structure organizes Individual Agents into Squads, Squad Groups, and a Commander that assigns tactical roles. The policy uses a relational actor-critic to reason over variable-sized sets of entities and legal commands. This allows the same model to control changing army sizes without fixed observation or action slots. Training combines behaviour cloning, PPO, curriculum learning, and a frozen opponent league. A general policy is compared with a hierarchy of policies specializing in capture, assault, flanking, reserve, and ranged support. The general policy wins 61.1% of its main evaluation battles and adapts effectively to a second simulation after fine-tuning. Individual specialists develop recognizable strengths, but the fixed hierarchy does not consistently combine them into a stronger controller. Overall, the project demonstrates the potential of reusable command learning while identifying coordination as the central challenge for hierarchical tactical AI.