Tetris-99++: MARL Evolutionary Learning via PSRO & SESIL Application

Tetris-99++ is a complex and interesting spin-off of legacy Tetris. It introduces Tetris in a multiplayer setting and a mechanic that allows interaction of two Tetris boards playing in parallel. The mechanic that makes Tetris-99++ so mind-boggling to look at is having control over another player’s board’s next object. This level of control of the game can create cooperation and competition within the same run of the environment. Understanding the be- haviour of the agents becomes so intriguing. A combination of two complex algorithms, PSRO and SESIL, is used to simulate evolution- like improvement within a population of agents similar to beings in nature. Having a rather difficult environment like Tetris-99++ challenges these complex algorithms to see if they can benefit from their mutation and offspring features to evolve through a learning curve that is less steep than using plain Independent Q-Learning models. To do so, the new customer environment is required to be standardised and packaged in a standard library like PettingZoo to allow leveraging of models and algorithms that support the Pet- tingZoo library. RLLib and EasyMarl are two libraries that provide models and algorithms like DQN, Independent Q-Learning, PSRO, and SESIL to use against our standard custom PettingZoo environ- ment, Tetris-99++. Matching the behaviour of the agents within the environment to their immediate return displays a good observation of agents’ thinking mindset within different models. There are lots of configurations and changes that can be made to this new custom environment to view the effect on learning and outcomes within different models and algorithms, such that it opens up many doors to new Multi-Agent Reinforcement Learning projects and research. GitHub Repo: https://github.com/TheSlenk/tetris-marl