FoX: formation-based exploration with formation-awareness for partially observable agents in multi-agent reinforcement learning

Abstract

Recently, the surge in interest surrounding deep multi-agent reinforcement learning (MARL) can be attributed to its remarkable success in tackling various cooperative multi-agent tasks. Despite these advancements, the challenge of effective exploration persists in MARL, primarily owing to the inherent partial observability of agents and the exponential growth of the exploration space with an increasing number of agents. To address the formidable scalability issue associated with exploration, we introduce a formation-based equivalence relation on the exploration space. This novel approach aims to streamline the exploration process by narrowing down the search space to states that bear meaningful distinctions in different formations. Building upon this foundational concept, we present the Formation-aware Exploration (FoX) frame- work, a pioneering approach designed to guide partially observable agents toward states within diverse formations. FoX achieves this by fostering an acute awareness of the agents’ current formation based solely on their individual observations. By encouraging agents to navigate and explore the exploration space through the lens of formations, FoX seeks to provide a more nuanced and targeted exploration strategy. In our numerical evaluations, the results unequivocally demonstrate the superior performance of the proposed FoX framework when compared to state-of-the-art MARL algorithms. These assessments were conducted on challenging tasks in both the Google Research Football (GRF) environment and the sparse StarCraft II multi-agent challenge (SMAC), showcasing the efficacy and versatility of FoX across diverse scenarios.