Latent Universal Self-Organization

Abstract

Vector-quantised (VQ) autoencoders are a useful way to convert continuous data into discrete repre- sentations, which can help in finding patterns, reducing data size, and detecting unusual behaviours. However, these models often face a key problem called codebook collapse, where only a small portion of the available discrete codes are used. This limits the models ability to fully represent the variety and structure in the data. While several methods such as commitment loss, exponential moving averages, and Gumbel-based approximations have been proposed to address this, they often depend on sensitive tuning and do not give much control over how the internal space is shaped. In this work, we take a different perspective. We treat the latent space not just as a container for features, but as a kind of virtual cosmos a structured space that can organise itself based on simple, universal principles. Specifically, we bring in three ideas: gravity, which encourages representations to gather around stable centres; diffusion, which spreads them out to avoid overcrowding; and entropy, which supports balanced use of all available codes. We build these ideas into a framework called Latent Universal Self-Organization (LUSO). In LUSO, the latent space is divided into fixed regions, each with its own centre. These regions help guide the overall layout of the space. Within each region, a learnable codebook captures the local details. To bring everything together, we introduce a training objective called fractal loss, which encourages both global balance across regions and local clarity within each region. Overall, this approach offers not just a technical solution to codebook collapse, but also a new way of thinking about latent space as something that can organise itself through simple rules, much like systems in nature do. LUSO helps build more balanced, meaningful, and reliable representations, especially in tasks that require identifying and understanding patterns in complex data.