CONTROLLABLE REPRESENTATION: TOWARDS FINE-GRAINED FOCUSING IMAGE FROM INSTRUCTION

Abstract

Contrastive Language-Image Pre-training (CLIP) has proven pivotal in extracting significant content information from images across a variety of tasks. By aligning textual and visual modalities, it facilitates comprehensive image understanding, capturing even minute details, including those extraneous to specific tasks. However, CLIP lacks reasoning capabilities and cannot be directly guided through text instructions. In contrast, large-scale pre-training and instruction tuning have demonstrated success in developing general-purpose language models with extensive competencies. Yet, the integration of these capabilities into image representation tasks remains underexplored. In this paper, we propose an efficient and innovative method for representing images that align with open-ended instructions by utilizing a pre-trained image encoder and a frozen text-only large language model (LLM). Our approach maps between the embedding spaces of these components, enabling the seamless integration of textual and visual information. The method processes images and corresponding open text instructions as input, producing image representations consistent with these instructions. By keeping both the LLM and vision encoder frozen and training only three lightweight linear layers, InstructionClip achieves remarkable efficiency, leveraging just 10,000 generated data points containing three types of semantic relational information: action, color, and background. Unlike traditional models, InstructionClip introduces a mechanism that enables the model to adapt to user-provided instructions, even those unseen during training. By focusing on relationships such as object actions, image backgrounds, and object colors, the model successfully demonstrates its ability to interpret complex and specific search intents. We construct a controlled evaluation dataset tailored to these relationships, allowing us to rigorously assess the model's performance under instruction-driven scenarios. Through systematic experiments on our test dataset, InstructionClip shows a clear advantage over baseline models like CLIP, which primarily rely on visual similarity. The results highlight InstructionClip’s ability to retrieve images that adhere to semantic relations explicitly described in the instructions, showcasing its flexibility and adaptability. This work underscores the potential for instruction-driven multimodal systems to be applied in real-world scenarios where user-defined control over image representations is critical. By addressing the limitations of existing approaches, InstructionClip opens up new possibilities for rich, multi-faceted interactions with image data guided by user-defined instructions.