MECA: Modular editing via customized expert networks and adaptors in large language models

Abstract

Updating language models with new information through targeted edits without resorting to expensive full model retraining remains a critical challenge, particularly when aiming to preserve pre-existing capabilities. In this work, we introduce Modular Editing via Customized expert networks and Adaptors) (MECA), a unified framework that selectively integrates new knowledge into language models. MECA employs a module-level deferral router to evaluate whether incoming queries fall within the scope of existing edit requests. Queries are then dynamically routed to either customized editing experts or key-value adaptors. This modular strategy ensures that updates are localized, thereby mitigating risks of unintended alterations on unrelated outputs. We validate our approach on sequential editing tasks using Llama2-7B, Llama2-13B and Falcon 11B, benchmarked across two diverse datasets ZsRE and Hallucination. Experimental results show that MECA consistently outperforms several stateof-the-art knowledge editing techniques, achieving improved integration of new information while preserving the model's original performance. Our analysis further demonstrates that the deferral routing mechanism for selecting modules effectively balances editing precision with overall model stability.