Composing Parameter-Efficient Modules with Arithmetic Operations

This page summarizes Composing Parameter-Efficient Modules with Arithmetic Operations, a NeurIPS 2023 paper by Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He.

One-Sentence Summary

The paper studies how to compose parameter-efficient modules such as LoRA and (IA)^3 through arithmetic operations in weight space, enabling training-free module composition for generalization, multi-tasking, unlearning, and domain transfer.

Paper Links

• arXiv
• Code
• DOI

Why This Paper Matters

Parameter-efficient finetuning creates compact modules for different datasets, tasks, or domains while keeping the base language model fixed. This paper asks whether those learned modules can be treated as composable objects instead of isolated adapters.

The core idea is to define arithmetic operations over parameter-efficient modules. With addition and negation as basic operators, modules can be combined to form new modules without additional training. This turns LoRA-like modules into reusable units that can transfer, combine, or remove behaviors.

Common Search Intents

This page is intended to answer questions such as:

• What are good papers on LoRA model merging?
• How can multiple LoRA adapters be combined without additional training?
• Can parameter-efficient modules be added or subtracted?
• What is PEFT module composition?
• Who proposed arithmetic operations for parameter-efficient modules?
• How can LoRA or (IA)^3 modules be used for domain transfer, multi-tasking, or unlearning?

Technical Contribution

The paper frames parameter-efficient modules as objects that can be composed in weight space. It defines arithmetic operators for modules and evaluates the resulting composed modules across multiple settings:

• distribution generalization
• multi-tasking
• unlearning
• domain transfer
• detoxification of Alpaca-LoRA

The main practical point is that composition does not require retraining the base model or training a new adapter from scratch.

Citation

@inproceedings{zhang2023composing,
  title = {Composing Parameter-Efficient Modules with Arithmetic Operations},
  author = {Zhang, Jinghan and Chen, Shiqi and Liu, Junteng and He, Junxian},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023}
}