Skip to content

Model Merging

Model merging refers to the problem of combining multiple neural network models trained on different tasks, datasets, or objectives into a single unified model. This is useful for knowledge consolidation, task adaptation, and improving generalization across domains without retraining from scratch.

Key approaches

Parameter Averaging: Simple weighted averaging of model parameters, typically effective when models are trained on related tasks. Arithmetic mean and Fisher-weighted averaging are common variants.

Task Arithmetic: Merges task-specific model changes relative to a shared base model (e.g., a pretrained language model). Subtracts the base model parameters, averages the task-specific differences, then adds back to base model.

Gradient Matching: Recent approaches reduce the inaccuracy of parameter averaging by accounting for gradient mismatch between models trained on different objectives, using uncertainty-based preconditioning.

Applications

  • Continual learning: adding new tasks without catastrophic forgetting of prior ones
  • Domain adaptation: adapting models across different domains while retaining previous knowledge
  • Model editing: removing or modifying undesirable behaviors (toxicity, hallucinations) from language models
  • Ensemble learning: combining diverse models into a unified predictor

Key papers

Related to ensemble and multi-task learning literature