The Dangers of Synthetic Data: Catastrophic Forgetting in LLM Skill Retrieval

As LLM agents become more dependent on dynamically retrieved skills, the process of selecting appropriate skills from extensive databases has emerged as a significant hurdle. This study introduces a skill routing system that encompasses over 34,000 skills and examines the effectiveness of skill retrieval when utilizing both limited real-world supervision and synthetic data. Findings indicate that while fine-tuning with synthetic data enhances retrieval accuracy within the training distribution, it simultaneously leads to catastrophic forgetting of skills outside this distribution. To address this issue, various strategies inspired by continual learning were assessed, including embedding-anchor regularization, Learning without Forgetting (LwF), Elastic Weight Consolidation (EWC), and L2-initialization. The evaluation revealed that these methods not only preserved performance in retrieving out-of-distribution skills but also boosted retrieval accuracy for synthetic in-distribution skills by nearly 14% for a specific retriever model. This research offers valuable insights and a practical framework for effectively fine-tuning models with limited supervision.