Exploring Metacognitive Retrieval-Augmented Generation: A Reproducibility Study

The recent advancements in Retrieval Augmented Generation (RAG) have emphasized multi-retrieval methods to address intricate tasks, particularly multi-hop question answering. However, these systems often face challenges in determining when sufficient information has been collected. To tackle this issue, researchers have proposed the Metacognitive Retrieval Augmented Generation (MetaRAG) framework, which draws on metacognitive principles to enhance the reasoning capabilities of Large Language Models (LLMs). In this reproducibility study, we aim to replicate the original experimental setup of MetaRAG while also expanding our analysis in two key areas: first, we assess the impact of PointWise and ListWise rerankers; second, we conduct a comparison with SIM-RAG, which utilizes a simplified critic model to manage retrieval cessation. Our findings indicate that while MetaRAG demonstrates notable improvements over traditional RAG and reasoning-based benchmarks, the absolute performance scores are lower than previously reported. This discrepancy can be attributed to challenges related to closed-source LLM updates, incomplete implementation details, and unavailable prompts. Ultimately, our research shows that MetaRAG can be partially reproduced, benefits significantly from reranking strategies, and proves to be more resilient than SIM-RAG when additional retrieval features are incorporated.