SimpleWikiSearch: An Offline Wikipedia Framework for Enhanced Search

Agentic search systems based on large language models (LLMs) are often assessed without considering the full context in which they operate. The effectiveness of these systems is influenced by various factors including the Wikipedia dataset used, the preprocessing methods applied, and the retrieval mechanisms in place. Unfortunately, many of these elements are not clearly defined, complicating the comparison of results and the reproducibility of findings. To address this issue, we introduce SimpleWikiSearch, a comprehensive framework designed to clarify and standardize the search environment. This system utilizes a complete English Wikipedia dataset, processes it for clarity, and organizes it into manageable segments. It also establishes both keyword and dense retrieval indexes, along with a straightforward tool interface featuring functions like search, open_url, and submit_answer. We present baseline performance metrics across six question-answering datasets utilizing open-source LLMs, and we include a random sample of 300 for comparative analysis with proprietary models. SimpleWikiSearch serves as a specialized tool for agentic search evaluations, emphasizing the importance of a well-defined reference setup over the development of new algorithms. The associated code and data will be accessible at the provided GitHub repository.