External Benchmark of Causal Recall: Testing AI Assistant Memory on YDB from Yandex

This piece discusses the evaluation of Yandex's AI assistant, particularly its ability to retrieve past solutions when addressing bugs. The key focus is on the assistant's memory recall rather than generating new responses. The study analyzed a dataset of tickets to measure the assistant's performance in recalling previous resolutions. The results revealed that traditional semantic search, which relies on word meaning, successfully identifies relevant past fixes only 38% of the time. In contrast, leveraging explicit causal relationships between issues and their corresponding pull requests significantly improved recall rates to 87%. This indicates that while semantic similarity is limited, understanding direct connections enhances the assistant's effectiveness in problem-solving.