OpenSanctions Pairs: Advancing Entity Matching with LLMs

The OpenSanctions Pairs dataset presents a significant advancement in entity matching by providing a comprehensive benchmark derived from real-world international sanctions data. This dataset comprises 755,540 labeled pairs collected from 293 diverse sources across 31 countries, featuring multilingual and cross-script names, as well as attributes that may be noisy or incomplete. The study evaluates a traditional rule-based matching system, specifically the nomenklatura RegressionV1 algorithm, against various large language models (LLMs) in both zero-shot and few-shot contexts. The findings reveal that LLMs significantly surpass the rule-based approach, achieving F1 scores as high as 98.95% with GPT-4o and 98.23% using an open-source model, DeepSeek-R1-Distill-Qwen-14B. While the DSPy MIPROv2 prompt optimization leads to consistent improvements, the inclusion of in-context examples does not enhance performance and may even hinder it. An analysis of errors indicates that the rule-based system tends to over-match, resulting in high false positives, while LLMs struggle primarily with cross-script transliteration and minor discrepancies in identifiers and dates. These insights suggest that the field is nearing a performance ceiling in pairwise matching, highlighting the need to focus on other components of the matching pipeline, such as blocking and clustering.