Introducing PulseBench-Tab: A Comprehensive Multilingual Benchmark for Table Extraction

PulseBench-Tab is a newly developed benchmark designed to evaluate the extraction of tables from document images in various languages. This resource features 1,820 tables that have been meticulously annotated by humans, covering 9 different languages and 4 writing systems, including Latin, CJK, Arabic, and Cyrillic. The tables are sourced from 380 authentic documents, such as financial statements, governmental reports, and regulatory filings. They vary significantly in size, with cell counts ranging from 2 to 1,183, and nearly half of them (48.1%) include merged or spanning cells. To facilitate evaluation, we introduce T-LAG (Table Logical Adjacency Graph), an innovative metric that represents tables as directed graphs based on cell adjacencies. This metric allows for the assessment of both structural integrity and content accuracy through optimal bipartite matching, yielding a comprehensive evaluation score. We have tested 9 different table extraction systems, both commercial and open-source, using this benchmark, and provided detailed results for each language. The complete dataset, evaluation code, and outputs from all tested systems are accessible to the public.