Meta AI Introduces NeuralBench: A Comprehensive Open-Source Benchmarking Tool for NeuroAI Models

The assessment of AI models utilizing brain signals has historically been inconsistent, with various research teams employing different preprocessing methods and datasets. This inconsistency complicates the determination of which models perform best across tasks. To address this issue, Meta AI has unveiled NeuralBench, an open-source framework designed for benchmarking NeuroAI models. The inaugural version, NeuralBench-EEG v1.0, stands out as the most extensive open benchmark available, encompassing 36 tasks, 94 datasets, and nearly 9,500 subjects, along with over 13,600 hours of EEG data. NeuralBench is structured around three main Python packages: NeuralFetch for data acquisition, NeuralSet for data preparation, and NeuralTrain for training models. This modular design allows users to easily execute tasks through a command-line interface. The framework's first release focuses on EEG and includes various task categories such as cognitive decoding and clinical applications. Notably, the results indicate that while foundation models show promise, task-specific models often perform comparably well, highlighting the need for further exploration in challenging areas like cognitive decoding.