Self-Indexing Attention for Efficient Sparse Long-Context LLM Inference

The challenge of sparse long-context inference lies in the need for effective token retrieval during both the prefill and decoding phases. Traditional approaches often utilize distinct retrieval methods for each phase, which limits the potential for reusing retrieval representations. To address this, we present Self-Indexing Attention, a framework that operates without the need for training and is based on a unified transform-domain sign-magnitude representation. This innovative approach allows for the creation of a token-level index that can be reused for both grouped prefill selection and decoding retrieval. Importantly, this representation is also compatible with external KV-cache compression, eliminating the need for separate indexing metadata. The 1-bit indexing system facilitates efficient retrieval through bitwise operations, which are well-supported by contemporary accelerators. Our evaluations show that at a 5% attention density, Self-Indexing Attention performs comparably to dense attention on benchmarks such as LongBench and RULER, achieving impressive speedups of up to 6.1 times for prefill and 10.3 times for decode operations. Further testing with TurboQuant and DeepSeekV4-Flash confirms its effectiveness in low-bit KV-cache compression and with pretrained sparse-attention indexers.