The team at MoonMath AI has introduced an open-source bf16 forward attention kernel specifically designed for AMD's MI300X GPU, utilizing HIP rather than assembly language. This kernel, licensed under MIT, has demonstrated superior performance compared to AMD’s AITER v3 across all evaluated shapes and rounding modes. The kernel operates exclusively on the MI300X, which is part of AMD's CDNA3 architecture.
The attention mechanism, crucial to transformer models, is executed efficiently through this kernel, which processes inputs in BSHD or BHSD layouts without requiring transposition. While it supports various sequence lengths, it does not accommodate causal masking or variable-length batching.
A notable feature of this kernel is its use of one-instruction assembly wrappers, allowing for opcode selection while the compiler manages register allocation. The kernel's architecture employs eight waves per block and optimizes memory placement, significantly enhancing performance. Benchmarking results indicate that the MoonMath kernel achieves a geometric mean speedup of 1.18× over AITER v3, demonstrating its effectiveness in real-world applications such as video diffusion.