Accelerating Convolutions: My Journey

After completing my article on NormIs-1, I decided to delve deeper into optimizing the model. While the architecture performed reasonably well in terms of intelligence metrics, its speed was a significant concern. The main culprit was the Depthwise Convolution, which involved numerous memory accesses. A small convolution block was positioned between the attention block and the Feed Forward Network (FFN), causing a bottleneck in computations. My goal was to enhance this specific area. Developing a full language model with comprehensive attention mechanisms was not feasible due to issues like unstable training, exploding loss, or a malfunctioning DataLoader. Instead, I opted for a simpler approach: a lightweight CNN combined with a custom MLX kernel for inference, alongside speed benchmarks to evaluate performance.