Innovative Late Materialization for Enhanced Sequence Training in Scalable Recommendation Systems

In the realm of deep learning recommendation systems, the demand for processing ultra-long user interaction histories is on the rise. Traditional methods, particularly the 'Fat Row' approach, face significant challenges due to excessive data redundancy. This redundancy becomes even more pronounced in multi-tenant environments where models with varying sequence length requirements operate on a shared dataset. To address these issues, we introduce a versioned late materialization strategy. This method optimizes storage by keeping user interaction histories in a single, normalized, and immutable format, allowing for on-the-fly reconstruction of sequences during training through lightweight versioned pointers. Our approach maintains consistency between online and offline data through a dual-protocol system, effectively preventing data leakage during both streaming and batch training. Additionally, we implement a read-optimized storage layer that enhances data retrieval for diverse model requirements. By employing disaggregated data preprocessing and optimizing input/output operations, we minimize the latency associated with sequence reconstruction, ensuring that training remains GPU-bound. This innovative system not only reduces resource consumption in training data infrastructure but also facilitates significant improvements in model performance, laying the groundwork for advanced architectures like HSTU and ULTRA-HSTU.