The Reformer - Pushing the limits of language modeling

Google's Reformer replaced quadratic self-attention with LSH attention, cutting complexity from O(n²) to O(n log n). Reversible residual layers eliminated per-layer activation caching, making 1M-token sequences trainable on a single GPU.

Most RAG pipelines still chunk at 512 tokens because teams default to standard transformers — not because short chunks improve retrieval quality. Longformer and BigBird productionized these ideas. Running GPT-4 Turbo at 128K context for full-document retrieval costs 40x more per doc than using an efficient-attention model built for it. Chunking is a workaround, not an architecture decision.