Large Language Models: A New Moore’s Law?

the idea that LLMs represent a new phase of scaling, a new 'Moore's Law,' is pure hype wrapped in a mathematical observation. it’s a correlation observed across scaling experiments, not a physical law. it just means we can scale parameter counts faster than we can scale compute, which is what the industry is doing.

what it really means is that scaling compute and data is the path forward, not some magic architectural leap. the actual bottleneck remains the quality of the training data and the efficiency of the inference pipeline.

don't get excited about the narrative; focus on optimizing the efficiency of the next trillion parameters we train.