While scaling pre-training may have plateaued, training test time compute could open up a new paradigm for scaling laws, unlocking previously unimaginable capabilities.

Instead of just scaling up model size during training, researchers are likely to focus on scaling the compute available for the model's Chain of Thought.

When I was in college, the idea that scaling compute in transformers could lead to artificial general intelligence was considered crazy, but I eventually recognized that scaling does improve performance.

The diminishing returns in pre-training mean that significant increases in intelligence will require exponentially more compute resources, making it harder to achieve breakthroughs.

In AI, every approach has performed better when more scale is applied, especially as foundational problems are addressed over time.

The current focus in AI development is on scaling models effectively, as the foundational challenges of reasoning and computation have been addressed.

The scaling laws became a foundational principle for AI development after being popularized by researchers, including those at OpenAI and Google DeepMind, who emphasized that it's not just about making models larger, but also about training them on sufficient data.

In 2025, reasoning will become the defining focus in AI development, as pre-training reaches diminishing returns.

OpenAI's new class of reasoning models hints at a potential new direction for scaling AI models.