Scaling Law: At the Crossroads of Scaling Laws

Overview

Pretraining Scaling Law Bottleneck and the Future of RL

Scaling Law tells us that more compute boosts model performance, but pre-training Scaling Law is fading due to exhausted high-quality data. RL Scaling Law has some initial validation but is limited by compute for future validation. Future breakthroughs in self-play RL could significantly increase compute demand. However, without sufficient compute, we can't even verify whether self-play reinforcement learning can take us to the next stage of the scaling law. In essence, we're pouring hundreds of billions in capital expenditure into what is essentially a “gamble“.

OpenAI's Next-Gen Models

GPT-5's future is unclear; Orion isn't GPT-5. OpenAI may combine GPT-4o's multimodal abilities with O3's reasoning for a GPT-4.5-level model. With the limitation of high-quality language data, OpenAI will focus less on pre-training at this stage and more on improving reasoning and applications.

O-Series Model Limitations