High-quality training data is crucial for the success of Deep Research, as the quality of the input data significantly influences the model's output quality.

The diminishing returns in pre-training mean that significant increases in intelligence will require exponentially more compute resources, making it harder to achieve breakthroughs.

The last decade focused on large-scale supervised learning, where increasing data and model size led to improved performance.

Training larger models requires substantial computing power, which translates to more GPUs running for extended periods and consuming increasing amounts of energy.

The advancements in AI depend significantly on the amount of computing power we allocate and the timing of unpredictable breakthroughs.

The scaling of compute resources will vary significantly depending on the problem, with some requiring up to 10,000 times more compute than others, which allows for more capable models.

The long tail of relevant concepts in data means that simply gathering data passively requires significantly more compute to achieve incremental improvements in model performance.

The era of supervised learning relied heavily on human-labeled data, but recent advancements allow us to train models without requiring explicit human input, unlocking new possibilities.

Training AI models can be likened to a recipe with three main ingredients: the model itself, the data it is trained on, and the compute power used for training.