Writer started in 2020, before GPT-2 existed, focusing on statistical models and later transitioning to transformer models after the 2019 paper on transformers was published.

The current AI boom is largely due to the Transformer architecture, and I believe there will be future breakthroughs of similar significance, though I can't predict them.

The analogy of Transformers being the new transistors highlights how AI models can be packaged and utilized in ways that are accessible to billions, similar to the evolution of desktop and mobile interfaces.

The last two years have been dominated by large AI models, which have proven to be popular and effective, but we are now entering an era focused on complex workflows around these models.

The current focus in AI development is on scaling models effectively, as the foundational challenges of reasoning and computation have been addressed.

New methods have been developed to observe an AI model's internal thought processes and how concepts are interconnected to form logical circuits.

The most cited paper in science is a deep learning paper from 2015, highlighting that significant contributions to AI can come from anywhere, not just Silicon Valley.

Language models are currently the primary focus of AI development, which was not anticipated 15 years ago when we were exploring physical grounding in AI.

The trend in machine learning is towards convergence, with many different types of models shifting to a more unified transformer model.