The core idea behind GPT-1 was that using a transformer to predict the next word would provide enough signal to generate coherent text, a concept that seemed obvious in hindsight but was initially doubted.

The foundational paper on attention mechanisms in 2017 led to the development of transformer models, which are now the basis for many AI applications.

OpenAI's release of GPT-2 in November 2019 marked a significant milestone with 1.5 billion parameters, followed by GPT-3, which was over 100 times larger and more capable than its predecessor.

In early 2023, I realized AI development was outpacing our ability to control it, influenced by the impressive capabilities of ChatGPT and my work on analog computation at Google.

The term "GPT wrapper" emerged to describe generative AI applications built on top of ChatGPT, which raised concerns about the sustainability of such models as OpenAI continues to innovate and dominate the market.

The evolution of language models, particularly with the introduction of GPT-3, has demonstrated a significant leap in their ability to perform language tasks, even though they do not possess real-world knowledge.

We started working with OpenAI on language models in 2019 and with the Department of Defense on AI applications in 2020, well before the recent AI craze.

With the advent of 10 trillion parameter models, we could see a leap in innovation similar to the transition from GPT-2 to GPT-3, which sparked a new era of AI companies.

Despite criticisms, OpenAI and DeepMind have been transparent about their methodologies, making it easier for others to replicate models like GPT-3 if they can secure enough compute and talent.