Implementing multi-query attention resulted in a fivefold reduction in GPU cache usage, which is crucial for efficiently generating responses.

We faced unique challenges in scaling our platform, particularly with GPU local attention caching, which led to unexpected negative results during testing.

We built a tool to streamline the process of iterating on our prompting layer, allowing us to run experiments and assess the impact of even trivial changes on performance and user engagement.

I deliberately designed our product with attention to detail, ensuring that I am the primary user, which helps in creating a better experience.

I realized that many project management tools are slow, which led us to focus on building a tool that is never slow.

Improving latency can be costly, so we explore alternatives like pruning the number of parameters served at runtime.

We built Linear with a local-first architecture, allowing for a faster user experience by syncing data directly to the client, avoiding the delays of traditional web apps.

Our observation was that we had been doing a lot of abstract explorations into multi-touch technology before we could create a flat, user-friendly device. When designing products, we realized that most devices required a tradeoff between small buttons and small displays, which made them difficult to use. Our approach allowed for a larger display without compromising on button functionality, enabling contextual buttons that could be larger or smaller based on the app's needs.

The future of inference involves optimizing across three dimensions: quality, speed, and user concurrency, which relates to cost.