SC
How to Scale AI Application Inference 100x ft. Fireworks’ Lin Qiao
@ Sequoia Capital
05/19/25
Takeaways
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
The alignment process in product design is critical and involves multiple phases, from ideation to scaling after achieving product-market fit.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
Achieving sustainable scale in AI applications requires aligning models with real-world use cases and optimizing user behavior throughout the product design process.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
Many app developers currently rely on off-the-shelf models, but infusing product knowledge into these models through prompt engineering is a new and necessary skill.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
Successful applications often incorporate a data flying wheel, which is essential for standing out in the market.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
Aligning the data distribution in your application workload with the training data distribution of the model is crucial, but it is challenging due to assumptions made by researchers about user needs.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
The inference system design is not one-size-fits-all; it must utilize production data for maximum optimization, and combining post-training and inference can drive significant innovation and acceleration in AI applications, ultimately reducing inference costs by 10 to 100 times.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
The future of inference involves optimizing across three dimensions: quality, speed, and user concurrency, which relates to cost.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
The future of scaling inference will focus on heavy customization tailored to specific applications, transforming it into a multi-dimensional optimization problem.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
The current high inference costs resemble an iceberg, where reducing these costs can enable applications to scale into sustainable businesses.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
Achieving this reduction in inference costs requires solving a complex three-dimensional optimization problem involving quality, speed, and cost.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
To optimize AI applications, we need to predict multiple tokens at once, align hardware selection with application needs, and implement various tuning mechanisms for quality.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
At Fireworks, we manage the complexity of AI application scaling by utilizing a virtual cloud infrastructure that ensures high quality and reliability in GPU selection.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
We have developed a self-serve platform that allows developers to easily tune AI applications for speed and quality, facilitating rapid scaling and experimentation.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
A food chain company scaled an AI feature from one shop to a thousand shops in three months using our service, demonstrating the potential for fast scaling in AI applications.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
A software development company expanded its AI feature rollout from 100,000 developers to 25 million in three months, highlighting the fascinating speed of scaling in AI.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
As companies scale their AI applications, it's crucial to find the optimal balance across quality, cost, and speed to achieve sustainable growth.
LQ
Lin Qiao
05/19/25
@ Sequoia Capital
I encourage everyone to share the next generation of inference challenges they are facing so we can discuss solutions together.