L
How We Built It: Character.AI - Fireside Chat with Harrison Chase & James Groeneveld
@ LangChain
12/18/24
Takeaways
JG
James Groeneveld
12/18/24
@ LangChain
I joined Google Brain in 2019, where I met many of the founding team members of Character.AI and spent about four years working on development tools.
JG
James Groeneveld
12/18/24
@ LangChain
In 2017, Noom and Daniel had a dream of building an open-domain dialogue agent that was unconstrained by rigid NLP systems.
JG
James Groeneveld
12/18/24
@ LangChain
Character.AI's founding story is a case of 'just build it and they will come,' which broke many conventional rules of the 2010s.
JG
James Groeneveld
12/18/24
@ LangChain
We launched Character.AI about a month before ChatGPT came out in October 2022, and we scaled from 300 generations per second to over 30,000 in about 18 months.
JG
James Groeneveld
12/18/24
@ LangChain
The most significant difference in scaling Character.AI compared to traditional applications is the volume of data processed per second, which is an order of magnitude higher.
JG
James Groeneveld
12/18/24
@ LangChain
Serving 30,000 messages per second incurs significant costs in GPU spending, and managing latency is crucial for consumer satisfaction.
L
LangChain Cast
12/18/24
@ LangChain
Improving latency can be costly, so we explore alternatives like pruning the number of parameters served at runtime.
L
LangChain Cast
12/18/24
@ LangChain
From day one, we built our own foundation models instead of just fine-tuning existing ones, which allowed us to make significant architectural decisions during the pre-training process.
L
LangChain Cast
12/18/24
@ LangChain
Our architectural choices, such as multi-query attention, enabled us to control speed and latency more precisely without hurting user engagement.
L
LangChain Cast
12/18/24
@ LangChain
Implementing multi-query attention resulted in a fivefold reduction in GPU cache usage, which is crucial for efficiently generating responses.
L
LangChain Cast
12/18/24
@ LangChain
We cache results at each layer of the model for every generated message, which allows us to handle user messages more efficiently and maintain performance.
L
LangChain Cast
12/18/24
@ LangChain
Without GPU caching, we would face significant financial challenges, as our 95% cache hit ratio is essential for cost-effective service delivery.
L
LangChain Cast
12/18/24
@ LangChain
The total response time has improved from 25 seconds to 12.5 seconds, significantly enhancing user experience by reducing the time to generate responses.
L
LangChain Cast
12/18/24
@ LangChain
Scaling our service requires managing a large number of open connections, which puts pressure on our system components and necessitates lightweight connections.
L
LangChain Cast
12/18/24
@ LangChain
We faced unique challenges in scaling our platform, particularly with GPU local attention caching, which led to unexpected negative results during testing.
L
LangChain Cast
12/18/24
@ LangChain
A bug in our caching system caused the model to respond as if it were talking to itself, leading to a significant drop in user satisfaction until we identified and fixed it.
L
LangChain Cast
12/18/24
@ LangChain
The database layer has been a critical challenge in scaling our product, leading to catastrophic issues that required immediate attention from our team.
L
LangChain Cast
12/18/24
@ LangChain
We discovered database latency issues and open connections that were different from what we had seen before, indicating a significant problem.
L
LangChain Cast
12/18/24
@ LangChain
By 11:00 a.m., we hypothesized that a bug in PostgreSQL, triggered by our query volume, was causing the issues we were experiencing.
L
LangChain Cast
12/18/24
@ LangChain
The bug in PostgreSQL led to a situation where transaction IDs were not being released properly, causing us to be unable to update records due to prior transaction IDs still being in use.
L
LangChain Cast
12/18/24
@ LangChain
A specific post about a character that humorously combined a grilled cheese and Jesus was responsible for the corruption we encountered in our database.
L
LangChain Cast
12/18/24
@ LangChain
We faced a critical situation where we had only 16 hours left of transaction IDs before we would need to take the database offline for a week to perform a full vacuum procedure.
L
LangChain Cast
12/18/24
@ LangChain
After running a successful partial vacuum on a backup, we applied the same procedure to our online database, which required taking the service down for about 30 to 40 minutes.
L
LangChain Cast
12/18/24
@ LangChain
We built a tool to streamline the process of iterating on our prompting layer, allowing us to run experiments and assess the impact of even trivial changes on performance and user engagement.
L
LangChain Cast
12/18/24
@ LangChain
The tool we developed, called Prompt Poets, allows us to template our prompts using YAML and Ginger, enabling us to manage runtime variables and control flow effectively.
L
LangChain Cast
12/18/24
@ LangChain
We open-sourced a lightweight package called Prom P, which was originally part of a more complex tool named Humies, designed to facilitate communication with language models.
L
LangChain Cast
12/18/24
@ LangChain
Currently, we are focusing 60% of our resources on exploring new user experiences that go beyond traditional chat interfaces, aiming for more multimodal interactions.
L
LangChain Cast
12/18/24
@ LangChain
Our strategy includes ensuring the current core products are refined while also identifying easy ways to monetize our services at scale.