The way Clio works is by processing real-world conversations with a language model to extract high-level summaries and group related answers, allowing us to understand user intent without reading raw conversations.

We validate Clio's effectiveness by generating synthetic conversation data and checking if it can reconstruct known distributions of conversation topics without prior guidance, and Clio accurately reconstructs the ground truth distribution, which is one of the many ways we know it performs well.

Clio enables us to evaluate the relevance of our models in real-world scenarios, ensuring they provide accurate and unbiased information.

Clio allows us to analyze real-world usage of our models, helping us design more thoughtful evaluations that match real-world use cases.

Clio has already provided insights into user interactions with Claude, informing our evaluations and product safety considerations.

Clio allows us to analyze refusal rates for specific queries, helping us identify over-refusals and under-refusals in AI responses.

Clio stands for Claude insights and observations, and it's a tool that helps us understand the different use cases for which people are using Claude, showing high-level aggregate clusters of usage.

Clio has shown that people use Claude in subjective settings, such as seeking relationship or health advice, which raises important questions about the values AI should represent in these contexts.

During the development of Clio, we recognized the importance of understanding how users interact with AI during elections, including both factual and subjective inquiries.