Clio allows us to analyze real-world usage of our models, helping us design more thoughtful evaluations that match real-world use cases.

We validate Clio's effectiveness by generating synthetic conversation data and checking if it can reconstruct known distributions of conversation topics without prior guidance, and Clio accurately reconstructs the ground truth distribution, which is one of the many ways we know it performs well.

Clio stands for Claude insights and observations, and it's a tool that helps us understand the different use cases for which people are using Claude, showing high-level aggregate clusters of usage.

Clio enables us to evaluate the relevance of our models in real-world scenarios, ensuring they provide accurate and unbiased information.

Clio has already provided insights into user interactions with Claude, informing our evaluations and product safety considerations.

Clio helps us understand the risks associated with AI models by revealing how they are being used in various applications, including scientific research and cybersecurity.

Clio provides insights into its multilingual performance, showing it works effectively for both English and Georgian conversations.

Clio has shown that people use Claude in subjective settings, such as seeking relationship or health advice, which raises important questions about the values AI should represent in these contexts.

Clio allows us to analyze refusal rates for specific queries, helping us identify over-refusals and under-refusals in AI responses.