A
What do people use AI models for?
@ Anthropic
12/13/24
Takeaways
DG
Deep Ganguli
12/13/24
@ Anthropic
I'm driven by fundamental questions about how people use and are affected by the systems we build at Anthropic, and how we can use that understanding to make our systems safer.
ED
Esin Durmus
12/13/24
@ Anthropic
I'm interested in understanding how AI systems will impact society at large, including what values they should have and how we can evaluate those values in practice.
MM
Miles McCain
12/13/24
@ Anthropic
I care a lot about understanding how our systems are used in the wild and the impact that has on real people around the world.
AT
Alex Tamkin
12/13/24
@ Anthropic
Clio stands for Claude insights and observations, and it's a tool that helps us understand the different use cases for which people are using Claude, showing high-level aggregate clusters of usage.
ED
Esin Durmus
12/13/24
@ Anthropic
Before Clio, we used top-down approaches to assert types of harm we wanted to measure, such as discrimination in high-stakes decision-making scenarios.
ED
Esin Durmus
12/13/24
@ Anthropic
Clio allows us to analyze real-world usage of our models, helping us design more thoughtful evaluations that match real-world use cases.
MM
Miles McCain
12/13/24
@ Anthropic
The way Clio works is by processing real-world conversations with a language model to extract high-level summaries and group related answers, allowing us to understand user intent without reading raw conversations.
MM
Miles McCain
12/13/24
@ Anthropic
We ensure that our analysis maintains user privacy by avoiding any identifying information and applying quantitative aggregation minimums to our clusters.
ED
Esin Durmus
12/13/24
@ Anthropic
Clio has already provided insights into user interactions with Claude, informing our evaluations and product safety considerations.
AT
Alex Tamkin
12/13/24
@ Anthropic
I was surprised to find that many users utilize Claude for research and brainstorming in fields like quantum mechanics and biology, rather than just writing tasks.
MM
Miles McCain
12/13/24
@ Anthropic
I found a significant number of users asking for parenting advice, which I had never considered before, and it led me to create educational games with my kids using Claude.
ED
Esin Durmus
12/13/24
@ Anthropic
Clio has shown that people use Claude in subjective settings, such as seeking relationship or health advice, which raises important questions about the values AI should represent in these contexts.
AT
Alex Tamkin
12/13/24
@ Anthropic
We validate Clio's effectiveness by generating synthetic conversation data and checking if it can reconstruct known distributions of conversation topics without prior guidance, and Clio accurately reconstructs the ground truth distribution, which is one of the many ways we know it performs well.
A
Anthropic Cast
12/13/24
@ Anthropic
Clio provides insights into its multilingual performance, showing it works effectively for both English and Georgian conversations.
A
Anthropic Cast
12/13/24
@ Anthropic
Anthropic has a strong safety mission, which includes responsible scaling policies to identify and mitigate catastrophic risks associated with AI models.
A
Anthropic Cast
12/13/24
@ Anthropic
Clio enhances our top-down safety approach by allowing us to identify blind spots in user traffic that may not have been anticipated when establishing policies.
A
Anthropic Cast
12/13/24
@ Anthropic
We observed suspicious activities, such as attempts to write spam emails and articles, which we flagged to our trust and safety team for further action.
A
Anthropic Cast
12/13/24
@ Anthropic
Clio helps us understand the risks associated with AI models by revealing how they are being used in various applications, including scientific research and cybersecurity.
A
Anthropic Cast
12/13/24
@ Anthropic
People often engage with Claude during moments of extreme personal crisis, highlighting the need for responsible AI interactions in sensitive contexts.
A
Anthropic Cast
12/13/24
@ Anthropic
Clio allows us to analyze refusal rates for specific queries, helping us identify over-refusals and under-refusals in AI responses.
A
Anthropic Cast
12/13/24
@ Anthropic
During the development of Clio, we recognized the importance of understanding how users interact with AI during elections, including both factual and subjective inquiries.
A
Anthropic Cast
12/13/24
@ Anthropic
Clio enables us to evaluate the relevance of our models in real-world scenarios, ensuring they provide accurate and unbiased information.
A
Anthropic Cast
12/13/24
@ Anthropic
We aim to share our findings about AI usage to foster societal resilience and prepare for the transformative potential of this technology.
A
Anthropic Cast
12/13/24
@ Anthropic
Clio's detailed documentation allows other organizations to replicate our methods and contribute to a broader understanding of AI technologies.
A
Anthropic Cast
12/13/24
@ Anthropic
We are excited to explore how Clio can help us understand the subjective use cases of AI and ensure models represent diverse viewpoints.
A
Anthropic Cast
12/13/24
@ Anthropic
Clio will help us assess the economic impacts of AI technology and its role in augmenting or replacing tasks in various fields.