How Clio Reveals Real World Claude Use Safely

TL;DR

Clio analyzes real world Claude usage without exposing private data and shows how users interact with the system. It uses a hierarchy of use cases derived from conversations, while applying privacy checks to prevent identifying information from leaking. The result is a privacy preserving, bottom up view of risks, benefits, and future directions.

Transcript

• All right, let's start off with a round of quick introduction. So, I'll start. I am Deep Ganguli, I'm a research scientist on the Societal Impacts team. I'm really driven by fundamental questions like how are people using and affected by the systems that we are building at Anthropic, and how do we use that understanding to make our systems safer.... Read More

Key Insights

• Clio is designed to understand real world use cases of Claude while preserving user privacy.
• The system starts from many real conversations and summaries them to identify user intent clusters.
• A second model explains what is happening in each cluster to provide high level interpretations.
• A privacy filter removes anything that could identify individuals before results are exposed.
• Clio aggregates results only after ensuring minimum numbers of distinct sources, protecting privacy.
• The approach bridges laboratory evaluations and real world use by grounding analysis in actual interactions.
• Clio helps tailor evaluations to specific real world use cases rather than hypothetical scenarios.
• Trust and safety considerations drive the design of Clio from the outset.
• More videos from this channel:
  • How to apply the 4 Ds of AI fluency in collaboration
  • How to scale enterprise AI end to end effectively
  • How Does Claude Use an Internal Mental Workspace?
• More from this channel: Anthropic

Questions & Answers

Q: What is Clio and what problem does it solve?

Clio stands for Claude insights and observations and is a tool that lets researchers understand the different use cases people have for Claude at a high level. It solves the problem of measuring real world usage without compromising user privacy by automating analysis of conversations and grouping them into meaningful clusters that reflect user intent and potential risks and benefits.

Q: How does Clio process real world conversations?

Clio starts with a large set of real world conversations and uses a language model to extract a private high level summary of each conversation. It then groups related answers into clusters according to user intent and uses another model to explain what is happening in each cluster, creating a hierarchy of use cases.

Q: How does Clio ensure privacy during analysis?

Before exposing results, Clio applies a privacy check to remove anything that could identify individuals. It uses a defense in depth strategy and requires a minimum number of unique organizations and conversations to form a cluster, ensuring that private data cannot be traced back to people while still providing useful insights.

Q: What did the team learn about evaluating AI harms with Clio?

The team learned that bottom up, real world usage data helps tailor evaluations to actual use cases, rather than relying solely on top down hypotheses about harms. This leads to more thoughtful evaluations that reflect how models are used in practice, such as discrimination, persuasion, and misinformation in real contexts.

Q: What is the mindset behind building Clio according to Alex?

Alex emphasizes designing safeguards that would make him comfortable as a Claude user. The team sought to avoid creating a tool that feels invasive or like spying, so they built privacy protections and ethical considerations into the architecture from the start, aligning with user expectations and responsible data use.

Q: What does Clio reveal about the kinds of conversations Claude is used for?

Clio reveals high level use cases across a range of domains, from understanding historical topics to assisting with experiments design. By clustering conversations, it shows the diverse ways Claude is used, highlighting risks and benefits and pointing toward future directions without exposing individual user content.

Q: How does Clio contribute to safety and trust in AI systems?

By exposing usage patterns and potential risks in a privacy-preserving way, Clio informs safety evaluations, helps identify where misuses or harmful outcomes could arise, and supports better risk mitigation strategies. This builds trust by showing that the system can study real world usage responsibly.

Q: What are the practical applications of Clio findings?

Findings from Clio can guide internal evaluations of model behavior, informing governance and safety practices. They can help prioritize areas for improvement, tailor monitoring to real world use cases, and support conversations with stakeholders about how Claude is being used and how to keep users safe in practical deployments.

Summary & Key Takeaways

• Clio collects real world Claude conversations and processes them to build a hierarchical map of user intents, revealing how the model is used in practice.

• Privacy and ethics are central, with multiple safeguards ensuring that no identifying information is exposed during analysis.

• The findings help guide evaluations, risk assessment, and improvements to safety and usefulness of Claude in real world settings.