Anthropic recently released its AI Fluency Index, analyzing 9,830 multi-turn Claude conversations to understand how people actually collaborate with AI. The findings are worth a close read for anyone leading a product team.
The encouraging signal: iteration correlates strongly with better behavior. Conversations that included iteration and refinement were 5.6x more likely to question the model’s reasoning, 4x more likely to identify missing context, and showed roughly double the number of observable fluency behaviors overall. People who stay in the conversation with AI, pushing, refining, clarifying, demonstrate stronger evaluative instincts than those who accept the first answer and move on.
That tracks with what we see working with product teams. The PMs who treat AI as a thinking partner rather than an answer machine consistently produce better work.
But there’s a second pattern in Anthropic’s data that should give product leaders pause.
When AI produces polished artifacts (code, documents, apps, interactive tools) users become more directive at the outset but less evaluative afterward. Specifically: a 5.2 percentage point drop in identifying missing context, a 3.7 point drop in checking facts, and a 3.1 point drop in questioning reasoning.
The more complete the output appears, the less likely people are to interrogate it.
Where this gets risky inside product organizations
We’ve all experienced this. A PM shares an AI-generated persona that reads coherently. A PRFAQ sounds strategic. A roadmap reads with confidence. A spec appears ready for execution. And because the artifact looks polished, it quietly gains authority it hasn’t earned.
The risk here is not speed. It’s unexamined confidence.
AI can compress hours of thinking into minutes. It cannot compress the responsibility of deciding well. And Anthropic’s data suggests that the better AI gets at producing finished-looking work, the more we need to actively protect our ability to question it.
This is a leadership problem, not an individual contributor problem. Asking a PM to “be more critical of AI output” without building that expectation into how the team works is like asking someone to slow down on a highway with no speed limit. The environment has to reinforce the behavior.
The question for product leaders is not “Are my PMs iterating with AI?” It’s “Have we built safeguards that preserve rigor as AI becomes embedded across discovery, prioritization, and delivery?”
Three practical ways to institutionalize AI fluency
- Make context a first-class input
AI output quality rises or falls based on the clarity of the inputs it receives. If context is fragmented, incomplete, or inconsistent, the output will be too. We talk about this constantly with the teams we work with, and it remains one of the highest-leverage things a leader can do.
There are three lightweight ways to institutionalize this.
- First, standardize AI-ready foundations. Every AI-driven artifact should trace back to a clear problem, persona, prioritized needs, and defined outcome. These become reusable inputs that travel across prompts and tools.
- Second, create a context-conducive environment. Centralize research artifacts, strategy documents, usage data, and past experiment results so PMs can easily incorporate them into their AI workflows.
- Third, use prompt libraries that force connection. Prompts should require explicit reference to customer needs, constraints, and intended outcomes, not simply “draft X.”
When context is easy to access and reuse, disciplined AI usage becomes the default rather than the exception.
- Use a shared skeptic prompt for polished output
Anthropic’s data shows that evaluation drops when artifacts are created. One way to counteract that is to establish a standard “polish trigger” prompt that any PM can run against AI output before it influences a decision:
- What assumptions is this making that were not explicitly provided?
- Where might this conflict with our stated customer needs or strategy?
- If this were wrong, what would be the most likely reason?
These three questions apply across most product management AI use cases, from personas to roadmaps to specs. This is not about slowing teams down. It is about ensuring that fluency does not quietly replace discernment.
- Publicly showcase one speed-plus-rigor example per quarter
Culture follows what gets amplified. If the only AI stories that get celebrated internally are about speed (“AI helped us ship faster”), teams will optimize for speed. If we start highlighting stories about speed and judgment together (“AI helped us move faster and avoid a flawed decision”), we reinforce that both matter.
In internal forums, teams share what AI accelerated, what context they provided, where they challenged the output, what they validated externally, and what decision changed as a result.
For example: “We used AI to cluster 28 interviews in 30 minutes. It surfaced five needs. Two were driven primarily by power users. We ran six additional interviews. One need was proven to be invalid. We avoided building a feature that would have impacted less than 8% of our base.”
That narrative reinforces that speed and rigor are strongest when combined intentionally.
The bigger implication
AI fluency is becoming a leadership responsibility.
Anthropic’s data points to a tension every product leader should care about: iteration improves fluency, but polished artifacts can reduce scrutiny. The more complete AI output looks, the more deliberate teams need to be about questioning what sits underneath it.
We have seen a version of this before with Agile. The teams that benefited most were not the ones that simply adopted ceremonies. They were the ones whose leaders changed how decisions were made, how learning surfaced, and how teams were expected to operate.
AI is creating a similar moment. Leaders can’t just encourage AI adoption. They have to create the practices, tools, and shared expectations that make rigorous AI use repeatable: reusable context libraries, skeptic prompts, validation habits, artifact review norms, and clear standards for when AI output is ready to influence a decision.
The goal is not to slow teams down. It is to make sure AI-driven speed compounds learning instead of scaling shallow confidence.
Product teams do not need less leadership in an AI-enabled world. They need leaders who build the infrastructure for better AI-assisted judgment, so fluency becomes a durable operating advantage instead of another round of productivity theater.
Want help putting this into practice?
At Product Rebels, we work with product leaders and teams navigating exactly this shift: using AI to accelerate delivery without losing product discipline or customer trust.
Is your team experiencing challenges in implementing AI into your product operating model or struggling in establishing the practices that enable the best outcomes from AI product building?
Schedule 30 minutes with us so we can learn a little bit about you and explore how we can help.
About Product Rebels
Product Rebels helps product leaders bring their teams from good to great. We work with established product organizations that already know the basics of product management but want to operate at a higher level. Our focus is not Product Management 101. It’s helping teams build strong customer foundations and outcome-oriented ways of working that consistently translate into better results; for customers and for the business.
We partner with leaders and teams to change how product work actually happens day to day: how customer insight is gathered and shared, how problems are framed, how tradeoffs are made, and how learning compounds over time. AI is infused throughout these practices as an accelerator, helping teams synthesize learning faster, explore more options, and move with greater confidence without sacrificing judgment or customer connection.
The result is product teams that don’t just ship more, they build the right things, make better decisions under pressure, and deliver meaningful, sustained impact.