There’s a piece of data buried in the public disclosures around AI-driven insurance customer service that deserves more attention than it usually gets. It doesn’t come from a vendor pitch deck. It comes from an independent review of customer complaints, and it points to something insurers rolling out conversational AI need to take seriously: the automation itself usually isn’t what makes customers angry. Not being able to reach a human is.
The Data Behind That Claim
A review of more than 4,500 customer reviews across the App Store, Google Play, Trustpilot, and the Better Business Bureau, conducted by Dimension Labs, found that when a policyholder couldn’t reach a human after something went wrong, the likelihood of a negative outcome increased by 29 percentage points. That’s a substantial swing, and it’s a more useful signal than the general satisfaction scores insurers usually look at, because it isolates the specific moment where trust breaks down.
This matters because it cuts against the instinct a lot of insurers have when deploying conversational AI, which is to treat automation coverage as the main success metric. If the AI resolves 70% of interactions without human involvement, that number looks good on a dashboard. But it says nothing about what happens to the other 30%, and the data suggests that’s exactly where customer relationships get damaged if the escalation path isn’t well designed.
A Real Example of the Tension
Lemonade’s own FY2025 disclosures are a useful case study here, precisely because the company has been public about its numbers in both directions. On one hand, 96% of First Notice of Loss events at Lemonade start with its AI claims agent, and 55% of claims resolve end to end without a human ever getting involved. That’s a genuinely strong automation result. On the other hand, the same public data on customer complaints shows the vulnerability that comes with heavy automation: when something doesn’t fit the AI’s script and a customer needs a human, the experience of not being able to get one becomes the defining moment of that customer’s relationship with the company, regardless of how well the automated 55% performed.
Why Insurance Makes This Worse Than It Would Be Elsewhere
Escalation failures sting more in insurance than in most other industries, for a simple reason: the conversations happen after something has already gone wrong in the policyholder’s life. A car accident, a house fire, a death in the family. These aren’t neutral customer service moments where a bit of automation friction is a minor annoyance. They’re moments where tone and access to a real person carry as much weight as speed. An AI system that’s fast but leaves someone stuck when they need a human during one of these moments does more reputational damage than a slower system that gets the escalation right.
Escalation Isn’t a Feature, It’s a Design Requirement
The practical lesson from this data is that escalation design has to be treated as a load-bearing part of any conversational AI deployment, not an edge case handled after the main system is built. That means documenting specific thresholds for when the AI hands off to a human, rather than leaving it to the AI’s internal confidence scoring alone. It means the handoff itself needs to carry the full conversation history, so the policyholder isn’t forced to repeat everything they already explained to the bot. And it means the human agents on the receiving end of that handoff need to actually be staffed and available, not a theoretical safety net that doesn’t hold up under real volume.
This is also increasingly a regulatory expectation, not just good practice. The NAIC Model Bulletin, now formally adopted in 25 states plus the District of Columbia, specifically requires human-in-the-loop processes for consequential decisions. Regulators have effectively caught up to what the customer complaint data already showed: automation without a reliable human backstop creates real risk, both to the customer and to the insurer.
The Deployment Choice This Points Toward
This is a big part of why agent-assist conversational AI, which runs alongside a live human agent rather than replacing them entirely, tends to be an easier and lower-risk starting point for insurers. It surfaces compliance prompts and next-best-action suggestions to the agent in real time, but the human keeps full decision authority throughout. There’s no escalation gap to design around, because a human was never out of the loop in the first place. Autonomous conversational AI, which interacts directly with policyholders without a human present for routine cases, can still work well, but it demands a much more deliberate escalation design, precisely because the whole point is to remove the human from most interactions.
What This Means Practically
For any insurer building out conversational AI, the takeaway from this data is straightforward: don’t measure success purely by containment rate. A 50 to 70% containment rate on well-scoped workflows is a reasonable industry benchmark, but it should be paired with a clear answer to a harder question, which is what happens to the policyholder in the remaining percentage when the AI can’t resolve their issue. If that answer isn’t a fast, well-staffed human handoff, the containment number is masking a trust problem rather than solving one.
