How do you evaluate AI support quality beyond deflection rate?

Leadership asks whether the AI is working, and the number you can defend is deflection or containment. What you cannot say is whether the customers behind it got a correct answer, finished the task, trusted the experience, or came back with the same problem. A deflected conversation can hide a customer who retried, reopened a ticket, escalated by email, churned, or acted on an incorrect answer.

Deflection records where the work went. Whether the customer got a correct answer is a separate question, and your vendor's "resolution" figure does not settle it: each vendor sets its own window, judge, and stance on accuracy, and for some it is the unit you pay for.

Our recommendation: keep deflection as an efficiency line and build your own scorecard around it. Rank answer correctness and verified resolution above efficiency, check each resolution against repeat contact in a fixed window per issue type, and segment by issue risk. Then run one bounded 30-day experiment with a decision rule you write before the data comes in.

TL;DR

• Deflection and containment are cheap, real-time efficiency metrics and weak quality metrics.
• Vendor "resolution" numbers differ in window, judge, and accuracy criterion, and Intercom and Zendesk bill on them.
• Grade a small, stratified sample with reviewers who do not configure the bot; correctness and safety fails override other scores.
• Joining conversations to later outcomes shows association; only a holdout or staggered rollout shows cause.

What deflection can and cannot tell you

Salesforce's Agentforce module defines Deflection Rate as the "Share of sessions that ended without escalation to a support representative or another agent." Decagon, which sells an AI support agent, calls deflection "a routing statistic" and resolution "an outcome statistic," and says "they can move in opposite directions from the same change to an AI agent." The mechanism holds without the vendor: hiding the handoff raises deflection whether or not more customers get answers.

The denominator decides the number

• Deflection over all AI-touched sessions = sessions with no handoff ÷ all sessions where the AI replied.
• Deflection over real questions = sessions with no handoff ÷ (AI sessions minus greetings, test traffic, and sessions abandoned at the first message).

The first counts a customer who typed "hi" and left as a success. The second still counts everyone who left after a wrong answer. Vendors choose too: Intercom's Fin resolution count leaves out greetings, unanswered clarifying questions, and explicit requests for a human. Write your denominator down and keep it fixed.

A session without a handoff has three explanations

The customer was solved, gave up, or switched channels, and session data cannot tell which. SQM Group's survey method for phone first-call resolution counts a call only with "yes" to "Was your call resolved?", one call, and "no" to using another channel (SQM, Jan 19, 2021, phone contact centers). Your AI data needs the same check: join sessions to later contacts.

Vendor "resolution" is several different metrics

None of these can be compared across tools, and two are billing units. Sources, for the definitions only: Intercom Help, Zendesk's tiers, "introduced on May 18, 2026", and Ada's guide (June 17, 2026).

Drag, an inbox vendor that competes with these tools, gives an example (Aug 8, 2026) without saying whether the numbers come from a real queue: "A single support queue, measured all four ways, produces four numbers that can span thirty points."

• Confirmed resolution: 46%
• Assumed resolution: 68%
• Deflection: 80%
• Satisfaction: 74%

CSAT does not close the gap. Intercom's CX Score scores closed chat or email conversations that include "at least two customer replies and two agent/chatbot replies," and Intercom says shorter conversations may now be rated when there is "a clear signal to evaluate." By our reading, a customer who leaves Fin after one exchange can count as an assumed resolution and still get no quality score, which is where silent failures are likely to sit.

The objection: deflection and CSAT are good enough

Deflection is cheap and real-time, sampling is slow and noisy, and vendors already report resolution, so keep deflection as the efficiency line. Vendor resolution follows rules each vendor sets. Zendesk's billable Verified Resolution is graded by Zendesk's own LLM, and Intercom also bills for assumed resolutions, which it infers from customer silence. CSAT and conversation scores can miss the short exchanges where silent failures sit. The only QA price we found is for Lorikeet's automated QA product, Coach: "about $0.27–$0.30 per ticket." We found no measured cost for human sampling. NIST's voluntary Generative AI Profile, which abbreviates generative AI as GAI, advises: "Avoid extrapolating GAI system performance or capabilities from narrow, non-systematic, and anecdotal assessments" (MS-2.5-001), which favors a small systematic sample over ad hoc spot checks.

A measurement hierarchy

Rank metrics so a gain lower down never excuses a loss higher up:

1. Safety and answer correctness.
2. Task or issue resolution, verified against repeat contact.
3. Escalation quality and context preservation.
4. Customer effort, confidence, and trust.
5. Operational efficiency and cost.
6. Product, customer, and business outcomes.

Correctness comes first because a confident wrong answer looks like a success in every routing metric. NIST AI 600-1 (July 2024) defines confabulation as "The production of confidently stated but erroneous or false content ... by which users may be misled or deceived."

The concrete case is Moffatt v. Air Canada (2024 BCCRT 149). Air Canada argued it could not be held liable for what its chatbot said. The tribunal found the airline "did not take reasonable care to ensure its chatbot was accurate," a "negligent misrepresentation," and awarded "$650.88 CAD" plus interest and fees, according to Manatt's summary (Feb 28, 2024). It is a decision of British Columbia's Civil Resolution Tribunal, a small-claims body. A deflection count cannot separate that conversation from a correct one.

Efficiency sits fifth because speed and quality can move apart. In measured research on "5,172 customer support agents" (arXiv v2, Nov 6, 2024; an assistant for human agents, not a customer-facing bot), Brynjolfsson, Li and Raymond found AI assistance raised issues resolved per hour "by 15% on average," while the "most experienced and highest-skilled workers see small gains in speed and small declines in quality."

Metric definitions, formulas, and gaming risks

The formulas are our recommendation; keep each definition fixed for any comparison.

Each efficiency metric can rise without helping a customer; pair it with a quality metric.

A QA rubric for sampled AI conversations

Correctness and policy or safety are gating: a fail on either fails the conversation, whatever the other scores say.

How to sample

• Stratify by risk segment and oversample high-risk ones. Include assumed resolutions and short conversations your quality score may not rate.
• Use reviewers who do not configure the bot. NIST MS-1.3-003: "Verify those conducting structured human feedback exercises are not directly involved in system development tasks for the same GAI model."
• Calibrate weekly: two reviewers grade a shared subset, and each disagreement becomes a rubric note.
• Open every cited article. NIST MS-2.5-003: "Review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities."
• Size per segment: if you would act on a failure that occurs once in N conversations, reading about 3N from that segment gives roughly a 95% chance of seeing it at least once (our arithmetic: 1 minus (1 minus 1/N) to the power 3N is close to 95%). Twig, which sells AI support, suggests "typically 50-100 interactions per week" (vendor guidance).

When the failure is the article

The rubric line "cited documentation was current" separates two fixes. When sampled failures trace to a missing or stale article, route them to the documentation owner: the fix is a new or corrected article. Gartner's measured figure, reported by destinationCRM (Sep 12, 2024), covers self-service in general, not AI accuracy: "in 43 percent of cases, customers couldn't find content relevant to their issues."

EkLine works on that layer: Docs Agent drafts and edits articles in a Confluence or Pylon knowledge base from sources such as support threads. A person reviews the diff under View All Changes and clicks Update KB, which "overwrites the live page immediately," so nothing goes live without that approval. Two scheduled agents help: "Improve Pylon knowledge base" reviews recent Pylon tickets for questions the knowledge base should have answered and drafts articles for your review, and "Confluence Knowledge Base Drift Review" detects Confluence pages that recent documentation changes made stale and drafts corrections. Knowledge base management is available on request.

Before, a stale-article cluster from your sample waits in a backlog; after, a drafted correction waits for a reviewer. EkLine drafts the article fix; your scorecard still decides whether anything improved. Start with the tutorial on turning recurring support questions into a knowledge base article, whose last step includes "Watch whether the question returns."

Join conversations to what happened next

Fix each signal's key and window before you look at results.

Set each window by how long a customer takes to discover a wrong answer. SQM sets phone windows by call reason: a claim "might need 30 days" and a status inquiry "might only need 5 days." Vendor guidance runs shorter: Decagon says "Recontact rate within 48 or 72 hours is the cheapest check on an inflated resolution figure," and Zendesk's blog uses 24 to 72 hours. One practitioner, Tian Pan, recommends "Repeat-contact rate at 7, 30, and 90 days."

These joins measure association. Customers who stay with the AI can differ from those who ask for a person first, so lower churn among them can reflect who they were. To show cause, use a holdout (eligible conversations in a segment randomly routed to the AI or the human path) or a staggered rollout (segments switched on in an order fixed in advance, compared with segments still waiting). Both need the same definitions on each side, a primary metric and window chosen before launch, and time for lagging outcomes to appear.

Segment before you average

Report every metric by issue type, customer tier, product area, risk (money, account access, security, and policy commitments in the high tier), and channel. Averages hide harmful failures in small high-risk segments: if password resets dominate your volume, the blended rate can rise while answers about a changed API endpoint or a refund policy get worse. NIST says organizations "may additionally wish to allocate risk management resources relative to the severity and likelihood of negative impacts" (AI 600-1, p. 3), and SQM's per-reason windows apply the same logic to time. No source we found measures how often this happens; it follows from how averages work.

An example balanced scorecard

Illustrative layout, fields and formulas only. Values come from your own human and AI baselines. The only published averages we found are vendor figures on the vendor's own definition.

A 30-day plan

Owners: the support leader (decision rule), Support Ops (definitions and instrumentation), reviewers outside the bot team (sample), and the knowledge base owner (documentation failures).

A rule might read: expand in a segment when its sample shows no gated fails and repeat contact is no worse than the human baseline; pull back on any high-risk safety fail or repeat contact worse than the baseline; hold otherwise.

Replace the deflection target with one experiment

An illustrative bounded experiment:

• Hypothesis: AI answers to refund-policy questions are correct and hold as well as human answers.
• Segment: refund-policy questions in chat, all tiers (high risk).
• Primary signal: repeat contact on the same issue within 30 days, a claim-type window.
• Guardrail: gated correctness and policy pass rate in a weekly stratified sample.
• Window: 30 days of conversations, plus 30 days for the last window to close.
• Decision rule: written in week 1 and applied as written.

Bring that segment to your next leadership review, with deflection beside it as the efficiency line.

---

Sources

• Fin AI Agent outcomes, Intercom Help
• About automated resolution tiers, Zendesk
• Monitor Agent Metrics and Scores, Salesforce Trailhead
• What is automated resolution in AI customer service, Ada
• Resolution rate vs deflection rate, Decagon
• CX Score, Intercom Help
• AI Support Agent Resolution Rates: What Vendors Claim vs What Production Shows, Drag
• First Call Resolution Rate Calculation, SQM Group
• Only 14% of Customer Service Issues Are Fully Resolved in Self-Service, destinationCRM
• Generative AI at Work, Brynjolfsson, Li and Raymond
• NIST AI 600-1: AI RMF Generative AI Profile
• AI Gone Wild: Airline Has to Honor a Refund Policy Its Chatbot Fabricated, Manatt
• The Deflection Metric That Lied, Tian Pan
• Ticket deflection vs. resolution: Metrics that matter, Zendesk
• Resolution rate vs deflection rate in AI support, Lorikeet
• What Metrics Should You Track After Launching AI Customer Support?, Twig
• Manage a Confluence or Pylon knowledge base, EkLine docs
• Scheduled agents, EkLine docs
• Tutorial: turn recurring support questions into a knowledge base article, EkLine docs