How to measure an AI service agent: the metrics beyond "resolved"

To know whether an AI agent helps the customer, measure how many problems were actually resolved, not how many conversations the bot closed. We recommend six numbers: confirmed resolution, 7-day recontact, score separated for AI and person, handoff quality, cost per resolved case and requests per human agent.

Why “resolved” misleads

Most AI service dashboards show containment: the share of conversations that ended without reaching a person. The number rises when the bot resolves, and also when the customer gives up, switches channels or comes back the next day. High containment can be success or flight, and the dashboard doesn’t tell them apart.

In our reading, this is the most common mistake of those starting to automate: measuring what’s easy to count. The Klarna case illustrates the risk, as we cover in the Klarna lesson. The company reported volume and speed, and then went back to prioritizing the quality of human support.

The 6 metrics

1. Confirmed resolution

Divide the conversations where the customer didn’t return for the same reason by the total conversations handled by AI. It answers the question containment doesn’t: does the bot resolve or just close?

2. 7-day recontact

Count the customers who come back about the same topic within a week. If the number rises, containment is hiding a problem. A 7-day window is our suggestion and can vary with your buying cycle.

3. Customer score, separated by AI and by person

Run a short survey at the end of service and record AI and agents in distinct columns, by demand type. In the CX Brain and AntennasBI survey (1,003 Brazilians, 2024), human service got an average score of 9.3 and AI 7.0. A single average hides that gap and keeps you from seeing which topic AI lags on.

4. Handoff quality to a person

Look at how many cases pass, how long the customer waits to be served and how many times they had to repeat the problem. A high handoff rate isn’t a failure. A handoff with no history is. It’s the point where the hybrid model usually breaks, and we explain the design in hybrid service: human + AI.

5. Cost per resolved case

Divide the total service cost, with AI, agents and tools, by the actually resolved cases, using metric 1. Cost per conversation almost always falls when AI comes in. Cost per resolved case shows whether the savings survive recontact.

6. Requests per human agent

Record how many times the customer asks for a person and at what point in the conversation. If the requests cluster around one topic, that topic needs to leave AI or be rewritten.

Model dashboard

Metric	How to calculate	Frequency	Alert sign
Confirmed resolution	cases without recontact ÷ AI cases	weekly	falls two weeks in a row
7-day recontact	returning customers ÷ served customers	weekly	rises two weeks in a row
AI × person score	average by demand type	monthly	AI below the person on any type
Handoff quality	% of handoffs with history and average wait	weekly	customer repeated the problem
Cost per resolved case	total cost ÷ resolved cases	monthly	cost per conversation falls and this doesn’t
Requests per human	requests ÷ conversations, by topic	weekly	concentration on one topic

We don’t set numeric targets because they depend on your product, ticket and channel. Measure for four weeks before fixing any of them.

What Klarna reported

In February 2024, Klarna stated that its AI assistant resolved queries in under 2 minutes, versus 11 minutes before, with 25% fewer repeat contacts and satisfaction equal to that of agents. In 2025, according to tech.co, the company began prioritizing human support too. They are the company’s own numbers, without independent verification, and they serve to show that time and recontact already entered the account.

How to start in 4 weeks

1. Week 1: choose a flow and define what counts as the “same reason” for recontact.
2. Week 2: turn on the score survey with AI and person separated.
3. Week 3: read a sample of “resolved” conversations and compare with the dashboard.
4. Week 4: calculate cost per resolved case and decide what to expand, adjust or return to a person.

To automate the most-used channel more safely, see AI on WhatsApp: 10 precautions before automating.

How NorthStar fits in

We build the dashboard together with the service design: what AI resolves, when it passes to a person and how each metric is collected in your systems. See how we work in custom sales and service automation and on the sales and service with AI page.

Frequently asked

What's the difference between containment and resolution?

Containment is the share of conversations that end without reaching a person. Confirmed resolution is the share where the customer didn’t return for the same reason. The first can rise when the customer gives up. The second shows whether the problem was resolved.

Which metric should I look at first?

The 7-day recontact. It’s easy to calculate with the data you already have and quickly shows whether containment is hiding a problem.

How do I compare the AI score with the person's?

Apply the same survey to both and separate by demand type. In the CX Brain survey (2024), the average score was 9.3 for human and 7.0 for AI, which indicates the comparison is worth it.

Is there a market target for these metrics?

We didn’t find a reliable reference, so we don’t suggest numbers. Measure your base for four weeks and use that result as a starting point.

Isn't cost per conversation enough?

No. It almost always falls, including when AI resolves poorly. Cost per resolved case accounts for the customers who come back.

Sources

• Consumidor Moderno, on the CX Brain + AntennasBI survey, Sep/2024. consumidormoderno.com.br
• Tech.co. Klarna reverses AI overhaul, 20/05/2025. tech.co