Confidence Score
A confidence score is a value between 0 and 1 that an AI model uses to express how sure it is about its answer. In many chatbots the score decides whether the AI responds itself, asks a clarifying question, or hands off to a human agent.
What is a confidence score?
A confidence score is the quantified self-assessment of an AI model: how sure am I about this answer? Values typically range from 0 (uncertain) to 1 (very sure) and represent a probability, not an absolute measure.
In many messaging bots, the confidence score steers what happens after a user question: high confidence → bot answers, medium → clarifying follow-up, low → handoff to a human agent.
How a confidence score arises
Different bot types produce scores differently:
| Bot type | How confidence is measured |
|---|---|
| Keyword bot (autoresponder) | Match or no match — no real confidence score. Rule fires or it doesn't. |
| NLU bot (intent classification) | Probability that the user input belongs to intent X. Output: probability distribution across all intents. |
| RAG bot (retrieval-augmented generation) | Similarity between user question and the best-matching knowledge base chunk, combined with model self-assessment. |
| AI agent | Multi-step: confidence at tool selection, confidence at result, weighted total. |
Thresholds
In practice you work with thresholds that steer system behavior:
- ≥ 0.80 — AI answers on its own. High certainty.
- 0.50 to 0.80 — AI asks a clarifying follow-up ("Did you mean X or Y?").
- < 0.50 — handoff to human agent.
Exact thresholds depend on the model and the use case. For safety-critical topics (health, legal, finance) thresholds run higher.
Confidence score and handoff logic
In SendSeven, AI agents in chat hand over by rules, not by score: for example when the customer asks for a person, a defined keyword comes up, the maximum number of bot replies is reached, or the agent can't find an answer several times in a row (default 2), so the customer doesn't get stuck in a loop. Every reply is checked against your sources before it is sent. Email agents do use a threshold: they send automatically only above it (default 80) and otherwise save a draft for the team.
Keyword-based bots (autoresponders) have no real confidence score at all. They either match or return a fallback text.
Common causes of low confidence
- Question outside the knowledge base: Customer asks about a topic that hasn't been ingested.
- Ambiguous phrasing: "I have a problem with the app" fits 15 topics.
- Wrong language: User writes in English, bot trained in German.
- Spelling/slang: Typos or jargon the model doesn't recognize.
- Multi-intent: Customer asks two questions at once — model doesn't know which to answer first.
Confidence visibility
In SendSeven, the team sees a confidence badge on AI Suggestions (reply suggestions in the inbox) and in AI Search: green from 70, yellow from 40 to 69, red below 40. A low score shows which suggestion to check before sending and often points to a gap in the knowledge base.
Confidence vs. correctness
One important lesson: high confidence does not mean high correctness. A model can be very confident and still wrong ("hallucination"). Confidence is a signal, not a guarantee. So a productive setup always includes: a test set with ground truth, regular sampling, and a feedback loop from agents to knowledge-base maintainers.