Confidence Score

How it works

Confidence scores are generated during the retrieval step of a RAG system. After retrieving candidate passages from the knowledge base and drafting a response, the model scores the match between the retrieved context and the user's question. A high score (0.9+) means the AI found a direct, specific answer in the knowledge base. A low score (under 0.65) means the retrieval was weak — the question might be outside the knowledge base scope, or the customer's phrasing was ambiguous.

Confidence thresholds are configurable. A lower threshold (e.g., 60%) means the AI attempts more answers but risks more inaccurate responses. A higher threshold (e.g., 85%) means more escalations but higher accuracy on AI-handled conversations. The right threshold depends on your product complexity and the cost tolerance for AI errors.

Some systems use dynamic thresholds — a lower threshold for common topics where the AI has strong training data, and a higher threshold for rare or high-stakes topics like refunds or legal matters.

When you'll encounter it

Confidence scores appear in AI support dashboards, escalation logs, and knowledge-gap reports. You'll configure confidence thresholds during onboarding and tune them as you review the first weeks of escalations. A pattern of high-confidence answers with incorrect content means the knowledge base needs updating; a pattern of low-confidence escalations on topics that should be covered means retrieval is failing.