Add an evidence-based confidence score (0–100) — procedure

Procedure for adding an evidence-based 0–100 confidence score to responses while preserving fail-closed sentinel behavior.

Use this procedure to add an evidence-based confidence score

Choose this procedure when every non-sentinel response must end with a numeric confidence line that reflects correctness plus evidential support.

Required output

Every non-sentinel response ends with a confidence line

Every non-sentinel response must end with a final line: Confidence: <0–100>/100

The format is fixed.

Meaning

Confidence is evidence-based, not probabilistic

Confidence reflects correctness + evidential support.

It is not a statistical probability.

Fail-closed rule

Sentinel-only responses stay sentinel-only

If an active evidence policy requires a sentinel-only fail-closed response, output exactly the sentinel and stop.

Do not append a confidence score after the sentinel.

Canonical links

Open the linked policy, prompt file, and adjacent references that implement this procedure.

Related

Policy and prompt files for this guide

Open the policy linked to this guide and any prompt files used with it.

Add an evidence-based confidence score (0–100) — policy

Prompt files (1) Hide prompt files

• System confidence-score.system.txt

Choose how you will enforce the confidence line

Pick one mode first, then apply the matching setup.

Option 1 — System prompt template

Recommended when you want the confidence line enforced through a dedicated system prompt file.

Jump to Option 1 →

Option 2 — Manual response contract

Use this when you want to add the confidence rules directly into your own policy or template stack.

Jump to Option 2 →

Option 3 — Full workflow

Use this when confidence scoring should run as part of the Fact-Checking Kit workflow.

Jump to Option 3 →

Set up the procedure

Complete these steps before you run the scoring flow.

Step 1

Choose the enforcement mode

Decide whether you will enforce confidence scoring through the dedicated system prompt, a manual response contract, or the full Fact-Checking Kit workflow.

Step 2

Install the template when using Option 1

If you use Option 1, install confidence-score.system.txt in the runtime.

Step 3

Preserve fail-closed compatibility

Ensure compatibility with evidence-boundary fail-closed behavior.

If an evidence policy triggers a sentinel-only response, output exactly the sentinel and stop. Recommended enforcement: instruction-hierarchy-and-evidence-boundary.system.txt

Step 4

Apply the scoring policy

Apply the scoring rules from the linked policy.

Evidence-based confidence score (0–100) — policy

Verify the setup

Use these smoke tests to confirm that confidence scoring is working correctly.

Smoke test 1 — Adequate admissible evidence

Ask a factual question where you provide adequate admissible evidence.

Expected: the response ends with Confidence: <0–100>/100.

Smoke test 2 — Sentinel-only fail-closed case

Trigger a sentinel-only fail-closed case under your active evidence boundary.

Expected: output exactly the sentinel and stop, with no confidence line.

Choose the exact mode

Each option maps to one enforcement model for adding the confidence line.

Option 1

System prompt template (recommended)

Enforce the confidence line through confidence-score.system.txt.

Best when you want a dedicated enforcement layer in the runtime.

Open system prompt

Policy Procedure

Example: “Summarize what the attached logs show about the error and what is NOT proven.” You must provide logs or excerpts and the active evidence boundary.

Option 2

Manual response contract

Add the confidence line and its rules directly into your own policy or template stack.

Best when you already maintain a custom contract and want to layer scoring into it explicitly.

Open policy

Template reference Procedure

Example: “Assess claim X and state confidence.” You must provide admissible evidence under the active policy.

Option 3

Full workflow (Fact-Checking Kit)

Run confidence scoring as part of the broader fact-checking workflow.

Best when confidence should be embedded in a full verification run rather than added as a standalone contract.

Open procedure

Policy System prompt

Example: “Verify whether claim X is supported; fail closed if not.” Output must end with a confidence line unless a sentinel-only response is required.

Common mistakes

These are the most common failure points for this procedure.

Confidence after sentinel

Appending a confidence line after a sentinel-only fail-closed response.

Wrong format

Using non-numeric formats instead of Confidence: <0–100>/100.

Wrong meaning

Treating confidence as probability instead of evidence-weighted analytic confidence.

Inflated confidence

Reporting high confidence when evidence is indirect, weak, or conflicting.

The policy requires evidence-strength caps.

Related libraries

Open the matching library when you need the broader policy set, adjacent procedures, prompt files, or the content map.

Policies

Browse the rule set behind evidence-based confidence scoring.

Open index →

How-to

Browse the full procedural library and adjacent verification workflows.

Open index →

Prompt library

Open the prompt files used to enforce this procedure in a runtime.

Open library →

Content map

Open the broader site structure and adjacent pages.

Open page →