Use these manual checks before treating a score as evidence. This page does not execute the scorer or validate an answer. The linked articles describe historical implementation records.
A resolved marker still needs a supported claim
First locate the answer span that makes the claim. Follow its marker to the exact cited passage, then compare the claim with that passage. A marker that resolves to a retrieved fragment can still accompany an unsupported statement.
| Record | Text | Manual observation |
|---|---|---|
| Passage A | Inspect the filter every 12 days. This note does not specify a replacement interval. | Inspection interval is stated. |
| Answer | Replace the filter every 12 days [A]. | The marker resolves. The replacement claim exceeds the passage. |
- Save the exact answer span and source revision.
- Record marker resolution separately from claim support.
- Mark the replacement claim unsupported in this fictional pair. Keep a reviewer decision separate from the parser result.
- Repeat with a supported answer: "Inspect the filter every 12 days [A]." A useful check must distinguish this pair.
Read Valid Citations, Unsupported Claims: What Fifteen Questions Revealed About Our RAG for the historical case and its sampling limits. Its selected questions do not establish a general model error rate.
Open the existing stale-evidence fixture to inspect source freshness as a separate dimension. A current source can still fail to support a claim.
Keep the quantity check beside its blind spot
For each answer quantity, retain its value and unit together. Compare against quantities in the cited passages. Preserve unit-prefix case where it changes scale. Pair membership alone does not bind a quantity to the correct subject.
| Passage | Answer | Expected manual finding |
|---|---|---|
| LED A consumes 8 mW. | LED A consumes 8 MW. | Changed unit prefix. Unsupported pair. |
| LED A consumes 8 mW. LED B consumes 20 mW. | LED A consumes 20 mW. | Pair occurs in source. Subject assignment is wrong. |
| LED A consumes 8 mW. | LED A consumes eight milliwatts. | Meaning is equivalent. Parser handling must be measured separately. |
{"record_id":"manual-quantity-example","fixture_origin":"synthetic",
"parser_observation":"NOT_RUN","review_status":"NOT_REVIEWED",
"model_run":false,"release_eligible":false}
Read What Our Number-and-Unit Scorer Could Actually Detect for the implemented scorer's normalization and metric boundaries. Its parser fixture counts are not live model accuracy.
When recording a rate, name the eligible population, numerator and excluded errors. Keep an empty applicable population distinct from a measured zero.
What to retain after review
Copy the answer, cited passage and immutable revisions into a review record. Keep syntax resolution, quantity matching and human claim adjudication as separate fields. A complete record does not establish production effectiveness.
Return to the evidence inventory to document evidence references and gaps. The original inventory remains a draft preparation tool.