cited-rag — Evaluation

Software Quality Engineering report · v0.1.0 · QA certified 2026-11-16 (board cited-rag, task t_e42b7406) · Scope: PRODUCT_BRIEF.md §2 · Verdict: PASS 6/6

Current release: PyPI · cited-rag v0.1.1 (70 tests passing) is the published version. The 52-question SQE evaluation below was measured on v0.1.0 (QA-certified baseline, 2026-11-16); the retrieval, citation and refusal pipeline is unchanged in v0.1.1, so every metric below remains identical on v0.1.1.

Local RAG you can trust. Zero hallucination by design: every claim is cited ([file:L1-L2]) or the answer is refused with Insufficient evidence. Stdlib-only, offline-first.

1. Reproducible protocol

One command reproduces every number below from the bundled golden set — run it, don't trust this table:

pip install cited-rag
cited-rag index ./docs --db ./index.db
cited-rag eval --golden eval/golden.jsonl

2. Results (measured, not claimed)

MetricGate (C4)MeasuredVerdict
Recall@5≥ 0.801.00PASS
MRR≥ 0.600.93PASS
Correct-refusal (off-topic)≥ 0.901.00 (12/12)PASS
n (golden)≥ 50 Q/A52 (40 + 12)PASS
pytestgreen70 passed (v0.1.1; 59 at QA baseline)PASS
Index perf< 60 s / 50 docs12 docs in ~0.4 s *PASS *

* Reserve (non-blocking, QA-flagged): examples/docs ships 12 docs, not 50 — the 60 s budget holds with huge margin (0.4 s) but load at exactly 50 docs was not measured as-is. Raw eval output: Recall@5=1.00 MRR=0.93 correct-refusal=1.00 (12/12) n=52.

3. Release gates C1–C6

GateRequirementEvidenceVerdict
C1 offline stdlib-onlyindex + ask, no external depdependencies = []; index+ask OK offlinePASS
C2 exact citations100% claims cited, exact offsets3/3 phrases verbatim [01-bm25-ranking.md:L1-L20], fidelity 1.00PASS
C3 refusalInsufficient evidence, exit 0, no invented citations12/12 off-topic → refusalPASS
C4 evalRecall@5 ≥ 0.80, MRR ≥ 0.60, refusal ≥ 0.90, ≥ 50 Q/A1.00 / 0.93 / 1.00, n = 52PASS
C5 LLM OFF by defaultopt-in --llm only, citation format unchangedstore_true, default=False; no network outside adapterPASS
C6 tests + perf + docspytest green, index < 60 s, 3-command quickstart59 passed (QA) / 70 on v0.1.1 / 0.4 s / README quickstartPASS *

* Same 50-docs reserve as §2. Blocking gates C2/C3/C4 all PASS → YES to ship, tag cited-rag-v0.1.0.

4. Honest limitations

5. Artifacts & links

Package

PyPI · cited-rag v0.1.1 (70 tests)
pip install cited-rag

Source

GitHub · shenn667/cited-rag
CI badge, MIT, Trusted Publishing (no stored token).

Dataset

HF · SHENNN667/cited-rag-golden-52
Golden set (52 Q). Published under namespace SHENNN667 (no write rights on shenn667 — the lowercase shenn667/... URL returns 401, use this link). Fallback: eval/golden.jsonl in the repo.

Article

dev.to · Zero-Dependency Verified Agents
2026-11-18 — the 3-brick loop (context → cited answer → CI gate).

Registry

Glama · agent-context-bridge
MCP context pillar listed on Glama. Note: cited-rag itself is a stdlib CLI and has no separate Glama listing.

6. Reproduce it now

git clone https://github.com/shenn667/cited-rag.git
cd cited-rag
pip install cited-rag   # or: pip install -e .
cited-rag index examples/docs
cited-rag ask "What are the default BM25 parameters?" --k 5
cited-rag ask "What is the best sourdough recipe?" --k 5   # -> Insufficient evidence
cited-rag eval --golden eval/golden.jsonl
# expected: Recall@5=1.00  MRR=0.93  correct-refusal=1.00 (12/12)  n=52