Skip to content

Retrieval effectiveness evals

Small, honest benchmark for the agent-context architecture. Metric: correctness per token/tool call, not recall volume. Rerun after any context-architecture change and at release audits (see releasing checklist) — never per-PR.

Protocol

For each task, run a fresh agent session and record:

  • tool calls until first correct edit
  • relevant-file recall (touched files ∩ golden files)
  • wrong-file reads (confusables opened, irrelevant dirs browsed)
  • approx context tokens loaded before the first edit

Compare against the previous run of the same task. Two consecutive regressions on any task → revisit the router/index wording for that area, or (only with evidence grep can't fix it) reconsider retrieval infrastructure.

Tasks

Golden lists state what a strong engineer needs; confusables are what a weak run wrongly opens.

1. Bug fix — duplicate transaction recorded twice

  • Golden: docs/architecture/ingestion-pipeline.md §Cross-source dedupe, TransactionRepository.resolveCrossSourceDuplicate, ADR-0020
  • Confusable: parser/NotificationKey dedupe (different mechanism), UnmatchedJsonCodec

2. Feature — add a currency with unusual minor digits (JPY-style)

  • Golden: codebase-map §Add a currency recipe, common/Currency.kt, docs/architecture/money.md
  • Confusable: MoneyFormatter vs MoneyText roles; hardcoding digits in UI

3. Refactor — rename a classification keyword constant

  • Golden: docs/architecture/classification.md, KeywordClassifier.DEFAULT_KEYWORDS, settings category_keywords JSON replace-not-merge rule (codebase-map recipe)
  • Confusable: Correction overrides (corrections table — different seam)

4. DB change — add a column to transactions

  • Golden: codebase-map §DB migration procedure, data-layer.md, latest app/schemas/*.json
  • Confusable: editing released migrations retroactively (forbidden); SQL aggregation instead of in-memory filtering (ADR-0006)

5. Cross-module — change capture gating for one bank package

  • Golden: ingestion-pipeline.md routing table, NotificationFilter, tracked-packages CSV (ADR-0008), parser.md starter rules + fixtures
  • Confusable: BankRuleSets debug-shell rules; NotificationKey formula

6. Dependency bump — ComposeCharts upgrade

  • Golden: tooling.md (build stack, icon-catalog regen), ui.md §Charts, ADR-0028/0029
  • Confusable: hand-editing generated MaterialIconCatalog.kt (generated — regenerate instead); ADR-0013 (superseded chart approach)

Negative pairs (baked into glossary)

ExportCodec vs UnmatchedExportCodec; MoneyText vs MoneyFormatter. Glossary disambiguation notes are the standing negative test surface.