---
title: Code index vs grep: what we built, the two lane trials and the verdict
date: 2026-10-19
tags: [tools, code-index, discovery, trial, tokens]
---

# Code index vs grep: results of the two lane trials (measured 2026-10-17 to 2026-10-19)

**Question.** Lanes burn about 2x the 16M tokens/day target, and part of that is agents hunting through the repo with grep-style searches. Would a local search index, either pushed into the order or queried by the agent, find the right files faster and cut tokens?

**Short answer.** The index finds the right files far better than grep in an offline benchmark (hybrid 81% vs grep 39% at hit@5). In real lanes it did **not** reduce tokens, minutes or phases in either trial. Recommendation: switch trial B off, keep the tool and the file-name lookup available, and look for the token burn elsewhere (see "What next"). The original slowness was fixed by ignore files, not the index.

## Background: why we looked at search at all

The box became very slow on 2026-10-17 because agents repeatedly ran `rg --files` from the top of a drive holding 1,320,034 files. One file search from the root took 8.1 s wall / 67.0 CPU-s (median of 3). A root `.rgignore` only got it to 6.7 s / 54.3 CPU-s. The remainder was one folder with 8,030 files but 394,890 directories (about 253 per agent run); a second `.rgignore` skipping it gave 0.3 s / about 2.2 CPU-s, roughly 30x less CPU. Caveats: ignore files only apply to searches starting at that folder, and the directories still exist. The fix was not benchmarked for tokens, only for time and CPU.

## 1. What was built (`_codeindex`)

| Piece | What it does |
|---|---|
| `code.db` | SQLite FTS5 plus sqlite-vector (Apache-2.0, v1.1.2, win_amd64, sha256 checked against PyPI) with the MiniLM ONNX model already cached by chromadb (384-d). About 24.5k chunks: git-tracked text of one repo, a notes vault, and about 6.4k agent reports and plans. |
| Exclusions | Sensitive areas and patterns are never indexed. |
| `codeindex.py` | `build` (incremental by file hash, below-normal priority, 4 cores max), `search "q" --mode hybrid\|fts\|vec`, `files-like NAME`. |
| `discover.py`, `discovery_hook.py` | Trial A: build a "Discovery" section for a lane order. |
| `code-search.py`, `slim.db.gz`, `apply_query_trial.ps1` | Trial B: a slim trigram FTS5 copy (113 MB gzipped) staged into the container so the agent can query it itself. |
| `evaluate.py`, `evaluate_trigram.py` | Retrieval benchmark. |
| `trial_report.py` | Compares the two arms from the trial logs. |

## 2. Offline retrieval benchmark

150 past commit subjects, each turned into a query; success = the files that commit changed appear in the top results. The grep baseline is a simulation of agent keyword grep, not measured agents.

| Method | hit@5 | hit@10 |
|---|---|---|
| grep (simulated) | 39% | 47% |
| FTS5 words | 69% | 75% |
| vector | 65% | 78% |
| **hybrid (FTS + vector)** | **81%** | **89%** |

Trigram index (trial B, re-run on the grown corpus the same day): hit@10 trigram **71%** vs word FTS 59%, hybrid 64%, grep 40%. Query time: 8 ms (FTS) to about 520 ms (hybrid; model load dominates).

So on the benchmark the index is clearly better at "which file is this about". The trials ask the harder question: does that save an agent real work.

## 3. Trial A: push a Discovery section into the order

- **Design:** The lane runner appends a generated "Discovery" section to the staged copy of the order for about half of task families (sha1 of salt + base, salt `2026-10-18-a`). Fails open. The order file in the lanes folder is never edited.
- **Window:** started 2026-10-17 23:44, stopped on 2026-10-18 after 25 task families.
- **Result (final numbers, `trial_log.jsonl`):**

| Arm | Families | Median tokens | Median minutes | Mean phases | Done/landed |
|---|---|---|---|---|---|
| Discovery | 14 | 372,713 | 23.8 | 11.5 | 6 |
| Control | 11 | 371,620 | 26.0 | 5.9 | 5 |

- **Reading:** tokens are identical, minutes are 2 min lower, but phases per family are almost double with Discovery. No token win. (An early read of the same trial, before the last families finished, looked worse for Discovery: 277k vs 205k tokens. The final numbers are neutral; either way there is no gain.)
- **Failures:** one lane hit a Discovery timeout (`TimeoutExpired`, 20 s), logged and failed open as designed.
- **Decision (2026-10-18):** disabled.

## 4. Trial B: let the agent query the index

- **Design:** instead of pushing text, stage the query tool and slim database and add a short optional note. Salt `2026-10-18-q`, started 2026-10-18 16:49, log `trial_log_query.jsonl`.
- **Result so far (37 families, `python trial_report.py trial_log_query.jsonl`):**

| Arm | Families | Median tokens | Median minutes | Mean phases | Done/landed |
|---|---|---|---|---|---|
| Query tool | 20 | 451,494 | 26.9 | 6.15 | 6 |
| Control | 17 | 422,929 | 23.8 | 7.24 | 6 |

- **Reading:** the query arm used about 7% more tokens and 3 more minutes at the median, with slightly fewer phases. That is within the noise for these sample sizes, but there is no sign of a win, which is what the trial needed to show to justify keeping it.

## 5. Caveats (why this is "direction only")

1. **Task size varies a lot** between families. Medians of 14-20 families per arm can swing by tens of percent on a few large tasks.
2. **Confound in trial B:** the Codex version changed at 2026-10-18 16:45, shortly before trial B started, so the two trials are not on the same engine and should not be compared with each other.
3. **Tokens per family include verify and landing phases**, which the index cannot help with.
4. **"Done/landed" is low in both arms** (6 of 17-20), so many families were still running or blocked; unfinished families are not in the token medians.
5. The benchmark measures finding files from a commit-style query, not the way agents actually search mid-task (they often already know the file from the order).

## 6. Verdict and recommendation

| Question | Answer |
|---|---|
| Does the index find files better than grep? | Yes, clearly (81% vs 39% hit@5). |
| Did either trial cut tokens, minutes or phases? | No. A was neutral on tokens with more phases; B was about 7% worse on tokens. |
| Keep the trials running? | No. Both have enough families to say "no win". |
| Keep the tool? | Yes, as an optional lookup for planners and humans (`codeindex.py search`, `files-like`). It costs nothing when unused. |

**What next (where the token burn probably is):**
- Orders and re-runs: lanes parked for looping  and repeated verify/rework phases burn more than file hunting.
- Container lane staging and test reruns: 805 pool jobs in 24 h with a 28% failure rate (many are expected test failures, but each rerun costs a phase).
- Per-family token accounting by phase (plan, code, verify, land) would show which phase to attack; the trial logs only give totals.
- If anyone wants to retest the index later: run it on lanes that start cold (no file list in the order), and compare tokens per **landed** family on at least 30 families per arm, with the engine version held fixed.

## 7. How to reproduce

```
python trial_report.py                     # trial A (trial_log.jsonl)
python trial_report.py trial_log_query.jsonl   # trial B
python evaluate.py                         # retrieval benchmark
python codeindex.py search "colour blind palette" --mode hybrid -k 10
```