3.3 KiB
Benchmarks
Milestone 0 records measurements before changing compiler, storage, rendering, or response contracts.
Run the maintained smoke benchmark:
make benchmark-smoke
Run the 1,000-node generic baseline:
make benchmark
Run the Milestone 1 warm-operation counter and latency smoke gate:
make benchmark-m1-smoke
Run the maintained 1,000-node Milestone 1 benchmark:
make benchmark-m1
Run the Milestone 2 agent-workflow smoke and full gates:
make benchmark-m2-smoke
make benchmark-m2
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches only in a disposable temporary directory. It does not read another project, self-host DocForge, or mutate repository content.
milestone0-2026-07-29.json is the clean-tree baseline captured from commit
fd4759096e90edb13a745621aae4872f23079357. It uses compact sorted JSON for response sizes and
time.perf_counter_ns() for durations. The file is data, not a performance threshold. Later work
must explain fixture or environment changes before comparing results.
milestone1-2026-07-29.json is the clean-tree fast-core baseline captured from commit
6253c45a5eca01efa8c73ea3dfe4d85c55878ada. Unlike the historical baseline, the Milestone 1
harness enforces operation-specific p95 ceilings and fixed zero-work counter invariants. Its
human-readable interpretation is in
docs/MILESTONE_1_BASELINE.md.
milestone2-2026-07-29.json is the clean-tree agent-retrieval and client-integration baseline
captured from commit fb0df5e4a1c591c2a84788fd4814d98550f11863. It gates every warmup and
sample, reconstructs complete task-context and generation-diff collections across bounded pages,
records whether diagnostics were dropped for response budget, checks all hidden-work counters,
and measures isolated-process peak RSS. Its interpretation is in
docs/MILESTONE_2_BASELINE.md.
The generic fixture exposes whole-source scaling. It does not replace the incremental adapter equivalence tests and does not claim to measure a portable graph renderer, because Milestone 0 has no portable graph-planning or graph-rendering contract.
The Milestone 1 harness treats wall time and structured work counters as separate gates. Warm operations fail if they load a complete project, parse source files, reconstruct an adapter projection, extract adapter sources, build an index, prepare a render, construct rendered output, or hash complete rendered output. Its latency ceilings are the Milestone 1 targets, not claims about all hardware.
Every warmup and measured invocation is validated. The recorded counter ranges also require one
index synchronization for the synchronization operation, no hidden synchronization for reads and
status, one index check for each retrieval snapshot, and exactly one manager request for viewer
status. The 1,000-node run records bounded semantic summaries for exact errors, search, filtering,
backlinks, both traversal directions, paged context, render receipt states, and visualization
freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum.
process_peak_rss_kib is the cumulative main-process RUSAGE_SELF high-water mark and excludes the
detached viewer worker.