DocForge2/benchmarks

Benchmarks

Milestone 0 records measurements before changing compiler, storage, rendering, or response contracts.

Run the maintained smoke benchmark:

make benchmark-smoke

Run the 1,000-node generic baseline:

make benchmark

Run the Milestone 1 warm-operation counter and latency smoke gate:

make benchmark-m1-smoke

Run the maintained 1,000-node Milestone 1 benchmark:

make benchmark-m1

Run the Milestone 2 agent-workflow smoke and full gates:

make benchmark-m2-smoke
make benchmark-m2

Run the Milestone 3 independent-projection smoke and maintained full gates:

make benchmark-m3-smoke
make benchmark-m3
make benchmark-m3-full

Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates:

make benchmark-m4-smoke
make benchmark-m4-full
make adoption-m4

The benchmark creates canonical sources, derived state, changesets, rendered output, and caches only in a disposable temporary directory. It does not read another project, self-host DocForge, or mutate repository content.

milestone0-2026-07-29.json is the clean-tree baseline captured from commit fd4759096e90edb13a745621aae4872f23079357. It uses compact sorted JSON for response sizes and time.perf_counter_ns() for durations. The file is data, not a performance threshold. Later work must explain fixture or environment changes before comparing results.

milestone1-2026-07-29.json is the clean-tree fast-core baseline captured from commit 6253c45a5eca01efa8c73ea3dfe4d85c55878ada. Unlike the historical baseline, the Milestone 1 harness enforces operation-specific p95 ceilings and fixed zero-work counter invariants. Its human-readable interpretation is in docs/MILESTONE_1_BASELINE.md.

milestone2-2026-07-29.json is the clean-tree agent-retrieval and client-integration baseline captured from commit fb0df5e4a1c591c2a84788fd4814d98550f11863. It gates every warmup and sample, reconstructs complete task-context and generation-diff collections across bounded pages, records whether diagnostics were dropped for response budget, checks all hidden-work counters, and measures isolated-process peak RSS. Its interpretation is in docs/MILESTONE_2_BASELINE.md.

milestone3-2026-07-29.json is the clean-tree independent-projection baseline captured from commit f5dccb5e1c312121f1af63780162f593d9363b98. It measures versioned manual and graph planning, in-process and detached rendering, production cold/warm/forced-full behavior, fragment cache sweeps, add/change/delete/reorder equivalence, portable publication, receipt-only status, traced memory, detached worker peak memory, and response size. Its interpretation is in docs/MILESTONE_3_BASELINE.md.

milestone4-2026-07-29.json is the clean-tree adapter SDK and reference-Python baseline captured from commit 95271dcf2e48045b9d3aed9b9ea09c7fc155692c. It measures cold and warm incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and bounded response size. Its interpretation is in docs/MILESTONE_4_BASELINE.md.

The generic fixtures expose whole-source and projection scaling. They do not replace incremental adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and 999 edges; portable version 1 deliberately excludes Logic.

The Milestone 1 harness treats wall time and structured work counters as separate gates. Warm operations fail if they load a complete project, parse source files, reconstruct an adapter projection, extract adapter sources, build an index, prepare a render, construct rendered output, or hash complete rendered output. Its latency ceilings are the Milestone 1 targets, not claims about all hardware.

Every warmup and measured invocation is validated. The recorded counter ranges also require one index synchronization for the synchronization operation, no hidden synchronization for reads and status, one index check for each retrieval snapshot, and exactly one manager request for viewer status. The 1,000-node run records bounded semantic summaries for exact errors, search, filtering, backlinks, both traversal directions, paged context, render receipt states, and visualization freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum. process_peak_rss_kib is the cumulative main-process RUSAGE_SELF high-water mark and excludes the detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use per-operation tracemalloc peaks and detached worker receipt peaks instead. Milestone 4 uses per-operation tracemalloc peaks and a cumulative benchmark-process high-water gate.