108 lines
4.8 KiB
Markdown
108 lines
4.8 KiB
Markdown
# Benchmarks
|
|
|
|
Milestone 0 records measurements before changing compiler, storage, rendering, or response
|
|
contracts.
|
|
|
|
Run the maintained smoke benchmark:
|
|
|
|
```bash
|
|
make benchmark-smoke
|
|
```
|
|
|
|
Run the 1,000-node generic baseline:
|
|
|
|
```bash
|
|
make benchmark
|
|
```
|
|
|
|
Run the Milestone 1 warm-operation counter and latency smoke gate:
|
|
|
|
```bash
|
|
make benchmark-m1-smoke
|
|
```
|
|
|
|
Run the maintained 1,000-node Milestone 1 benchmark:
|
|
|
|
```bash
|
|
make benchmark-m1
|
|
```
|
|
|
|
Run the Milestone 2 agent-workflow smoke and full gates:
|
|
|
|
```bash
|
|
make benchmark-m2-smoke
|
|
make benchmark-m2
|
|
```
|
|
|
|
Run the Milestone 3 independent-projection smoke and maintained full gates:
|
|
|
|
```bash
|
|
make benchmark-m3-smoke
|
|
make benchmark-m3
|
|
make benchmark-m3-full
|
|
```
|
|
|
|
Run the Milestone 4 adapter SDK smoke, maintained full, and fresh-wheel adoption gates:
|
|
|
|
```bash
|
|
make benchmark-m4-smoke
|
|
make benchmark-m4-full
|
|
make adoption-m4
|
|
```
|
|
|
|
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches
|
|
only in a disposable temporary directory. It does not read another project, self-host DocForge, or
|
|
mutate repository content.
|
|
|
|
`milestone0-2026-07-29.json` is the clean-tree baseline captured from commit
|
|
`fd4759096e90edb13a745621aae4872f23079357`. It uses compact sorted JSON for response sizes and
|
|
`time.perf_counter_ns()` for durations. The file is data, not a performance threshold. Later work
|
|
must explain fixture or environment changes before comparing results.
|
|
|
|
`milestone1-2026-07-29.json` is the clean-tree fast-core baseline captured from commit
|
|
`6253c45a5eca01efa8c73ea3dfe4d85c55878ada`. Unlike the historical baseline, the Milestone 1
|
|
harness enforces operation-specific p95 ceilings and fixed zero-work counter invariants. Its
|
|
human-readable interpretation is in
|
|
[`docs/MILESTONE_1_BASELINE.md`](../docs/MILESTONE_1_BASELINE.md).
|
|
|
|
`milestone2-2026-07-29.json` is the clean-tree agent-retrieval and client-integration baseline
|
|
captured from commit `fb0df5e4a1c591c2a84788fd4814d98550f11863`. It gates every warmup and
|
|
sample, reconstructs complete task-context and generation-diff collections across bounded pages,
|
|
records whether diagnostics were dropped for response budget, checks all hidden-work counters,
|
|
and measures isolated-process peak RSS. Its interpretation is in
|
|
[`docs/MILESTONE_2_BASELINE.md`](../docs/MILESTONE_2_BASELINE.md).
|
|
|
|
`milestone3-2026-07-29.json` is the clean-tree independent-projection baseline captured from commit
|
|
`f5dccb5e1c312121f1af63780162f593d9363b98`. It measures versioned manual and graph planning,
|
|
in-process and detached rendering, production cold/warm/forced-full behavior, fragment cache
|
|
sweeps, add/change/delete/reorder equivalence, portable publication, receipt-only status, traced
|
|
memory, detached worker peak memory, and response size. Its interpretation is in
|
|
[`docs/MILESTONE_3_BASELINE.md`](../docs/MILESTONE_3_BASELINE.md).
|
|
|
|
`milestone4-2026-07-29.json` is the clean-tree adapter SDK and reference-Python baseline captured
|
|
from commit `95271dcf2e48045b9d3aed9b9ea09c7fc155692c`. It measures cold and warm
|
|
incremental builds, complete/incremental graph-plus-Logic equivalence, corrupt extraction-cache
|
|
recovery, corrupt-index recovery, parser and extraction counters, traced and process memory, and
|
|
bounded response size. Its interpretation is in
|
|
[`docs/MILESTONE_4_BASELINE.md`](../docs/MILESTONE_4_BASELINE.md).
|
|
|
|
The generic fixtures expose whole-source and projection scaling. They do not replace incremental
|
|
adapter equivalence tests. Milestone 3's portable fixture contains 1,000 Nodes/Flow/Web nodes and
|
|
999 edges; portable version 1 deliberately excludes Logic.
|
|
|
|
The Milestone 1 harness treats wall time and structured work counters as separate gates. Warm
|
|
operations fail if they load a complete project, parse source files, reconstruct an adapter
|
|
projection, extract adapter sources, build an index, prepare a render, construct rendered output,
|
|
or hash complete rendered output. Its latency ceilings are the Milestone 1 targets, not claims
|
|
about all hardware.
|
|
|
|
Every warmup and measured invocation is validated. The recorded counter ranges also require one
|
|
index synchronization for the synchronization operation, no hidden synchronization for reads and
|
|
status, one index check for each retrieval snapshot, and exactly one manager request for viewer
|
|
status. The 1,000-node run records bounded semantic summaries for exact errors, search, filtering,
|
|
backlinks, both traversal directions, paged context, render receipt states, and visualization
|
|
freshness. The reported p95 uses the nearest-rank method; with ten samples it is the maximum.
|
|
`process_peak_rss_kib` is the cumulative main-process `RUSAGE_SELF` high-water mark and excludes the
|
|
detached viewer worker. It is diagnostic and not operation-local. Milestone 3 memory gates use
|
|
per-operation `tracemalloc` peaks and detached worker receipt peaks instead. Milestone 4 uses
|
|
per-operation `tracemalloc` peaks and a cumulative benchmark-process high-water gate.
|