DocForge2/docs/MILESTONE_1_BASELINE.md

# DocForge2 Milestone 1 baseline

Milestone 1 removes repeated whole-project work from routine warm reads while retaining complete
loading and deep validation as recovery and equivalence oracles. The maintained machine-readable
evidence is
[`benchmarks/milestone1-2026-07-29.json`](../benchmarks/milestone1-2026-07-29.json), captured from
clean commit `6253c45a5eca01efa8c73ea3dfe4d85c55878ada`.

## Environment and method

- Platform: x86-64 Linux 7.1.3 with glibc 2.43.
- Python: CPython 3.14.6.
- Fixture: 1,000 Markdown files, 1,000 nodes, and 999 dependency edges.
- Samples: ten measured invocations after validated warmups.
- Duration clock: `time.perf_counter_ns()`.
- Percentile: nearest rank, so p95 is the maximum with ten samples.
- Response size: UTF-8 bytes of compact, sorted JSON.
- Process memory: cumulative main-process `RUSAGE_SELF` high-water mark.

All canonical sources, caches, indexes, changesets, renders, and viewer state were created in a
disposable temporary directory. The run did not read WorldForge, ScrapeStation, legacy DocForge
indexes, or production MCP state.

The benchmark validates every warmup and measured result. It also fails when a routine warm
operation performs a project load, parses canonical source, rebuilds an adapter projection,
extracts adapter sources, builds an index, prepares a render, constructs rendered output, or hashes
complete rendered output.

## Maintained 1,000-node results

| Operation | Median | p95 | Gate | Response |
|---|---:|---:|---:|---:|
| Warm no-change synchronization | 9.192 ms | 9.324 ms | 100 ms | 1,570 B |
| Exact node | 17.887 ms | 18.577 ms | 50 ms | 1,487 B |
| Missing-node error | 18.170 ms | 18.320 ms | 50 ms | 1,130 B |
| Search, limit 20 | 19.518 ms | 19.884 ms | 100 ms | 11,164 B |
| Filter, limit 20 | 18.274 ms | 19.534 ms | 100 ms | 8,572 B |
| Backlinks, limit 20 | 18.039 ms | 18.455 ms | 100 ms | 1,210 B |
| Dependencies, depth 8 | 18.117 ms | 19.061 ms | 100 ms | 2,559 B |
| Impact, depth 8 | 18.059 ms | 18.716 ms | 100 ms | 2,553 B |
| Context, 32,000-token budget, page 20 | 25.395 ms | 25.867 ms | 250 ms | 13,152 B |
| Current render receipt status | 18.969 ms | 19.613 ms | 50 ms | 1,603 B |
| Stale render receipt status | 18.910 ms | 19.087 ms | 50 ms | 1,551 B |
| Missing render receipt status | 17.986 ms | 18.746 ms | 50 ms | 1,352 B |
| Corrupt render receipt status | 18.007 ms | 18.511 ms | 50 ms | 1,352 B |
| Current visualization status | 9.875 ms | 10.836 ms | 50 ms | 1,446 B |
| Stale visualization status | 10.050 ms | 10.472 ms | 50 ms | 1,438 B |
| Not-running visualization status | 0.258 ms | 0.288 ms | 50 ms | 1,008 B |
| Unavailable visualization status | 0.053 ms | 0.081 ms | 50 ms | 1,138 B |

Every measured p95 passed its maintained ceiling. Exact retrieval is 15.4 times faster than the
Milestone 0 median. Warm synchronization is 15.5 times faster. The paged context response is 17.2
times faster and 19.6 times smaller than the inherited full response.

The cumulative process peak was 940,116 KiB. This is not an operation-local steady-state value. It
includes fixture construction, all benchmark phases, and Python allocator high-water behavior. It
excludes the detached visualization worker. Milestone 0's isolated subprocess measurements remain
the better evidence for per-operation steady-state memory until a maintained operation-local memory
harness is added.

## Work-proof counters

Routine retrieval and status operations recorded:

- Zero complete project loads.
- Zero canonical files or bytes parsed.
- Zero adapter projection loads and source extractions.
- Zero index builds.
- Zero render preparations, output bytes constructed, or complete output bytes hashed.
- One index check and two cheap source-generation checks for each pinned retrieval.
- One viewer-manager request for each running visualization-status query.

No-change synchronization recorded one synchronization and no build. Receipt and visualization
status recorded no hidden synchronization. The counter contract is fixed, schema-validated, and
executed by the repository gate.

## Meaning of the result

The Milestone 0 evidence showed that SQLite queries were already fast after a generation was
pinned. Milestone 1 confirms that repeated source discovery, parsing, and validation were the
dominant cost. A versioned source-generation receipt, immutable SQLite read snapshot, and bounded
indexed operations remove that cost without changing graph authority or storage.

The evidence still does not justify replacing SQLite. Complete project loading, complete index
checking, full adapter projection, and deep render validation remain independent truth and recovery
oracles.

## Known limits

- The context compiler still materializes its bounded selected graph before transport pagination.
  A streaming planner requires separate scale evidence.
- Generic stat identities are cheap publication proofs, not cryptographic integrity scans.
- Legacy non-incremental adapters may not provide a cheap generation identity.
- Incremental adapter manifest, invalidation, extraction, and assembly are not yet measured at
  1,000-source scale.
- Pagination cursors detect corruption and stale generations. They are not authenticated
  authorization tokens.
- An individually oversized context entry is returned as explicit hash-identified omission
  evidence. Targeted retrieval is required for its content.
- The maintained process peak is cumulative and excludes detached worker memory.
- Manual and graph render plans do not exist until Milestone 3.