102 lines
5.4 KiB
Markdown
102 lines
5.4 KiB
Markdown
# DocForge2 Milestone 1 baseline
|
|
|
|
Milestone 1 removes repeated whole-project work from routine warm reads while retaining complete
|
|
loading and deep validation as recovery and equivalence oracles. The maintained machine-readable
|
|
evidence is
|
|
[`benchmarks/milestone1-2026-07-29.json`](../benchmarks/milestone1-2026-07-29.json), captured from
|
|
clean commit `6253c45a5eca01efa8c73ea3dfe4d85c55878ada`.
|
|
|
|
## Environment and method
|
|
|
|
- Platform: x86-64 Linux 7.1.3 with glibc 2.43.
|
|
- Python: CPython 3.14.6.
|
|
- Fixture: 1,000 Markdown files, 1,000 nodes, and 999 dependency edges.
|
|
- Samples: ten measured invocations after validated warmups.
|
|
- Duration clock: `time.perf_counter_ns()`.
|
|
- Percentile: nearest rank, so p95 is the maximum with ten samples.
|
|
- Response size: UTF-8 bytes of compact, sorted JSON.
|
|
- Process memory: cumulative main-process `RUSAGE_SELF` high-water mark.
|
|
|
|
All canonical sources, caches, indexes, changesets, renders, and viewer state were created in a
|
|
disposable temporary directory. The run did not read WorldForge, ScrapeStation, legacy DocForge
|
|
indexes, or production MCP state.
|
|
|
|
The benchmark validates every warmup and measured result. It also fails when a routine warm
|
|
operation performs a project load, parses canonical source, rebuilds an adapter projection,
|
|
extracts adapter sources, builds an index, prepares a render, constructs rendered output, or hashes
|
|
complete rendered output.
|
|
|
|
## Maintained 1,000-node results
|
|
|
|
| Operation | Median | p95 | Gate | Response |
|
|
|---|---:|---:|---:|---:|
|
|
| Warm no-change synchronization | 9.192 ms | 9.324 ms | 100 ms | 1,570 B |
|
|
| Exact node | 17.887 ms | 18.577 ms | 50 ms | 1,487 B |
|
|
| Missing-node error | 18.170 ms | 18.320 ms | 50 ms | 1,130 B |
|
|
| Search, limit 20 | 19.518 ms | 19.884 ms | 100 ms | 11,164 B |
|
|
| Filter, limit 20 | 18.274 ms | 19.534 ms | 100 ms | 8,572 B |
|
|
| Backlinks, limit 20 | 18.039 ms | 18.455 ms | 100 ms | 1,210 B |
|
|
| Dependencies, depth 8 | 18.117 ms | 19.061 ms | 100 ms | 2,559 B |
|
|
| Impact, depth 8 | 18.059 ms | 18.716 ms | 100 ms | 2,553 B |
|
|
| Context, 32,000-token budget, page 20 | 25.395 ms | 25.867 ms | 250 ms | 13,152 B |
|
|
| Current render receipt status | 18.969 ms | 19.613 ms | 50 ms | 1,603 B |
|
|
| Stale render receipt status | 18.910 ms | 19.087 ms | 50 ms | 1,551 B |
|
|
| Missing render receipt status | 17.986 ms | 18.746 ms | 50 ms | 1,352 B |
|
|
| Corrupt render receipt status | 18.007 ms | 18.511 ms | 50 ms | 1,352 B |
|
|
| Current visualization status | 9.875 ms | 10.836 ms | 50 ms | 1,446 B |
|
|
| Stale visualization status | 10.050 ms | 10.472 ms | 50 ms | 1,438 B |
|
|
| Not-running visualization status | 0.258 ms | 0.288 ms | 50 ms | 1,008 B |
|
|
| Unavailable visualization status | 0.053 ms | 0.081 ms | 50 ms | 1,138 B |
|
|
|
|
Every measured p95 passed its maintained ceiling. Exact retrieval is 15.4 times faster than the
|
|
Milestone 0 median. Warm synchronization is 15.5 times faster. The paged context response is 17.2
|
|
times faster and 19.6 times smaller than the inherited full response.
|
|
|
|
The cumulative process peak was 940,116 KiB. This is not an operation-local steady-state value. It
|
|
includes fixture construction, all benchmark phases, and Python allocator high-water behavior. It
|
|
excludes the detached visualization worker. Milestone 0's isolated subprocess measurements remain
|
|
the better evidence for per-operation steady-state memory until a maintained operation-local memory
|
|
harness is added.
|
|
|
|
## Work-proof counters
|
|
|
|
Routine retrieval and status operations recorded:
|
|
|
|
- Zero complete project loads.
|
|
- Zero canonical files or bytes parsed.
|
|
- Zero adapter projection loads and source extractions.
|
|
- Zero index builds.
|
|
- Zero render preparations, output bytes constructed, or complete output bytes hashed.
|
|
- One index check and two cheap source-generation checks for each pinned retrieval.
|
|
- One viewer-manager request for each running visualization-status query.
|
|
|
|
No-change synchronization recorded one synchronization and no build. Receipt and visualization
|
|
status recorded no hidden synchronization. The counter contract is fixed, schema-validated, and
|
|
executed by the repository gate.
|
|
|
|
## Meaning of the result
|
|
|
|
The Milestone 0 evidence showed that SQLite queries were already fast after a generation was
|
|
pinned. Milestone 1 confirms that repeated source discovery, parsing, and validation were the
|
|
dominant cost. A versioned source-generation receipt, immutable SQLite read snapshot, and bounded
|
|
indexed operations remove that cost without changing graph authority or storage.
|
|
|
|
The evidence still does not justify replacing SQLite. Complete project loading, complete index
|
|
checking, full adapter projection, and deep render validation remain independent truth and recovery
|
|
oracles.
|
|
|
|
## Known limits
|
|
|
|
- The context compiler still materializes its bounded selected graph before transport pagination.
|
|
A streaming planner requires separate scale evidence.
|
|
- Generic stat identities are cheap publication proofs, not cryptographic integrity scans.
|
|
- Legacy non-incremental adapters may not provide a cheap generation identity.
|
|
- Incremental adapter manifest, invalidation, extraction, and assembly are not yet measured at
|
|
1,000-source scale.
|
|
- Pagination cursors detect corruption and stale generations. They are not authenticated
|
|
authorization tokens.
|
|
- An individually oversized context entry is returned as explicit hash-identified omission
|
|
evidence. Targeted retrieval is required for its content.
|
|
- The maintained process peak is cumulative and excludes detached worker memory.
|
|
- Manual and graph render plans do not exist until Milestone 3.
|
|
|