86 lines
4 KiB
Markdown
86 lines
4 KiB
Markdown
|
# Milestone 2 baseline
|
||
|
|
|
||
|
|
## Scope and method
|
||
|
|
|
||
|
|
This baseline records the agent-retrieval and client-integration behavior added in Milestone 2.
|
||
|
|
It was captured on 2026-07-29 from clean candidate commit
|
||
|
|
`fb0df5e4a1c591c2a84788fd4814d98550f11863`.
|
||
|
|
|
||
|
|
The maintained command was:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
.venv/bin/python tools/milestone2_benchmark.py \
|
||
|
|
--nodes 1000 \
|
||
|
|
--samples 10 \
|
||
|
|
--output /tmp/docforge-milestone2-final.json
|
||
|
|
```
|
||
|
|
|
||
|
|
The fixture contains 1,000 Markdown nodes and 999 edges in a direct fan-in around one focus node.
|
||
|
|
The configured MCP response limit is 200,000 characters. Durations use
|
||
|
|
`time.perf_counter_ns()` and nearest-rank p95. Peak memory uses an isolated child process and
|
||
|
|
`RUSAGE_SELF`. Every warmup and measured invocation is validated.
|
||
|
|
|
||
|
|
Environment:
|
||
|
|
|
||
|
|
- Linux 7.1.3-200.nobara.fc44.x86_64.
|
||
|
|
- CPython 3.14.6.
|
||
|
|
- x86_64.
|
||
|
|
- Ten warm samples after one warmup.
|
||
|
|
- Isolated memory ceiling: 262,144 KiB.
|
||
|
|
|
||
|
|
The complete machine-readable result is
|
||
|
|
[`benchmarks/milestone2-2026-07-29.json`](../benchmarks/milestone2-2026-07-29.json).
|
||
|
|
|
||
|
|
## Results
|
||
|
|
|
||
|
|
| Operation | Median | p95 | Limit | Maximum response |
|
||
|
|
|---|---:|---:|---:|---:|
|
||
|
|
| Read bootstrap | 9.406 ms | 9.884 ms | 100 ms | 7,495 B |
|
||
|
|
| No-AST bootstrap | 9.153 ms | 9.379 ms | 100 ms | 8,796 B |
|
||
|
|
| Task diagnostic page | 73.326 ms | 93.115 ms | 500 ms | 7,190 B |
|
||
|
|
| Task complete traversal | 703.561 ms | 721.847 ms | 2,500 ms | 151,172 B/page |
|
||
|
|
| Generation diagnostic page | 40.961 ms | 41.650 ms | 100 ms | 66,516 B |
|
||
|
|
| Generation maximum page | 55.565 ms | 59.184 ms | 100 ms | 199,566 B |
|
||
|
|
| Generation complete traversal | 418.607 ms | 425.315 ms | 500 ms | 66,516 B/page |
|
||
|
|
| Codex configuration preview | 314.365 ms | 364.383 ms | 500 ms | 2,627 B |
|
||
|
|
| Claude configuration preview | 314.326 ms | 364.365 ms | 500 ms | 2,748 B |
|
||
|
|
| OpenClaw configuration preview | 314.401 ms | 364.532 ms | 500 ms | 2,869 B |
|
||
|
|
| Codex doctor | 0.421 ms | 0.556 ms | 100 ms | 3,595 B |
|
||
|
|
| Claude doctor | 0.364 ms | 0.446 ms | 100 ms | 3,669 B |
|
||
|
|
| OpenClaw doctor | 0.384 ms | 0.484 ms | 100 ms | 3,602 B |
|
||
|
|
|
||
|
|
Isolated peak RSS was 86,448 KiB.
|
||
|
|
|
||
|
|
Task traversal returned 108 evidence records and 892 explicit omissions across 11 pages. One
|
||
|
|
individually oversized focus record became a response-limit surrogate bound to the original record
|
||
|
|
hash. The remaining omissions were token-budget evidence. The benchmark verified every unique
|
||
|
|
subject, reconstructed the original collection hash, and matched the exact 1,000-node fixture.
|
||
|
|
|
||
|
|
Generation traversal returned all 1,000 changed-node details across 10 pages. It reconstructed the
|
||
|
|
stored retained-collection hash. The maximum generation page approached the response limit and
|
||
|
|
proved that optional diagnostics were dropped before the primary result.
|
||
|
|
|
||
|
|
## Structured-work gates
|
||
|
|
|
||
|
|
Configuration preview and doctor performed zero project loads, source parses, adapter projection
|
||
|
|
loads, adapter extraction, index checks, synchronization, index builds, render preparation,
|
||
|
|
rendered-byte construction or hashing, and viewer-manager requests.
|
||
|
|
|
||
|
|
Task-context pages performed exactly one index check and two cheap source-generation checks. They
|
||
|
|
performed none of the hidden work above. Generation-diff pages performed exactly two cheap
|
||
|
|
source-generation checks and no index check or hidden work.
|
||
|
|
|
||
|
|
## Measured limits and future notes
|
||
|
|
|
||
|
|
- Continuation is stateless and regenerates the task capsule for each page. The complete
|
||
|
|
11-page traversal remains within its gate, but later work can avoid repeated planning without
|
||
|
|
weakening generation binding.
|
||
|
|
- Configuration preview deliberately spends about 314 ms proving that the exact isolated
|
||
|
|
interpreter can import the MCP module. Discovery-only checks were rejected as unsafe.
|
||
|
|
- Claude configuration syntax is supported, but its timeout representation remains unverified.
|
||
|
|
Doctor therefore reports degraded rather than healthy.
|
||
|
|
- Doctor is a configuration inspector, not an MCP connection or SQLite integrity test.
|
||
|
|
- Legacy adapters without cheap source-generation identity report unknown for generation-diff
|
||
|
|
freshness.
|
||
|
|
- The results do not justify a storage rewrite. SQLite remains fast after one generation is pinned.
|