219 lines
10 KiB
Markdown
219 lines
10 KiB
Markdown
# DocForge2 Milestone 0 baseline
|
||
|
||
Milestone 0 measures the inherited implementation before redesign. The evidence supports keeping
|
||
SQLite and targeting repeated source discovery, parsing, graph validation, and index verification
|
||
in later milestones. It does not support a speculative storage rewrite.
|
||
|
||
The maintained machine-readable result is
|
||
[`benchmarks/milestone0-2026-07-29.json`](../benchmarks/milestone0-2026-07-29.json). The harness is
|
||
[`tools/milestone0_baseline.py`](../tools/milestone0_baseline.py).
|
||
|
||
## Environment and method
|
||
|
||
- Repository revision: `fd4759096e90edb13a745621aae4872f23079357`.
|
||
- Working tree during the recorded run: clean.
|
||
- Platform: x86-64 Linux 7.1.3 with glibc 2.43.
|
||
- Python: CPython 3.14.6.
|
||
- Pytest: 9.1.1.
|
||
- Ruff: 0.16.0.
|
||
- Pyright: 1.1.411.
|
||
- Node.js: 22.22.2.
|
||
- npm: 10.9.7.
|
||
- Duration clock: `time.perf_counter_ns()`.
|
||
- Response size: UTF-8 bytes of compact, sorted JSON.
|
||
- Standalone memory: GNU `/usr/bin/time -v` maximum resident set size.
|
||
- Warning policy: repository-configured warnings as errors.
|
||
|
||
Two generic fixtures and the existing incremental contract fixture were measured. The small
|
||
`alpha` fixture has three nodes and two edges. The generated scale fixture has 1,000 Markdown
|
||
files, 1,000 nodes, 999 dependency edges, one depth-8 context profile with a 32,000-token budget,
|
||
and one manual view. All fixtures and derived artifacts were disposable and confined to `/tmp`.
|
||
No WorldForge, ScrapeStation, production project, or self-hosted DocForge data was used.
|
||
|
||
The committed 1,000-node run used ten warm samples and three cold samples. The supplementary audit
|
||
used more repetitions for short operations and separately launched processes for representative
|
||
memory and startup measurements.
|
||
|
||
## Repository-native gates
|
||
|
||
The Milestone 0 aggregate is:
|
||
|
||
```bash
|
||
make gate
|
||
```
|
||
|
||
It composes formatting, Python lint, HTML/CSS/JavaScript lint, strict types, compilation, public
|
||
contract tests, the complete warning-strict test suite, lock validation, npm dependency validation,
|
||
package building, and a disposable benchmark smoke run.
|
||
|
||
The candidate gate passed with:
|
||
|
||
- Ruff formatting and lint clean across 50 files.
|
||
- Web HTML, rendered-manual HTML, CSS, and JavaScript lint clean.
|
||
- Pyright reporting zero errors, warnings, or informational diagnostics.
|
||
- Python compilation clean.
|
||
- Public-contract gate: 8 tests and 42 schema subtests passed.
|
||
- Complete suite: 95 tests and 44 subtests passed.
|
||
- `uv lock --check` and `npm ls --all` passed.
|
||
- Wheel and source distribution built successfully.
|
||
|
||
Additional audit checks passed: `git diff --check`, parse validation for all five published JSON
|
||
schemas, and `git fsck --full`.
|
||
|
||
## Three-node baseline
|
||
|
||
These measurements show fixed overhead. They are not evidence of scale behavior.
|
||
|
||
| Operation | Median | p95 | Compact response |
|
||
|---|---:|---:|---:|
|
||
| Project open | 0.624 ms | 0.647 ms | — |
|
||
| Load, parse, validate, and fingerprint | 1.367 ms | 1.540 ms | — |
|
||
| Full index build | 4.177 ms | 4.409 ms | 665 B |
|
||
| Full index check | 1.794 ms | 1.868 ms | 665 B |
|
||
| Warm no-change synchronize | 1.812 ms | 2.075 ms | 783 B |
|
||
| Exact node | 3.800 ms | 4.095 ms | 704 B |
|
||
| Search | 3.903 ms | 4.375 ms | 1,240 B |
|
||
| Context | 5.080 ms | 5.452 ms | 1,847 B |
|
||
| Atomic manual render | 3.542 ms | 3.825 ms | 753 B |
|
||
| Render status | 1.941 ms | 2.190 ms | 756 B |
|
||
| MCP bootstrap | 3.174 ms | 3.458 ms | 2,458 B |
|
||
| MCP exact node | 3.920 ms | 4.419 ms | 894 B |
|
||
| MCP context | 5.326 ms | 5.616 ms | 2,037 B |
|
||
| Changeset registration | 2.572 ms | 3.351 ms | 1,047 B |
|
||
| Changeset validation | 2.642 ms | 2.928 ms | 1,003 B |
|
||
| Changeset diff | 2.751 ms | 3.167 ms | 1,745 B |
|
||
| Exact-hash apply and refresh | 15.989 ms | 17.558 ms | 3,455 B |
|
||
|
||
The rendered manual was 2,043 bytes. The multi-operation process peaked at 68,644 KiB RSS.
|
||
|
||
## 1,000-node maintained baseline
|
||
|
||
| Operation | Median | p95 | Compact response |
|
||
|---|---:|---:|---:|
|
||
| Project open | 0.584 ms | 0.607 ms | — |
|
||
| Load, parse, validate, and fingerprint | 128.650 ms | 132.255 ms | — |
|
||
| Cold synchronize from missing index | 692.605 ms | 696.071 ms | 870 B |
|
||
| Full index build | 277.172 ms | 289.531 ms | 661 B |
|
||
| Full index check | 148.778 ms | 157.888 ms | 661 B |
|
||
| Warm no-change synchronize | 142.479 ms | 145.926 ms | 779 B |
|
||
| Exact node | 286.306 ms | 304.226 ms | 696 B |
|
||
| Search, limit 20 | 288.793 ms | 301.227 ms | 10,312 B |
|
||
| Dependencies, depth 8 | 287.791 ms | 291.129 ms | 1,650 B |
|
||
| Impact, depth 8 | 288.280 ms | 290.636 ms | 1,650 B |
|
||
| Context, 32,000-token budget | 436.897 ms | 453.055 ms | 258,034 B |
|
||
| Atomic manual render | 280.007 ms | 283.380 ms | 757 B |
|
||
| Current manual render status | 150.591 ms | 155.833 ms | 760 B |
|
||
| MCP bootstrap | 273.410 ms | 277.901 ms | 1,720 B |
|
||
| MCP exact node | 287.094 ms | 298.913 ms | 886 B |
|
||
| MCP search, limit 20 | 289.737 ms | 307.090 ms | 10,502 B |
|
||
| MCP context | 434.853 ms | 447.168 ms | 258,224 B |
|
||
| MCP render status | 150.758 ms | 164.194 ms | 950 B |
|
||
| Changeset registration | 212.281 ms | 215.227 ms | 1,013 B |
|
||
| Changeset validation | 148.522 ms | 162.789 ms | 969 B |
|
||
| Changeset diff | 148.230 ms | 151.704 ms | 1,575 B |
|
||
| Exact-hash apply and refresh | 1,253.231 ms | 1,253.231 ms | 3,509 B |
|
||
|
||
The generated manual was 583,150 bytes. The static viewer HTML, CSS, and JavaScript totaled
|
||
105,244 bytes. The 258,224-byte MCP context result exceeds the normal 200,000-character project
|
||
limit. Under the normal policy it correctly becomes a structured `result_too_large` error rather
|
||
than a partial response.
|
||
|
||
The maintained harness reports a cumulative process high-water mark of 528,228 KiB. This includes
|
||
the entire multi-operation run and its child-process startup samples. The operation-isolated audit
|
||
is more useful for steady-state memory:
|
||
|
||
| Standalone operation | Peak RSS |
|
||
|---|---:|
|
||
| MCP import and `--help` | 65,568 KiB |
|
||
| Cold synchronization | 43,004 KiB |
|
||
| Exact CLI lookup | 39,504 KiB |
|
||
| Manual render | 39,632 KiB |
|
||
| Context compilation | 41,164 KiB |
|
||
| In-memory MCP connection and bootstrap | 79,516 KiB |
|
||
| Full 1,000-node operation harness | 78,308 KiB |
|
||
|
||
## Startup baseline
|
||
|
||
| Fresh-process operation | Median | Output |
|
||
|---|---:|---:|
|
||
| CLI `info`, 3 nodes | 80.139 ms | 386 B |
|
||
| CLI `info`, 1,000 nodes | 214.045 ms | 407 B |
|
||
| CLI exact lookup, 1,000 nodes | 369.681 ms | 814 B |
|
||
| `docforge-mcp --help` | 299.278 ms | 536 B |
|
||
| In-memory MCP create, connect, and list | 24.919 ms | 31 tools |
|
||
| In-memory MCP bootstrap, 1,000 nodes | 273.410 ms | 1,720 B |
|
||
|
||
## Incremental adapter baseline
|
||
|
||
The repository's existing two-source incremental test loader is contract evidence, not a scale
|
||
benchmark.
|
||
|
||
| Operation | Median | Maximum | Response |
|
||
|---|---:|---:|---:|
|
||
| Cold incremental build | 2.567 ms | 2.567 ms | 892 B |
|
||
| Warm build | 1.785 ms | 1.864 ms | 892 B |
|
||
| Warm no-change synchronize | 0.192 ms | 0.328 ms | 802 B |
|
||
| Exact node | 0.463 ms | 0.512 ms | 574 B |
|
||
| Full/incremental equivalence oracle | 0.203 ms | 0.341 ms | 187 B |
|
||
|
||
The cold build reparsed both sources. The warm build reported two cache hits and no invalidation or
|
||
reparse. This confirms that the incremental state and attestation path can avoid extraction. A
|
||
scaled manifest, invalidation, extraction, assembly, and publication benchmark remains missing.
|
||
|
||
## Rendering and live graph baseline
|
||
|
||
Milestone 0 has a supported `generic_html` manual renderer and a generation-pinned live graph
|
||
viewer. It does not have `ManualRenderPlan`, `GraphViewPlan`, or a portable graph renderer.
|
||
|
||
The 1,000-node manual render takes 280.007 ms and emits 583,150 bytes. Render status takes
|
||
150.591 ms because it recompiles the complete manual in memory before comparing the expected hash.
|
||
|
||
Once one validated index generation is pinned, the live viewer shows the actual SQLite read cost:
|
||
|
||
| Operation | Median | Response |
|
||
|---|---:|---:|
|
||
| Snapshot pin including validation | 143.761 ms | — |
|
||
| Overview | 1.078 ms | 977 B |
|
||
| Search, limit 20 | 1.363 ms | 10,423 B |
|
||
| Neighborhood, depth 8 | 0.425 ms | 4,891 B |
|
||
| Convergence web, depth 8 | 0.396 ms | 5,811 B |
|
||
|
||
## Measured bottlenecks
|
||
|
||
1. Generic `Project.load()` walks the source set twice, parses and validates every source, rereads
|
||
captured files for mutation detection, hashes the generation, and queries the Git revision.
|
||
2. Exact index retrieval validates twice. At 1,000 files it performs about 2,000 Markdown
|
||
front-matter parses around one bounded SQLite query.
|
||
3. Context compilation validates three times and performs about 3,000 source parses.
|
||
4. The fast attestation path applies to incremental projects. `verify_rows=False` does not make a
|
||
generic-project check cheap.
|
||
5. Full index publication intentionally reloads sources to detect concurrent mutation.
|
||
6. Missing-index synchronization compounds failed validation, locked revalidation, build, and
|
||
final validation.
|
||
7. Render status recompiles the complete manual to derive its expected hash.
|
||
8. Large response construction becomes material before SQLite retrieval does.
|
||
|
||
The profiler corroborated these paths. In the scale fixture, pinned SQLite operations remain about
|
||
0.4–1.4 ms while ordinary exact retrieval remains about 286 ms. Later optimization should first
|
||
remove redundant full-project work and introduce stable request snapshots or cheap source
|
||
generations. The evidence does not justify replacing SQLite.
|
||
|
||
## Recorded gaps
|
||
|
||
Milestone 0 deliberately records these missing measurements and gates:
|
||
|
||
- Compiler stages are not separately timed.
|
||
- There is no scaled incremental adapter fixture.
|
||
- No threshold policy yet defines acceptable regressions.
|
||
- There is no zero-source-parse assertion for routine warm reads.
|
||
- Per-tool MCP response-size budgets are not individually frozen.
|
||
- Manual planning is not separated from rendering.
|
||
- Portable graph planning and rendering do not exist.
|
||
- Recovery timing is not maintained for every corruption and degraded-refresh path.
|
||
- End-to-end stdio MCP request latency is not maintained beyond startup.
|
||
- Only Python 3.14 was exercised in this environment.
|
||
- The package declares MIT metadata but has no tracked standalone `LICENSE`, `COPYING`, or
|
||
`NOTICE` file.
|
||
|
||
These are inputs to later milestones. They are not permission to expand Milestone 0 into a
|
||
compiler, renderer, storage, or packaging redesign.
|