administrator/DocForge2
1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions Pages

Record Milestone 0 performance baseline

Browse source
This commit is contained in:
Andraxion 2026-07-29 03:20:09 -04:00
parent fd4759096e
commit cb59a822a8
5 changed files with 542 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

219
docs/MILESTONE_0_BASELINE.md Normal file
View file

@ -0,0 +1,219 @@
# DocForge2 Milestone 0 baseline
Milestone 0 measures the inherited implementation before redesign. The evidence supports keeping
SQLite and targeting repeated source discovery, parsing, graph validation, and index verification
in later milestones. It does not support a speculative storage rewrite.
The maintained machine-readable result is
[`benchmarks/milestone0-2026-07-29.json`](../benchmarks/milestone0-2026-07-29.json). The harness is
[`tools/milestone0_baseline.py`](../tools/milestone0_baseline.py).
## Environment and method
- Repository revision: `fd4759096e90edb13a745621aae4872f23079357`.
- Working tree during the recorded run: clean.
- Platform: x86-64 Linux 7.1.3 with glibc 2.43.
- Python: CPython 3.14.6.
- Pytest: 9.1.1.
- Ruff: 0.16.0.
- Pyright: 1.1.411.
- Node.js: 22.22.2.
- npm: 10.9.7.
- Duration clock: `time.perf_counter_ns()`.
- Response size: UTF-8 bytes of compact, sorted JSON.
- Standalone memory: GNU `/usr/bin/time -v` maximum resident set size.
- Warning policy: repository-configured warnings as errors.
Two generic fixtures and the existing incremental contract fixture were measured. The small
`alpha` fixture has three nodes and two edges. The generated scale fixture has 1,000 Markdown
files, 1,000 nodes, 999 dependency edges, one depth-8 context profile with a 32,000-token budget,
and one manual view. All fixtures and derived artifacts were disposable and confined to `/tmp`.
No WorldForge, ScrapeStation, production project, or self-hosted DocForge data was used.
The committed 1,000-node run used ten warm samples and three cold samples. The supplementary audit
used more repetitions for short operations and separately launched processes for representative
memory and startup measurements.
## Repository-native gates
The Milestone 0 aggregate is:
```bash
make gate
```
It composes formatting, Python lint, HTML/CSS/JavaScript lint, strict types, compilation, public
contract tests, the complete warning-strict test suite, lock validation, npm dependency validation,
package building, and a disposable benchmark smoke run.
The candidate gate passed with:
- Ruff formatting and lint clean across 50 files.
- Web HTML, rendered-manual HTML, CSS, and JavaScript lint clean.
- Pyright reporting zero errors, warnings, or informational diagnostics.
- Python compilation clean.
- Public-contract gate: 8 tests and 42 schema subtests passed.
- Complete suite: 95 tests and 44 subtests passed.
- `uv lock --check` and `npm ls --all` passed.
- Wheel and source distribution built successfully.
Additional audit checks passed: `git diff --check`, parse validation for all five published JSON
schemas, and `git fsck --full`.
## Three-node baseline
These measurements show fixed overhead. They are not evidence of scale behavior.
| Operation | Median | p95 | Compact response |
|---|---:|---:|---:|
| Project open | 0.624 ms | 0.647 ms | — |
| Load, parse, validate, and fingerprint | 1.367 ms | 1.540 ms | — |
| Full index build | 4.177 ms | 4.409 ms | 665 B |
| Full index check | 1.794 ms | 1.868 ms | 665 B |
| Warm no-change synchronize | 1.812 ms | 2.075 ms | 783 B |
| Exact node | 3.800 ms | 4.095 ms | 704 B |
| Search | 3.903 ms | 4.375 ms | 1,240 B |
| Context | 5.080 ms | 5.452 ms | 1,847 B |
| Atomic manual render | 3.542 ms | 3.825 ms | 753 B |
| Render status | 1.941 ms | 2.190 ms | 756 B |
| MCP bootstrap | 3.174 ms | 3.458 ms | 2,458 B |
| MCP exact node | 3.920 ms | 4.419 ms | 894 B |
| MCP context | 5.326 ms | 5.616 ms | 2,037 B |
| Changeset registration | 2.572 ms | 3.351 ms | 1,047 B |
| Changeset validation | 2.642 ms | 2.928 ms | 1,003 B |
| Changeset diff | 2.751 ms | 3.167 ms | 1,745 B |
| Exact-hash apply and refresh | 15.989 ms | 17.558 ms | 3,455 B |
The rendered manual was 2,043 bytes. The multi-operation process peaked at 68,644 KiB RSS.
## 1,000-node maintained baseline
| Operation | Median | p95 | Compact response |
|---|---:|---:|---:|
| Project open | 0.584 ms | 0.607 ms | — |
| Load, parse, validate, and fingerprint | 128.650 ms | 132.255 ms | — |
| Cold synchronize from missing index | 692.605 ms | 696.071 ms | 870 B |
| Full index build | 277.172 ms | 289.531 ms | 661 B |
| Full index check | 148.778 ms | 157.888 ms | 661 B |
| Warm no-change synchronize | 142.479 ms | 145.926 ms | 779 B |
| Exact node | 286.306 ms | 304.226 ms | 696 B |
| Search, limit 20 | 288.793 ms | 301.227 ms | 10,312 B |
| Dependencies, depth 8 | 287.791 ms | 291.129 ms | 1,650 B |
| Impact, depth 8 | 288.280 ms | 290.636 ms | 1,650 B |
| Context, 32,000-token budget | 436.897 ms | 453.055 ms | 258,034 B |
| Atomic manual render | 280.007 ms | 283.380 ms | 757 B |
| Current manual render status | 150.591 ms | 155.833 ms | 760 B |
| MCP bootstrap | 273.410 ms | 277.901 ms | 1,720 B |
| MCP exact node | 287.094 ms | 298.913 ms | 886 B |
| MCP search, limit 20 | 289.737 ms | 307.090 ms | 10,502 B |
| MCP context | 434.853 ms | 447.168 ms | 258,224 B |
| MCP render status | 150.758 ms | 164.194 ms | 950 B |
| Changeset registration | 212.281 ms | 215.227 ms | 1,013 B |
| Changeset validation | 148.522 ms | 162.789 ms | 969 B |
| Changeset diff | 148.230 ms | 151.704 ms | 1,575 B |
| Exact-hash apply and refresh | 1,253.231 ms | 1,253.231 ms | 3,509 B |
The generated manual was 583,150 bytes. The static viewer HTML, CSS, and JavaScript totaled
105,244 bytes. The 258,224-byte MCP context result exceeds the normal 200,000-character project
limit. Under the normal policy it correctly becomes a structured `result_too_large` error rather
than a partial response.
The maintained harness reports a cumulative process high-water mark of 528,228 KiB. This includes
the entire multi-operation run and its child-process startup samples. The operation-isolated audit
is more useful for steady-state memory:
| Standalone operation | Peak RSS |
|---|---:|
| MCP import and `--help` | 65,568 KiB |
| Cold synchronization | 43,004 KiB |
| Exact CLI lookup | 39,504 KiB |
| Manual render | 39,632 KiB |
| Context compilation | 41,164 KiB |
| In-memory MCP connection and bootstrap | 79,516 KiB |
| Full 1,000-node operation harness | 78,308 KiB |
## Startup baseline
| Fresh-process operation | Median | Output |
|---|---:|---:|
| CLI `info`, 3 nodes | 80.139 ms | 386 B |
| CLI `info`, 1,000 nodes | 214.045 ms | 407 B |
| CLI exact lookup, 1,000 nodes | 369.681 ms | 814 B |
| `docforge-mcp --help` | 299.278 ms | 536 B |
| In-memory MCP create, connect, and list | 24.919 ms | 31 tools |
| In-memory MCP bootstrap, 1,000 nodes | 273.410 ms | 1,720 B |
## Incremental adapter baseline
The repository's existing two-source incremental test loader is contract evidence, not a scale
benchmark.
| Operation | Median | Maximum | Response |
|---|---:|---:|---:|
| Cold incremental build | 2.567 ms | 2.567 ms | 892 B |
| Warm build | 1.785 ms | 1.864 ms | 892 B |
| Warm no-change synchronize | 0.192 ms | 0.328 ms | 802 B |
| Exact node | 0.463 ms | 0.512 ms | 574 B |
| Full/incremental equivalence oracle | 0.203 ms | 0.341 ms | 187 B |
The cold build reparsed both sources. The warm build reported two cache hits and no invalidation or
reparse. This confirms that the incremental state and attestation path can avoid extraction. A
scaled manifest, invalidation, extraction, assembly, and publication benchmark remains missing.
## Rendering and live graph baseline
Milestone 0 has a supported `generic_html` manual renderer and a generation-pinned live graph
viewer. It does not have `ManualRenderPlan`, `GraphViewPlan`, or a portable graph renderer.
The 1,000-node manual render takes 280.007 ms and emits 583,150 bytes. Render status takes
150.591 ms because it recompiles the complete manual in memory before comparing the expected hash.
Once one validated index generation is pinned, the live viewer shows the actual SQLite read cost:
| Operation | Median | Response |
|---|---:|---:|
| Snapshot pin including validation | 143.761 ms | — |
| Overview | 1.078 ms | 977 B |
| Search, limit 20 | 1.363 ms | 10,423 B |
| Neighborhood, depth 8 | 0.425 ms | 4,891 B |
| Convergence web, depth 8 | 0.396 ms | 5,811 B |
## Measured bottlenecks
1. Generic `Project.load()` walks the source set twice, parses and validates every source, rereads
captured files for mutation detection, hashes the generation, and queries the Git revision.
2. Exact index retrieval validates twice. At 1,000 files it performs about 2,000 Markdown
front-matter parses around one bounded SQLite query.
3. Context compilation validates three times and performs about 3,000 source parses.
4. The fast attestation path applies to incremental projects. `verify_rows=False` does not make a
generic-project check cheap.
5. Full index publication intentionally reloads sources to detect concurrent mutation.
6. Missing-index synchronization compounds failed validation, locked revalidation, build, and
final validation.
7. Render status recompiles the complete manual to derive its expected hash.
8. Large response construction becomes material before SQLite retrieval does.
The profiler corroborated these paths. In the scale fixture, pinned SQLite operations remain about
0.4–1.4 ms while ordinary exact retrieval remains about 286 ms. Later optimization should first
remove redundant full-project work and introduce stable request snapshots or cheap source
generations. The evidence does not justify replacing SQLite.
## Recorded gaps
Milestone 0 deliberately records these missing measurements and gates:
- Compiler stages are not separately timed.
- There is no scaled incremental adapter fixture.
- No threshold policy yet defines acceptable regressions.
- There is no zero-source-parse assertion for routine warm reads.
- Per-tool MCP response-size budgets are not individually frozen.
- Manual planning is not separated from rendering.
- Portable graph planning and rendering do not exist.
- Recovery timing is not maintained for every corruption and degraded-refresh path.
- End-to-end stdio MCP request latency is not maintained beyond startup.
- Only Python 3.14 was exercised in this environment.
- The package declares MIT metadata but has no tracked standalone `LICENSE`, `COPYING`, or
`NOTICE` file.
These are inputs to later milestones. They are not permission to expand Milestone 0 into a
compiler, renderer, storage, or packaging redesign.