administrator/DocForge2
1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions Pages
DocForge2/docs/MILESTONE_0_BASELINE.md

10 KiB
Raw Blame History

DocForge2 Milestone 0 baseline

Milestone 0 measures the inherited implementation before redesign. The evidence supports keeping SQLite and targeting repeated source discovery, parsing, graph validation, and index verification in later milestones. It does not support a speculative storage rewrite.

The maintained machine-readable result is benchmarks/milestone0-2026-07-29.json. The harness is tools/milestone0_baseline.py.

Environment and method

  • Repository revision: fd4759096e90edb13a745621aae4872f23079357.
  • Working tree during the recorded run: clean.
  • Platform: x86-64 Linux 7.1.3 with glibc 2.43.
  • Python: CPython 3.14.6.
  • Pytest: 9.1.1.
  • Ruff: 0.16.0.
  • Pyright: 1.1.411.
  • Node.js: 22.22.2.
  • npm: 10.9.7.
  • Duration clock: time.perf_counter_ns().
  • Response size: UTF-8 bytes of compact, sorted JSON.
  • Standalone memory: GNU /usr/bin/time -v maximum resident set size.
  • Warning policy: repository-configured warnings as errors.

Two generic fixtures and the existing incremental contract fixture were measured. The small alpha fixture has three nodes and two edges. The generated scale fixture has 1,000 Markdown files, 1,000 nodes, 999 dependency edges, one depth-8 context profile with a 32,000-token budget, and one manual view. All fixtures and derived artifacts were disposable and confined to /tmp. No WorldForge, ScrapeStation, production project, or self-hosted DocForge data was used.

The committed 1,000-node run used ten warm samples and three cold samples. The supplementary audit used more repetitions for short operations and separately launched processes for representative memory and startup measurements.

Repository-native gates

The Milestone 0 aggregate is:

make gate

It composes formatting, Python lint, HTML/CSS/JavaScript lint, strict types, compilation, public contract tests, the complete warning-strict test suite, lock validation, npm dependency validation, package building, and a disposable benchmark smoke run.

The candidate gate passed with:

  • Ruff formatting and lint clean across 50 files.
  • Web HTML, rendered-manual HTML, CSS, and JavaScript lint clean.
  • Pyright reporting zero errors, warnings, or informational diagnostics.
  • Python compilation clean.
  • Public-contract gate: 8 tests and 42 schema subtests passed.
  • Complete suite: 95 tests and 44 subtests passed.
  • uv lock --check and npm ls --all passed.
  • Wheel and source distribution built successfully.

Additional audit checks passed: git diff --check, parse validation for all five published JSON schemas, and git fsck --full.

Three-node baseline

These measurements show fixed overhead. They are not evidence of scale behavior.

Operation Median p95 Compact response
Project open 0.624 ms 0.647 ms —
Load, parse, validate, and fingerprint 1.367 ms 1.540 ms —
Full index build 4.177 ms 4.409 ms 665 B
Full index check 1.794 ms 1.868 ms 665 B
Warm no-change synchronize 1.812 ms 2.075 ms 783 B
Exact node 3.800 ms 4.095 ms 704 B
Search 3.903 ms 4.375 ms 1,240 B
Context 5.080 ms 5.452 ms 1,847 B
Atomic manual render 3.542 ms 3.825 ms 753 B
Render status 1.941 ms 2.190 ms 756 B
MCP bootstrap 3.174 ms 3.458 ms 2,458 B
MCP exact node 3.920 ms 4.419 ms 894 B
MCP context 5.326 ms 5.616 ms 2,037 B
Changeset registration 2.572 ms 3.351 ms 1,047 B
Changeset validation 2.642 ms 2.928 ms 1,003 B
Changeset diff 2.751 ms 3.167 ms 1,745 B
Exact-hash apply and refresh 15.989 ms 17.558 ms 3,455 B

The rendered manual was 2,043 bytes. The multi-operation process peaked at 68,644 KiB RSS.

1,000-node maintained baseline

Operation Median p95 Compact response
Project open 0.584 ms 0.607 ms —
Load, parse, validate, and fingerprint 128.650 ms 132.255 ms —
Cold synchronize from missing index 692.605 ms 696.071 ms 870 B
Full index build 277.172 ms 289.531 ms 661 B
Full index check 148.778 ms 157.888 ms 661 B
Warm no-change synchronize 142.479 ms 145.926 ms 779 B
Exact node 286.306 ms 304.226 ms 696 B
Search, limit 20 288.793 ms 301.227 ms 10,312 B
Dependencies, depth 8 287.791 ms 291.129 ms 1,650 B
Impact, depth 8 288.280 ms 290.636 ms 1,650 B
Context, 32,000-token budget 436.897 ms 453.055 ms 258,034 B
Atomic manual render 280.007 ms 283.380 ms 757 B
Current manual render status 150.591 ms 155.833 ms 760 B
MCP bootstrap 273.410 ms 277.901 ms 1,720 B
MCP exact node 287.094 ms 298.913 ms 886 B
MCP search, limit 20 289.737 ms 307.090 ms 10,502 B
MCP context 434.853 ms 447.168 ms 258,224 B
MCP render status 150.758 ms 164.194 ms 950 B
Changeset registration 212.281 ms 215.227 ms 1,013 B
Changeset validation 148.522 ms 162.789 ms 969 B
Changeset diff 148.230 ms 151.704 ms 1,575 B
Exact-hash apply and refresh 1,253.231 ms 1,253.231 ms 3,509 B

The generated manual was 583,150 bytes. The static viewer HTML, CSS, and JavaScript totaled 105,244 bytes. The 258,224-byte MCP context result exceeds the normal 200,000-character project limit. Under the normal policy it correctly becomes a structured result_too_large error rather than a partial response.

The maintained harness reports a cumulative process high-water mark of 528,228 KiB. This includes the entire multi-operation run and its child-process startup samples. The operation-isolated audit is more useful for steady-state memory:

Standalone operation Peak RSS
MCP import and --help 65,568 KiB
Cold synchronization 43,004 KiB
Exact CLI lookup 39,504 KiB
Manual render 39,632 KiB
Context compilation 41,164 KiB
In-memory MCP connection and bootstrap 79,516 KiB
Full 1,000-node operation harness 78,308 KiB

Startup baseline

Fresh-process operation Median Output
CLI info, 3 nodes 80.139 ms 386 B
CLI info, 1,000 nodes 214.045 ms 407 B
CLI exact lookup, 1,000 nodes 369.681 ms 814 B
docforge-mcp --help 299.278 ms 536 B
In-memory MCP create, connect, and list 24.919 ms 31 tools
In-memory MCP bootstrap, 1,000 nodes 273.410 ms 1,720 B

Incremental adapter baseline

The repository's existing two-source incremental test loader is contract evidence, not a scale benchmark.

Operation Median Maximum Response
Cold incremental build 2.567 ms 2.567 ms 892 B
Warm build 1.785 ms 1.864 ms 892 B
Warm no-change synchronize 0.192 ms 0.328 ms 802 B
Exact node 0.463 ms 0.512 ms 574 B
Full/incremental equivalence oracle 0.203 ms 0.341 ms 187 B

The cold build reparsed both sources. The warm build reported two cache hits and no invalidation or reparse. This confirms that the incremental state and attestation path can avoid extraction. A scaled manifest, invalidation, extraction, assembly, and publication benchmark remains missing.

Rendering and live graph baseline

Milestone 0 has a supported generic_html manual renderer and a generation-pinned live graph viewer. It does not have ManualRenderPlan, GraphViewPlan, or a portable graph renderer.

The 1,000-node manual render takes 280.007 ms and emits 583,150 bytes. Render status takes 150.591 ms because it recompiles the complete manual in memory before comparing the expected hash.

Once one validated index generation is pinned, the live viewer shows the actual SQLite read cost:

Operation Median Response
Snapshot pin including validation 143.761 ms —
Overview 1.078 ms 977 B
Search, limit 20 1.363 ms 10,423 B
Neighborhood, depth 8 0.425 ms 4,891 B
Convergence web, depth 8 0.396 ms 5,811 B

Measured bottlenecks

  1. Generic Project.load() walks the source set twice, parses and validates every source, rereads captured files for mutation detection, hashes the generation, and queries the Git revision.
  2. Exact index retrieval validates twice. At 1,000 files it performs about 2,000 Markdown front-matter parses around one bounded SQLite query.
  3. Context compilation validates three times and performs about 3,000 source parses.
  4. The fast attestation path applies to incremental projects. verify_rows=False does not make a generic-project check cheap.
  5. Full index publication intentionally reloads sources to detect concurrent mutation.
  6. Missing-index synchronization compounds failed validation, locked revalidation, build, and final validation.
  7. Render status recompiles the complete manual to derive its expected hash.
  8. Large response construction becomes material before SQLite retrieval does.

The profiler corroborated these paths. In the scale fixture, pinned SQLite operations remain about 0.4–1.4 ms while ordinary exact retrieval remains about 286 ms. Later optimization should first remove redundant full-project work and introduce stable request snapshots or cheap source generations. The evidence does not justify replacing SQLite.

Recorded gaps

Milestone 0 deliberately records these missing measurements and gates:

  • Compiler stages are not separately timed.
  • There is no scaled incremental adapter fixture.
  • No threshold policy yet defines acceptable regressions.
  • There is no zero-source-parse assertion for routine warm reads.
  • Per-tool MCP response-size budgets are not individually frozen.
  • Manual planning is not separated from rendering.
  • Portable graph planning and rendering do not exist.
  • Recovery timing is not maintained for every corruption and degraded-refresh path.
  • End-to-end stdio MCP request latency is not maintained beyond startup.
  • Only Python 3.14 was exercised in this environment.
  • The package declares MIT metadata but has no tracked standalone LICENSE, COPYING, or NOTICE file.

These are inputs to later milestones. They are not permission to expand Milestone 0 into a compiler, renderer, storage, or packaging redesign.