administrator/DocForge2
1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions Pages

Record Milestone 0 performance baseline

Browse source
This commit is contained in:
Andraxion 2026-07-29 03:20:09 -04:00
parent fd4759096e
commit cb59a822a8
5 changed files with 542 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

19
ACTIVE_SLICE.md
View file

@ -1,15 +1,12 @@
# Active slice
# Active milestone
```text
Slice: DFG-23 process-stable adapter implementation boundary
Goal: Prevent a long-running project server from synchronizing with adapter code or configuration that changed after the adapter object was imported.
In scope: Language-neutral implementation roots/files/suffixes; project-local Python package inference; descriptor fingerprinting; bounded change evidence; restart-required MCP remediation; staged/unstaged deletion contract guidance.
Out of scope: In-process Python module reloading; MCP self-restart; project-specific Git enumeration; canonical source mutation; deployment or publication.
Done when: Every MCP operation rejects added, changed, deleted, missing, or unsafe adapter implementation files before synchronization; derived files outside the declared boundary remain ignored; descriptor changes require restart; deletion semantics remain manifest-owned and staging-independent; the complete DocForge quality gate passes.
Owners: DocForge owns implementation-boundary confinement, fingerprinting, bounded diagnostics, MCP preflight, and restart remediation. Adapters own the declared boundary and current source-manifest enumeration. Canonical sources and Git staging remain outside this lifecycle guard.
Proof: The focused adapter suite passed 15 tests, including inferred and explicit implementation boundaries, descriptor changes, additions, edits, deletions, ignored derived files, and exact MCP remediation. Strict Pyright passed with no errors or warnings. Ruff lint and formatting, Python compilation, and the HTML/CSS/JavaScript quality gate passed. The complete warning-strict suite passed 87 tests and 2 subtests.
Milestone: 0 — successor foundation and measured baseline
Goal: Seed DocForge2 from the most advanced local lineage without breaking DocForge v1 contracts.
In scope: Complete Git lineage; verified no-AST work; adapter lifecycle safeguards; compatibility guarantees; repository-native contract and quality gates; cold/warm, memory, rendering, and response-size baselines; public successor migration; fresh-clone verification.
Out of scope: Storage redesign; compiler or renderer redesign; portable render plans; self-hosting; production MCP repointing; WorldForge or ScrapeStation changes; tags and releases.
Done when: administrator/DocForge2 is public and seeded from the advanced clean tree; administrator/DocForge remains intact; origin and legacy identify the successor and v1 remotes; all gates and a fresh-clone proof pass; the recorded baseline identifies measured bottlenecks without speculative optimization.
Status: Candidate. Source integration, contracts, gates, and measurements are complete. Public repository creation, remote migration, and fresh-clone verification remain.
```
**Next gate:** Prove Worldforge's C++ integration against the generic incremental and assembly
contracts. Extract a reusable language frontend only after a second consumer demonstrates which
behavior is genuinely shared.
No later milestone is active.

4
README.md
View file

@ -131,6 +131,8 @@ DocForge describes them as a source graph.
- [Core contract](docs/CONTRACT.md) — invariants and security boundary.
- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter,
schema, changeset, rendering, and no-AST guarantees.
- [Milestone 0 baseline](docs/MILESTONE_0_BASELINE.md) — validation evidence, cold and warm
performance, memory, rendering and response sizes, bottlenecks, and missing coverage.
- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary.
- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle.
- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical
@ -154,4 +156,6 @@ make gate
Focused entry points are available as `make contract`, `make test`, `make type`,
`make benchmark-smoke`, and `make benchmark`.
The committed 1,000-node baseline and its measurement method are under `benchmarks/`.
See [AGENTS.md](AGENTS.md) before changing core boundaries.

29
benchmarks/README.md Normal file
View file

@ -0,0 +1,29 @@
# Benchmarks
Milestone 0 records measurements before changing compiler, storage, rendering, or response
contracts.
Run the maintained smoke benchmark:
```bash
make benchmark-smoke
```
Run the 1,000-node generic baseline:
```bash
make benchmark
```
The benchmark creates canonical sources, derived state, changesets, rendered output, and caches
only in a disposable temporary directory. It does not read another project, self-host DocForge, or
mutate repository content.
`milestone0-2026-07-29.json` is the clean-tree baseline captured from commit
`fd4759096e90edb13a745621aae4872f23079357`. It uses compact sorted JSON for response sizes and
`time.perf_counter_ns()` for durations. The file is data, not a performance threshold. Later work
must explain fixture or environment changes before comparing results.
The generic fixture exposes whole-source scaling. It does not replace the incremental adapter
equivalence tests and does not claim to measure a portable graph renderer, because Milestone 0 has
no portable graph-planning or graph-rendering contract.

282
benchmarks/milestone0-2026-07-29.json Normal file
View file

@ -0,0 +1,282 @@
{
"benchmark": "docforge2_milestone0",
"environment": {
"implementation": "CPython",
"machine": "x86_64",
"platform": "Linux-7.1.3-200.nobara.fc44.x86_64-x86_64-with-glibc2.43",
"python": "3.14.6"
},
"fixture": {
"context_budget_tokens": 32000,
"edge_count": 999,
"kind": "synthetic_generic",
"node_count": 1000,
"source_file_count": 1000,
"traversal_depth": 8
},
"known_gaps": [
"Generic warm reads still parse canonical source files.",
"Compiler stages are not separately instrumented.",
"Scaled incremental extraction is not measured by this generic fixture.",
"Manual planning is not separated from rendering.",
"Portable graph planning and rendering do not exist in Milestone 0.",
"Per-operation peak RSS requires an external process harness."
],
"method": {
"clock": "time.perf_counter_ns",
"cold_samples": 3,
"memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss",
"response_size": "UTF-8 bytes of compact sorted JSON",
"samples": 10
},
"operations": {
"changeset_diff": {
"max_ms": 151.704,
"median_ms": 148.23,
"min_ms": 145.184,
"p95_ms": 151.704,
"response_bytes": 1575,
"samples": 10
},
"changeset_register": {
"max_ms": 215.227,
"median_ms": 212.281,
"min_ms": 210.251,
"p95_ms": 215.227,
"response_bytes": 1013,
"samples": 10
},
"changeset_validate": {
"max_ms": 162.789,
"median_ms": 148.522,
"min_ms": 145.624,
"p95_ms": 162.789,
"response_bytes": 969,
"samples": 10
},
"cli_exact_startup": {
"max_ms": 370.763,
"median_ms": 369.681,
"min_ms": 368.637,
"p95_ms": 370.763,
"response_bytes": 814,
"samples": 3
},
"cli_info_startup": {
"max_ms": 215.402,
"median_ms": 214.045,
"min_ms": 212.262,
"p95_ms": 215.402,
"response_bytes": 407,
"samples": 3
},
"cold_synchronize": {
"max_ms": 696.071,
"median_ms": 692.605,
"min_ms": 689.365,
"p95_ms": 696.071,
"response_bytes": 870,
"samples": 3
},
"context_32k": {
"max_ms": 453.055,
"median_ms": 436.897,
"min_ms": 427.958,
"p95_ms": 453.055,
"response_bytes": 258034,
"samples": 10
},
"dependencies_depth_8": {
"max_ms": 291.129,
"median_ms": 287.791,
"min_ms": 286.129,
"p95_ms": 291.129,
"response_bytes": 1650,
"samples": 10
},
"exact_hash_apply_and_refresh": {
"max_ms": 1253.231,
"median_ms": 1253.231,
"min_ms": 1253.231,
"p95_ms": 1253.231,
"response_bytes": 3509,
"samples": 1
},
"exact_node": {
"max_ms": 304.226,
"median_ms": 286.306,
"min_ms": 283.013,
"p95_ms": 304.226,
"response_bytes": 696,
"samples": 10
},
"full_index_build": {
"max_ms": 289.531,
"median_ms": 277.172,
"min_ms": 276.361,
"p95_ms": 289.531,
"response_bytes": 661,
"samples": 3
},
"full_index_check": {
"max_ms": 157.888,
"median_ms": 148.778,
"min_ms": 143.19,
"p95_ms": 157.888,
"response_bytes": 661,
"samples": 10
},
"impact_depth_8": {
"max_ms": 290.636,
"median_ms": 288.28,
"min_ms": 285.26,
"p95_ms": 290.636,
"response_bytes": 1650,
"samples": 10
},
"manual_render": {
"max_ms": 283.38,
"median_ms": 280.007,
"min_ms": 279.406,
"p95_ms": 283.38,
"response_bytes": 757,
"samples": 3
},
"manual_render_status": {
"max_ms": 155.833,
"median_ms": 150.591,
"min_ms": 148.979,
"p95_ms": 155.833,
"response_bytes": 760,
"samples": 10
},
"mcp_bootstrap": {
"max_ms": 277.901,
"median_ms": 273.41,
"min_ms": 271.248,
"p95_ms": 277.901,
"response_bytes": 1720,
"samples": 10
},
"mcp_context_32k": {
"max_ms": 447.168,
"median_ms": 434.853,
"min_ms": 430.848,
"p95_ms": 447.168,
"response_bytes": 258224,
"samples": 10
},
"mcp_exact_node": {
"max_ms": 298.913,
"median_ms": 287.094,
"min_ms": 284.413,
"p95_ms": 298.913,
"response_bytes": 886,
"samples": 10
},
"mcp_import_and_help": {
"max_ms": 300.449,
"median_ms": 299.278,
"min_ms": 297.873,
"p95_ms": 300.449,
"response_bytes": 536,
"samples": 3
},
"mcp_render_status": {
"max_ms": 164.194,
"median_ms": 150.758,
"min_ms": 147.522,
"p95_ms": 164.194,
"response_bytes": 950,
"samples": 10
},
"mcp_search_limit_20": {
"max_ms": 307.09,
"median_ms": 289.737,
"min_ms": 284.139,
"p95_ms": 307.09,
"response_bytes": 10502,
"samples": 10
},
"project_load": {
"max_ms": 132.255,
"median_ms": 128.65,
"min_ms": 127.489,
"p95_ms": 132.255,
"samples": 10
},
"project_open": {
"max_ms": 0.607,
"median_ms": 0.584,
"min_ms": 0.568,
"p95_ms": 0.607,
"samples": 10
},
"search_limit_20": {
"max_ms": 301.227,
"median_ms": 288.793,
"min_ms": 286.57,
"p95_ms": 301.227,
"response_bytes": 10312,
"samples": 10
},
"viewer_neighborhood_depth_8": {
"max_ms": 0.46,
"median_ms": 0.425,
"min_ms": 0.41,
"p95_ms": 0.46,
"response_bytes": 4891,
"samples": 10
},
"viewer_overview": {
"max_ms": 1.124,
"median_ms": 1.078,
"min_ms": 1.048,
"p95_ms": 1.124,
"response_bytes": 977,
"samples": 10
},
"viewer_search_limit_20": {
"max_ms": 1.389,
"median_ms": 1.363,
"min_ms": 1.344,
"p95_ms": 1.389,
"response_bytes": 10423,
"samples": 10
},
"viewer_snapshot_pin": {
"max_ms": 143.9,
"median_ms": 143.761,
"min_ms": 143.569,
"p95_ms": 143.9,
"samples": 3
},
"viewer_web_depth_8": {
"max_ms": 0.899,
"median_ms": 0.396,
"min_ms": 0.353,
"p95_ms": 0.899,
"response_bytes": 5811,
"samples": 10
},
"warm_no_change_synchronize": {
"max_ms": 145.926,
"median_ms": 142.479,
"min_ms": 140.151,
"p95_ms": 145.926,
"response_bytes": 779,
"samples": 10
}
},
"process_peak_rss_kib": 528228,
"schema_version": 1,
"sizes": {
"context_compact_bytes": 258034,
"manual_artifact_bytes": 583150,
"static_viewer_assets_bytes": 105244
},
"source": {
"dirty": false,
"revision": "fd4759096e90edb13a745621aae4872f23079357"
}
}

219
docs/MILESTONE_0_BASELINE.md Normal file
View file

@ -0,0 +1,219 @@
# DocForge2 Milestone 0 baseline
Milestone 0 measures the inherited implementation before redesign. The evidence supports keeping
SQLite and targeting repeated source discovery, parsing, graph validation, and index verification
in later milestones. It does not support a speculative storage rewrite.
The maintained machine-readable result is
[`benchmarks/milestone0-2026-07-29.json`](../benchmarks/milestone0-2026-07-29.json). The harness is
[`tools/milestone0_baseline.py`](../tools/milestone0_baseline.py).
## Environment and method
- Repository revision: `fd4759096e90edb13a745621aae4872f23079357`.
- Working tree during the recorded run: clean.
- Platform: x86-64 Linux 7.1.3 with glibc 2.43.
- Python: CPython 3.14.6.
- Pytest: 9.1.1.
- Ruff: 0.16.0.
- Pyright: 1.1.411.
- Node.js: 22.22.2.
- npm: 10.9.7.
- Duration clock: `time.perf_counter_ns()`.
- Response size: UTF-8 bytes of compact, sorted JSON.
- Standalone memory: GNU `/usr/bin/time -v` maximum resident set size.
- Warning policy: repository-configured warnings as errors.
Two generic fixtures and the existing incremental contract fixture were measured. The small
`alpha` fixture has three nodes and two edges. The generated scale fixture has 1,000 Markdown
files, 1,000 nodes, 999 dependency edges, one depth-8 context profile with a 32,000-token budget,
and one manual view. All fixtures and derived artifacts were disposable and confined to `/tmp`.
No WorldForge, ScrapeStation, production project, or self-hosted DocForge data was used.
The committed 1,000-node run used ten warm samples and three cold samples. The supplementary audit
used more repetitions for short operations and separately launched processes for representative
memory and startup measurements.
## Repository-native gates
The Milestone 0 aggregate is:
```bash
make gate
```
It composes formatting, Python lint, HTML/CSS/JavaScript lint, strict types, compilation, public
contract tests, the complete warning-strict test suite, lock validation, npm dependency validation,
package building, and a disposable benchmark smoke run.
The candidate gate passed with:
- Ruff formatting and lint clean across 50 files.
- Web HTML, rendered-manual HTML, CSS, and JavaScript lint clean.
- Pyright reporting zero errors, warnings, or informational diagnostics.
- Python compilation clean.
- Public-contract gate: 8 tests and 42 schema subtests passed.
- Complete suite: 95 tests and 44 subtests passed.
- `uv lock --check` and `npm ls --all` passed.
- Wheel and source distribution built successfully.
Additional audit checks passed: `git diff --check`, parse validation for all five published JSON
schemas, and `git fsck --full`.
## Three-node baseline
These measurements show fixed overhead. They are not evidence of scale behavior.
| Operation | Median | p95 | Compact response |
|---|---:|---:|---:|
| Project open | 0.624 ms | 0.647 ms | — |
| Load, parse, validate, and fingerprint | 1.367 ms | 1.540 ms | — |
| Full index build | 4.177 ms | 4.409 ms | 665 B |
| Full index check | 1.794 ms | 1.868 ms | 665 B |
| Warm no-change synchronize | 1.812 ms | 2.075 ms | 783 B |
| Exact node | 3.800 ms | 4.095 ms | 704 B |
| Search | 3.903 ms | 4.375 ms | 1,240 B |
| Context | 5.080 ms | 5.452 ms | 1,847 B |
| Atomic manual render | 3.542 ms | 3.825 ms | 753 B |
| Render status | 1.941 ms | 2.190 ms | 756 B |
| MCP bootstrap | 3.174 ms | 3.458 ms | 2,458 B |
| MCP exact node | 3.920 ms | 4.419 ms | 894 B |
| MCP context | 5.326 ms | 5.616 ms | 2,037 B |
| Changeset registration | 2.572 ms | 3.351 ms | 1,047 B |
| Changeset validation | 2.642 ms | 2.928 ms | 1,003 B |
| Changeset diff | 2.751 ms | 3.167 ms | 1,745 B |
| Exact-hash apply and refresh | 15.989 ms | 17.558 ms | 3,455 B |
The rendered manual was 2,043 bytes. The multi-operation process peaked at 68,644 KiB RSS.
## 1,000-node maintained baseline
| Operation | Median | p95 | Compact response |
|---|---:|---:|---:|
| Project open | 0.584 ms | 0.607 ms | — |
| Load, parse, validate, and fingerprint | 128.650 ms | 132.255 ms | — |
| Cold synchronize from missing index | 692.605 ms | 696.071 ms | 870 B |
| Full index build | 277.172 ms | 289.531 ms | 661 B |
| Full index check | 148.778 ms | 157.888 ms | 661 B |
| Warm no-change synchronize | 142.479 ms | 145.926 ms | 779 B |
| Exact node | 286.306 ms | 304.226 ms | 696 B |
| Search, limit 20 | 288.793 ms | 301.227 ms | 10,312 B |
| Dependencies, depth 8 | 287.791 ms | 291.129 ms | 1,650 B |
| Impact, depth 8 | 288.280 ms | 290.636 ms | 1,650 B |
| Context, 32,000-token budget | 436.897 ms | 453.055 ms | 258,034 B |
| Atomic manual render | 280.007 ms | 283.380 ms | 757 B |
| Current manual render status | 150.591 ms | 155.833 ms | 760 B |
| MCP bootstrap | 273.410 ms | 277.901 ms | 1,720 B |
| MCP exact node | 287.094 ms | 298.913 ms | 886 B |
| MCP search, limit 20 | 289.737 ms | 307.090 ms | 10,502 B |
| MCP context | 434.853 ms | 447.168 ms | 258,224 B |
| MCP render status | 150.758 ms | 164.194 ms | 950 B |
| Changeset registration | 212.281 ms | 215.227 ms | 1,013 B |
| Changeset validation | 148.522 ms | 162.789 ms | 969 B |
| Changeset diff | 148.230 ms | 151.704 ms | 1,575 B |
| Exact-hash apply and refresh | 1,253.231 ms | 1,253.231 ms | 3,509 B |
The generated manual was 583,150 bytes. The static viewer HTML, CSS, and JavaScript totaled
105,244 bytes. The 258,224-byte MCP context result exceeds the normal 200,000-character project
limit. Under the normal policy it correctly becomes a structured `result_too_large` error rather
than a partial response.
The maintained harness reports a cumulative process high-water mark of 528,228 KiB. This includes
the entire multi-operation run and its child-process startup samples. The operation-isolated audit
is more useful for steady-state memory:
| Standalone operation | Peak RSS |
|---|---:|
| MCP import and `--help` | 65,568 KiB |
| Cold synchronization | 43,004 KiB |
| Exact CLI lookup | 39,504 KiB |
| Manual render | 39,632 KiB |
| Context compilation | 41,164 KiB |
| In-memory MCP connection and bootstrap | 79,516 KiB |
| Full 1,000-node operation harness | 78,308 KiB |
## Startup baseline
| Fresh-process operation | Median | Output |
|---|---:|---:|
| CLI `info`, 3 nodes | 80.139 ms | 386 B |
| CLI `info`, 1,000 nodes | 214.045 ms | 407 B |
| CLI exact lookup, 1,000 nodes | 369.681 ms | 814 B |
| `docforge-mcp --help` | 299.278 ms | 536 B |
| In-memory MCP create, connect, and list | 24.919 ms | 31 tools |
| In-memory MCP bootstrap, 1,000 nodes | 273.410 ms | 1,720 B |
## Incremental adapter baseline
The repository's existing two-source incremental test loader is contract evidence, not a scale
benchmark.
| Operation | Median | Maximum | Response |
|---|---:|---:|---:|
| Cold incremental build | 2.567 ms | 2.567 ms | 892 B |
| Warm build | 1.785 ms | 1.864 ms | 892 B |
| Warm no-change synchronize | 0.192 ms | 0.328 ms | 802 B |
| Exact node | 0.463 ms | 0.512 ms | 574 B |
| Full/incremental equivalence oracle | 0.203 ms | 0.341 ms | 187 B |
The cold build reparsed both sources. The warm build reported two cache hits and no invalidation or
reparse. This confirms that the incremental state and attestation path can avoid extraction. A
scaled manifest, invalidation, extraction, assembly, and publication benchmark remains missing.
## Rendering and live graph baseline
Milestone 0 has a supported `generic_html` manual renderer and a generation-pinned live graph
viewer. It does not have `ManualRenderPlan`, `GraphViewPlan`, or a portable graph renderer.
The 1,000-node manual render takes 280.007 ms and emits 583,150 bytes. Render status takes
150.591 ms because it recompiles the complete manual in memory before comparing the expected hash.
Once one validated index generation is pinned, the live viewer shows the actual SQLite read cost:
| Operation | Median | Response |
|---|---:|---:|
| Snapshot pin including validation | 143.761 ms | — |
| Overview | 1.078 ms | 977 B |
| Search, limit 20 | 1.363 ms | 10,423 B |
| Neighborhood, depth 8 | 0.425 ms | 4,891 B |
| Convergence web, depth 8 | 0.396 ms | 5,811 B |
## Measured bottlenecks
1. Generic `Project.load()` walks the source set twice, parses and validates every source, rereads
captured files for mutation detection, hashes the generation, and queries the Git revision.
2. Exact index retrieval validates twice. At 1,000 files it performs about 2,000 Markdown
front-matter parses around one bounded SQLite query.
3. Context compilation validates three times and performs about 3,000 source parses.
4. The fast attestation path applies to incremental projects. `verify_rows=False` does not make a
generic-project check cheap.
5. Full index publication intentionally reloads sources to detect concurrent mutation.
6. Missing-index synchronization compounds failed validation, locked revalidation, build, and
final validation.
7. Render status recompiles the complete manual to derive its expected hash.
8. Large response construction becomes material before SQLite retrieval does.
The profiler corroborated these paths. In the scale fixture, pinned SQLite operations remain about
0.4–1.4 ms while ordinary exact retrieval remains about 286 ms. Later optimization should first
remove redundant full-project work and introduce stable request snapshots or cheap source
generations. The evidence does not justify replacing SQLite.
## Recorded gaps
Milestone 0 deliberately records these missing measurements and gates:
- Compiler stages are not separately timed.
- There is no scaled incremental adapter fixture.
- No threshold policy yet defines acceptable regressions.
- There is no zero-source-parse assertion for routine warm reads.
- Per-tool MCP response-size budgets are not individually frozen.
- Manual planning is not separated from rendering.
- Portable graph planning and rendering do not exist.
- Recovery timing is not maintained for every corruption and degraded-refresh path.
- End-to-end stdio MCP request latency is not maintained beyond startup.
- Only Python 3.14 was exercised in this environment.
- The package declares MIT metadata but has no tracked standalone `LICENSE`, `COPYING`, or
`NOTICE` file.
These are inputs to later milestones. They are not permission to expand Milestone 0 into a
compiler, renderer, storage, or packaging redesign.