DocForge2/DEVELOPMENT_NOTES.md

DocForge2 development notes

This is the running implementation record for DocForge2. It records what is active, what was measured, what changed, what failed, why architectural decisions were made, and which ideas were deferred. Stable user and compatibility contracts still belong in dedicated documentation.

Working rules

• Only one milestone is active at a time.
• main remains the last fully verified milestone.
• Active implementation occurs on dev.
• Every milestone begins from direct repository evidence and ends with focused tests, the complete repository gate, updated measurements, documentation closeout, and a clean pushed state.
• WorldForge, ScrapeStation, legacy DocForge, and production MCP bindings remain out of scope.
• DocForge2 does not self-host during this program.
• Release tags and Forgejo releases require Rob's explicit approval.

Milestone 0 — complete

Milestone 0 established the public successor, preserved the complete lineage and v1 tag, integrated the no-AST and adapter-lifecycle work, froze compatibility guarantees, added repository-native quality and contract gates, and recorded cold/warm performance, memory, rendering, and response sizes.

The central measurement was decisive: a 1,000-node warm exact lookup took about 286 ms while the generation-pinned SQLite query path took about 0.4–1.4 ms. Repeated whole-source loading and validation, not SQLite, is the first optimization target.

Milestone 1 — active: fast, observable core

Outcome

Warm retrieval should disappear into normal tool overhead. Routine reads must not parse project sources. Status must not render or rebuild hidden work. Results must remain bounded independently of project size.

Starting evidence

• Generic Project.load() walks, captures, parses, rereads, validates, hashes, and checks Git for the complete source set.
• Exact retrieval validates twice around one bounded SQLite query.
• Context compilation performs three full project loads.
• Render status recompiles the complete manual.
• Incremental adapters already prove that manifest attestation can make no-change synchronization and exact retrieval sub-millisecond on a tiny fixture.
• Pinned viewer queries prove the current SQLite schema can serve bounded reads quickly.

Current work

1. Audit request-scoped immutable snapshot and persistent-generation options.
2. Audit result receipts, pagination, bounded response contracts, and side-effect-free status.
3. Audit graph validation complexity, indexed traversal, profiling, and zero-source-parse proofs.
4. Reconcile the audits into the smallest additive design that preserves v1 behavior.
5. Implement and measure coherent slices, committing only after their gates pass.

Work log

Linear dependency validation

The inherited dependency-cycle preparation scanned every edge once for every node. The graph validator now constructs dependency adjacency in one edge pass and sorts each adjacency list before the existing deterministic depth-first cycle check.

A 2,000-node regression test counts complete edge-collection iteration passes and caps them at four. The focused correctness and bounded-pass tests pass, and the configured strict source type gate is clean.

One validation command initially included tests/test_core.py in a direct Pyright invocation. Repository Pyright intentionally covers src and tools, so that command reported existing untyped test-result indexing rather than a source defect. Rerunning the repository-configured type gate produced zero diagnostics.

Initial design constraints

• Full rebuild remains the recovery and equivalence oracle.
• Canonical content remains authoritative.
• Existing one-method load_projection() adapters remain unchanged.
• No-AST adapters remain first-class.
• Indexes, source-generation receipts, and caches remain disposable.
• Cheap reads may trust only identity-bound, versioned, corruption-checked receipts.
• Any optimization must fail closed on source mutation and must preserve stale-read refusal.

Future ideas and suggestions

These are notes, not commitments:

• A stable source-generation provider may deserve a public adapter capability only after both the generic project and one incremental adapter prove the same boundary.
• Profiling receipts could eventually feed the human-facing project control panel, but Milestone 1 should expose structured data before adding UI.
• Large context and changeset payloads may need cursor pagination or compact immutable receipts. The choice should follow actual client workflows rather than generic pagination machinery.