DocForge2/DEVELOPMENT_NOTES.md

# DocForge2 development notes

This is the running implementation record for DocForge2. It records what is active, what was
measured, what changed, what failed, why architectural decisions were made, and which ideas were
deferred. Stable user and compatibility contracts still belong in dedicated documentation.

## Working rules

- Only one milestone is active at a time.
- `main` remains the last fully verified milestone.
- Active implementation occurs on `dev`.
- Every milestone begins from direct repository evidence and ends with focused tests, the complete
  repository gate, updated measurements, documentation closeout, and a clean pushed state.
- WorldForge, ScrapeStation, legacy DocForge, and production MCP bindings remain out of scope.
- DocForge2 does not self-host during this program.
- Release tags and Forgejo releases require Rob's explicit approval.

## Milestone 0 — complete

Milestone 0 established the public successor, preserved the complete lineage and v1 tag, integrated
the no-AST and adapter-lifecycle work, froze compatibility guarantees, added repository-native
quality and contract gates, and recorded cold/warm performance, memory, rendering, and response
sizes.

The central measurement was decisive: a 1,000-node warm exact lookup took about 286 ms while the
generation-pinned SQLite query path took about 0.4–1.4 ms. Repeated whole-source loading and
validation, not SQLite, is the first optimization target.

## Milestone 1 — active: fast, observable core

### Outcome

Warm retrieval should disappear into normal tool overhead. Routine reads must not parse project
sources. Status must not render or rebuild hidden work. Results must remain bounded independently
of project size.

### Starting evidence

- Generic `Project.load()` walks, captures, parses, rereads, validates, hashes, and checks Git for
  the complete source set.
- Exact retrieval validates twice around one bounded SQLite query.
- Context compilation performs three full project loads.
- Render status recompiles the complete manual.
- Incremental adapters already prove that manifest attestation can make no-change synchronization
  and exact retrieval sub-millisecond on a tiny fixture.
- Pinned viewer queries prove the current SQLite schema can serve bounded reads quickly.

### Current work

1. Audit request-scoped immutable snapshot and persistent-generation options.
2. Audit result receipts, pagination, bounded response contracts, and side-effect-free status.
3. Audit graph validation complexity, indexed traversal, profiling, and zero-source-parse proofs.
4. Reconcile the audits into the smallest additive design that preserves v1 behavior.
5. Implement and measure coherent slices, committing only after their gates pass.

### Work log

#### Linear dependency validation

The inherited dependency-cycle preparation scanned every edge once for every node. The graph
validator now constructs dependency adjacency in one edge pass and sorts each adjacency list before
an iterative deterministic depth-first cycle check. The iterative stack also removes recursion
depth as a failure mode on large valid graphs. The same edge pass now rejects missing sources as
well as missing targets.

A 10,000-node regression test counts complete edge-collection iteration passes and caps them at
four. The focused correctness and bounded-pass tests pass, and the configured strict source type
gate is clean.

One validation command initially included `tests/test_core.py` in a direct Pyright invocation.
Repository Pyright intentionally covers `src` and `tools`, so that command reported existing
untyped test-result indexing rather than a source defect. Rerunning the repository-configured type
gate produced zero diagnostics.

#### Persistent generic source generations

Generic projects now persist a version-1 source-generation receipt only after complete source
loading and fully verified index publication. The receipt binds the explicit generic source
contract, project and root identity, adapter, revision, source hash, every canonical/authority/
descriptor regular-file identity, and every source-membership directory identity.

A normal warm check reads no canonical source bytes. It validates the known directories and files
directly using device, inode, mode, size, nanosecond modification time, and nanosecond change time.
Directory identities detect add, delete, and rename operations without an `rglob`. Any missing,
malformed, incompatible, foreign, or dirty receipt becomes a cache miss and falls back to the full
canonical load and row-verification oracle. Successful fallback verification repairs the disposable
receipt.

The receipt is deliberately generic-project behavior. Incremental adapter manifests retain
authority over generated or specialist source identities. A one-method legacy adapter continues to
work even when it cannot provide a cheap generation.

#### Request-scoped immutable reads

Index reads now use one read-only SQLite transaction pinned to one verified file signature and one
source identity. Existence checks and queries share that connection. Before returning, the request
rechecks the index signature and current cheap source generation. A concurrent source or index
change fails closed.

Context compilation hydrates nodes and edges from the pinned derived snapshot while retaining
profiles from the immutable descriptor. It no longer loads or parses canonical sources. The public
full `Project.load()` and deep `ProjectIndex.check()` behavior remains the recovery and equivalence
oracle.

Focused tests prove that fresh-process-style generic reads can run exact, search, filter,
backlinks, dependency, impact, context, and no-change synchronization operations while
`Project.load()` is forbidden. They also prove a final source-generation change is rejected before
return and missing/corrupt receipts fall back and repair.

On the maintained 1,000-file fixture, the current work-in-progress measurements are:

| Operation | Milestone 0 median | Milestone 1 WIP median |
|---|---:|---:|
| Warm no-change synchronize | 142.479 ms | 20.007 ms |
| Exact node | 286.306 ms | 40.277 ms |
| Search, limit 20 | 288.793 ms | 41.455 ms |
| Dependencies, depth 8 | 287.791 ms | 41.551 ms |
| Context, 32k | 436.897 ms | 46.245 ms |
| MCP exact node | 287.094 ms | 40.051 ms |
| MCP context, 32k | 434.853 ms | 46.454 ms |

The three-sample WIP run is directional, not the final Milestone 1 baseline. The final evidence run
will use the maintained sample counts and committed clean-tree revision.

#### Read-only audit reconciliation

The three Milestone 1 audits agreed on the main architecture:

- Keep complete loading and deep checking as independent truth oracles.
- Trust only versioned, identity-bound disposable generation receipts.
- Use one pinned read transaction and retain a final dirty check.
- Hydrate context from the current index.
- Replace full-edge traversal scans with bounded indexed frontier reads.
- Add compact success receipts before allowing large mutations to report post-write size errors.
- Replace hidden render-status rendering with a receipt comparison.
- Add algorithmic counters and parse-count gates alongside wall-clock thresholds.

One audit identified a correctness risk beyond latency: a large mutating MCP operation can commit
successfully and then be replaced by `result_too_large`. This must be fixed in Milestone 1 so
exactly-once operations never report a false failure after mutation.

#### Bounded indexed retrieval

Search, metadata filtering, backlinks, dependency traversal, and impact traversal now query one
extra row beyond the requested bound and report `limit` plus `truncated`. Backlinks, dependency,
and impact APIs accept the same additive `limit` option through Python, CLI, and MCP surfaces.
Omitted limits are capped by the project `max_results` policy.

Traversal no longer loads the complete edge table and repeatedly scans it. It performs
deterministically ordered frontier queries through the existing source primary key or target index.
Each request also has a deterministic edge-examination budget derived from its result limit. The
response includes `examined_edges` and `examined_edges_limit` counters so algorithmic work can be
asserted independently of machine timing. `truncated` is true when either another unique result
exists or the work budget prevents proving completeness. A focused core, CLI, MCP, Ruff, and
Pyright gate passes for this work-in-progress slice.

### Initial design constraints

- Full rebuild remains the recovery and equivalence oracle.
- Canonical content remains authoritative.
- Existing one-method `load_projection()` adapters remain unchanged.
- No-AST adapters remain first-class.
- Indexes, source-generation receipts, and caches remain disposable.
- Cheap reads may trust only identity-bound, versioned, corruption-checked receipts.
- Any optimization must fail closed on source mutation and must preserve stale-read refusal.

### Future ideas and suggestions

These are notes, not commitments:

- A stable source-generation provider may deserve a public adapter capability only after both the
  generic project and one incremental adapter prove the same boundary.
- Profiling receipts could eventually feed the human-facing project control panel, but Milestone 1
  should expose structured data before adding UI.
- Large context and changeset payloads may need cursor pagination or compact immutable receipts.
  The choice should follow actual client workflows rather than generic pagination machinery.