Document incremental adapter indexing design

toreview.md

@@ -0,0 +1,93 @@
+# Incremental Adapter Indexing — Design Review Notes
+
+## Current behavior
+
+DocForge adapters currently return a complete project projection. A build:
+
+1. Enumerates and captures every source selected by the adapter.
+2. Re-extracts every node and relationship.
+3. Calculates source, node, edge, and content hashes.
+4. Writes a complete temporary SQLite index, including full-text search data.
+5. Runs an integrity check.
+6. Verifies that canonical source did not change during the build.
+7. Atomically replaces the previous index.
+
+The hashes currently provide identity, stale-state detection, proposal safety, and
+publication consistency. They are not used as a per-source extraction cache.
+DocForge changesets record proposed manual-node operations, but changesets are
+separate from adapter source extraction and index construction.
+
+Ani-web's current adapter build produces roughly 3,700 nodes and 6,200
+relationships in about 10 seconds. Its generated adapter index is approximately
+9.4 MB. This is inexpensive at development-slice or commit cadence. It would be
+wasteful, though still initially tolerable for most SSDs, to rebuild continuously
+on every file-save event.
+
+## Proposed direction
+
+Incremental indexing should be a backward-compatible, opt-in DocForge capability,
+not a private Ani-web optimization.
+
+DocForge core should provide:
+
+- Persistent per-source fingerprints.
+- Versioned cached extraction records.
+- A contract for added, changed, deleted, and affected sources.
+- Cache invalidation when the adapter version, graph schema, extraction rules, or
+  project descriptor changes.
+- Whole-candidate graph validation before publication.
+- Atomic publication so readers see only a complete old or complete new graph.
+- A deterministic full-rebuild fallback.
+- Reporting that distinguishes cache hits, reparsed sources, invalidated sources,
+  rebuilt relationships, and total build time.
+
+Adapters opting into incremental extraction should:
+
+1. Compare current source fingerprints with the last validated build.
+2. Reparse only added and changed sources.
+3. Remove cached facts belonging to deleted sources.
+4. Identify reverse dependencies and other globally affected sources.
+5. Recompute all relationships whose evidence may have changed.
+6. Assemble a complete candidate projection from cached and refreshed facts.
+7. Validate that candidate as strictly as a full rebuild.
+8. Publish only after source-stability and integrity checks pass.
+
+The SQLite index may still be rewritten completely because it is small and atomic
+replacement is simple and safe. The primary optimization is avoiding repeated
+parsing and static relationship discovery, not avoiding every database page write.
+
+## Correctness requirements
+
+- Incremental and full builds from the same revision must produce identical
+  canonical projections, hashes, node sets, edge sets, and query results.
+- Deleted and renamed sources must leave no orphaned nodes, edges, backlinks, or
+  full-text-search rows.
+- A changed symbol, import, route, plugin declaration, SQL reference, test, or
+  operational unit must invalidate every relationship that can depend on it.
+- Uncertain dependency impact must trigger broader invalidation or a full rebuild.
+  It must never retain a possibly stale edge.
+- Interrupted or failed builds must leave the previous validated index intact.
+- Existing adapters must continue using the complete-projection contract without
+  modification until they explicitly opt in.
+
+## Suggested implementation sequence
+
+1. Add generic cache and invalidation primitives to DocForge without changing the
+   existing adapter behavior.
+2. Add contract tests for cache identity, invalidation, deletion, interruption,
+   and atomic publication.
+3. Implement incremental extraction in the Ani-web adapter as the first adopter.
+4. Run full and incremental builds against identical revisions and compare their
+   complete outputs.
+5. Measure wall time, parsed-source counts, cache hit rate, database writes, and
+   index size before enabling automatic rebuilds.
+6. Add a debounced watcher only after equivalence and failure-recovery tests pass.
+
+## Recommended rebuild policy before this exists
+
+- Rebuild after a coherent development slice.
+- Rebuild before graph-dependent documentation work when the index is stale.
+- Rebuild before validation and rendering.
+- Do not rebuild on every keystroke or un-debounced save event.
+- If automatic rebuilding is added first, debounce and coalesce changes for at
+  least 30–60 seconds.