DocForge2/docs/INCREMENTAL_INDEXING.md

Incremental Adapter Indexing

DocForge Release 1 adapters return one complete immutable projection. That contract remains supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing unchanged files while preserving the same validated, atomically published graph.

Release 1 compatibility

The incremental interface is additive:

• An existing adapter implementing only load_projection() continues to work unchanged.
• Existing generic projects, descriptors, canonical sources, changesets, and indexes require no migration.
• Only adapters implementing both load_manifest() and extract_source() use the incremental path.
• Incremental adapters must still implement load_projection() for clean rebuilds and equivalence testing.
• Existing adapters receive identical correctness behavior but no incremental speedup until they opt in.

Safety model

Incremental indexing is an extraction optimization. It does not weaken publication:

1. The adapter returns a cheap, deterministic AdapterManifest.
2. DocForge compares every source fingerprint and extractor version with the last cache generation.
3. Added, changed, deleted, and reverse-dependent sources are invalidated.
4. The adapter reparses only invalidated sources.
5. DocForge assembles cached and refreshed contributions into a complete candidate projection.
6. The complete graph passes the same validation as a full adapter projection.
7. DocForge rereads the manifest to prove sources remained stable.
8. The extraction cache and SQLite graph are published with atomic file replacement.

An interrupted extraction never replaces the last validated SQLite index. A malformed, incompatible, or missing cache is a cache miss, not a partial graph.

Adapter contract

An incremental loader implements all three methods:

class MyAdapter:
    def load_manifest(self) -> AdapterManifest: ...
    def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
    def load_projection(self) -> AdapterProjection: ...

load_projection() remains the deterministic full-rebuild fallback and equivalence oracle.

Each AdapterSource declares:

• A stable source ID.
• A safe project-relative source path.
• A SHA-256 content fingerprint.
• An extractor version.
• Other source IDs whose changes can alter this source's extracted facts.

Each AdapterSourceProjection owns:

• Its primary graph nodes.
• Its primary graph relationships, including cross-source relationships owned by that source.
• Optional function-scoped logic projections.

Ownership must be deterministic. Two sources may not produce the same primary node or the same function logic projection.

Invalidation

DocForge invalidates a source when:

• It is new.
• Its fingerprint changed.
• Its extractor version changed.
• A declared dependency was added, changed, or deleted.
• Any source in its reverse-dependency chain was invalidated.

Deleted sources are omitted from the candidate projection. Their cached dependency declarations remain available long enough to invalidate surviving dependents.

If an adapter cannot precisely describe the affected sources, it should declare broader dependencies or change its adapter/extractor version. Incorrectly retaining a stale relationship is never an acceptable optimization.

Build reporting

build and reindex include an extraction report:

{
  "build": {
    "mode": "incremental",
    "cache_hits": 391,
    "reparsed_sources": 3,
    "invalidated_sources": 3,
    "deleted_sources": 0,
    "total_sources": 394,
    "cache_hit_ids": ["..."],
    "reparsed_source_ids": ["..."]
  }
}

Adapters can call AdapterProject.verify_incremental_equivalence() in release and contract tests. The check compares project identity, revision, source hash, nodes, and relationships against load_projection().

Manual changes and relationships

Changesets remain an approval queue, not a compiler queue. Compilation never silently applies a proposal.

After explicit application:

1. The project-owned applier updates canonical files.
2. Changed manual files receive new fingerprints.
3. Incremental extraction reparses those files and affected dependents.
4. The complete candidate graph is validated and published.
5. Declared renders are regenerated.

docforge_propose_relationship_update queues relationship-only additions and removals without rewriting node content. It is still bound to the changeset's complete base source hash and the anchor node's expected content hash.

Lazy logic boundary

LogicProjection stores control flow separately from the primary architecture graph. It is owned by one function or method node and one source extraction.

Logic nodes can represent entries, conditions, basic blocks, calls, merges, loops, returns, and raises. Logic edges retain relation, display label, and deterministic ordinal. Adapters may leave logic empty until they implement a language analyzer.

This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes, Flow, Web, ordinary search, or architectural traversal. The Logic tab and docforge_get_logic request one function-scoped projection on demand. The built-in Python analyzer covers conditions, short-circuit booleans, loops, match, exception paths, returns, and raises. It reports possible static paths; it does not claim runtime branch outcomes.

Full rebuilds

Full rebuilds remain mandatory as a fallback and equivalence oracle. Change the adapter version, extractor version, or cache schema whenever old cached facts are no longer valid. Removing the confined extraction cache also forces a clean reparse without affecting canonical files.