DocForge2/docs/INCREMENTAL_INDEXING.md

Incremental Adapter Indexing

DocForge Release 1 adapters return one complete immutable projection. That contract remains supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing unchanged files while preserving the same validated, atomically published graph.

Release 1 compatibility

The incremental interface is additive:

• An existing adapter implementing only load_projection() continues to work unchanged.
• Existing generic projects, descriptors, canonical sources, changesets, and indexes require no migration.
• Only adapters implementing both load_manifest() and extract_source() use the incremental path.
• Incremental adapters must still implement load_projection() for clean rebuilds and equivalence testing.
• Existing adapters receive identical correctness behavior but no incremental speedup until they opt in.

Safety model

Incremental indexing is an extraction optimization. It does not weaken publication:

1. The adapter returns a cheap, deterministic AdapterManifest.
2. DocForge compares every source fingerprint and extractor version with the last cache generation.
3. Added, changed, deleted, and reverse-dependent sources are invalidated.
4. The adapter reparses only invalidated sources.
5. DocForge assembles cached and refreshed contributions into a complete candidate projection.
6. The complete graph passes the same validation as a full adapter projection.
7. DocForge rereads the manifest to prove sources remained stable.
8. The extraction cache and SQLite graph are published with atomic file replacement.

An interrupted extraction never replaces the last validated SQLite index. A malformed, incompatible, or missing cache is a cache miss, not a partial graph.

Adapter contract

An incremental loader implements all three methods:

class MyAdapter:
    def load_manifest(self) -> AdapterManifest: ...
    def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
    def load_projection(self) -> AdapterProjection: ...

load_projection() remains the deterministic full-rebuild fallback and equivalence oracle.

Each AdapterSource declares:

• A stable source ID.
• A safe project-relative source path.
• A SHA-256 content fingerprint.
• An extractor version.
• Other source IDs whose changes can alter this source's extracted facts.

Each AdapterSourceProjection owns:

• Its primary graph nodes.
• Its primary graph relationships, including cross-source relationships owned by that source.
• Optional function-scoped logic projections.

Ownership must be deterministic. Two sources may not produce the same primary node or the same function logic projection.

Some language tools emit overlapping raw evidence before ownership can be resolved. An incremental loader may additionally implement:

def assemble_projection(
    manifest: AdapterManifest,
    contributions: tuple[AdapterSourceProjection, ...],
) -> AdapterAssembly: ...

DocForge caches and invalidates the source contributions normally, then passes the complete current contribution set to this pure assembly step. The assembler must deterministically select or merge overlapping evidence and return one valid final graph and Logic set. It may not read hidden source state or create a second extraction cache. The final identity, revision, and source hash must match the manifest exactly.

Without an assembler, the stricter default remains in force: two contributions may not publish the same primary node or Logic owner.

Invalidation

DocForge invalidates a source when:

• It is new.
• Its fingerprint changed.
• Its extractor version changed.
• A declared dependency was added, changed, or deleted.
• Any source in its reverse-dependency chain was invalidated.

Deleted sources are omitted from the candidate projection. Their cached dependency declarations remain available long enough to invalidate surviving dependents.

If an adapter cannot precisely describe the affected sources, it should declare broader dependencies or change its adapter/extractor version. Incorrectly retaining a stale relationship is never an acceptable optimization.

Build reporting

build and reindex include an extraction report:

{
  "build": {
    "mode": "incremental",
    "cache_hits": 391,
    "reparsed_sources": 3,
    "invalidated_sources": 3,
    "deleted_sources": 0,
    "total_sources": 394,
    "cache_hit_ids": ["..."],
    "reparsed_source_ids": ["..."]
  }
}

Adapters can call AdapterProject.verify_incremental_equivalence() in release and contract tests. The check compares project identity, revision, source hash, nodes, and relationships against load_projection().

Manual changes and relationships

Changesets remain an approval queue, not a compiler queue. Compilation never silently applies a proposal.

After explicit application:

1. The project-owned applier updates canonical files.
2. Changed manual files receive new fingerprints.
3. Incremental extraction reparses those files and affected dependents.
4. The complete candidate graph is validated and published.
5. Declared renders are regenerated.

docforge_propose_relationship_update queues relationship-only additions and removals without rewriting node content. It is still bound to the changeset's complete base source hash and the anchor node's expected content hash.

Lazy logic boundary

LogicProjection stores control flow separately from the primary architecture graph. It is owned by one function or method node and one source extraction.

Logic nodes can represent entries, conditions, basic blocks, calls, convergence points, loops, returns, and raises. Logic edges retain relation, display label, and deterministic ordinal. Adapters may leave logic empty until they implement a language analyzer.

This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes, Flow, Web, ordinary search, or architectural traversal. The Logic tab and docforge_get_logic request one function-scoped projection on demand. The built-in analyzers cover Python, JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while extracting a changed source contribution; ordinary graph reads do not load or execute them. A grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers report possible static paths; they do not claim runtime branch outcomes.

Full rebuilds

Full rebuilds remain mandatory as a fallback and equivalence oracle. Change the adapter version, extractor version, or cache schema whenever old cached facts are no longer valid. Removing the confined extraction cache also forces a clean reparse without affecting canonical files.