DocForge2/docs/INCREMENTAL_INDEXING.md

# Incremental Adapter Indexing

DocForge Release 1 adapters return one complete immutable projection. That contract remains
supported. The incremental compiler adds an opt-in source-scoped contract that avoids reparsing
unchanged files while preserving the same validated, atomically published graph.

## Release 1 compatibility

The incremental interface is additive:

- An existing adapter implementing only `load_projection()` continues to work unchanged.
- Existing generic projects, descriptors, canonical sources, changesets, and indexes require no
  migration.
- Only adapters implementing both `load_manifest()` and `extract_source()` use the incremental
  path.
- Incremental adapters must still implement `load_projection()` for clean rebuilds and equivalence
  testing.
- Existing adapters receive identical correctness behavior but no incremental speedup until they
  opt in.

## Safety model

Incremental indexing is an extraction optimization. It does not weaken publication:

1. The adapter returns a cheap, deterministic `AdapterManifest`.
2. DocForge compares every source fingerprint and extractor version with the last cache generation.
3. Added, changed, deleted, and reverse-dependent sources are invalidated.
4. The adapter reparses only invalidated sources.
5. DocForge assembles cached and refreshed contributions into a complete candidate projection.
6. The complete graph passes the same validation as a full adapter projection.
7. DocForge rereads the manifest to prove sources remained stable.
8. The extraction cache and SQLite graph are published with atomic file replacement.

An interrupted extraction never replaces the last validated SQLite index. A malformed,
incompatible, or missing cache is a cache miss, not a partial graph.

## Adapter contract

An incremental loader implements all three methods:

```python
class MyAdapter:
    def load_manifest(self) -> AdapterManifest: ...
    def extract_source(self, source: AdapterSource) -> AdapterSourceProjection: ...
    def load_projection(self) -> AdapterProjection: ...
```

`load_projection()` remains the deterministic full-rebuild fallback and equivalence oracle.

Each `AdapterSource` declares:

- A stable source ID.
- A safe project-relative source path.
- A SHA-256 content fingerprint.
- An extractor version.
- Other source IDs whose changes can alter this source's extracted facts.

Each `AdapterSourceProjection` owns:

- Its primary graph nodes.
- Its primary graph relationships, including cross-source relationships owned by that source.
- Optional function-scoped logic projections.

Ownership must be deterministic. Two sources may not produce the same primary node or the same
function logic projection.

Some language tools emit overlapping raw evidence before ownership can be resolved. An incremental
loader may additionally implement:

```python
def assemble_projection(
    manifest: AdapterManifest,
    contributions: tuple[AdapterSourceProjection, ...],
) -> AdapterAssembly: ...
```

DocForge caches and invalidates the source contributions normally, then passes the complete current
contribution set to this pure assembly step. The assembler must deterministically select or merge
overlapping evidence and return one valid final graph and Logic set. It may not read hidden source
state or create a second extraction cache. The final identity, revision, and source hash must match
the manifest exactly.

Without an assembler, the stricter default remains in force: two contributions may not publish the
same primary node or Logic owner.

## Invalidation

DocForge invalidates a source when:

- It is new.
- Its fingerprint changed.
- Its extractor version changed.
- A declared dependency was added, changed, or deleted.
- Any source in its reverse-dependency chain was invalidated.

Deleted sources are omitted from the candidate projection. Their cached dependency declarations
remain available long enough to invalidate surviving dependents.

If an adapter cannot precisely describe the affected sources, it should declare broader
dependencies or change its adapter/extractor version. Incorrectly retaining a stale relationship
is never an acceptable optimization.

## Build reporting

`build` and `reindex` include an extraction report:

```json
{
  "build": {
    "mode": "incremental",
    "cache_hits": 391,
    "reparsed_sources": 3,
    "invalidated_sources": 3,
    "deleted_sources": 0,
    "total_sources": 394,
    "cache_hit_ids": ["..."],
    "reparsed_source_ids": ["..."]
  }
}
```

Adapters can call `AdapterProject.verify_incremental_equivalence()` in release and contract tests.
The check compares project identity, revision, source hash, nodes, and relationships against
`load_projection()`.

## Manual changes and relationships

Changesets remain an approval queue, not a compiler queue. Compilation never silently applies a
proposal.

After explicit application:

1. The project-owned applier updates canonical files.
2. Changed manual files receive new fingerprints.
3. Incremental extraction reparses those files and affected dependents.
4. The complete candidate graph is validated and published.
5. Declared renders are regenerated.

`docforge_propose_relationship_update` queues relationship-only additions and removals without
rewriting node content. It is still bound to the changeset's complete base source hash and the
anchor node's expected content hash.

## Lazy logic boundary

`LogicProjection` stores control flow separately from the primary architecture graph. It is owned
by one function or method node and one source extraction.

Logic nodes can represent entries, conditions, basic blocks, calls, convergence points, loops,
returns, and raises. Logic edges retain relation, display label, and deterministic ordinal.
Adapters may leave
logic empty until they implement a language analyzer.

This boundary prevents thousands of boolean expressions and basic blocks from polluting Nodes,
Flow, Web, ordinary search, or architectural traversal. The Logic tab and `docforge_get_logic`
request one function-scoped projection on demand. The built-in analyzers cover Python,
JavaScript, and C++. Python uses the standard-library AST. JavaScript and C++ share pinned
Tree-sitter infrastructure with thin language-aware control-flow profiles. Parsers run only while
extracting a changed source contribution; ordinary graph reads do not load or execute them. A
grammar alone supplies syntax, not control-flow meaning, so each new language still needs a small
semantic profile for its branch, loop, case, exception, and termination constructs. All analyzers
report possible static paths; they do not claim runtime branch outcomes.

## Full rebuilds

Full rebuilds remain mandatory as a fallback and equivalence oracle. Change the adapter version,
extractor version, or cache schema whenever old cached facts are no longer valid. Removing the
confined extraction cache also forces a clean reparse without affecting canonical files.