administrator/DocForge2
1
0
Fork 0
Code Issues Pull requests Projects Releases 2 Packages Wiki Activity Actions Pages

Establish Milestone 0 compatibility and quality gates

Browse source
This commit is contained in:
Andraxion 2026-07-29 03:12:30 -04:00
parent 15a913003c
commit 8ebb78a71d
15 changed files with 1114 additions and 27 deletions
Download patch file Download diff file Expand all files Collapse all files

52
Makefile Normal file
View file

@ -0,0 +1,52 @@
PYTHON := .venv/bin/python
PYRIGHT := pyright
UV := uv
NPM := npm
PYTHONPYCACHEPREFIX := /tmp/docforge-quality-pycache
PYTEST_BASETEMP := /tmp/docforge-quality-pytest
.PHONY: benchmark benchmark-smoke build compile contract dependencies format-check gate lint lock test type
format-check:
$(PYTHON) -m ruff format --check src tests tools
lint:
$(PYTHON) -m ruff check src tests tools
$(NPM) run lint:web
type:
$(PYRIGHT) --pythonpath .venv/bin/python
compile:
PYTHONPYCACHEPREFIX=$(PYTHONPYCACHEPREFIX) $(PYTHON) -m compileall -q src tests tools
contract:
PYTHONPYCACHEPREFIX=$(PYTHONPYCACHEPREFIX) $(PYTHON) -m pytest -q \
-p no:cacheprovider --basetemp=$(PYTEST_BASETEMP) \
tests/test_public_contract.py \
tests/test_adapter_contract.py::AdapterContractTests::test_no_ast_index_policy_rejects_logic_publication \
tests/test_adapter_contract.py::AdapterContractTests::test_no_ast_index_accepts_legacy_and_non_logic_incremental_adapters \
tests/test_adapter_contract.py::AdapterContractTests::test_no_ast_rejects_preexisting_logic_index_and_viewer_snapshot \
tests/test_mcp_server.py::DocForgeMcpTests::test_no_ast_binding_preserves_adapter_and_blocks_logic
test:
PYTHONPYCACHEPREFIX=$(PYTHONPYCACHEPREFIX) $(PYTHON) -m pytest -q \
-p no:cacheprovider --basetemp=$(PYTEST_BASETEMP)
lock:
$(UV) lock --check
dependencies:
$(NPM) ls --all
build:
$(UV) build
benchmark-smoke:
$(PYTHON) tools/milestone0_baseline.py --nodes 25 --samples 1 --cold-samples 1 \
--output /tmp/docforge-milestone0-smoke.json > /dev/null
benchmark:
$(PYTHON) tools/milestone0_baseline.py --nodes 1000 --samples 10 --cold-samples 3
gate: format-check lint type compile contract test lock dependencies build benchmark-smoke

18
README.md
View file

@ -86,8 +86,8 @@ hiding a step inserts an explicit omitted-path bridge so downstream control flow
Requirements are Python 3.12+, `uv`, and Node.js/npm.
```bash
git clone forgejo@repo.andraxion.net:administrator/DocForge.git /absolute/path/DocForge
cd /absolute/path/DocForge
git clone forgejo@repo.andraxion.net:administrator/DocForge2.git /absolute/path/DocForge2
cd /absolute/path/DocForge2
uv sync --group dev
npm ci
@ -129,6 +129,8 @@ DocForge describes them as a source graph.
- [User manual](docs/USER_MANUAL.md) — features, setup, visualization, CLI, MCP, apply, adapters,
and troubleshooting.
- [Core contract](docs/CONTRACT.md) — invariants and security boundary.
- [Milestone 0 compatibility](docs/COMPATIBILITY.md) — preserved package, CLI, MCP, adapter,
schema, changeset, rendering, and no-AST guarantees.
- [MCP contract](docs/MCP_CONTRACT.md) — exact tool and process boundary.
- [Viewer manager](docs/VIEWER_MANAGER.md) — native service setup and lifecycle.
- [Adapter decision](docs/APPLICATION_DECISION.md) — why custom adapters own canonical
@ -143,13 +145,13 @@ DocForge describes them as a source graph.
## Development
Run the complete repository-native gate:
```bash
npx pyright
npm run lint:web
uv run ruff check src tests tools
uv run ruff format --check src tests tools
uv run python -m compileall -q src tests tools
uv run pytest -q
make gate
```
Focused entry points are available as `make contract`, `make test`, `make type`,
`make benchmark-smoke`, and `make benchmark`.
See [AGENTS.md](AGENTS.md) before changing core boundaries.

159
docs/COMPATIBILITY.md Normal file
View file

@ -0,0 +1,159 @@
# DocForge2 Milestone 0 compatibility contract
Milestone 0 establishes DocForge2 as the successor repository without renaming or replacing the
working DocForge interfaces. Compatibility changes require an explicit decision, a contract-test
update, and migration guidance.
The compatibility gate is:
```bash
make contract
```
The complete repository gate is:
```bash
make gate
```
## Distribution and Python imports
The Python distribution and import package remain `docforge`.
The installed executable names remain:
- `docforge`
- `docforge-mcp`
- `docforge-viewer-manager`
The top-level imports recorded by `docforge.__all__` remain supported. The documented adapter,
model, index, rendering, application, and MCP factory names imported from these submodules also
remain supported:
- `docforge.adapter_contract`
- `docforge.application`
- `docforge.index`
- `docforge.mcp_server`
- `docforge.models`
- `docforge.render_contract`
Names beginning with an underscore are implementation details. New public names may be added
without breaking this contract.
## CLI and MCP surfaces
Existing `docforge` command names and arguments remain supported. Existing `docforge-mcp` tool
names and arguments remain supported. Additive commands, tools, and response fields are allowed.
Removing or changing an existing name, required argument, stable error code, or safety boundary
requires an explicit compatibility decision.
MCP results retain:
- A structured `status`.
- Project and source identity when available.
- Stable structured domain errors.
- A bounded content warning.
- Staleness information.
- The configured output-size limit.
The result schema describes the common envelope. Operation-specific fields are additive and remain
bounded by the configured tool-output limit.
## Versioned data contracts
Milestone 0 preserves:
- Project descriptor schema version 1.
- Node schema version 1.
- Edge schema version 1.
- Changeset schema version 1.
- Result-envelope schema version 1.
- SQLite index schema version 2.
- Index-attestation schema version 1.
- Incremental extraction-cache schema version 1.
Indexes, attestations, extraction caches, previews, and rendered artifacts are disposable. A schema
change may rebuild them. Canonical project content and stored proposals may not be silently
rewritten to satisfy a new implementation.
## Adapter compatibility
An adapter implementing only:
```python
load_projection()
```
remains first-class. Incremental manifests, source extraction, deterministic assembly, Logic
projection, and proposal or application support are optional capabilities. Incremental adapters
must retain `load_projection()` as their independent clean-build and equivalence oracle.
Project adapters remain explicitly composed. Generic DocForge does not discover arbitrary adapter
modules or choose a project globally.
## Preserved no-AST binding
`docforge-mcp --project-root /project --no-ast` is a stable shorthand for the
`preserve-no-ast` binding policy.
The binding:
- Keeps one-method complete-projection adapters working.
- Keeps non-AST incremental fingerprinting and caching working.
- Rejects nonempty function-Logic publication.
- Rejects a pre-existing index containing function Logic.
- Blocks `docforge_get_logic`.
- Prevents the live viewer from pinning an index containing Logic.
- Applies the same restriction during hash-bound canonical-application refresh.
- Reports the effective policy through bootstrap and contract results.
DocForge does not inspect arbitrary adapter source to prove which parser implementation it uses.
The no-AST binding is an owner-selected process policy backed by Logic publication and retrieval
enforcement. It is not a filesystem sandbox and cannot stop an unrelated process with repository
write access from changing adapter code.
## Changesets and application
The following guarantees remain stable:
1. Registration writes one complete proposal atomically.
2. Proposal identity includes its project, root, base revision, canonical source hash, writer, and
ordered operations.
3. Validation and diff inspection precede application.
4. Append, rebase, abandonment, and application use exact current hashes.
5. Stale, conflicting, unauthorized, unsafe, or invalid proposals fail closed.
6. Canonical application is absent unless one startup-bound applier is configured.
7. Derived refresh failures produce an explicit degraded receipt after canonical application. They
do not make an applied proposal safe to apply twice.
## Rendering and visualization
The `generic_html` renderer remains the supported version-1 manual projection. It retains confined
paths, raw-HTML suppression, fixed template tokens, deterministic identities, atomic replacement,
and side-effect-free status.
The live graph viewer remains a read-only consumer of a generation-pinned validated index. It does
not become project authority or MCP retrieval authority.
`ManualRenderPlan`, `GraphViewPlan`, a portable graph renderer, and independently packaged
renderers are later-milestone direction. Milestone 0 does not claim that those contracts exist.
## Safety boundary
DocForge remains bound to one explicit project root. It rejects absolute paths, root escapes, and
symbolic-link escapes. Documentation text remains untrusted data. Normal MCP operation exposes no
arbitrary filesystem access, renderer execution, shell command, Git mutation, deployment,
publication, or project switching.
## Recorded weaknesses, not compatibility promises
Milestone 0 records rather than redesigns these areas:
- Generic warm reads still repeat whole-project discovery, parsing, and validation.
- Tree-sitter and the JavaScript and C++ grammars remain mandatory installation dependencies even
when their runtime modules are unused.
- Several version strings and defaults remain duplicated.
- Large changeset results and context responses need compact receipt or pagination contracts.
- Manual planning is not separated from rendering.
- There is no portable graph-planning or graph-rendering contract.
- DocForge2 does not self-host its bootstrap documentation.

11
docs/MCP_CONTRACT.md
View file

@ -189,10 +189,13 @@ extraction. Under this binding:
- `docforge_get_logic` returns `adapter_policy_forbids_logic`;
- a nonempty Logic projection is rejected before index publication;
- a pre-existing index containing Logic is rejected before any read or live-viewer snapshot;
- hash-bound canonical application refresh uses the same policy-bound index;
- `adapter_ast_upgrade` and `function_logic_extraction` appear as excluded operations; and
- changing the policy requires changing the process configuration and starting a new MCP process.
The policy governs the DocForge binding and conforming MCP clients. DocForge still exposes no
filesystem sandbox and cannot prevent an unrelated process with direct repository write access
from editing adapter files. Repository permissions and project instructions remain responsible for
that broader boundary.
The policy governs the DocForge binding and conforming MCP clients. DocForge can enforce published
and indexed Logic, but it does not inspect arbitrary adapter source to prove which parser
implementation the adapter uses. DocForge still exposes no filesystem sandbox and cannot prevent
an unrelated process with direct repository write access from editing adapter files. Repository
permissions and project instructions remain responsible for that broader boundary.

6
docs/USER_MANUAL.md
View file

@ -675,6 +675,12 @@ fingerprinting and caching when those mechanisms do not add AST analysis. The ad
therefore benefit from current synchronization, proposals, application, rendering, and graph tools
without a source-analysis rewrite.
The binding rejects a pre-existing index containing Logic before reads or live visualization. A
configured canonical application service also refreshes through the same no-AST index policy.
DocForge does not inspect arbitrary adapter source to prove which parsing library it uses, so
repository permissions and project instructions remain responsible for adapter implementation
changes outside this process boundary.
## Troubleshooting
### `adapter_restart_required`

6
pyproject.toml
View file

@ -19,7 +19,11 @@ dependencies = [
]
[dependency-groups]
dev = ["pytest>=9.1,<10", "ruff>=0.15,<1"]
dev = [
"jsonschema>=4.25,<5",
"pytest>=9.1,<10",
"ruff>=0.15,<1",
]
[project.scripts]
docforge = "docforge.cli:main"

20
schemas/result.schema.json
View file

@ -12,13 +12,29 @@
"revision": { "type": "string" },
"source_hash": { "type": "string", "pattern": "^[0-9a-f]{64}$" },
"adapter": { "type": "string" }
}
},
"additionalProperties": true
},
{
"type": "object",
"required": ["status", "error"],
"properties": {
"status": { "const": "error" },
"project_id": { "type": "string" },
"project_root_fingerprint": {
"type": "string",
"pattern": "^[0-9a-f]{16}$"
},
"adapter": { "type": "string" },
"server_version": { "type": "string" },
"revision": { "type": "string" },
"source_hash": {
"type": ["string", "null"],
"pattern": "^[0-9a-f]{64}$"
},
"content_warning": { "type": "string" },
"staleness": { "enum": ["current", "stale", "unknown"] },
"synchronization": { "type": "object" },
"error": {
"type": "object",
"required": ["code", "message", "details"],
@ -30,7 +46,7 @@
"additionalProperties": false
}
},
"additionalProperties": false
"additionalProperties": true
}
]
}

3
src/docforge/application.py
View file

@ -352,12 +352,13 @@ class CanonicalApplicationService:
*,
applier_id: str | None,
applier: CanonicalApplier | None,
index: ProjectIndex | None = None,
) -> None:
self.project = project
self.applier_id = applier_id
self.applier = applier
self.changesets = ChangesetStore(project, applier_id)
self.index = ProjectIndex(project)
self.index = index or ProjectIndex(project)
self.rendering = RenderService(project, self.changesets)
@property

22
src/docforge/index.py
View file

@ -393,16 +393,20 @@ class ProjectIndex:
def _logic_projections(self) -> tuple[LogicProjection, ...]:
if isinstance(self.project, LogicProject):
projections = self.project.logic_projections()
if projections and not self.allow_logic:
self._require_logic_allowed(len(projections))
return projections
return ()
def _require_logic_allowed(self, projection_count: int) -> None:
if projection_count and not self.allow_logic:
raise DocForgeError(
"adapter_policy_forbids_logic",
(
"This index preserves a no-AST adapter and refuses function-Logic "
"publication"
"publication or retrieval"
),
logic_projection_count=projection_count,
)
return projections
return ()
def check(self, *, verify_rows: bool = True) -> dict[str, object]:
if isinstance(self.project, IncrementalStateProject):
@ -491,6 +495,14 @@ class ProjectIndex:
raise DocForgeError(
"stale_index", "Derived index does not match canonical source", field=key
)
try:
logic_projection_count = int(metadata["logic_projection_count"])
except (KeyError, ValueError) as error:
raise DocForgeError(
"invalid_index",
"Derived index has invalid Logic metadata",
) from error
self._require_logic_allowed(logic_projection_count)
current_signature = self._index_signature()
if not verify_rows and (
current_signature == self._verified_index_signature or self._attestation_matches()
@ -503,7 +515,7 @@ class ProjectIndex:
"edge_hash": metadata["edge_hash"],
"edge_count": int(metadata["edge_count"]),
"logic_hash": metadata["logic_hash"],
"logic_projection_count": int(metadata["logic_projection_count"]),
"logic_projection_count": logic_projection_count,
"logic_node_count": int(metadata["logic_node_count"]),
"logic_edge_count": int(metadata["logic_edge_count"]),
"status": "ok",

1
src/docforge/mcp_server.py
View file

@ -123,6 +123,7 @@ class DocForgeService:
self.project,
applier_id=canonical_applier_id,
applier=canonical_applier,
index=self.index,
)
self.visualization = ViewerManagerClient(self.index)
self.context_provider = context_provider

62
tests/test_adapter_contract.py
View file

@ -47,6 +47,7 @@ from docforge.models import (
RenderConfig,
RenderView,
)
from docforge.viewer_manager import ViewerManagerClient
from docforge.visualization import VisualizationIndexSnapshot
@ -236,6 +237,13 @@ class OverlappingIncrementalLoader(IncrementalLoader):
return self.assemble_projection(manifest, contributions).projection
class NonLogicIncrementalLoader(IncrementalLoader):
"""Exercise non-AST incremental caching without publishing function Logic."""
def extract_source(self, source: AdapterSource) -> AdapterSourceProjection:
return replace(super().extract_source(source), logic=())
class AdapterContractTests(unittest.TestCase):
def projection(self, root: Path) -> AdapterProjection:
foundation = Node(
@ -561,6 +569,60 @@ class AdapterContractTests(unittest.TestCase):
self.assertEqual("adapter_policy_forbids_logic", captured.exception.code)
self.assertFalse(project.descriptor.index_path.exists())
def test_no_ast_index_accepts_legacy_and_non_logic_incremental_adapters(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
legacy = AdapterProject(
Loader(self.projection(root)),
cache_root=root / ".cache" / "legacy-no-ast",
)
legacy_index = ProjectIndex(legacy, allow_logic=False)
self.assertEqual(2, legacy_index.build()["node_count"])
self.assertEqual(
"guide.workflow",
legacy_index.get_node("guide.workflow")["node"]["node_id"],
)
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
loader = NonLogicIncrementalLoader(root)
project = AdapterProject(
loader,
cache_root=root / ".cache" / "incremental-no-ast",
)
index = ProjectIndex(project, allow_logic=False)
first = index.build()
self.assertEqual(2, first["build"]["reparsed_sources"])
loader.extract_calls.clear()
second = index.build()
self.assertEqual([], loader.extract_calls)
self.assertEqual(2, second["build"]["cache_hits"])
self.assertEqual(0, second["build"]["reparsed_sources"])
self.assertEqual(0, second["logic_projection_count"])
def test_no_ast_rejects_preexisting_logic_index_and_viewer_snapshot(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
project = AdapterProject(
IncrementalLoader(root),
cache_root=root / ".cache" / "preexisting-logic",
)
ProjectIndex(project).build()
preserved = ProjectIndex(project, allow_logic=False)
with self.assertRaises(DocForgeError) as checked:
preserved.check(verify_rows=False)
self.assertEqual("adapter_policy_forbids_logic", checked.exception.code)
with self.assertRaises(DocForgeError) as viewed:
ViewerManagerClient(
preserved,
state_path=root / ".cache" / "viewer-state.json",
).start()
self.assertEqual("adapter_policy_forbids_logic", viewed.exception.code)
def test_fast_incremental_reads_reverify_a_changed_index_file(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()

6
tests/test_mcp_server.py
View file

@ -81,7 +81,11 @@ class DocForgeMcpTests(unittest.IsolatedAsyncioTestCase):
async def test_no_ast_binding_preserves_adapter_and_blocks_logic(self) -> None:
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture("alpha", Path(directory))
ProjectIndex(Project.open(root)).build()
project = Project.open(root)
ProjectIndex(project).build()
service = DocForgeService(project, no_ast=True)
self.assertIs(service.index, service.application.index)
self.assertFalse(service.application.index.allow_logic)
async with create_connected_server_and_client_session(
create_server(root, no_ast=True), raise_exceptions=True
) as session:

270
tests/test_public_contract.py Normal file
View file

@ -0,0 +1,270 @@
from __future__ import annotations
import argparse
import hashlib
import importlib
import json
import shutil
import subprocess
import sys
import tempfile
import tomllib
import unittest
from pathlib import Path
from jsonschema import Draft202012Validator
import docforge
from docforge.changeset_contract import canonical_bytes, document_hash
from docforge.changesets import ChangesetStore
from docforge.cli import _parser
from docforge.index import ProjectIndex
from docforge.mcp_server import (
ALL_TOOLS,
APPLICATION_TOOLS,
PROPOSAL_TOOLS,
READ_TOOLS,
SERVER_VERSION,
DocForgeService,
)
from docforge.project import Project
ROOT = Path(__file__).resolve().parents[1]
FIXTURES = ROOT / "tests" / "fixtures"
SCHEMAS = ROOT / "schemas"
PUBLIC_IMPORTS = {
"docforge": (
"CanonicalApplier",
"CanonicalApplicationService",
"DocForgeError",
"GenericCanonicalApplier",
"Project",
),
"docforge.adapter_contract": (
"AdapterAssembly",
"AdapterEdge",
"AdapterImplementation",
"AdapterLoader",
"AdapterManifest",
"AdapterNode",
"AdapterProject",
"AdapterProjection",
"AdapterProjectSettings",
"AdapterSource",
"AdapterSourceProjection",
"IncrementalAdapterAssembler",
"IncrementalAdapterLoader",
),
"docforge.application": (
"CanonicalApplier",
"CanonicalApplicationService",
"GenericCanonicalApplier",
),
"docforge.index": ("ProjectIndex",),
"docforge.mcp_server": (
"create_project_server",
"create_read_only_server",
"create_server",
),
"docforge.models": (
"Edge",
"LogicEdge",
"LogicNode",
"LogicProjection",
"Node",
"ProjectDescriptor",
"ProjectService",
"ProjectSnapshot",
),
"docforge.render_contract": (
"GenericHtmlRenderer",
"PreparedRender",
"Renderer",
"renderer_for",
),
}
EXPECTED_ENTRY_POINTS = {
"docforge": "docforge.cli:main",
"docforge-mcp": "docforge.mcp_server:main",
"docforge-viewer-manager": "docforge.viewer_manager:main",
}
EXPECTED_CLI_COMMANDS = {
"apply",
"backlinks",
"build",
"check",
"context",
"dependencies",
"filter",
"impact",
"info",
"onboard",
"preview",
"reindex",
"render",
"render-status",
"search",
"show",
"sync",
"validate",
"validate-index",
"visualization-status",
"visualization-stop",
"visualize",
}
EXPECTED_MCP_TOOLS = {
"docforge_abandon_changeset",
"docforge_apply_changeset",
"docforge_backlinks",
"docforge_bootstrap",
"docforge_create_changeset",
"docforge_dependencies",
"docforge_filter_nodes",
"docforge_get_changeset",
"docforge_get_changeset_diff",
"docforge_get_context",
"docforge_get_contract",
"docforge_get_logic",
"docforge_get_node",
"docforge_impact",
"docforge_list_changesets",
"docforge_preview_changeset",
"docforge_project_info",
"docforge_propose_node_create",
"docforge_propose_node_delete",
"docforge_propose_node_move",
"docforge_propose_node_update",
"docforge_propose_relationship_update",
"docforge_rebase_changeset",
"docforge_register_changes",
"docforge_render_status",
"docforge_search",
"docforge_stop_visualization",
"docforge_sync",
"docforge_validate_changeset",
"docforge_validate_project",
"docforge_visualization_status",
"docforge_visualize",
}
class PublicContractTests(unittest.TestCase):
def copy_fixture(self, destination: Path) -> Path:
root = destination / "alpha"
shutil.copytree(FIXTURES / "alpha", root)
return root
@staticmethod
def schema(name: str) -> dict[str, object]:
return json.loads((SCHEMAS / name).read_text(encoding="utf-8"))
def test_distribution_version_entry_points_and_imports_are_stable(self) -> None:
project = tomllib.loads((ROOT / "pyproject.toml").read_text(encoding="utf-8"))["project"]
self.assertEqual("docforge", project["name"])
self.assertEqual(docforge.__version__, project["version"])
self.assertEqual(docforge.__version__, SERVER_VERSION)
scripts = project["scripts"]
for name, target in EXPECTED_ENTRY_POINTS.items():
self.assertEqual(target, scripts[name])
for module_name, names in PUBLIC_IMPORTS.items():
module = importlib.import_module(module_name)
for name in names:
with self.subTest(module=module_name, name=name):
self.assertTrue(hasattr(module, name))
def test_cli_and_mcp_names_remain_additively_compatible(self) -> None:
parser = _parser()
commands = next(
action for action in parser._actions if isinstance(action, argparse._SubParsersAction)
)
self.assertLessEqual(EXPECTED_CLI_COMMANDS, set(commands.choices))
self.assertLessEqual(
EXPECTED_MCP_TOOLS,
set((*ALL_TOOLS, *APPLICATION_TOOLS)),
)
self.assertEqual((*READ_TOOLS, *PROPOSAL_TOOLS), ALL_TOOLS)
completed = subprocess.run(
[sys.executable, "-m", "docforge.mcp_server", "--help"],
cwd=ROOT,
check=True,
capture_output=True,
text=True,
)
self.assertIn("--project-root", completed.stdout)
self.assertIn("--no-ast", completed.stdout)
def test_published_schemas_validate_their_current_contract_examples(self) -> None:
for path in sorted(SCHEMAS.glob("*.json")):
with self.subTest(schema=path.name):
Draft202012Validator.check_schema(json.loads(path.read_text(encoding="utf-8")))
descriptor = tomllib.loads(
(FIXTURES / "alpha" / ".docforge" / "project.toml").read_text(encoding="utf-8")
)
Draft202012Validator(self.schema("project.schema.json")).validate(descriptor)
with tempfile.TemporaryDirectory() as directory:
root = self.copy_fixture(Path(directory))
project = Project.open(root)
snapshot = project.load()
node = snapshot.nodes[0]
node_metadata = {
"schema_version": 1,
"id": node.node_id,
"title": node.title,
"family": node.family,
"authority": node.authority,
"status": node.status,
"tags": list(node.tags),
"summary": node.summary,
}
Draft202012Validator(self.schema("node.schema.json")).validate(node_metadata)
edge = snapshot.edges[0]
Draft202012Validator(self.schema("edge.schema.json")).validate(
{
"source_id": edge.source_id,
"relation": edge.relation,
"target_id": edge.target_id,
}
)
store = ChangesetStore(project, "alpha-editor")
store.create("public-contract")
changeset = json.loads(
(root / ".docforge" / "changesets" / "public-contract.json").read_text(
encoding="utf-8"
)
)
Draft202012Validator(self.schema("changeset.schema.json")).validate(changeset)
service = DocForgeService(project)
ProjectIndex(project).build()
success = service.project_info()
error = service.invoke(lambda: service.index.get_node("missing.public-contract-node"))
result_validator = Draft202012Validator(self.schema("result.schema.json"))
result_validator.validate(success)
result_validator.validate(error)
def test_changeset_hash_is_exact_canonical_json_sha256(self) -> None:
document = {
"schema_version": 1,
"changeset_id": "hash-contract",
"project_id": "alpha-docs",
"root_fingerprint": "0" * 16,
"base_revision": "unversioned",
"base_source_hash": "1" * 64,
"creator": "alpha-editor",
"operations": [],
}
expected = hashlib.sha256(canonical_bytes(document)).hexdigest()
self.assertEqual(expected, document_hash(document))
if __name__ == "__main__":
unittest.main()

493
tools/milestone0_baseline.py Normal file
View file

@ -0,0 +1,493 @@
"""Reproducible Milestone 0 timing, memory, rendering, and response-size baseline."""
from __future__ import annotations
import argparse
import json
import math
import platform
import resource
import statistics
import subprocess
import sys
import tempfile
import time
from collections.abc import Callable
from pathlib import Path
from typing import cast
from docforge.application import CanonicalApplicationService, GenericCanonicalApplier
from docforge.changesets import ChangesetStore
from docforge.context import compile_context
from docforge.index import ProjectIndex
from docforge.mcp_server import DocForgeService
from docforge.project import Project
from docforge.rendering import RenderService
from docforge.visualization import VisualizationIndexSnapshot
ROOT = Path(__file__).resolve().parents[1]
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Measure DocForge against a disposable deterministic generic project."
)
parser.add_argument("--nodes", type=int, default=1000)
parser.add_argument("--samples", type=int, default=10)
parser.add_argument("--cold-samples", type=int, default=3)
parser.add_argument("--output", type=Path)
return parser
def _node_id(index: int) -> str:
return f"guide.node-{index:04d}"
def _write_project(root: Path, node_count: int) -> None:
content_root = root / "docs" / "content"
template_root = root / "docs" / "templates"
descriptor_root = root / ".docforge"
content_root.mkdir(parents=True)
template_root.mkdir(parents=True)
descriptor_root.mkdir(parents=True)
(root / "POLICY.md").write_text(
"# Synthetic benchmark policy\n\n"
"This disposable project measures repository-native DocForge operations.\n",
encoding="utf-8",
)
(template_root / "manual.html").write_text(
'<!doctype html><html lang="en"><head><meta charset="utf-8">'
"<title>{{ docforge_title }}</title></head>"
'<body data-project="{{ docforge_project_id }}" '
'data-view="{{ docforge_view_id }}"><main>{{ docforge_content }}</main></body></html>\n',
encoding="utf-8",
)
(descriptor_root / "project.toml").write_text(
f"""schema_version = 1
project_id = "synthetic-{node_count}"
title = "Synthetic {node_count} Node Baseline"
adapter = "generic"
[sources]
content_roots = ["docs/content"]
authority_files = ["POLICY.md"]
[derived]
cache_root = ".docforge/cache"
index = ".docforge/cache/index.sqlite3"
[changesets]
root = ".docforge/changesets"
[[changesets.writers]]
id = "benchmark-editor"
families = ["guide"]
operations = ["create", "update", "move", "delete"]
[render]
template_root = "docs/templates"
preview_root = ".docforge/previews"
[[render.views]]
id = "manual"
renderer = "generic_html"
template = "manual.html"
output = ".docforge/rendered/manual.html"
title = "Synthetic Manual"
families = ["guide"]
[graph]
allowed_relations = ["depends_on", "relates_to"]
[limits]
max_source_bytes = 100000
max_nodes = {max(node_count * 2, 100)}
max_query_chars = 200
max_results = 100
max_traversal_depth = 8
max_context_tokens = 32000
max_tool_output_chars = 5000000
max_changesets = 100
max_changeset_operations = 100
max_changeset_bytes = 1000000
max_render_views = 10
max_template_bytes = 1000000
max_render_bytes = 20000000
[[profiles]]
id = "active"
families = ["guide"]
statuses = ["active"]
required_nodes = ["{_node_id(node_count - 1)}"]
token_budget = 32000
dependency_depth = 8
""",
encoding="utf-8",
)
for index in range(node_count):
relationships = f'depends_on = ["{_node_id(index - 1)}"]\n' if index > 0 else ""
(content_root / f"node-{index:04d}.md").write_text(
f"""+++
schema_version = 1
id = "{_node_id(index)}"
title = "Synthetic node {index:04d}"
family = "guide"
authority = "derived"
status = "active"
tags = ["synthetic", "batch-{index // 100:02d}"]
summary = "Synthetic measurement node {index:04d} for the repository-native baseline."
{relationships}+++
This deterministic benchmark content exists only in a disposable temporary directory.
""",
encoding="utf-8",
)
def _json_size(value: object) -> int | None:
if value is None:
return None
if isinstance(value, str):
return len(value.encode("utf-8"))
return len(
json.dumps(value, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")
)
def _measure(
operation: Callable[[], object],
*,
samples: int,
warmups: int = 1,
response_size: bool = True,
) -> tuple[dict[str, object], object]:
for _ in range(warmups):
operation()
durations: list[float] = []
last: object = None
for _ in range(samples):
started = time.perf_counter_ns()
last = operation()
durations.append((time.perf_counter_ns() - started) / 1_000_000)
ordered = sorted(durations)
p95_index = max(0, math.ceil(len(ordered) * 0.95) - 1)
result: dict[str, object] = {
"samples": samples,
"median_ms": round(statistics.median(ordered), 3),
"p95_ms": round(ordered[p95_index], 3),
"min_ms": round(ordered[0], 3),
"max_ms": round(ordered[-1], 3),
}
if response_size:
result["response_bytes"] = _json_size(last)
return result, last
def _run(command: list[str]) -> str:
return subprocess.run(
command,
cwd=ROOT,
check=True,
capture_output=True,
text=True,
).stdout
def _git(command: list[str]) -> str:
return subprocess.run(
["git", *command],
cwd=ROOT,
check=True,
capture_output=True,
text=True,
).stdout.strip()
def _benchmark(root: Path, node_count: int, samples: int, cold_samples: int) -> dict[str, object]:
target = _node_id(node_count - 1)
project = Project.open(root)
index = ProjectIndex(project)
operations: dict[str, object] = {}
operations["project_open"], _ = _measure(
lambda: Project.open(root),
samples=samples,
response_size=False,
)
operations["project_load"], _ = _measure(
project.load,
samples=samples,
response_size=False,
)
operations["full_index_build"], _ = _measure(
index.build,
samples=max(1, cold_samples),
warmups=0,
)
def cold_synchronize() -> dict[str, object]:
index.path.unlink(missing_ok=True)
index.attestation_path.unlink(missing_ok=True)
return ProjectIndex(project).synchronize()
operations["cold_synchronize"], _ = _measure(
cold_synchronize,
samples=cold_samples,
warmups=0,
)
index = ProjectIndex(project)
index.synchronize()
operations["full_index_check"], _ = _measure(index.check, samples=samples)
operations["warm_no_change_synchronize"], _ = _measure(
index.synchronize,
samples=samples,
)
operations["exact_node"], _ = _measure(
lambda: index.get_node(target),
samples=samples,
)
operations["search_limit_20"], _ = _measure(
lambda: index.search("Synthetic measurement", limit=20),
samples=samples,
)
operations["dependencies_depth_8"], _ = _measure(
lambda: index.dependencies(target, depth=8),
samples=samples,
)
operations["impact_depth_8"], _ = _measure(
lambda: index.impact(_node_id(0), depth=8),
samples=samples,
)
operations["context_32k"], context = _measure(
lambda: compile_context(index, "active", 32000),
samples=samples,
)
renderer = RenderService(project)
operations["manual_render"], _ = _measure(
lambda: renderer.render("manual"),
samples=max(1, cold_samples),
warmups=0,
)
operations["manual_render_status"], _ = _measure(
lambda: renderer.status("manual"),
samples=samples,
)
operations["viewer_snapshot_pin"], _ = _measure(
lambda: VisualizationIndexSnapshot(index, index.check()),
samples=max(1, cold_samples),
response_size=False,
)
snapshot = VisualizationIndexSnapshot(index, index.check())
operations["viewer_overview"], _ = _measure(snapshot.overview, samples=samples)
operations["viewer_search_limit_20"], _ = _measure(
lambda: snapshot.search(
query="Synthetic measurement",
family=None,
kind=None,
language=None,
capability=None,
limit=20,
),
samples=samples,
)
operations["viewer_neighborhood_depth_8"], _ = _measure(
lambda: snapshot.node(target, depth=8, limit=100),
samples=samples,
)
operations["viewer_web_depth_8"], _ = _measure(
lambda: snapshot.web(target, depth=8, limit=100),
samples=samples,
)
service = DocForgeService(project)
operations["mcp_bootstrap"], _ = _measure(service.bootstrap, samples=samples)
operations["mcp_exact_node"], _ = _measure(
lambda: service.invoke(lambda: service.index.get_node(target)),
samples=samples,
)
operations["mcp_search_limit_20"], _ = _measure(
lambda: service.invoke(lambda: service.index.search("Synthetic measurement", limit=20)),
samples=samples,
)
operations["mcp_context_32k"], _ = _measure(
lambda: service.invoke(lambda: compile_context(service.index, "active", 32000)),
samples=samples,
)
operations["mcp_render_status"], _ = _measure(
lambda: service.render_status("manual"),
samples=samples,
)
store = ChangesetStore(project, "benchmark-editor")
registration_counter = 0
def register() -> dict[str, object]:
nonlocal registration_counter
registration_counter += 1
return store.register(
f"benchmark-{registration_counter:03d}",
[
{
"operation": "update",
"node_id": target,
"metadata": {
"summary": (
"Synthetic measurement node updated only inside a benchmark proposal."
)
},
"rationale": "Measure atomic registration without changing canonical sources.",
}
],
)
registration_samples = min(samples, 10)
operations["changeset_register"], registered = _measure(
register,
samples=registration_samples,
warmups=0,
)
changeset_id = f"benchmark-{registration_counter:03d}"
operations["changeset_validate"], _ = _measure(
lambda: store.validate(changeset_id),
samples=samples,
)
operations["changeset_diff"], _ = _measure(
lambda: store.diff(changeset_id),
samples=samples,
)
if not isinstance(registered, dict):
raise RuntimeError("Changeset registration returned an invalid result")
registered_result = cast(dict[str, object], registered)
application = CanonicalApplicationService(
project,
applier_id="benchmark-editor",
applier=GenericCanonicalApplier(project),
)
operations["exact_hash_apply_and_refresh"], _ = _measure(
lambda: application.apply(changeset_id, str(registered_result["changeset_hash"])),
samples=1,
warmups=0,
)
operations["cli_info_startup"], _ = _measure(
lambda: _run(
[
sys.executable,
"-m",
"docforge.cli",
"--project-root",
str(root),
"info",
]
),
samples=max(1, cold_samples),
warmups=0,
)
operations["cli_exact_startup"], _ = _measure(
lambda: _run(
[
sys.executable,
"-m",
"docforge.cli",
"--project-root",
str(root),
"show",
target,
]
),
samples=max(1, cold_samples),
warmups=0,
)
operations["mcp_import_and_help"], _ = _measure(
lambda: _run([sys.executable, "-m", "docforge.mcp_server", "--help"]),
samples=max(1, cold_samples),
warmups=0,
)
manual_path = root / ".docforge" / "rendered" / "manual.html"
static_asset_bytes = sum(
path.stat().st_size
for path in (
ROOT / "src" / "docforge" / "assets" / "graph.html",
ROOT / "src" / "docforge" / "assets" / "graph.css",
ROOT / "src" / "docforge" / "assets" / "graph.js",
)
)
return {
"fixture": {
"kind": "synthetic_generic",
"node_count": node_count,
"edge_count": node_count - 1,
"source_file_count": node_count,
"context_budget_tokens": 32000,
"traversal_depth": 8,
},
"operations": operations,
"sizes": {
"context_compact_bytes": _json_size(context),
"manual_artifact_bytes": manual_path.stat().st_size,
"static_viewer_assets_bytes": static_asset_bytes,
},
"process_peak_rss_kib": int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss),
}
def main() -> int:
arguments = _parser().parse_args()
if arguments.nodes < 2:
raise SystemExit("--nodes must be at least 2")
if arguments.samples < 1 or arguments.cold_samples < 1:
raise SystemExit("sample counts must be positive")
with tempfile.TemporaryDirectory(prefix="docforge-milestone0-") as directory:
root = Path(directory).resolve()
_write_project(root, arguments.nodes)
measurement = _benchmark(
root,
arguments.nodes,
arguments.samples,
arguments.cold_samples,
)
status = _git(["status", "--porcelain"])
result: dict[str, object] = {
"schema_version": 1,
"benchmark": "docforge2_milestone0",
"source": {
"revision": _git(["rev-parse", "HEAD"]),
"dirty": bool(status),
},
"environment": {
"platform": platform.platform(),
"machine": platform.machine(),
"python": platform.python_version(),
"implementation": platform.python_implementation(),
},
"method": {
"clock": "time.perf_counter_ns",
"memory": "resource.getrusage(RUSAGE_SELF).ru_maxrss",
"response_size": "UTF-8 bytes of compact sorted JSON",
"samples": arguments.samples,
"cold_samples": arguments.cold_samples,
},
**measurement,
"known_gaps": [
"Generic warm reads still parse canonical source files.",
"Compiler stages are not separately instrumented.",
"Scaled incremental extraction is not measured by this generic fixture.",
"Manual planning is not separated from rendering.",
"Portable graph planning and rendering do not exist in Milestone 0.",
"Per-operation peak RSS requires an external process harness.",
],
}
encoded = json.dumps(result, sort_keys=True, indent=2) + "\n"
if arguments.output is not None:
output = arguments.output.resolve()
output.parent.mkdir(parents=True, exist_ok=True)
output.write_text(encoded, encoding="utf-8")
sys.stdout.write(encoded)
return 0
if __name__ == "__main__":
raise SystemExit(main())

2
uv.lock generated
View file

@ -218,6 +218,7 @@ dependencies = [
[package.dev-dependencies]
dev = [
{ name = "jsonschema" },
{ name = "pytest" },
{ name = "ruff" },
]
@ -233,6 +234,7 @@ requires-dist = [
[package.metadata.requires-dev]
dev = [
{ name = "jsonschema", specifier = ">=4.25,<5" },
{ name = "pytest", specifier = ">=9.1,<10" },
{ name = "ruff", specifier = ">=0.15,<1" },
]