docs(ocr): archive ingestion integration spec

openspec/config.yaml

@@ -5,12 +5,20 @@ context: |
   Tech stack: Node.js 22 (ESM), TypeScript 5.8 strict, Express 4, tsx runner
   Architecture: modular service — src/modules/{ingest,process,parsers,retrieve,answer,catalog,embeddings,vectorstore,logs}, src/api (OpenAPI), src/shared
   Data stores: PostgreSQL (pg) + Qdrant (@qdrant/js-client-rest) + OpenAI embeddings
-  Testing: node:test + node:assert/strict executed via `npm test` (NODE_ENV=test tsx --test tests/**/*.test.ts)
+  Testing: node:test + node:assert/strict via `npm test`; OCR Python service via `ocr-service/.venv/bin/python -m pytest ocr-service/tests` (repo-local venv)
   Quality: `npm run check` (tsc --noEmit, strict), `npm run build` (tsc emit to dist/)
   Deploy: Docker + EasyPanel; migrations are manual scripts (migrate:lifecycle, migrate:legacy:lifecycle)
   No linter, formatter, or coverage tooling is configured.
 
-strict_tdd: true
+strict_tdd: false
+
+# Fail-closed correction (2026-09-21, sdd-init guard before archive):
+# explicit strict_tdd: true is preserved as intent, but two in-scope projects
+# (./ and ocr-service/) have no single explicit workspace-level test command
+# that covers the complete set. Per decision gate, strict_tdd fails closed to
+# false. Downstream TDD execution requires per-project commands:
+# - ./ → npm test
+# - ocr-service/ → ocr-service/.venv/bin/python -m pytest ocr-service/tests
 
 testing:
   projects:
@@ -30,6 +38,21 @@ testing:
         type_checker: true
         type_checker_command: npm run check
         formatter: false
+    - path: ocr-service/
+      stack: Python 3.11 + FastAPI 0.116 + PaddleOCR (uvicorn, httpx)
+      test_command: ocr-service/.venv/bin/python -m pytest ocr-service/tests
+      test_framework: pytest 8.4 (FastAPI TestClient)
+      layers:
+        unit: true
+        integration: true
+        e2e: false
+      coverage:
+        available: false
+        command: ""
+      quality:
+        linter: false
+        type_checker: false
+        formatter: false
 
 rules:
   proposal:
@@ -50,9 +73,13 @@ rules:
       - Follow existing code patterns (ESM imports with .js extensions, strict types)
       - Never read or expose .env* files, llaves, or backups/ contents
     tdd: true
-    test_command: npm test
+    test_commands:
+      - npm test
+      - ocr-service/.venv/bin/python -m pytest ocr-service/tests
   verify:
-    test_command: npm test
+    test_commands:
+      - npm test
+      - ocr-service/.venv/bin/python -m pytest ocr-service/tests
     build_command: npm run build
     coverage_threshold: 0
   archive:

openspec/specs/ocr-ingest-orchestration/spec.md

@@ -0,0 +1,94 @@
+# OCR Ingest Orchestration Specification
+
+## Purpose
+
+Define routing, lifecycle, recovery, and release behavior for OCR ingestion without changing native ingestion.
+
+## Requirements
+
+### Requirement: Native and OCR Routing
+
+The system MUST preserve the synchronous Point 2 path when all PDF pages have sufficient native text. Before accepting OCR work, it MUST retain the original and manifest durably and create a non-active version with a null source content hash.
+
+#### Scenario: Native PDF remains synchronous
+
+- GIVEN a PDF whose pages satisfy native detection
+- WHEN it is ingested
+- THEN the system MUST return the existing `201` response and MUST NOT contact OCR
+- AND lifecycle and retrieval behavior MUST remain unchanged
+
+#### Scenario: OCR work is accepted asynchronously
+
+- GIVEN a valid scanned or mixed PDF requiring OCR
+- WHEN ingestion is accepted
+- THEN it MUST return `202` with source/version identity, `state: indexing`, `phase: ocr_queued`, and `statusUrl`
+- AND `reviewUrl` MUST be null and `activated` MUST be false
+
+### Requirement: Version-Level Progress Gate
+
+The system MUST report authenticated document/page progress using `native_extracting`, `ocr_queued`, `ocr_running`, `review_required`, `indexing`, `ready`, `active`, `failed`, or `rejected`. A version SHALL reach `review_required` only after every document is native-complete or OCR-complete.
+
+#### Scenario: Multi-document candidate completes
+
+- GIVEN a version with native and OCR documents
+- WHEN all non-blank pages pass their applicable extraction gates
+- THEN the version MUST become `review_required` with a review URL
+- AND no document MAY be omitted from reported progress
+
+#### Scenario: One document fails
+
+- GIVEN any requested page fails or violates integrity or quality gates
+- WHEN version progress is evaluated
+- THEN the version MUST become `failed` with an actionable error
+- AND it MUST produce no embeddings or partial activation
+
+### Requirement: Idempotent Concurrent Dispatch
+
+The system MUST maintain one job per version/document, prevent concurrent ownership, and reuse its remote idempotency key during recovery.
+
+#### Scenario: Duplicate ingestion while pending
+
+- GIVEN the same source and original have a non-terminal OCR version
+- WHEN ingestion is retried
+- THEN it MUST return that version and job without duplicates
+
+#### Scenario: Lease recovery
+
+- GIVEN one job has a live lease and another has an expired lease
+- WHEN dispatch or reconciliation runs concurrently
+- THEN the live job MUST remain untouched and the expired job MUST be redispatched once with its original key
+- AND an existing remote job MUST be recovered rather than duplicated
+
+### Requirement: Fail-Closed Availability
+
+OCR candidates MUST remain invisible until approved and active. OCR, catalog, embedding, or vector failures MUST preserve the previous active version.
+
+#### Scenario: OCR is unavailable
+
+- GIVEN OCR cannot complete after allowed transient retries
+- WHEN the job is processed
+- THEN the candidate MUST fail closed without activation or engine substitution
+- AND the prior active version MUST remain retrievable
+
+#### Scenario: Catalog is unavailable during retrieval
+
+- GIVEN active-version resolution is unavailable
+- WHEN retrieval is requested
+- THEN the system MUST return `503` and MUST NOT query without an active-version filter
+
+### Requirement: Controlled Enablement and Rollback
+
+OCR MUST be enabled only after its private service is ready, RAG is deployed disabled, and native behavior is verified. Disabling it MUST stop new OCR work while preserving native ingestion and candidate invisibility.
+
+#### Scenario: Release gate passes
+
+- GIVEN OCR is ready and `npm run check`, `npm run build`, `npm test`, and offline OCR tests pass
+- WHEN OCR is enabled after native production verification
+- THEN new eligible ingestions MAY use the OCR path
+
+#### Scenario: Operational rollback
+
+- GIVEN OCR processing or acceptance fails after deployment
+- WHEN OCR is disabled or the RAG release is rolled back
+- THEN native ingestion MUST continue and existing candidates MUST remain non-retrievable
+- AND the previous active corpus MUST require no embedding recomputation

openspec/specs/ocr-processing/spec.md

@@ -0,0 +1,105 @@
+# OCR Processing Specification
+
+## Purpose
+
+Define safe OCR behavior.
+
+## Requirements
+
+### Requirement: Per-Page Native Detection
+
+The system MUST evaluate PDF pages independently. Native text is sufficient only when `N >= 120`, `A >= 80`, `W >= 20`, and `R <= 0.01`, under `pdf-detection-v1`.
+
+#### Scenario: Textual pages bypass OCR
+
+- GIVEN every page meets native thresholds
+- WHEN extraction completes
+- THEN native text MUST be selected and no OCR job MAY be created
+
+#### Scenario: Mixed PDF selects exact pages
+
+- GIVEN only some pages fail any native threshold
+- WHEN detection completes
+- THEN OCR MUST receive those unique, ordered, one-based pages
+- AND `N == 0` MUST always request OCR inspection
+
+### Requirement: Blank and OCR Quality Gates
+
+A page MAY be blank only with `inkCoverage < 0.015` and fewer than 10 OCR characters. Non-blank OCR MUST meet `nonWhitespaceCharacters >= 40`, median `>= 0.80`, p10 `>= 0.50`, and low-confidence ratio `<= 0.20`.
+
+#### Scenario: Verified blank contributes no text
+
+- GIVEN both blank conditions hold
+- WHEN the page is classified
+- THEN it MUST be blank and contribute no candidate text
+
+#### Scenario: Visible ink fails closed
+
+- GIVEN visible ink and OCR fails the quality gate
+- WHEN processing completes
+- THEN the page MUST NOT be blank and the version MUST fail with `OCR_QUALITY_BLOCKED`
+
+### Requirement: Deterministic Candidate Composition
+
+The system MUST use native text or OCR lines, never both per page. It SHALL order lines by top, left, and original index; pages ascending; and separate non-blank pages with `--- Page N ---`.
+
+#### Scenario: Candidate is reproducible
+
+- GIVEN identical validated page records
+- WHEN the candidate is composed repeatedly
+- THEN canonical text, ordering, and hashes MUST be identical
+- AND raw OCR and native extraction MUST remain available for audit
+
+### Requirement: Risk Tokens Remain Uncorrected
+
+The system MUST mark `\b[A-Za-z]{2,}[A-Za-z0-9_-]*\d[A-Za-z0-9_-]*\b` tokens and elevate ambiguous, differing, unique, or context-adjacent ones. It MUST NOT auto-correct casing or `O/0`, `I/1/l`, or `S/5`.
+
+#### Scenario: Known code ambiguities are flagged
+
+- GIVEN OCR emits `CBGO4a`, `FATo7`, `DSAuo8`, or `NSAvo6`
+- WHEN risk analysis runs
+- THEN each token MUST be shown for human review unchanged
+
+### Requirement: Private Idempotent OCR Service
+
+Private PaddleOCR `3.4.0` with PaddlePaddle CPU `3.2.2` MUST require bearer authorization, allowlisted jobs, and deterministic identity for identical hash, config, and pages. It MUST NOT be Internet-accessible.
+
+#### Scenario: Idempotent submission
+
+- GIVEN valid requests share a key and payload
+- WHEN both are submitted
+- THEN both MUST identify one job and result without duplicate processing
+
+#### Scenario: Conflicting or invalid submission
+
+- GIVEN a reused key conflicts, authentication fails, or configuration is disallowed
+- WHEN submission occurs
+- THEN OCR MUST return the contracted `409`, `401/403`, or `400` class
+
+#### Scenario: Result integrity mismatch
+
+- GIVEN returned integrity fields differ from the request
+- WHEN RAG validates the result
+- THEN processing MUST fail without a reviewable candidate
+
+### Requirement: Operational Limits and Health
+
+OCR MUST enforce 50 MiB, 100 pages, 200 DPI, 25 megapixels/page, one concurrent job, one replica, queue depth three, 60 seconds/page, 15 minutes total, and 3 CPU/5 GiB maximum. Readiness MUST await pinned models; polling SHALL back off 2–15 seconds.
+
+#### Scenario: Unsafe input is rejected before render
+
+- GIVEN a PDF is oversized, encrypted, corrupt, too long, or decompression-unsafe
+- WHEN it is accepted for processing
+- THEN it MUST be rejected before full render without changing the active version
+
+#### Scenario: Queue or transient service pressure
+
+- GIVEN the queue is full or OCR is transiently unavailable
+- WHEN RAG submits work
+- THEN `429` MUST remain retryable; connection/`502`/`503` failures MUST allow at most two same-key resubmissions
+
+#### Scenario: Deterministic failure is not retried
+
+- GIVEN OCR returns `400`, `401`, `403`, `413`, `422`, or deterministic render failure
+- WHEN RAG handles the response
+- THEN it MUST NOT retry or switch engines

openspec/specs/ocr-review-workflow/spec.md

@@ -0,0 +1,106 @@
+# OCR Review Workflow Specification
+
+## Purpose
+
+Define authenticated review, atomic correction, activation, and retention.
+
+## Requirements
+
+### Requirement: Mandatory Authenticated Review
+
+OCR versions MUST remain `review_required` and non-retrievable until approved. Status, review, images, approval, and rejection MUST require the administrator token.
+
+#### Scenario: Reviewer inspects a candidate
+
+- GIVEN all documents passed the version gate
+- WHEN an authorized reviewer opens review
+- THEN it MUST show image, native text, raw OCR, boxes, confidence, differences, and risks
+
+#### Scenario: Unauthorized or premature access
+
+- GIVEN credentials fail or the version is not reviewable
+- WHEN review or decision is requested
+- THEN it MUST reject the request without exposing artifacts or changing state
+
+### Requirement: Atomic Optimistic Corrections
+
+Approval MUST validate `candidateSha256`, unique line identities, and every `expectedLineSha256` before changes. Reviewed text SHALL be immutable.
+
+#### Scenario: Valid corrections commit together
+
+- GIVEN hashes are current and correction targets are unique
+- WHEN approval is submitted
+- THEN all replacements MUST commit together with reviewer, time, and resulting hashes
+
+#### Scenario: Stale or conflicting correction
+
+- GIVEN any candidate, line, identity, or box is stale or duplicated
+- WHEN approval is submitted
+- THEN the system MUST return `409` without corrections or state transition
+
+### Requirement: Approval and Activation Gate
+
+Approval MUST cover the version, derive final hashes, and move `review_required` to `indexing` before chunking. Activation SHALL use Point 2 verification and the expected active version.
+
+#### Scenario: Approved content becomes active
+
+- GIVEN corrections are valid and the expected active version is current
+- WHEN indexing and point verification succeed
+- THEN the new version MUST become active atomically and the previous one superseded
+- AND only reviewed content MAY be retrieved
+
+#### Scenario: Active version changes during review
+
+- GIVEN another version activates during review
+- WHEN approved indexing completes
+- THEN the candidate MUST remain `ready`, return `409 ACTIVE_VERSION_CHANGED`, and not replace it
+
+#### Scenario: Reusable content already exists
+
+- GIVEN final content, fingerprint, and metadata match a reusable version
+- WHEN approval checks the active-version precondition
+- THEN the candidate MUST be `rejected` with `DUPLICATE_REUSABLE_VERSION`
+- AND only the existing version MAY be activated
+
+### Requirement: Rejection and Failure Isolation
+
+Rejection MUST produce `rejected`, preserve temporary audit evidence, and generate no embeddings. Failed or rejected candidates SHALL never alter the active version.
+
+#### Scenario: Reviewer rejects candidate
+
+- GIVEN a version is `review_required`
+- WHEN an authorized reviewer rejects it
+- THEN it MUST become `rejected` with audit data and no indexing
+
+### Requirement: Retention Safety
+
+Private artifacts MUST use `0600` permissions and verifiable hashes. The system MUST retain review 30 days, failed/rejected artifacts 7 days, review images 7 days post-decision, OCR copies until transfer or 24 hours, and active artifacts until 30 days after supersession. Cleanup MUST be idempotent and never delete active versions.
+
+#### Scenario: Review expires
+
+- GIVEN a version remains `review_required` for 30 days
+- WHEN daily retention runs exclusively
+- THEN it MUST become `rejected` with `REVIEW_EXPIRED` before removal
+
+#### Scenario: Active artifacts are protected
+
+- GIVEN an active version is past a TTL
+- WHEN retention runs concurrently with review or purge
+- THEN its original, reviewed text, raw OCR, and manifest MUST remain
+- AND cleanup MUST not escape the version directory or expose a public URL
+
+#### Scenario: Interrupted cleanup resumes
+
+- GIVEN deletion stopped after retention entered its deleting state
+- WHEN cleanup runs again
+- THEN deletion MUST resume safely and finish in the deleted state without affecting other versions
+
+### Requirement: Human-Approved Production Acceptance
+
+The release MUST prove corrected content after approval; confidence SHALL NOT authorize activation.
+
+#### Scenario: FacturaTech acceptance succeeds
+
+- GIVEN the real FacturaTech PDF has been reviewed and its 34 entries approved
+- WHEN the version becomes active and retrieval is queried
+- THEN all 34 entries and `CBG04a`, `FAT07`, `DSAU08`, and `NSAV06` MUST be returned exactly