docs(sdd): define OCR integration contracts

openspec/changes/ocr-ingest-integration/design.md

@@ -0,0 +1,95 @@
+# Design: OCR Ingest Integration
+
+## Technical Approach
+
+Insert page-aware OCR before existing lifecycle indexing. Native PDFs retain synchronous `201`; OCR candidates persist originals, catalog rows, and jobs before returning `202`. PostgreSQL remains authoritative, Qdrant receives only reviewed content, and every failure preserves the active pointer.
+
+## Architecture Decisions
+
+| Decision | Choice and rationale |
+|---|---|
+| Extraction | Fixture-gate `pdfjs-dist`; use `pdf-parse` `pagerender` only if proven on Node 22. Aggregated-text slicing is forbidden. |
+| Execution | One in-process dispatcher with PostgreSQL leases; recovery needs no second Node deployment. |
+| OCR | Private FastAPI, one worker/replica, baked PaddleOCR 3.4.0/PaddlePaddle CPU 3.2.2 models. |
+| Ownership | RAG owns durable `0600` artifacts; OCR owns a 24-hour work copy; reviewed bytes are immutable. |
+
+## Data Flow and Concurrency
+
+`upload → disk stage → page detection → native 201 OR durable candidate/job → 202 → OCR → review_required → approve → index/verify → activation CAS`
+
+`findPendingOcrVersion(identity)` is distinct from `findReusableVersion` and joins versions, documents, and jobs using:
+
+```ts
+type PendingOcrIdentity={sourceId:string;originalManifestHash:Sha256;processingFingerprint:Sha256;metadataHash:Sha256}
+```
+
+It accepts `source_content_hash IS NULL` and states `pending|indexing|review_required`. Migration 002 adds a matching partial unique index. Under `withSourceLock`, ingestion validates source type/reference, performs this lookup, then either returns the persisted version/jobs or preallocates `versionId`, durably stages/fsyncs originals, and transactionally inserts one candidate plus `UNIQUE(version_id,document_id)` jobs. A uniqueness loss reloads the winner; the losing staging directory is orphan-swept. All duplicates return the same `202/statusUrl` and persisted activation intent.
+
+Approval starts one transaction locking source and candidate. Before artifact writes, corrections, or transition, it requires `request.expectedActiveVersionId === candidate.base_active_version_id === source.active_version_id`; otherwise `409 ACTIVE_VERSION_CHANGED` and zero mutation. Candidate/line hashes and unique targets are then validated atomically.
+
+After final hashes, source-locked `findReusableVersion(finalContent,fingerprint,metadata)` runs. A match rejects the candidate: `DUPLICATE_REUSABLE_VERSION`. Persisted `candidate.activate_requested === false` preserves the active pointer; only true activates the `ready|superseded|active` version via expected-active CAS. CAS failure returns `409 ACTIVE_VERSION_CHANGED` and rolls back the reusable branch. Otherwise, approval persists corrections/hashes and indexes to `ready`. Only persisted `candidate.activate_requested === true` invokes Point-2 activation with the original expected-active CAS; false succeeds in `ready` without pointer change. CAS failure returns `409 ACTIVE_VERSION_CHANGED`, leaving the candidate `ready`.
+
+Jobs use `queued→running→succeeded|failed`; expired leases return to `queued` with unchanged remote key/ID. Versions use `pending→indexing→review_required→indexing→ready→active`, plus fail/reject/purge paths. Retention uses version locks and `present→retention_deleting→retention_deleted`; active artifacts are never TTL-deleted.
+
+## Interfaces / Contracts
+
+`OcrAccepted` is the OCR `202`; all `/ingestions/*` routes require lifecycle bearer authentication.
+
+```ts
+type OcrAccepted={accepted:true;sourceId:string;versionId:UUID;versionNumber:number;state:"indexing";phase:"ocr_queued";statusUrl:string;reviewUrl:null;activated:false}
+type IngestionStatus={sourceId:string;versionId:UUID;state:VersionState;phase:"native_extracting"|"ocr_queued"|"ocr_running"|"review_required"|"indexing"|"ready"|"active"|"failed"|"rejected";activated:boolean;documents:{documentId:string;state:"native_complete"|"ocr_queued"|"ocr_running"|"ocr_complete"|"failed";completedPages:number;totalPages:number;pages:{page:number;method:"native"|"ocr"|"blank";state:"pending"|"native_complete"|"ocr_queued"|"ocr_running"|"ocr_complete"|"blank"|"failed";errorCode?:string}[]}[];error:{code:string;message:string;retryable:boolean}|null;statusUrl:string;reviewUrl:string|null}
+type Review={versionId:UUID;candidateSha256:Sha256;baseActiveVersionId:UUID|null;documents:{documentId:string;pages:{page:number;imageUrl:string;nativeText:string;ocr:{text:string;lines:{lineId:string;text:string;confidence:number;bbox:number[]}[]};candidateText:string;differences:string[];risks:string[]}[]}[]}
+type Approve={candidateSha256:Sha256;expectedActiveVersionId:UUID|null;reviewedBy:string;corrections:{documentId:string;page:number;lineId:string;expectedLineSha256:Sha256;replacementText:string}[]}
+type Reject={candidateSha256:Sha256;reviewedBy:string;reason:string}
+```
+
+Reject requires current candidate hash and `review_required`; success returns `{versionId,state:"rejected",activated:false}`. Approve returns `{versionId,state,activated,activatedVersionId?,errorCode?}`; hash/state/precondition conflicts are `409`.
+
+Authenticated `POST /v1/jobs` uses multipart `file`/`request` and `Idempotency-Key=<documentSha256>:<configVersion>:<pagesSha256>`; `GET /v1/jobs/:id` returns status and `GET /v1/jobs/:id/result` returns result:
+
+```ts
+type Submit={documentSha256:Sha256;pages:number[];languages:["es","en"];dpi:200;engine:"paddleocr";engineVersion:"3.4.0";runtimeVersion:"3.2.2";configVersion:"ocr-v1";returnLayout:true}
+type Ack={jobId:string;status:"queued";documentSha256:Sha256;requestedPages:number[];configVersion:"ocr-v1";createdAt:ISODate}
+type JobStatus={jobId:string;status:"queued"|"running"|"succeeded"|"failed";completedPages:number;totalPages:number;error:{code:string;message:string}|null}
+type Result={schemaVersion:"1";jobId:string;documentSha256:Sha256;engine:{name:"paddleocr";version:"3.4.0";runtime:"paddlepaddle-3.2.2";device:"cpu";configVersion:"ocr-v1";dpi:200};pages:{page:number;width:number;height:number;processingMs:number;text:string;metrics:Metrics;lines:OcrLine[]}[]}
+```
+
+RAG verifies job/document/config/exact ordered page set/schema before persistence. `DELETE /v1/jobs/:id` is authenticated, idempotent `204`. Network/502/503 retry twice with the same key; `429` remains queued; polling backs off 2–15 seconds. 400/401/403/409/413/422/500, deterministic render failures, and integrity mismatches are terminal and fail closed.
+
+## Files
+
+Modify `src/modules/{ingest/service,parsers/parser-registry,catalog/repository,catalog/reconciler}.ts`, `src/{app,api/openapi,config/env}.ts`, shared types/IDs, playground, Docker/docs; add migration 002, `src/modules/ocr/{client,dispatcher,detection,composition,artifacts,review,indexing,retention}.ts`, `ocr-service/**`, fixtures, and tests.
+
+## Requirement / Scenario Coverage
+
+The 17 rows assign all 34 spec scenarios exactly once.
+
+| Requirement | Component | RED scenarios |
+|---|---|---|
+| O1 Routing | ingest/artifacts | native-201; OCR-202 |
+| O2 Progress | repository/API | all-documents; one-fails |
+| O3 Dispatch | repository/dispatcher | duplicate-pending; lease-recovery |
+| O4 Availability | retrieval/dispatcher | OCR-down; catalog-down |
+| O5 Enablement | config/deploy | release-gate; rollback |
+| P1 Detection | parser/detection | textual; exact-mixed-pages |
+| P2 Quality | detection | verified-blank; visible-ink-blocked |
+| P3 Composition | composition | reproducible-candidate |
+| P4 Risks | composition/review | known-ambiguities |
+| P5 OCR service | client/service | idempotent; invalid/conflict; integrity-mismatch |
+| P6 Limits | service/client | unsafe-input; pressure; deterministic-failure |
+| V1 Review auth | API/review | inspect; unauthorized/premature |
+| V2 Corrections | review/repository | atomic-commit; stale-conflict |
+| V3 Approval | review/indexing | new-activate=true/false; active-race; reusable-activate=true/false |
+| V4 Rejection | review | reject-no-index |
+| V5 Retention | retention/artifacts | expiry; active-protected; resume-delete |
+| V6 Acceptance | production | FacturaTech-34-and-codes |
+
+## Testing, Threats, Rollout
+
+Strict TDD: Node unit/integration fixtures plus offline pytest; run `npm test`, check, build, then human production acceptance. Threat matrix: documentation-like routing is applicable—only `.pdf` enters OCR; `requirements.txt`/executable `.md` parse but never execute, while `CMakeLists.txt`, MDX, and `README.sh` remain unsupported; RED-test each. Git selection, commit, push, and PR-command rows are N/A (no VCS automation). Private HTTP is applicable: RED-test auth, allowlist, limits, retry classes, and integrity.
+
+Deploy OCR ready → migration 002 → RAG with OCR disabled → native verification → enable. Rollback disables OCR; candidates remain invisible. Auto-chain cohesive code+test work units at ≤400 authored lines; report unavoidable `size:exception` after one split.
+
+## Open Questions
+
+None.

openspec/changes/ocr-ingest-integration/specs/ocr-ingest-orchestration/spec.md

@@ -0,0 +1,94 @@
+# OCR Ingest Orchestration Specification
+
+## Purpose
+
+Define routing, lifecycle, recovery, and release behavior for OCR ingestion without changing native ingestion.
+
+## Requirements
+
+### Requirement: Native and OCR Routing
+
+The system MUST preserve the synchronous Point 2 path when all PDF pages have sufficient native text. Before accepting OCR work, it MUST retain the original and manifest durably and create a non-active version with a null source content hash.
+
+#### Scenario: Native PDF remains synchronous
+
+- GIVEN a PDF whose pages satisfy native detection
+- WHEN it is ingested
+- THEN the system MUST return the existing `201` response and MUST NOT contact OCR
+- AND lifecycle and retrieval behavior MUST remain unchanged
+
+#### Scenario: OCR work is accepted asynchronously
+
+- GIVEN a valid scanned or mixed PDF requiring OCR
+- WHEN ingestion is accepted
+- THEN it MUST return `202` with source/version identity, `state: indexing`, `phase: ocr_queued`, and `statusUrl`
+- AND `reviewUrl` MUST be null and `activated` MUST be false
+
+### Requirement: Version-Level Progress Gate
+
+The system MUST report authenticated document/page progress using `native_extracting`, `ocr_queued`, `ocr_running`, `review_required`, `indexing`, `ready`, `active`, `failed`, or `rejected`. A version SHALL reach `review_required` only after every document is native-complete or OCR-complete.
+
+#### Scenario: Multi-document candidate completes
+
+- GIVEN a version with native and OCR documents
+- WHEN all non-blank pages pass their applicable extraction gates
+- THEN the version MUST become `review_required` with a review URL
+- AND no document MAY be omitted from reported progress
+
+#### Scenario: One document fails
+
+- GIVEN any requested page fails or violates integrity or quality gates
+- WHEN version progress is evaluated
+- THEN the version MUST become `failed` with an actionable error
+- AND it MUST produce no embeddings or partial activation
+
+### Requirement: Idempotent Concurrent Dispatch
+
+The system MUST maintain one job per version/document, prevent concurrent ownership, and reuse its remote idempotency key during recovery.
+
+#### Scenario: Duplicate ingestion while pending
+
+- GIVEN the same source and original have a non-terminal OCR version
+- WHEN ingestion is retried
+- THEN it MUST return that version and job without duplicates
+
+#### Scenario: Lease recovery
+
+- GIVEN one job has a live lease and another has an expired lease
+- WHEN dispatch or reconciliation runs concurrently
+- THEN the live job MUST remain untouched and the expired job MUST be redispatched once with its original key
+- AND an existing remote job MUST be recovered rather than duplicated
+
+### Requirement: Fail-Closed Availability
+
+OCR candidates MUST remain invisible until approved and active. OCR, catalog, embedding, or vector failures MUST preserve the previous active version.
+
+#### Scenario: OCR is unavailable
+
+- GIVEN OCR cannot complete after allowed transient retries
+- WHEN the job is processed
+- THEN the candidate MUST fail closed without activation or engine substitution
+- AND the prior active version MUST remain retrievable
+
+#### Scenario: Catalog is unavailable during retrieval
+
+- GIVEN active-version resolution is unavailable
+- WHEN retrieval is requested
+- THEN the system MUST return `503` and MUST NOT query without an active-version filter
+
+### Requirement: Controlled Enablement and Rollback
+
+OCR MUST be enabled only after its private service is ready, RAG is deployed disabled, and native behavior is verified. Disabling it MUST stop new OCR work while preserving native ingestion and candidate invisibility.
+
+#### Scenario: Release gate passes
+
+- GIVEN OCR is ready and `npm run check`, `npm run build`, `npm test`, and offline OCR tests pass
+- WHEN OCR is enabled after native production verification
+- THEN new eligible ingestions MAY use the OCR path
+
+#### Scenario: Operational rollback
+
+- GIVEN OCR processing or acceptance fails after deployment
+- WHEN OCR is disabled or the RAG release is rolled back
+- THEN native ingestion MUST continue and existing candidates MUST remain non-retrievable
+- AND the previous active corpus MUST require no embedding recomputation

openspec/changes/ocr-ingest-integration/specs/ocr-processing/spec.md

@@ -0,0 +1,105 @@
+# OCR Processing Specification
+
+## Purpose
+
+Define safe OCR behavior.
+
+## Requirements
+
+### Requirement: Per-Page Native Detection
+
+The system MUST evaluate PDF pages independently. Native text is sufficient only when `N >= 120`, `A >= 80`, `W >= 20`, and `R <= 0.01`, under `pdf-detection-v1`.
+
+#### Scenario: Textual pages bypass OCR
+
+- GIVEN every page meets native thresholds
+- WHEN extraction completes
+- THEN native text MUST be selected and no OCR job MAY be created
+
+#### Scenario: Mixed PDF selects exact pages
+
+- GIVEN only some pages fail any native threshold
+- WHEN detection completes
+- THEN OCR MUST receive those unique, ordered, one-based pages
+- AND `N == 0` MUST always request OCR inspection
+
+### Requirement: Blank and OCR Quality Gates
+
+A page MAY be blank only with `inkCoverage < 0.015` and fewer than 10 OCR characters. Non-blank OCR MUST meet `nonWhitespaceCharacters >= 40`, median `>= 0.80`, p10 `>= 0.50`, and low-confidence ratio `<= 0.20`.
+
+#### Scenario: Verified blank contributes no text
+
+- GIVEN both blank conditions hold
+- WHEN the page is classified
+- THEN it MUST be blank and contribute no candidate text
+
+#### Scenario: Visible ink fails closed
+
+- GIVEN visible ink and OCR fails the quality gate
+- WHEN processing completes
+- THEN the page MUST NOT be blank and the version MUST fail with `OCR_QUALITY_BLOCKED`
+
+### Requirement: Deterministic Candidate Composition
+
+The system MUST use native text or OCR lines, never both per page. It SHALL order lines by top, left, and original index; pages ascending; and separate non-blank pages with `--- Page N ---`.
+
+#### Scenario: Candidate is reproducible
+
+- GIVEN identical validated page records
+- WHEN the candidate is composed repeatedly
+- THEN canonical text, ordering, and hashes MUST be identical
+- AND raw OCR and native extraction MUST remain available for audit
+
+### Requirement: Risk Tokens Remain Uncorrected
+
+The system MUST mark `\b[A-Za-z]{2,}[A-Za-z0-9_-]*\d[A-Za-z0-9_-]*\b` tokens and elevate ambiguous, differing, unique, or context-adjacent ones. It MUST NOT auto-correct casing or `O/0`, `I/1/l`, or `S/5`.
+
+#### Scenario: Known code ambiguities are flagged
+
+- GIVEN OCR emits `CBGO4a`, `FATo7`, `DSAuo8`, or `NSAvo6`
+- WHEN risk analysis runs
+- THEN each token MUST be shown for human review unchanged
+
+### Requirement: Private Idempotent OCR Service
+
+Private PaddleOCR `3.4.0` with PaddlePaddle CPU `3.2.2` MUST require bearer authorization, allowlisted jobs, and deterministic identity for identical hash, config, and pages. It MUST NOT be Internet-accessible.
+
+#### Scenario: Idempotent submission
+
+- GIVEN valid requests share a key and payload
+- WHEN both are submitted
+- THEN both MUST identify one job and result without duplicate processing
+
+#### Scenario: Conflicting or invalid submission
+
+- GIVEN a reused key conflicts, authentication fails, or configuration is disallowed
+- WHEN submission occurs
+- THEN OCR MUST return the contracted `409`, `401/403`, or `400` class
+
+#### Scenario: Result integrity mismatch
+
+- GIVEN returned integrity fields differ from the request
+- WHEN RAG validates the result
+- THEN processing MUST fail without a reviewable candidate
+
+### Requirement: Operational Limits and Health
+
+OCR MUST enforce 50 MiB, 100 pages, 200 DPI, 25 megapixels/page, one concurrent job, one replica, queue depth three, 60 seconds/page, 15 minutes total, and 3 CPU/5 GiB maximum. Readiness MUST await pinned models; polling SHALL back off 2–15 seconds.
+
+#### Scenario: Unsafe input is rejected before render
+
+- GIVEN a PDF is oversized, encrypted, corrupt, too long, or decompression-unsafe
+- WHEN it is accepted for processing
+- THEN it MUST be rejected before full render without changing the active version
+
+#### Scenario: Queue or transient service pressure
+
+- GIVEN the queue is full or OCR is transiently unavailable
+- WHEN RAG submits work
+- THEN `429` MUST remain retryable; connection/`502`/`503` failures MUST allow at most two same-key resubmissions
+
+#### Scenario: Deterministic failure is not retried
+
+- GIVEN OCR returns `400`, `401`, `403`, `413`, `422`, or deterministic render failure
+- WHEN RAG handles the response
+- THEN it MUST NOT retry or switch engines

openspec/changes/ocr-ingest-integration/specs/ocr-review-workflow/spec.md

@@ -0,0 +1,106 @@
+# OCR Review Workflow Specification
+
+## Purpose
+
+Define authenticated review, atomic correction, activation, and retention.
+
+## Requirements
+
+### Requirement: Mandatory Authenticated Review
+
+OCR versions MUST remain `review_required` and non-retrievable until approved. Status, review, images, approval, and rejection MUST require the administrator token.
+
+#### Scenario: Reviewer inspects a candidate
+
+- GIVEN all documents passed the version gate
+- WHEN an authorized reviewer opens review
+- THEN it MUST show image, native text, raw OCR, boxes, confidence, differences, and risks
+
+#### Scenario: Unauthorized or premature access
+
+- GIVEN credentials fail or the version is not reviewable
+- WHEN review or decision is requested
+- THEN it MUST reject the request without exposing artifacts or changing state
+
+### Requirement: Atomic Optimistic Corrections
+
+Approval MUST validate `candidateSha256`, unique line identities, and every `expectedLineSha256` before changes. Reviewed text SHALL be immutable.
+
+#### Scenario: Valid corrections commit together
+
+- GIVEN hashes are current and correction targets are unique
+- WHEN approval is submitted
+- THEN all replacements MUST commit together with reviewer, time, and resulting hashes
+
+#### Scenario: Stale or conflicting correction
+
+- GIVEN any candidate, line, identity, or box is stale or duplicated
+- WHEN approval is submitted
+- THEN the system MUST return `409` without corrections or state transition
+
+### Requirement: Approval and Activation Gate
+
+Approval MUST cover the version, derive final hashes, and move `review_required` to `indexing` before chunking. Activation SHALL use Point 2 verification and the expected active version.
+
+#### Scenario: Approved content becomes active
+
+- GIVEN corrections are valid and the expected active version is current
+- WHEN indexing and point verification succeed
+- THEN the new version MUST become active atomically and the previous one superseded
+- AND only reviewed content MAY be retrieved
+
+#### Scenario: Active version changes during review
+
+- GIVEN another version activates during review
+- WHEN approved indexing completes
+- THEN the candidate MUST remain `ready`, return `409 ACTIVE_VERSION_CHANGED`, and not replace it
+
+#### Scenario: Reusable content already exists
+
+- GIVEN final content, fingerprint, and metadata match a reusable version
+- WHEN approval checks the active-version precondition
+- THEN the candidate MUST be `rejected` with `DUPLICATE_REUSABLE_VERSION`
+- AND only the existing version MAY be activated
+
+### Requirement: Rejection and Failure Isolation
+
+Rejection MUST produce `rejected`, preserve temporary audit evidence, and generate no embeddings. Failed or rejected candidates SHALL never alter the active version.
+
+#### Scenario: Reviewer rejects candidate
+
+- GIVEN a version is `review_required`
+- WHEN an authorized reviewer rejects it
+- THEN it MUST become `rejected` with audit data and no indexing
+
+### Requirement: Retention Safety
+
+Private artifacts MUST use `0600` permissions and verifiable hashes. The system MUST retain review 30 days, failed/rejected artifacts 7 days, review images 7 days post-decision, OCR copies until transfer or 24 hours, and active artifacts until 30 days after supersession. Cleanup MUST be idempotent and never delete active versions.
+
+#### Scenario: Review expires
+
+- GIVEN a version remains `review_required` for 30 days
+- WHEN daily retention runs exclusively
+- THEN it MUST become `rejected` with `REVIEW_EXPIRED` before removal
+
+#### Scenario: Active artifacts are protected
+
+- GIVEN an active version is past a TTL
+- WHEN retention runs concurrently with review or purge
+- THEN its original, reviewed text, raw OCR, and manifest MUST remain
+- AND cleanup MUST not escape the version directory or expose a public URL
+
+#### Scenario: Interrupted cleanup resumes
+
+- GIVEN deletion stopped after retention entered its deleting state
+- WHEN cleanup runs again
+- THEN deletion MUST resume safely and finish in the deleted state without affecting other versions
+
+### Requirement: Human-Approved Production Acceptance
+
+The release MUST prove corrected content after approval; confidence SHALL NOT authorize activation.
+
+#### Scenario: FacturaTech acceptance succeeds
+
+- GIVEN the real FacturaTech PDF has been reviewed and its 34 entries approved
+- WHEN the version becomes active and retrieval is queried
+- THEN all 34 entries and `CBG04a`, `FAT07`, `DSAU08`, and `NSAV06` MUST be returned exactly