8.8 KiB
Design: OCR Ingest Integration
Technical Approach
Insert page-aware OCR before existing lifecycle indexing. Native PDFs retain synchronous 201; OCR candidates persist originals, catalog rows, and jobs before returning 202. PostgreSQL remains authoritative, Qdrant receives only reviewed content, and every failure preserves the active pointer.
Architecture Decisions
| Decision | Choice and rationale |
|---|---|
| Extraction | Fixture-gate pdfjs-dist; use pdf-parse pagerender only if proven on Node 22. Aggregated-text slicing is forbidden. |
| Execution | One in-process dispatcher with PostgreSQL leases; recovery needs no second Node deployment. |
| OCR | Private FastAPI, one worker/replica, baked PaddleOCR 3.4.0/PaddlePaddle CPU 3.2.2 models. |
| Ownership | RAG owns durable 0600 artifacts; OCR owns a 24-hour work copy; reviewed bytes are immutable. |
Data Flow and Concurrency
upload → disk stage → page detection → native 201 OR durable candidate/job → 202 → OCR → review_required → approve → index/verify → activation CAS
findPendingOcrVersion(identity) is distinct from findReusableVersion and joins versions, documents, and jobs using:
type PendingOcrIdentity={sourceId:string;originalManifestHash:Sha256;processingFingerprint:Sha256;metadataHash:Sha256}
It accepts source_content_hash IS NULL and states pending|indexing|review_required. Migration 002 adds a matching partial unique index. Under withSourceLock, ingestion validates source type/reference, performs this lookup, then either returns the persisted version/jobs or preallocates versionId, durably stages/fsyncs originals, and transactionally inserts one candidate plus UNIQUE(version_id,document_id) jobs. A uniqueness loss reloads the winner; the losing staging directory is orphan-swept. All duplicates return the same 202/statusUrl and persisted activation intent.
Approval starts one transaction locking source and candidate. Before artifact writes, corrections, or transition, it requires request.expectedActiveVersionId === candidate.base_active_version_id === source.active_version_id; otherwise 409 ACTIVE_VERSION_CHANGED and zero mutation. Candidate/line hashes and unique targets are then validated atomically.
After final hashes, source-locked findReusableVersion(finalContent,fingerprint,metadata) runs. A match rejects the candidate: DUPLICATE_REUSABLE_VERSION. Persisted candidate.activate_requested === false preserves the active pointer; only true activates the ready|superseded|active version via expected-active CAS. CAS failure returns 409 ACTIVE_VERSION_CHANGED and rolls back the reusable branch. Otherwise, approval persists corrections/hashes and indexes to ready. Only persisted candidate.activate_requested === true invokes Point-2 activation with the original expected-active CAS; false succeeds in ready without pointer change. CAS failure returns 409 ACTIVE_VERSION_CHANGED, leaving the candidate ready.
Jobs use queued→running→succeeded|failed; expired leases return to queued with unchanged remote key/ID. Versions use pending→indexing→review_required→indexing→ready→active, plus fail/reject/purge paths. Retention uses version locks and present→retention_deleting→retention_deleted; active artifacts are never TTL-deleted.
Interfaces / Contracts
OcrAccepted is the OCR 202; all /ingestions/* routes require lifecycle bearer authentication.
type OcrAccepted={accepted:true;sourceId:string;versionId:UUID;versionNumber:number;state:"indexing";phase:"ocr_queued";statusUrl:string;reviewUrl:null;activated:false}
type IngestionStatus={sourceId:string;versionId:UUID;state:VersionState;phase:"native_extracting"|"ocr_queued"|"ocr_running"|"review_required"|"indexing"|"ready"|"active"|"failed"|"rejected";activated:boolean;documents:{documentId:string;state:"native_complete"|"ocr_queued"|"ocr_running"|"ocr_complete"|"failed";completedPages:number;totalPages:number;pages:{page:number;method:"native"|"ocr"|"blank";state:"pending"|"native_complete"|"ocr_queued"|"ocr_running"|"ocr_complete"|"blank"|"failed";errorCode?:string}[]}[];error:{code:string;message:string;retryable:boolean}|null;statusUrl:string;reviewUrl:string|null}
type Review={versionId:UUID;candidateSha256:Sha256;baseActiveVersionId:UUID|null;documents:{documentId:string;pages:{page:number;imageUrl:string;nativeText:string;ocr:{text:string;lines:{lineId:string;text:string;confidence:number;bbox:number[]}[]};candidateText:string;differences:string[];risks:string[]}[]}[]}
type Approve={candidateSha256:Sha256;expectedActiveVersionId:UUID|null;reviewedBy:string;corrections:{documentId:string;page:number;lineId:string;expectedLineSha256:Sha256;replacementText:string}[]}
type Reject={candidateSha256:Sha256;reviewedBy:string;reason:string}
Reject requires current candidate hash and review_required; success returns {versionId,state:"rejected",activated:false}. Approve returns {versionId,state,activated,activatedVersionId?,errorCode?}; hash/state/precondition conflicts are 409.
Authenticated POST /v1/jobs uses multipart file/request and Idempotency-Key=<documentSha256>:<configVersion>:<pagesSha256>; GET /v1/jobs/:id returns status and GET /v1/jobs/:id/result returns result:
type Submit={documentSha256:Sha256;pages:number[];languages:["es","en"];dpi:200;engine:"paddleocr";engineVersion:"3.4.0";runtimeVersion:"3.2.2";configVersion:"ocr-v1";returnLayout:true}
type Ack={jobId:string;status:"queued";documentSha256:Sha256;requestedPages:number[];configVersion:"ocr-v1";createdAt:ISODate}
type JobStatus={jobId:string;status:"queued"|"running"|"succeeded"|"failed";completedPages:number;totalPages:number;error:{code:string;message:string}|null}
type Result={schemaVersion:"1";jobId:string;documentSha256:Sha256;engine:{name:"paddleocr";version:"3.4.0";runtime:"paddlepaddle-3.2.2";device:"cpu";configVersion:"ocr-v1";dpi:200};pages:{page:number;width:number;height:number;processingMs:number;text:string;metrics:Metrics;lines:OcrLine[]}[]}
RAG verifies job/document/config/exact ordered page set/schema before persistence. DELETE /v1/jobs/:id is authenticated, idempotent 204. Network/502/503 retry twice with the same key; 429 remains queued; polling backs off 2–15 seconds. 400/401/403/409/413/422/500, deterministic render failures, and integrity mismatches are terminal and fail closed.
Files
Modify src/modules/{ingest/service,parsers/parser-registry,catalog/repository,catalog/reconciler}.ts, src/{app,api/openapi,config/env}.ts, shared types/IDs, playground, Docker/docs; add migration 002, src/modules/ocr/{client,dispatcher,detection,composition,artifacts,review,indexing,retention}.ts, ocr-service/**, fixtures, and tests.
Requirement / Scenario Coverage
The 17 rows assign all 34 spec scenarios exactly once.
| Requirement | Component | RED scenarios |
|---|---|---|
| O1 Routing | ingest/artifacts | native-201; OCR-202 |
| O2 Progress | repository/API | all-documents; one-fails |
| O3 Dispatch | repository/dispatcher | duplicate-pending; lease-recovery |
| O4 Availability | retrieval/dispatcher | OCR-down; catalog-down |
| O5 Enablement | config/deploy | release-gate; rollback |
| P1 Detection | parser/detection | textual; exact-mixed-pages |
| P2 Quality | detection | verified-blank; visible-ink-blocked |
| P3 Composition | composition | reproducible-candidate |
| P4 Risks | composition/review | known-ambiguities |
| P5 OCR service | client/service | idempotent; invalid/conflict; integrity-mismatch |
| P6 Limits | service/client | unsafe-input; pressure; deterministic-failure |
| V1 Review auth | API/review | inspect; unauthorized/premature |
| V2 Corrections | review/repository | atomic-commit; stale-conflict |
| V3 Approval | review/indexing | new-activate=true/false; active-race; reusable-activate=true/false |
| V4 Rejection | review | reject-no-index |
| V5 Retention | retention/artifacts | expiry; active-protected; resume-delete |
| V6 Acceptance | production | FacturaTech-34-and-codes |
Testing, Threats, Rollout
Strict TDD: Node unit/integration fixtures plus offline pytest; run npm test, check, build, then human production acceptance. Threat matrix: documentation-like routing is applicable—only .pdf enters OCR; requirements.txt/executable .md parse but never execute, while CMakeLists.txt, MDX, and README.sh remain unsupported; RED-test each. Git selection, commit, push, and PR-command rows are N/A (no VCS automation). Private HTTP is applicable: RED-test auth, allowlist, limits, retry classes, and integrity.
Deploy OCR ready → migration 002 → RAG with OCR disabled → native verification → enable. Rollback disables OCR; candidates remain invisible. Auto-chain cohesive code+test work units at ≤400 authored lines; report unavoidable size:exception after one split.
Open Questions
None.