95 lines
8.8 KiB
Markdown
95 lines
8.8 KiB
Markdown
# Design: OCR Ingest Integration
|
||
|
||
## Technical Approach
|
||
|
||
Insert page-aware OCR before existing lifecycle indexing. Native PDFs retain synchronous `201`; OCR candidates persist originals, catalog rows, and jobs before returning `202`. PostgreSQL remains authoritative, Qdrant receives only reviewed content, and every failure preserves the active pointer.
|
||
|
||
## Architecture Decisions
|
||
|
||
| Decision | Choice and rationale |
|
||
|---|---|
|
||
| Extraction | Fixture-gate `pdfjs-dist`; use `pdf-parse` `pagerender` only if proven on Node 22. Aggregated-text slicing is forbidden. |
|
||
| Execution | One in-process dispatcher with PostgreSQL leases; recovery needs no second Node deployment. |
|
||
| OCR | Private FastAPI, one worker/replica, baked PaddleOCR 3.4.0/PaddlePaddle CPU 3.2.2 models. |
|
||
| Ownership | RAG owns durable `0600` artifacts; OCR owns a 24-hour work copy; reviewed bytes are immutable. |
|
||
|
||
## Data Flow and Concurrency
|
||
|
||
`upload → disk stage → page detection → native 201 OR durable candidate/job → 202 → OCR → review_required → approve → index/verify → activation CAS`
|
||
|
||
`findPendingOcrVersion(identity)` is distinct from `findReusableVersion` and joins versions, documents, and jobs using:
|
||
|
||
```ts
|
||
type PendingOcrIdentity={sourceId:string;originalManifestHash:Sha256;processingFingerprint:Sha256;metadataHash:Sha256}
|
||
```
|
||
|
||
It accepts `source_content_hash IS NULL` and states `pending|indexing|review_required`. Migration 002 adds a matching partial unique index. Under `withSourceLock`, ingestion validates source type/reference, performs this lookup, then either returns the persisted version/jobs or preallocates `versionId`, durably stages/fsyncs originals, and transactionally inserts one candidate plus `UNIQUE(version_id,document_id)` jobs. A uniqueness loss reloads the winner; the losing staging directory is orphan-swept. All duplicates return the same `202/statusUrl` and persisted activation intent.
|
||
|
||
Approval starts one transaction locking source and candidate. Before artifact writes, corrections, or transition, it requires `request.expectedActiveVersionId === candidate.base_active_version_id === source.active_version_id`; otherwise `409 ACTIVE_VERSION_CHANGED` and zero mutation. Candidate/line hashes and unique targets are then validated atomically.
|
||
|
||
After final hashes, source-locked `findReusableVersion(finalContent,fingerprint,metadata)` runs. A match rejects the candidate: `DUPLICATE_REUSABLE_VERSION`. Persisted `candidate.activate_requested === false` preserves the active pointer; only true activates the `ready|superseded|active` version via expected-active CAS. CAS failure returns `409 ACTIVE_VERSION_CHANGED` and rolls back the reusable branch. Otherwise, approval persists corrections/hashes and indexes to `ready`. Only persisted `candidate.activate_requested === true` invokes Point-2 activation with the original expected-active CAS; false succeeds in `ready` without pointer change. CAS failure returns `409 ACTIVE_VERSION_CHANGED`, leaving the candidate `ready`.
|
||
|
||
Jobs use `queued→running→succeeded|failed`; expired leases return to `queued` with unchanged remote key/ID. Versions use `pending→indexing→review_required→indexing→ready→active`, plus fail/reject/purge paths. Retention uses version locks and `present→retention_deleting→retention_deleted`; active artifacts are never TTL-deleted.
|
||
|
||
## Interfaces / Contracts
|
||
|
||
`OcrAccepted` is the OCR `202`; all `/ingestions/*` routes require lifecycle bearer authentication.
|
||
|
||
```ts
|
||
type OcrAccepted={accepted:true;sourceId:string;versionId:UUID;versionNumber:number;state:"indexing";phase:"ocr_queued";statusUrl:string;reviewUrl:null;activated:false}
|
||
type IngestionStatus={sourceId:string;versionId:UUID;state:VersionState;phase:"native_extracting"|"ocr_queued"|"ocr_running"|"review_required"|"indexing"|"ready"|"active"|"failed"|"rejected";activated:boolean;documents:{documentId:string;state:"native_complete"|"ocr_queued"|"ocr_running"|"ocr_complete"|"failed";completedPages:number;totalPages:number;pages:{page:number;method:"native"|"ocr"|"blank";state:"pending"|"native_complete"|"ocr_queued"|"ocr_running"|"ocr_complete"|"blank"|"failed";errorCode?:string}[]}[];error:{code:string;message:string;retryable:boolean}|null;statusUrl:string;reviewUrl:string|null}
|
||
type Review={versionId:UUID;candidateSha256:Sha256;baseActiveVersionId:UUID|null;documents:{documentId:string;pages:{page:number;imageUrl:string;nativeText:string;ocr:{text:string;lines:{lineId:string;text:string;confidence:number;bbox:number[]}[]};candidateText:string;differences:string[];risks:string[]}[]}[]}
|
||
type Approve={candidateSha256:Sha256;expectedActiveVersionId:UUID|null;reviewedBy:string;corrections:{documentId:string;page:number;lineId:string;expectedLineSha256:Sha256;replacementText:string}[]}
|
||
type Reject={candidateSha256:Sha256;reviewedBy:string;reason:string}
|
||
```
|
||
|
||
Reject requires current candidate hash and `review_required`; success returns `{versionId,state:"rejected",activated:false}`. Approve returns `{versionId,state,activated,activatedVersionId?,errorCode?}`; hash/state/precondition conflicts are `409`.
|
||
|
||
Authenticated `POST /v1/jobs` uses multipart `file`/`request` and `Idempotency-Key=<documentSha256>:<configVersion>:<pagesSha256>`; `GET /v1/jobs/:id` returns status and `GET /v1/jobs/:id/result` returns result:
|
||
|
||
```ts
|
||
type Submit={documentSha256:Sha256;pages:number[];languages:["es","en"];dpi:200;engine:"paddleocr";engineVersion:"3.4.0";runtimeVersion:"3.2.2";configVersion:"ocr-v1";returnLayout:true}
|
||
type Ack={jobId:string;status:"queued";documentSha256:Sha256;requestedPages:number[];configVersion:"ocr-v1";createdAt:ISODate}
|
||
type JobStatus={jobId:string;status:"queued"|"running"|"succeeded"|"failed";completedPages:number;totalPages:number;error:{code:string;message:string}|null}
|
||
type Result={schemaVersion:"1";jobId:string;documentSha256:Sha256;engine:{name:"paddleocr";version:"3.4.0";runtime:"paddlepaddle-3.2.2";device:"cpu";configVersion:"ocr-v1";dpi:200};pages:{page:number;width:number;height:number;processingMs:number;text:string;metrics:Metrics;lines:OcrLine[]}[]}
|
||
```
|
||
|
||
RAG verifies job/document/config/exact ordered page set/schema before persistence. `DELETE /v1/jobs/:id` is authenticated, idempotent `204`. Network/502/503 retry twice with the same key; `429` remains queued; polling backs off 2–15 seconds. 400/401/403/409/413/422/500, deterministic render failures, and integrity mismatches are terminal and fail closed.
|
||
|
||
## Files
|
||
|
||
Modify `src/modules/{ingest/service,parsers/parser-registry,catalog/repository,catalog/reconciler}.ts`, `src/{app,api/openapi,config/env}.ts`, shared types/IDs, playground, Docker/docs; add migration 002, `src/modules/ocr/{client,dispatcher,detection,composition,artifacts,review,indexing,retention}.ts`, `ocr-service/**`, fixtures, and tests.
|
||
|
||
## Requirement / Scenario Coverage
|
||
|
||
The 17 rows assign all 34 spec scenarios exactly once.
|
||
|
||
| Requirement | Component | RED scenarios |
|
||
|---|---|---|
|
||
| O1 Routing | ingest/artifacts | native-201; OCR-202 |
|
||
| O2 Progress | repository/API | all-documents; one-fails |
|
||
| O3 Dispatch | repository/dispatcher | duplicate-pending; lease-recovery |
|
||
| O4 Availability | retrieval/dispatcher | OCR-down; catalog-down |
|
||
| O5 Enablement | config/deploy | release-gate; rollback |
|
||
| P1 Detection | parser/detection | textual; exact-mixed-pages |
|
||
| P2 Quality | detection | verified-blank; visible-ink-blocked |
|
||
| P3 Composition | composition | reproducible-candidate |
|
||
| P4 Risks | composition/review | known-ambiguities |
|
||
| P5 OCR service | client/service | idempotent; invalid/conflict; integrity-mismatch |
|
||
| P6 Limits | service/client | unsafe-input; pressure; deterministic-failure |
|
||
| V1 Review auth | API/review | inspect; unauthorized/premature |
|
||
| V2 Corrections | review/repository | atomic-commit; stale-conflict |
|
||
| V3 Approval | review/indexing | new-activate=true/false; active-race; reusable-activate=true/false |
|
||
| V4 Rejection | review | reject-no-index |
|
||
| V5 Retention | retention/artifacts | expiry; active-protected; resume-delete |
|
||
| V6 Acceptance | production | FacturaTech-34-and-codes |
|
||
|
||
## Testing, Threats, Rollout
|
||
|
||
Strict TDD: Node unit/integration fixtures plus offline pytest; run `npm test`, check, build, then human production acceptance. Threat matrix: documentation-like routing is applicable—only `.pdf` enters OCR; `requirements.txt`/executable `.md` parse but never execute, while `CMakeLists.txt`, MDX, and `README.sh` remain unsupported; RED-test each. Git selection, commit, push, and PR-command rows are N/A (no VCS automation). Private HTTP is applicable: RED-test auth, allowlist, limits, retry classes, and integrity.
|
||
|
||
Deploy OCR ready → migration 002 → RAG with OCR disabled → native verification → enable. Rollback disables OCR; candidates remain invisible. Auto-chain cohesive code+test work units at ≤400 authored lines; report unavoidable `size:exception` after one split.
|
||
|
||
## Open Questions
|
||
|
||
None.
|