rag-service/openspec/specs/ocr-ingest-orchestration/spec.md

OCR Ingest Orchestration Specification

Purpose

Define routing, lifecycle, recovery, and release behavior for OCR ingestion without changing native ingestion.

Requirements

Requirement: Native and OCR Routing

The system MUST preserve the synchronous Point 2 path when all PDF pages have sufficient native text. Before accepting OCR work, it MUST retain the original and manifest durably and create a non-active version with a null source content hash.

Scenario: Native PDF remains synchronous

• GIVEN a PDF whose pages satisfy native detection
• WHEN it is ingested
• THEN the system MUST return the existing 201 response and MUST NOT contact OCR
• AND lifecycle and retrieval behavior MUST remain unchanged

Scenario: OCR work is accepted asynchronously

• GIVEN a valid scanned or mixed PDF requiring OCR
• WHEN ingestion is accepted
• THEN it MUST return 202 with source/version identity, state: indexing, phase: ocr_queued, and statusUrl
• AND reviewUrl MUST be null and activated MUST be false

Requirement: Version-Level Progress Gate

The system MUST report authenticated document/page progress using native_extracting, ocr_queued, ocr_running, review_required, indexing, ready, active, failed, or rejected. A version SHALL reach review_required only after every document is native-complete or OCR-complete.

Scenario: Multi-document candidate completes

• GIVEN a version with native and OCR documents
• WHEN all non-blank pages pass their applicable extraction gates
• THEN the version MUST become review_required with a review URL
• AND no document MAY be omitted from reported progress

Scenario: One document fails

• GIVEN any requested page fails or violates integrity or quality gates
• WHEN version progress is evaluated
• THEN the version MUST become failed with an actionable error
• AND it MUST produce no embeddings or partial activation

Requirement: Idempotent Concurrent Dispatch

The system MUST maintain one job per version/document, prevent concurrent ownership, and reuse its remote idempotency key during recovery.

Scenario: Duplicate ingestion while pending

• GIVEN the same source and original have a non-terminal OCR version
• WHEN ingestion is retried
• THEN it MUST return that version and job without duplicates

Scenario: Lease recovery

• GIVEN one job has a live lease and another has an expired lease
• WHEN dispatch or reconciliation runs concurrently
• THEN the live job MUST remain untouched and the expired job MUST be redispatched once with its original key
• AND an existing remote job MUST be recovered rather than duplicated

Requirement: Fail-Closed Availability

OCR candidates MUST remain invisible until approved and active. OCR, catalog, embedding, or vector failures MUST preserve the previous active version.

Scenario: OCR is unavailable

• GIVEN OCR cannot complete after allowed transient retries
• WHEN the job is processed
• THEN the candidate MUST fail closed without activation or engine substitution
• AND the prior active version MUST remain retrievable

Scenario: Catalog is unavailable during retrieval

• GIVEN active-version resolution is unavailable
• WHEN retrieval is requested
• THEN the system MUST return 503 and MUST NOT query without an active-version filter

Requirement: Controlled Enablement and Rollback

OCR MUST be enabled only after its private service is ready, RAG is deployed disabled, and native behavior is verified. Disabling it MUST stop new OCR work while preserving native ingestion and candidate invisibility.

Scenario: Release gate passes

• GIVEN OCR is ready and npm run check, npm run build, npm test, and offline OCR tests pass
• WHEN OCR is enabled after native production verification
• THEN new eligible ingestions MAY use the OCR path

Scenario: Operational rollback

• GIVEN OCR processing or acceptance fails after deployment
• WHEN OCR is disabled or the RAG release is rolled back
• THEN native ingestion MUST continue and existing candidates MUST remain non-retrievable
• AND the previous active corpus MUST require no embedding recomputation