rag-service/openspec/changes/ocr-ingest-integration/apply-progress.md

Apply Progress: OCR Ingest Integration

Current State

• Mode: Strict TDD
• Delivery: Feature-branch-chain; Units 1–7 implemented; the maintainer approved size:exception for Unit 7's cohesive 506 functional authored lines and 585 total native-accounting lines
• Completed tasks: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2, 3.3, 3.4, 4.2, 4.3
• Overall task progress: 15/30 complete

Unit 1: Migration

Implementation Summary

• Added migrations/002_ocr_review.sql with durable OCR jobs, auditable page records, atomic review correction targets, and a partial unique index for non-terminal OCR identities.
• Added focused schema contract coverage in tests/catalog/migration-002.test.ts.
• Preserved migrations/001_knowledge_lifecycle.sql and all repository logic unchanged.

TDD Cycle Evidence

Task	Test File	Layer	Safety Net	RED	GREEN	TRIANGULATE	REFACTOR
1.2	tests/catalog/migration-002.test.ts	Schema contract	N/A (new files)	Valid RED: 4/4 failed with ENOENT because migration 002 did not exist. An earlier syntax-invalid run was corrected and is not counted as RED.	4/4 passed with migration 002 present.	4 independent schema behaviors cover jobs, pages, corrections, and pending identity uniqueness.	None needed; SQL already matches the minimal authoritative contract.

Test Summary

• Total tests written: 4
• Total tests passing: 4
• Layers used: Schema contract (4)
• Approval tests: None — no existing production file was modified
• Pure functions created: 0

Work Unit Evidence

Evidence	Result
Focused test command and exact result	npx --no-install tsx --test tests/catalog/migration-002.test.ts — exit 0; 4 tests, 4 passed, 0 failed.
Runtime harness command/scenario and exact result	N/A — the task forecast defines Unit 1 as schema-only, and the project has no PostgreSQL test harness or database-test dependency. Applying a live migration would require external database access outside this unit.
Rollback boundary	Delete migrations/002_ocr_review.sql and tests/catalog/migration-002.test.ts, revert task 1.2 to pending, and remove this Unit 1 progress/history entry. No existing migration or repository behavior must be reverted.

Additional Validation

• npm run check — exit 0.
• git diff --check — exit 0 after removing one trailing-space warning introduced in the history header.

Deviations

None — the migration follows the proposal, specifications, design, and closed OCR persistence contract.

Remaining Persistence Tasks

• 1.1 Complete repository-level RED coverage for duplicate pending ingestion, lease recovery, and review transitions in Unit 2.
• 1.2 Add migration 002 schema and pending OCR identity unique index.
• 1.3 Add catalog repository candidates and leases in Unit 2.
• 1.4 Refactor persistence code after Unit 2 reaches green.

Review Boundary

• Start: Migration 001 already supplies lifecycle states and parent catalog tables.
• End: Migration 002 and its focused schema contract test are green.
• Out of scope: Repository methods, dispatch behavior, runtime PostgreSQL migration, Unit 2, commits, pushes, PRs, deployment, and secrets.
• Rollback: Remove only the two Unit 1 implementation files and their progress metadata.
• Authored review budget: 225 additions/deletions, below the 400-line budget for this autonomous slice.

Unit 2: Repository

Implementation Summary

• Added pending OCR identity lookup, atomic queue claiming, expired-lease recovery, and guarded review transitions to CatalogRepository.
• Added a focused fake-pool repository harness that covers duplicate reuse, empty queues, remote identity preservation, and invalid transitions.
• Corrected the canonical test script so it executes both root-level and nested test suites.

TDD Cycle Evidence

Task	Test File	Layer	Safety Net	RED	GREEN	TRIANGULATE	REFACTOR
1.1	tests/catalog/repository-ocr.test.ts	Unit/fake pool	4/4 existing tests passed	0/6 with methods absent; then 5/6 while each completion gate was missing	6/6 passed after task 1.3	Found/not-found candidates, claimed/empty queue, expired lease, valid/invalid transition	Covered by 1.4
1.3	tests/catalog/repository-ocr.test.ts	Unit/fake pool	4/4 existing tests passed	Tests preceded production code	6/6 passed	Multiple inputs and state paths exercised	Shared OCR row mapping extracted only after green
1.4	tests/catalog/repository-ocr.test.ts	Approval refactor	6/6 before refactor	N/A — behavior-preserving approval baseline	6/6 after refactor	Existing six cases preserved	Reusable aliased-column projection removed SQL duplication

Work Unit Evidence

Evidence	Result
Focused test command and exact result	NODE_ENV=test npx --no-install tsx --test tests/catalog/repository-ocr.test.ts — exit 0; 6 tests passed, 0 failed. NODE_ENV=test npx --no-install tsx --test tests/catalog/*.test.ts — exit 0; 10 tests passed, 0 failed.
Runtime harness command/scenario and exact result	NODE_ENV=test npx --no-install tsx --test tests/*.test.ts — exit 0; 25 root tests passed, 0 failed. Canonical npm test — exit 0; 35 total tests passed, 0 failed, proving the 25 root and 10 nested tests execute together.
Rollback boundary	Revert Unit 2 changes in src/modules/catalog/repository.ts, delete tests/catalog/repository-ocr.test.ts, and revert the package.json test script; Unit 1 migration remains intact.

Validation and Boundary

• Corrective rerun for failed evidence revision sha256:9433c0cd6628f36f4c96a27d41c6d429f4303d038ffe928f297367c274f21c2f; fresh evidence revision sha256:a19119f84381447f54a6223e2a203acaf9d3ae08b6ae8f585e3bd52cd510909b.
• The failed canonical script expanded only nested tests after nested suites appeared. The minimal correction is NODE_ENV=test tsx --test tests/*.test.ts tests/**/*.test.ts.
• Focused repository suite passed 6/6; all catalog suites passed 10/10; explicit root suites passed 25/25; canonical npm test passed 35/35; npm run check and git diff --check exited 0.
• Native Unit 2 accounting was 511 changed lines. The maintainer approved size:exception exclusively for Unit 2; the corrective attempt has a separate 600-line remediation budget. All later units retain the normal 400-line budget.
• No design deviations. No Unit 3 work was started.

Corrective TDD Evidence

Correction	Safety Net / RED	GREEN	TRIANGULATE	REFACTOR
Canonical test discovery	Failed evidence revision proved canonical npm test executed only 10 nested tests while the explicit root command executed 25 tests.	Updated only the package test argument list; canonical npm test passed 35/35.	Direct root 25/25 and nested catalog 10/10 results sum to and identify the canonical 35/35 run.	None needed — one package-script argument was added.

Unit 3: Extraction

Implementation Summary

• Added a Node 22 fixture spike proving pdf-parse page callbacks backed by bundled PDF.js 2.0.550 preserve ordered pages, including a blank page.
• Added parsePdfPages with one-based page identity, raw native text, stable SHA-256 hashes, and fail-closed complete-page accounting.
• Preserved textual ingestion as data-only, excluded CMakeLists.txt, MDX, and shell lookalikes, and kept task 2.2 pending for Unit 4 detection/composition assertions.

TDD Cycle Evidence

Task	Test File	Layer	Safety Net	RED	GREEN	TRIANGULATE	REFACTOR
2.1	tests/parsers/pdf-pages.test.ts; scripts/spike-pdfjs.ts	Runtime spike	20/20 lifecycle service tests passed	Focused suite failed 0/1 because parsePdfPages did not exist; the spike and parser implementation were still absent.	Node 22.14.0 spike returned pages 1–3 exactly, including blank page 2.	Three pages prove non-empty, blank, and later-page ordering paths.	Replaced the over-budget direct dependency with the design-approved existing pdf-parse pagerender path; spike remained green.
2.3	tests/parsers/pdf-pages.test.ts	Unit/integration fixture	20/20 lifecycle service tests passed	Focused suite failed 0/1 on the missing export before production code.	Final focused run passed 5/5 after implementation.	Covered page hashes/blank page, native PDF composition, non-PDF rejection, data-only text parsing, and unsupported lookalikes.	Focused suite remained 5/5 after consolidating on the existing parser dependency.

Work Unit Evidence

Evidence	Result
Focused test command and exact result	NODE_ENV=test npx --no-install tsx --test tests/parsers/pdf-pages.test.ts — exit 0; 5 passed, 0 failed.
Runtime harness command/scenario and exact result	NODE_ENV=test npx --no-install tsx scripts/spike-pdfjs.ts — exit 0 on Node 22.14.0; PDF.js 2.0.550 returned three ordered pages with page 2 blank.
Rollback boundary	Revert src/modules/parsers/parser-registry.ts; delete scripts/spike-pdfjs.ts, tests/parsers/pdf-pages.test.ts, and tests/fixtures/ocr/native-three-pages.pdf; revert tasks 2.1/2.3 and this Unit 3 progress/history entry. Units 1–2 remain intact.

Validation and Boundary

• Canonical npm test passed 40/40; npm run check and git diff --check exited 0.
• Native authored slice before progress/history metadata: 225 additions/deletions; no package or lockfile delta remains.
• Start: Unit 2 persistence is complete. End: per-page native extraction and parser threat routing are green.
• Out of scope: detection thresholds, OCR selection, composition, risks, Unit 4 tasks 2.4/2.5, commits, pushes, PRs, deployment, and restricted paths.
• No specification deviation. The design-authorized pdf-parse pagerender fallback was selected only after its Node 22 fixture spike passed.

Unit 4: Detection and Composition

Implementation Summary

• Added pure per-page native detection under pdf-detection-v1, exact OCR page selection, verified-blank classification, and fail-closed OCR quality gates.
• Added deterministic candidate composition using exactly one source per page, ordered OCR lines, blank omission, canonical hashes, and unchanged risk tokens prioritized for review.
• Completed the remaining task 2.2 scenarios by combining the Unit 3 data-only/parser threat coverage with Unit 4 mixed-PDF, detection, blank, hash, and risk RED assertions.

TDD Cycle Evidence

Task	Test File	Layer	Safety Net	RED	GREEN	TRIANGULATE	REFACTOR
2.2	tests/ocr/detection.test.ts; tests/parsers/pdf-pages.test.ts	Unit/integration fixture	Unit 3 parser suite passed 5/5 before edits.	Focused Unit 4 suite failed before production with ERR_MODULE_NOT_FOUND for src/modules/ocr/detection.js.	Focused suite passed 5/5 after detection and composition were added.	Real generated mixed PDF selected pages 2 and 3; .txt/.md stayed data-only and CMake/MDX/shell lookalikes stayed unsupported.	None needed; routing remains separated between parser support and the pure PDF OCR planner.
2.4	tests/ocr/detection.test.ts	Unit	N/A (new production file)	Threshold, mixed-page, blank, and quality assertions preceded production code.	Exact native, blank, and OCR quality boundaries passed in the 5/5 focused suite.	Every native and OCR threshold has a passing boundary plus a failing neighbor; visible ink with empty OCR is blocked.	Pure functions and named policy version were present at green; final focused suite remained 5/5.
2.5	tests/ocr/detection.test.ts	Unit	N/A (new production file)	Ordered composition, blank omission, canonical hash, audit-source, and risk assertions preceded production code.	Deterministic composition and risk prioritization passed in the 5/5 focused suite.	Reordered pages, tied OCR boxes, blank middle page, repeated token, and four known ambiguous tokens exercise distinct paths.	Reused shared canonical JSON and SHA-256 utilities; final focused suite and type check remained green.

Test Summary

• Tests: 5 written and focused-passing; 45 canonical (unit 4, integration fixture 1)
• Approval tests: None — new production files; pure functions created: 6

Work Unit Evidence

Evidence	Result
Focused test command and exact result	NODE_ENV=test npx --no-install tsx --test tests/ocr/detection.test.ts — exit 0; 5 tests passed, 0 failed.
Runtime harness command/scenario and exact result	NODE_ENV=test npx --no-install tsx --test --test-name-pattern "parsed mixed PDF" tests/ocr/detection.test.ts — exit 0; 1 test passed, 0 failed. The harness generated a three-page PDF, parsed it through the Node 22 pdf-parse page callback, and selected only blank page 2 and insufficient page 3 for OCR.
Rollback boundary	Delete src/modules/ocr/detection.ts, src/modules/ocr/composition.ts, and tests/ocr/detection.test.ts; revert tasks 2.2/2.4/2.5 and this Unit 4 progress/history entry. Units 1–3 and their parser threat coverage remain intact.

Validation and Boundary

• Canonical npm test passed 45/45; npm run check, tracked git diff --check, and no-index whitespace checks for all three new files exited 0.
• Authored Unit 4 code and tests total 322 additions and 0 deletions, below the 400-line budget with no size exception.
• Start: Unit 3 page extraction and parser threat routing are complete. End: detection, blank/quality gates, deterministic composition, hashes, and risk tokens are green.
• Out of scope: OCR service, client, dispatcher, ingest routing, review APIs, commits, pushes, PRs, deployment, and restricted paths.
• No specification or design deviation.

Unit 5: OCR Service

Implementation and Correction Summary

• Added pinned Unit 5 FastAPI test dependencies and retained the ignored ocr-service/.venv environment.
• Added RED-first HTTP tests and a SQLite-backed private API for bearer auth, idempotency, allowlisting, upload/page limits, queue pressure, status, deletion, and health.
• Preserved failed evidence sha256:4a5c8ed8d8b302f7ff654d4f40fe0f13049bfdd29db4c76576f3847c0a9be15a: its harness returned 401/400/400; inspection showed & backgrounded the preceding shell AND-list, so request variables existed only in the child shell and authenticated curls sent an empty request form, reproduced as 400 INVALID_REQUEST.
• Corrected only the harness command boundary; the 381-line candidate implementation remained unchanged before progress persistence.

TDD Cycle Evidence

Task	Test File	Layer	Safety Net	RED	GREEN	TRIANGULATE	REFACTOR
3.1	ocr-service/tests/test_api.py	HTTP integration	N/A (new service)	Collection failed with ModuleNotFoundError: No module named 'app'; failed localhost runtime evidence then supplied correction RED.	8/8 focused tests and corrected curl 401/202/409 passed.	Auth variants, idempotency/conflict, allowlist, limits, pressure, integrity, and real multipart requests exercised distinct paths.	No production refactor: the defect was shell variable scope in the harness.
3.2	ocr-service/tests/test_api.py	HTTP integration/SQLite	N/A (new service)	Same missing-service RED covered the API; failed harness blocked completion.	8/8 focused tests and real Uvicorn harness passed.	Ready/not-ready health, queue capacity, status, deletion, and process cleanup exercised distinct paths.	Pure page hashing and centralized terminal errors retained; no correction needed.

Work Unit Evidence

• Focused: ocr-service/.venv/bin/python -m pytest ocr-service/tests -k auth — exit 0; 8 passed, 0 failed (one dependency deprecation warning).
• Runtime: local Uvicorn plus multipart curl — unauthorized 401 with UNAUTHORIZED, accepted 202 with queued identity, conflicting same key 409 with IDEMPOTENCY_CONFLICT.
• Gates: npm test 45/45, npm run check, Python compileall, tracked and untracked whitespace checks — all exit 0/equivalent clean.
• Cleanup: Uvicorn PID 268425 terminated and absent; temporary PDF, SQLite DB, and log removed; .venv retained.
• Rollback: remove .gitignore Unit 5 entries and ocr-service/, then revert tasks 3.1/3.2 and this Unit 5 progress/history block; Units 1–4 remain intact.
• Fresh evidence revision: sha256:0d8b2c57bbab967473eaf99a7ca900168554e64254fa80197b3626b4570b58d0; corrected final authored change count: 397 additions plus deletions, within 400.

Unit 6: OCR Rendering and Image

Implementation Summary

• Added deterministic 200 DPI PDF page rendering with a pre-render 25-megapixel guard, a PaddleOCR adapter, stable line identities, quality metrics, and the exact result schema.
• Added a deterministic fake-driven test seam and preserved ambiguous OCR tokens without correction.
• Added the private CPU image with PaddleOCR 3.4.0, PaddlePaddle CPU 3.2.2, baked Latin/mobile models, one Uvicorn worker, a non-root user, private job storage, healthcheck, and documented 3 CPU/5 GiB deployment limits.
• Added a Dockerfile-specific deny-by-default build context so restricted and unrelated repository paths are never sent to the Docker daemon.

TDD Cycle Evidence

Task	Test File	Layer	Safety Net	RED	GREEN	TRIANGULATE	REFACTOR
3.3	ocr-service/tests/test_render.py	Unit/integration fixture	Existing OCR API suite passed 8/8 before production edits.	Focused collection failed because app.engine did not exist.	Focused render suite passed 5/5; complete OCR suite passed 13/13.	Selected pages 1/3, invalid page and pixel-limit paths, repeatable fake results, and Paddle payload normalization exercise distinct behavior.	Named constants and pure metric/result transforms retained; focused suite remained 5/5.
3.4	ocr-service/tests/test_render.py	Static/container runtime	Existing OCR API suite passed 8/8 before production edits.	Docker contract was absent; deny-by-default context triangulation then failed until Dockerfile.dockerignore existed.	Image built with baked models; offline constrained container rendered and validated PNG/result output.	Static pin/runtime assertions plus real model build and --network none execution cover configuration and runtime paths.	Added missing OpenCV runtime libraries after the first build exposed libGL.so.1; exact corrected image passed.

Work Unit Evidence

Evidence	Result
Focused test command and exact result	ocr-service/.venv/bin/python -m pytest ocr-service/tests -k render -q — exit 0; 5 passed, 8 deselected, 1 dependency deprecation warning.
Runtime harness command/scenario and exact result	docker build --file ocr-service/Dockerfile --tag rag-ocr-service:unit6 . — corrected exact image exited 0 and baked both models. A docker run --rm --network none --cpus 3 --memory 5g ... harness loaded cached PaddleOCR 3.4.0/PaddlePaddle 3.2.2 models, rendered a 1700x2200 PNG (23,309 bytes), returned schema 1, one page, and one OCR line; host Pillow reopened it as PNG 1700x2200.
Rollback boundary	Remove ocr-service/{Dockerfile,Dockerfile.dockerignore,app/engine.py,app/models.py,app/render.py,tests/test_render.py} and revert Unit 6 changes in app/main.py, requirements.txt, and README.md; Units 1–5 remain intact.

Validation and Boundary

• RED was followed by GREEN and post-refactor reruns. One expected GREEN iteration corrected a test-side non-whitespace count from 16 to the mathematically correct 15.
• The first image build failed on missing libGL.so.1; adding the required slim-image runtime libraries produced a successful build and runtime harness. A later optional connectivity-check refactor rebuild exhausted Docker storage; that unvalidated one-line refactor was reverted, the validated Dockerfile bytes were restored, and the image tag was removed during cleanup.
• npm test passed 45/45; npm run check, Python compileall, tracked/untracked whitespace checks, and the full OCR pytest suite (13/13) passed.
• Start: Unit 5 private queue/API is complete. End: Unit 6 render/engine/schema/container behavior is complete. Unit 7 client work was not started.
• Native Unit 6 implementation/tests/docs are 397 authored additions plus deletions, within the 400-line budget; required SDD progress and workspace history metadata are administrative evidence outside that native slice.
• No specification or design deviation. The Docker image is private by deployment contract; CPU/RAM labels document limits that the platform must enforce.

Unit 7: OCR Client and Durable Artifacts

Implementation Summary

• Added a private OCR HTTP client that preserves the exact idempotency key across one initial submission and at most two transient resubmissions, treats 429 as retryable queue pressure without immediate resubmission, and treats deterministic failures as terminal.
• Added 2–15 second polling backoff plus strict acknowledgement, status, engine, job, document, ordered-page, metrics, line, and schema validation before a result can be persisted.
• Added durable 0600 original and canonical manifest writes, UUIDv5 document artifact identities, lexical path containment, verifiable hashes, and age-gated orphan sweeping that preserves retained, recent, non-UUID, and symlink entries.
• Left task 4.1 pending because Unit 7 proves only its retries/integrity clause; Unit 8 still owns flag-off 201, OCR 202/status, OCR-down fail-closed, and catalog-down 503 HTTP coverage.

TDD Cycle Evidence

Task	Test File	Layer	Safety Net	RED	GREEN	TRIANGULATE	REFACTOR
4.2	tests/ocr/client.test.ts	Unit/runtime fetch stub	Existing OCR detection suite passed 5/5; production file was new.	Focused suite failed with ERR_MODULE_NOT_FOUND for src/modules/ocr/client.js before production code existed.	Final focused suite passed 6/6; runtime retry scenario passed 1/1.	Network then 503 produced exactly three same-key attempts with 2s/4s backoff; 429 and 422 each stopped after one attempt; malformed acknowledgement/schema/pages failed integrity; polling used 2s/4s.	Adapted multipart bytes to a type-safe Uint8Array; focused suite and type check remained green.
4.3	tests/ocr/client.test.ts	Filesystem integration	Existing OCR detection suite passed 5/5; production file was new.	The same missing-module RED preceded artifact production code.	Private originals, canonical manifest, safe resolution, and sweep passed in the final 6/6 suite.	Two out-of-order documents proved canonical ordering/hashes; traversal was rejected; sweep removed only an old orphan while preserving retained/recent directories and a symlink.	A test-side assertion was corrected to inspect the retained dangling symlink with lstat; no production behavior changed, and the suite passed 6/6.

Test Summary

• Tests written and passing: 6 focused tests; unit/fetch-stub and filesystem integration layers.
• Approval tests: None — both production modules are new.
• RED: exit 1, module-not-found before either production file existed.
• GREEN/REFACTOR: exit 0, 6 passed, 0 failed.

Work Unit Evidence

Evidence	Result
Focused test command and exact result	NODE_ENV=test npx --no-install tsx --test tests/ocr/client.test.ts — exit 0; 6 passed, 0 failed.
Runtime harness command/scenario and exact result	NODE_ENV=test npx --no-install tsx --test --test-name-pattern "runtime retry stub" tests/ocr/client.test.ts — exit 0; 1 passed, 0 failed. Deterministic stub observed network failure, 503, then 202: exactly 3 submission attempts, one unchanged idempotency key, 2,000/4,000 ms backoffs, followed by a strictly validated two-page result.
Rollback boundary	Remove src/modules/ocr/client.ts, src/modules/ocr/artifacts.ts, and tests/ocr/client.test.ts; revert tasks 4.2/4.3 and this Unit 7 progress/history metadata. Units 1–6 remain intact.

Validation and Boundary

• npm test passed 51/51; npm run check, npm run build, tracked whitespace, and no-index whitespace checks for all three new files passed.
• Test-created rag-ocr-artifacts-* and rag-ocr-sweep-* directories were removed; no server, external OCR call, or persistent process was started; ocr-service/.venv was preserved.
• Start: Unit 6 OCR rendering and container runtime are complete. End: Unit 7 client retries/integrity and durable artifacts/orphan sweep are implemented and green.
• Out of scope and untouched: dispatcher, reconciler, ingest branch, app upload/status routes, Unit 8 tests, commits, pushes, PRs, native review, deployment, and restricted paths.
• The implementation/test slice is 506 authored additions (client 174, artifacts 129, tests 203). This exceeds the 400-line default after one honest cohesive assessment; no code-golf or second slicing pass was attempted. The maintainer explicitly approved size:exception, including 585 total native-accounting lines with required metadata.
• No specification or design deviation.

Approved Size-Exception Corrective Context

• The native attempt was reset after failed evidence revision sha256:dd224c3864adc8380dc7d9fd147cb448f5da8138277f3f346a35d7eeb3aca158; the failure was budget-only, not a functional defect.
• Corrective work unit unit-7-size-exception-validation uses attempt token sha256:0109cfd8d3b37a5672c65a37700269b89cb80e4e357a39f68a920779a1715c39 and must settle against the failed revision through the parent.
• Production and test files remain unchanged for the corrective rerun; only approval/reset evidence and fresh validation metadata may change.
• Task 4.1 remains pending, while tasks 4.2 and 4.3 remain complete. Unit 8 was not started.