From ba025fc93d6552bcf7984af276ad74c910fa3ec1 Mon Sep 17 00:00:00 2001 From: Paco POR-CORREO Date: Mon, 14 Sep 2026 14:20:40 +0200 Subject: [PATCH] docs(sdd): record OCR apply progress --- .../ocr-ingest-integration/apply-progress.md | 168 ++++++++++++++++++ 1 file changed, 168 insertions(+) create mode 100644 openspec/changes/ocr-ingest-integration/apply-progress.md diff --git a/openspec/changes/ocr-ingest-integration/apply-progress.md b/openspec/changes/ocr-ingest-integration/apply-progress.md new file mode 100644 index 0000000..db3b88e --- /dev/null +++ b/openspec/changes/ocr-ingest-integration/apply-progress.md @@ -0,0 +1,168 @@ +# Apply Progress: OCR Ingest Integration + +## Current State + +- **Mode:** Strict TDD +- **Delivery:** Feature-branch-chain, Units 1–4 complete; maintainer-approved `size:exception` for Unit 2 only +- **Completed tasks:** 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 2.5 +- **Overall task progress:** 9/30 complete + +## Unit 1: Migration + +### Implementation Summary + +- Added `migrations/002_ocr_review.sql` with durable OCR jobs, auditable page records, atomic review correction targets, and a partial unique index for non-terminal OCR identities. +- Added focused schema contract coverage in `tests/catalog/migration-002.test.ts`. +- Preserved `migrations/001_knowledge_lifecycle.sql` and all repository logic unchanged. + +### TDD Cycle Evidence + +| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR | +|---|---|---|---|---|---|---|---| +| 1.2 | `tests/catalog/migration-002.test.ts` | Schema contract | N/A (new files) | Valid RED: 4/4 failed with `ENOENT` because migration 002 did not exist. An earlier syntax-invalid run was corrected and is not counted as RED. | 4/4 passed with migration 002 present. | 4 independent schema behaviors cover jobs, pages, corrections, and pending identity uniqueness. | None needed; SQL already matches the minimal authoritative contract. | + +### Test Summary + +- **Total tests written:** 4 +- **Total tests passing:** 4 +- **Layers used:** Schema contract (4) +- **Approval tests:** None — no existing production file was modified +- **Pure functions created:** 0 + +### Work Unit Evidence + +| Evidence | Result | +|---|---| +| Focused test command and exact result | `npx --no-install tsx --test tests/catalog/migration-002.test.ts` — exit 0; 4 tests, 4 passed, 0 failed. | +| Runtime harness command/scenario and exact result | N/A — the task forecast defines Unit 1 as schema-only, and the project has no PostgreSQL test harness or database-test dependency. Applying a live migration would require external database access outside this unit. | +| Rollback boundary | Delete `migrations/002_ocr_review.sql` and `tests/catalog/migration-002.test.ts`, revert task 1.2 to pending, and remove this Unit 1 progress/history entry. No existing migration or repository behavior must be reverted. | + +### Additional Validation + +- `npm run check` — exit 0. +- `git diff --check` — exit 0 after removing one trailing-space warning introduced in the history header. + +### Deviations + +None — the migration follows the proposal, specifications, design, and closed OCR persistence contract. + +### Remaining Persistence Tasks + +- [x] 1.1 Complete repository-level RED coverage for duplicate pending ingestion, lease recovery, and review transitions in Unit 2. +- [x] 1.2 Add migration 002 schema and pending OCR identity unique index. +- [x] 1.3 Add catalog repository candidates and leases in Unit 2. +- [x] 1.4 Refactor persistence code after Unit 2 reaches green. + +### Review Boundary + +- **Start:** Migration 001 already supplies lifecycle states and parent catalog tables. +- **End:** Migration 002 and its focused schema contract test are green. +- **Out of scope:** Repository methods, dispatch behavior, runtime PostgreSQL migration, Unit 2, commits, pushes, PRs, deployment, and secrets. +- **Rollback:** Remove only the two Unit 1 implementation files and their progress metadata. +- **Authored review budget:** 225 additions/deletions, below the 400-line budget for this autonomous slice. + +## Unit 2: Repository + +### Implementation Summary + +- Added pending OCR identity lookup, atomic queue claiming, expired-lease recovery, and guarded review transitions to `CatalogRepository`. +- Added a focused fake-pool repository harness that covers duplicate reuse, empty queues, remote identity preservation, and invalid transitions. +- Corrected the canonical test script so it executes both root-level and nested test suites. + +### TDD Cycle Evidence + +| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR | +|---|---|---|---|---|---|---|---| +| 1.1 | `tests/catalog/repository-ocr.test.ts` | Unit/fake pool | 4/4 existing tests passed | 0/6 with methods absent; then 5/6 while each completion gate was missing | 6/6 passed after task 1.3 | Found/not-found candidates, claimed/empty queue, expired lease, valid/invalid transition | Covered by 1.4 | +| 1.3 | `tests/catalog/repository-ocr.test.ts` | Unit/fake pool | 4/4 existing tests passed | Tests preceded production code | 6/6 passed | Multiple inputs and state paths exercised | Shared OCR row mapping extracted only after green | +| 1.4 | `tests/catalog/repository-ocr.test.ts` | Approval refactor | 6/6 before refactor | N/A — behavior-preserving approval baseline | 6/6 after refactor | Existing six cases preserved | Reusable aliased-column projection removed SQL duplication | + +### Work Unit Evidence + +| Evidence | Result | +|---|---| +| Focused test command and exact result | `NODE_ENV=test npx --no-install tsx --test tests/catalog/repository-ocr.test.ts` — exit 0; 6 tests passed, 0 failed. `NODE_ENV=test npx --no-install tsx --test tests/catalog/*.test.ts` — exit 0; 10 tests passed, 0 failed. | +| Runtime harness command/scenario and exact result | `NODE_ENV=test npx --no-install tsx --test tests/*.test.ts` — exit 0; 25 root tests passed, 0 failed. Canonical `npm test` — exit 0; 35 total tests passed, 0 failed, proving the 25 root and 10 nested tests execute together. | +| Rollback boundary | Revert Unit 2 changes in `src/modules/catalog/repository.ts`, delete `tests/catalog/repository-ocr.test.ts`, and revert the `package.json` test script; Unit 1 migration remains intact. | + +### Validation and Boundary + +- Corrective rerun for failed evidence revision `sha256:9433c0cd6628f36f4c96a27d41c6d429f4303d038ffe928f297367c274f21c2f`; fresh evidence revision `sha256:a19119f84381447f54a6223e2a203acaf9d3ae08b6ae8f585e3bd52cd510909b`. +- The failed canonical script expanded only nested tests after nested suites appeared. The minimal correction is `NODE_ENV=test tsx --test tests/*.test.ts tests/**/*.test.ts`. +- Focused repository suite passed 6/6; all catalog suites passed 10/10; explicit root suites passed 25/25; canonical `npm test` passed 35/35; `npm run check` and `git diff --check` exited 0. +- Native Unit 2 accounting was 511 changed lines. The maintainer approved `size:exception` exclusively for Unit 2; the corrective attempt has a separate 600-line remediation budget. All later units retain the normal 400-line budget. +- No design deviations. No Unit 3 work was started. + +### Corrective TDD Evidence + +| Correction | Safety Net / RED | GREEN | TRIANGULATE | REFACTOR | +|---|---|---|---|---| +| Canonical test discovery | Failed evidence revision proved canonical `npm test` executed only 10 nested tests while the explicit root command executed 25 tests. | Updated only the package test argument list; canonical `npm test` passed 35/35. | Direct root 25/25 and nested catalog 10/10 results sum to and identify the canonical 35/35 run. | None needed — one package-script argument was added. | + +## Unit 3: Extraction + +### Implementation Summary + +- Added a Node 22 fixture spike proving `pdf-parse` page callbacks backed by bundled PDF.js `2.0.550` preserve ordered pages, including a blank page. +- Added `parsePdfPages` with one-based page identity, raw native text, stable SHA-256 hashes, and fail-closed complete-page accounting. +- Preserved textual ingestion as data-only, excluded `CMakeLists.txt`, MDX, and shell lookalikes, and kept task 2.2 pending for Unit 4 detection/composition assertions. + +### TDD Cycle Evidence + +| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR | +|---|---|---|---|---|---|---|---| +| 2.1 | `tests/parsers/pdf-pages.test.ts`; `scripts/spike-pdfjs.ts` | Runtime spike | 20/20 lifecycle service tests passed | Focused suite failed 0/1 because `parsePdfPages` did not exist; the spike and parser implementation were still absent. | Node 22.14.0 spike returned pages 1–3 exactly, including blank page 2. | Three pages prove non-empty, blank, and later-page ordering paths. | Replaced the over-budget direct dependency with the design-approved existing `pdf-parse` pagerender path; spike remained green. | +| 2.3 | `tests/parsers/pdf-pages.test.ts` | Unit/integration fixture | 20/20 lifecycle service tests passed | Focused suite failed 0/1 on the missing export before production code. | Final focused run passed 5/5 after implementation. | Covered page hashes/blank page, native PDF composition, non-PDF rejection, data-only text parsing, and unsupported lookalikes. | Focused suite remained 5/5 after consolidating on the existing parser dependency. | + +### Work Unit Evidence + +| Evidence | Result | +|---|---| +| Focused test command and exact result | `NODE_ENV=test npx --no-install tsx --test tests/parsers/pdf-pages.test.ts` — exit 0; 5 passed, 0 failed. | +| Runtime harness command/scenario and exact result | `NODE_ENV=test npx --no-install tsx scripts/spike-pdfjs.ts` — exit 0 on Node 22.14.0; PDF.js 2.0.550 returned three ordered pages with page 2 blank. | +| Rollback boundary | Revert `src/modules/parsers/parser-registry.ts`; delete `scripts/spike-pdfjs.ts`, `tests/parsers/pdf-pages.test.ts`, and `tests/fixtures/ocr/native-three-pages.pdf`; revert tasks 2.1/2.3 and this Unit 3 progress/history entry. Units 1–2 remain intact. | + +### Validation and Boundary + +- Canonical `npm test` passed 40/40; `npm run check` and `git diff --check` exited 0. +- Native authored slice before progress/history metadata: 225 additions/deletions; no package or lockfile delta remains. +- Start: Unit 2 persistence is complete. End: per-page native extraction and parser threat routing are green. +- Out of scope: detection thresholds, OCR selection, composition, risks, Unit 4 tasks 2.4/2.5, commits, pushes, PRs, deployment, and restricted paths. +- No specification deviation. The design-authorized `pdf-parse` pagerender fallback was selected only after its Node 22 fixture spike passed. + +## Unit 4: Detection and Composition + +### Implementation Summary + +- Added pure per-page native detection under `pdf-detection-v1`, exact OCR page selection, verified-blank classification, and fail-closed OCR quality gates. +- Added deterministic candidate composition using exactly one source per page, ordered OCR lines, blank omission, canonical hashes, and unchanged risk tokens prioritized for review. +- Completed the remaining task 2.2 scenarios by combining the Unit 3 data-only/parser threat coverage with Unit 4 mixed-PDF, detection, blank, hash, and risk RED assertions. + +### TDD Cycle Evidence + +| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR | +|---|---|---|---|---|---|---|---| +| 2.2 | `tests/ocr/detection.test.ts`; `tests/parsers/pdf-pages.test.ts` | Unit/integration fixture | Unit 3 parser suite passed 5/5 before edits. | Focused Unit 4 suite failed before production with `ERR_MODULE_NOT_FOUND` for `src/modules/ocr/detection.js`. | Focused suite passed 5/5 after detection and composition were added. | Real generated mixed PDF selected pages 2 and 3; `.txt`/`.md` stayed data-only and CMake/MDX/shell lookalikes stayed unsupported. | None needed; routing remains separated between parser support and the pure PDF OCR planner. | +| 2.4 | `tests/ocr/detection.test.ts` | Unit | N/A (new production file) | Threshold, mixed-page, blank, and quality assertions preceded production code. | Exact native, blank, and OCR quality boundaries passed in the 5/5 focused suite. | Every native and OCR threshold has a passing boundary plus a failing neighbor; visible ink with empty OCR is blocked. | Pure functions and named policy version were present at green; final focused suite remained 5/5. | +| 2.5 | `tests/ocr/detection.test.ts` | Unit | N/A (new production file) | Ordered composition, blank omission, canonical hash, audit-source, and risk assertions preceded production code. | Deterministic composition and risk prioritization passed in the 5/5 focused suite. | Reordered pages, tied OCR boxes, blank middle page, repeated token, and four known ambiguous tokens exercise distinct paths. | Reused shared canonical JSON and SHA-256 utilities; final focused suite and type check remained green. | + +### Test Summary + +- **Tests:** 5 written and focused-passing; 45 canonical (unit 4, integration fixture 1) +- **Approval tests:** None — new production files; **pure functions created:** 6 + +### Work Unit Evidence + +| Evidence | Result | +|---|---| +| Focused test command and exact result | `NODE_ENV=test npx --no-install tsx --test tests/ocr/detection.test.ts` — exit 0; 5 tests passed, 0 failed. | +| Runtime harness command/scenario and exact result | `NODE_ENV=test npx --no-install tsx --test --test-name-pattern "parsed mixed PDF" tests/ocr/detection.test.ts` — exit 0; 1 test passed, 0 failed. The harness generated a three-page PDF, parsed it through the Node 22 `pdf-parse` page callback, and selected only blank page 2 and insufficient page 3 for OCR. | +| Rollback boundary | Delete `src/modules/ocr/detection.ts`, `src/modules/ocr/composition.ts`, and `tests/ocr/detection.test.ts`; revert tasks 2.2/2.4/2.5 and this Unit 4 progress/history entry. Units 1–3 and their parser threat coverage remain intact. | + +### Validation and Boundary + +- Canonical `npm test` passed 45/45; `npm run check`, tracked `git diff --check`, and no-index whitespace checks for all three new files exited 0. +- Authored Unit 4 code and tests total 322 additions and 0 deletions, below the 400-line budget with no size exception. +- Start: Unit 3 page extraction and parser threat routing are complete. End: detection, blank/quality gates, deterministic composition, hashes, and risk tokens are green. +- Out of scope: OCR service, client, dispatcher, ingest routing, review APIs, commits, pushes, PRs, deployment, and restricted paths. +- No specification or design deviation.