273 lines
26 KiB
Markdown
273 lines
26 KiB
Markdown
# Apply Progress: OCR Ingest Integration
|
||
|
||
## Current State
|
||
|
||
- **Mode:** Strict TDD
|
||
- **Delivery:** Feature-branch-chain; Units 1–7 implemented; the maintainer approved `size:exception` for Unit 7's cohesive 506 functional authored lines and 585 total native-accounting lines
|
||
- **Completed tasks:** 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2, 3.3, 3.4, 4.2, 4.3
|
||
- **Overall task progress:** 15/30 complete
|
||
|
||
## Unit 1: Migration
|
||
|
||
### Implementation Summary
|
||
|
||
- Added `migrations/002_ocr_review.sql` with durable OCR jobs, auditable page records, atomic review correction targets, and a partial unique index for non-terminal OCR identities.
|
||
- Added focused schema contract coverage in `tests/catalog/migration-002.test.ts`.
|
||
- Preserved `migrations/001_knowledge_lifecycle.sql` and all repository logic unchanged.
|
||
|
||
### TDD Cycle Evidence
|
||
|
||
| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 1.2 | `tests/catalog/migration-002.test.ts` | Schema contract | N/A (new files) | Valid RED: 4/4 failed with `ENOENT` because migration 002 did not exist. An earlier syntax-invalid run was corrected and is not counted as RED. | 4/4 passed with migration 002 present. | 4 independent schema behaviors cover jobs, pages, corrections, and pending identity uniqueness. | None needed; SQL already matches the minimal authoritative contract. |
|
||
|
||
### Test Summary
|
||
|
||
- **Total tests written:** 4
|
||
- **Total tests passing:** 4
|
||
- **Layers used:** Schema contract (4)
|
||
- **Approval tests:** None — no existing production file was modified
|
||
- **Pure functions created:** 0
|
||
|
||
### Work Unit Evidence
|
||
|
||
| Evidence | Result |
|
||
|---|---|
|
||
| Focused test command and exact result | `npx --no-install tsx --test tests/catalog/migration-002.test.ts` — exit 0; 4 tests, 4 passed, 0 failed. |
|
||
| Runtime harness command/scenario and exact result | N/A — the task forecast defines Unit 1 as schema-only, and the project has no PostgreSQL test harness or database-test dependency. Applying a live migration would require external database access outside this unit. |
|
||
| Rollback boundary | Delete `migrations/002_ocr_review.sql` and `tests/catalog/migration-002.test.ts`, revert task 1.2 to pending, and remove this Unit 1 progress/history entry. No existing migration or repository behavior must be reverted. |
|
||
|
||
### Additional Validation
|
||
|
||
- `npm run check` — exit 0.
|
||
- `git diff --check` — exit 0 after removing one trailing-space warning introduced in the history header.
|
||
|
||
### Deviations
|
||
|
||
None — the migration follows the proposal, specifications, design, and closed OCR persistence contract.
|
||
|
||
### Remaining Persistence Tasks
|
||
|
||
- [x] 1.1 Complete repository-level RED coverage for duplicate pending ingestion, lease recovery, and review transitions in Unit 2.
|
||
- [x] 1.2 Add migration 002 schema and pending OCR identity unique index.
|
||
- [x] 1.3 Add catalog repository candidates and leases in Unit 2.
|
||
- [x] 1.4 Refactor persistence code after Unit 2 reaches green.
|
||
|
||
### Review Boundary
|
||
|
||
- **Start:** Migration 001 already supplies lifecycle states and parent catalog tables.
|
||
- **End:** Migration 002 and its focused schema contract test are green.
|
||
- **Out of scope:** Repository methods, dispatch behavior, runtime PostgreSQL migration, Unit 2, commits, pushes, PRs, deployment, and secrets.
|
||
- **Rollback:** Remove only the two Unit 1 implementation files and their progress metadata.
|
||
- **Authored review budget:** 225 additions/deletions, below the 400-line budget for this autonomous slice.
|
||
|
||
## Unit 2: Repository
|
||
|
||
### Implementation Summary
|
||
|
||
- Added pending OCR identity lookup, atomic queue claiming, expired-lease recovery, and guarded review transitions to `CatalogRepository`.
|
||
- Added a focused fake-pool repository harness that covers duplicate reuse, empty queues, remote identity preservation, and invalid transitions.
|
||
- Corrected the canonical test script so it executes both root-level and nested test suites.
|
||
|
||
### TDD Cycle Evidence
|
||
|
||
| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 1.1 | `tests/catalog/repository-ocr.test.ts` | Unit/fake pool | 4/4 existing tests passed | 0/6 with methods absent; then 5/6 while each completion gate was missing | 6/6 passed after task 1.3 | Found/not-found candidates, claimed/empty queue, expired lease, valid/invalid transition | Covered by 1.4 |
|
||
| 1.3 | `tests/catalog/repository-ocr.test.ts` | Unit/fake pool | 4/4 existing tests passed | Tests preceded production code | 6/6 passed | Multiple inputs and state paths exercised | Shared OCR row mapping extracted only after green |
|
||
| 1.4 | `tests/catalog/repository-ocr.test.ts` | Approval refactor | 6/6 before refactor | N/A — behavior-preserving approval baseline | 6/6 after refactor | Existing six cases preserved | Reusable aliased-column projection removed SQL duplication |
|
||
|
||
### Work Unit Evidence
|
||
|
||
| Evidence | Result |
|
||
|---|---|
|
||
| Focused test command and exact result | `NODE_ENV=test npx --no-install tsx --test tests/catalog/repository-ocr.test.ts` — exit 0; 6 tests passed, 0 failed. `NODE_ENV=test npx --no-install tsx --test tests/catalog/*.test.ts` — exit 0; 10 tests passed, 0 failed. |
|
||
| Runtime harness command/scenario and exact result | `NODE_ENV=test npx --no-install tsx --test tests/*.test.ts` — exit 0; 25 root tests passed, 0 failed. Canonical `npm test` — exit 0; 35 total tests passed, 0 failed, proving the 25 root and 10 nested tests execute together. |
|
||
| Rollback boundary | Revert Unit 2 changes in `src/modules/catalog/repository.ts`, delete `tests/catalog/repository-ocr.test.ts`, and revert the `package.json` test script; Unit 1 migration remains intact. |
|
||
|
||
### Validation and Boundary
|
||
|
||
- Corrective rerun for failed evidence revision `sha256:9433c0cd6628f36f4c96a27d41c6d429f4303d038ffe928f297367c274f21c2f`; fresh evidence revision `sha256:a19119f84381447f54a6223e2a203acaf9d3ae08b6ae8f585e3bd52cd510909b`.
|
||
- The failed canonical script expanded only nested tests after nested suites appeared. The minimal correction is `NODE_ENV=test tsx --test tests/*.test.ts tests/**/*.test.ts`.
|
||
- Focused repository suite passed 6/6; all catalog suites passed 10/10; explicit root suites passed 25/25; canonical `npm test` passed 35/35; `npm run check` and `git diff --check` exited 0.
|
||
- Native Unit 2 accounting was 511 changed lines. The maintainer approved `size:exception` exclusively for Unit 2; the corrective attempt has a separate 600-line remediation budget. All later units retain the normal 400-line budget.
|
||
- No design deviations. No Unit 3 work was started.
|
||
|
||
### Corrective TDD Evidence
|
||
|
||
| Correction | Safety Net / RED | GREEN | TRIANGULATE | REFACTOR |
|
||
|---|---|---|---|---|
|
||
| Canonical test discovery | Failed evidence revision proved canonical `npm test` executed only 10 nested tests while the explicit root command executed 25 tests. | Updated only the package test argument list; canonical `npm test` passed 35/35. | Direct root 25/25 and nested catalog 10/10 results sum to and identify the canonical 35/35 run. | None needed — one package-script argument was added. |
|
||
|
||
## Unit 3: Extraction
|
||
|
||
### Implementation Summary
|
||
|
||
- Added a Node 22 fixture spike proving `pdf-parse` page callbacks backed by bundled PDF.js `2.0.550` preserve ordered pages, including a blank page.
|
||
- Added `parsePdfPages` with one-based page identity, raw native text, stable SHA-256 hashes, and fail-closed complete-page accounting.
|
||
- Preserved textual ingestion as data-only, excluded `CMakeLists.txt`, MDX, and shell lookalikes, and kept task 2.2 pending for Unit 4 detection/composition assertions.
|
||
|
||
### TDD Cycle Evidence
|
||
|
||
| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 2.1 | `tests/parsers/pdf-pages.test.ts`; `scripts/spike-pdfjs.ts` | Runtime spike | 20/20 lifecycle service tests passed | Focused suite failed 0/1 because `parsePdfPages` did not exist; the spike and parser implementation were still absent. | Node 22.14.0 spike returned pages 1–3 exactly, including blank page 2. | Three pages prove non-empty, blank, and later-page ordering paths. | Replaced the over-budget direct dependency with the design-approved existing `pdf-parse` pagerender path; spike remained green. |
|
||
| 2.3 | `tests/parsers/pdf-pages.test.ts` | Unit/integration fixture | 20/20 lifecycle service tests passed | Focused suite failed 0/1 on the missing export before production code. | Final focused run passed 5/5 after implementation. | Covered page hashes/blank page, native PDF composition, non-PDF rejection, data-only text parsing, and unsupported lookalikes. | Focused suite remained 5/5 after consolidating on the existing parser dependency. |
|
||
|
||
### Work Unit Evidence
|
||
|
||
| Evidence | Result |
|
||
|---|---|
|
||
| Focused test command and exact result | `NODE_ENV=test npx --no-install tsx --test tests/parsers/pdf-pages.test.ts` — exit 0; 5 passed, 0 failed. |
|
||
| Runtime harness command/scenario and exact result | `NODE_ENV=test npx --no-install tsx scripts/spike-pdfjs.ts` — exit 0 on Node 22.14.0; PDF.js 2.0.550 returned three ordered pages with page 2 blank. |
|
||
| Rollback boundary | Revert `src/modules/parsers/parser-registry.ts`; delete `scripts/spike-pdfjs.ts`, `tests/parsers/pdf-pages.test.ts`, and `tests/fixtures/ocr/native-three-pages.pdf`; revert tasks 2.1/2.3 and this Unit 3 progress/history entry. Units 1–2 remain intact. |
|
||
|
||
### Validation and Boundary
|
||
|
||
- Canonical `npm test` passed 40/40; `npm run check` and `git diff --check` exited 0.
|
||
- Native authored slice before progress/history metadata: 225 additions/deletions; no package or lockfile delta remains.
|
||
- Start: Unit 2 persistence is complete. End: per-page native extraction and parser threat routing are green.
|
||
- Out of scope: detection thresholds, OCR selection, composition, risks, Unit 4 tasks 2.4/2.5, commits, pushes, PRs, deployment, and restricted paths.
|
||
- No specification deviation. The design-authorized `pdf-parse` pagerender fallback was selected only after its Node 22 fixture spike passed.
|
||
|
||
## Unit 4: Detection and Composition
|
||
|
||
### Implementation Summary
|
||
|
||
- Added pure per-page native detection under `pdf-detection-v1`, exact OCR page selection, verified-blank classification, and fail-closed OCR quality gates.
|
||
- Added deterministic candidate composition using exactly one source per page, ordered OCR lines, blank omission, canonical hashes, and unchanged risk tokens prioritized for review.
|
||
- Completed the remaining task 2.2 scenarios by combining the Unit 3 data-only/parser threat coverage with Unit 4 mixed-PDF, detection, blank, hash, and risk RED assertions.
|
||
|
||
### TDD Cycle Evidence
|
||
|
||
| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 2.2 | `tests/ocr/detection.test.ts`; `tests/parsers/pdf-pages.test.ts` | Unit/integration fixture | Unit 3 parser suite passed 5/5 before edits. | Focused Unit 4 suite failed before production with `ERR_MODULE_NOT_FOUND` for `src/modules/ocr/detection.js`. | Focused suite passed 5/5 after detection and composition were added. | Real generated mixed PDF selected pages 2 and 3; `.txt`/`.md` stayed data-only and CMake/MDX/shell lookalikes stayed unsupported. | None needed; routing remains separated between parser support and the pure PDF OCR planner. |
|
||
| 2.4 | `tests/ocr/detection.test.ts` | Unit | N/A (new production file) | Threshold, mixed-page, blank, and quality assertions preceded production code. | Exact native, blank, and OCR quality boundaries passed in the 5/5 focused suite. | Every native and OCR threshold has a passing boundary plus a failing neighbor; visible ink with empty OCR is blocked. | Pure functions and named policy version were present at green; final focused suite remained 5/5. |
|
||
| 2.5 | `tests/ocr/detection.test.ts` | Unit | N/A (new production file) | Ordered composition, blank omission, canonical hash, audit-source, and risk assertions preceded production code. | Deterministic composition and risk prioritization passed in the 5/5 focused suite. | Reordered pages, tied OCR boxes, blank middle page, repeated token, and four known ambiguous tokens exercise distinct paths. | Reused shared canonical JSON and SHA-256 utilities; final focused suite and type check remained green. |
|
||
|
||
### Test Summary
|
||
|
||
- **Tests:** 5 written and focused-passing; 45 canonical (unit 4, integration fixture 1)
|
||
- **Approval tests:** None — new production files; **pure functions created:** 6
|
||
|
||
### Work Unit Evidence
|
||
|
||
| Evidence | Result |
|
||
|---|---|
|
||
| Focused test command and exact result | `NODE_ENV=test npx --no-install tsx --test tests/ocr/detection.test.ts` — exit 0; 5 tests passed, 0 failed. |
|
||
| Runtime harness command/scenario and exact result | `NODE_ENV=test npx --no-install tsx --test --test-name-pattern "parsed mixed PDF" tests/ocr/detection.test.ts` — exit 0; 1 test passed, 0 failed. The harness generated a three-page PDF, parsed it through the Node 22 `pdf-parse` page callback, and selected only blank page 2 and insufficient page 3 for OCR. |
|
||
| Rollback boundary | Delete `src/modules/ocr/detection.ts`, `src/modules/ocr/composition.ts`, and `tests/ocr/detection.test.ts`; revert tasks 2.2/2.4/2.5 and this Unit 4 progress/history entry. Units 1–3 and their parser threat coverage remain intact. |
|
||
|
||
### Validation and Boundary
|
||
|
||
- Canonical `npm test` passed 45/45; `npm run check`, tracked `git diff --check`, and no-index whitespace checks for all three new files exited 0.
|
||
- Authored Unit 4 code and tests total 322 additions and 0 deletions, below the 400-line budget with no size exception.
|
||
- Start: Unit 3 page extraction and parser threat routing are complete. End: detection, blank/quality gates, deterministic composition, hashes, and risk tokens are green.
|
||
- Out of scope: OCR service, client, dispatcher, ingest routing, review APIs, commits, pushes, PRs, deployment, and restricted paths.
|
||
- No specification or design deviation.
|
||
|
||
## Unit 5: OCR Service
|
||
|
||
### Implementation and Correction Summary
|
||
|
||
- Added pinned Unit 5 FastAPI test dependencies and retained the ignored `ocr-service/.venv` environment.
|
||
- Added RED-first HTTP tests and a SQLite-backed private API for bearer auth, idempotency, allowlisting, upload/page limits, queue pressure, status, deletion, and health.
|
||
- Preserved failed evidence `sha256:4a5c8ed8d8b302f7ff654d4f40fe0f13049bfdd29db4c76576f3847c0a9be15a`: its harness returned `401/400/400`; inspection showed `&` backgrounded the preceding shell AND-list, so request variables existed only in the child shell and authenticated curls sent an empty `request` form, reproduced as `400 INVALID_REQUEST`.
|
||
- Corrected only the harness command boundary; the 381-line candidate implementation remained unchanged before progress persistence.
|
||
|
||
### TDD Cycle Evidence
|
||
|
||
| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 3.1 | `ocr-service/tests/test_api.py` | HTTP integration | N/A (new service) | Collection failed with `ModuleNotFoundError: No module named 'app'`; failed localhost runtime evidence then supplied correction RED. | 8/8 focused tests and corrected curl `401/202/409` passed. | Auth variants, idempotency/conflict, allowlist, limits, pressure, integrity, and real multipart requests exercised distinct paths. | No production refactor: the defect was shell variable scope in the harness. |
|
||
| 3.2 | `ocr-service/tests/test_api.py` | HTTP integration/SQLite | N/A (new service) | Same missing-service RED covered the API; failed harness blocked completion. | 8/8 focused tests and real Uvicorn harness passed. | Ready/not-ready health, queue capacity, status, deletion, and process cleanup exercised distinct paths. | Pure page hashing and centralized terminal errors retained; no correction needed. |
|
||
|
||
### Work Unit Evidence
|
||
|
||
- Focused: `ocr-service/.venv/bin/python -m pytest ocr-service/tests -k auth` — exit 0; 8 passed, 0 failed (one dependency deprecation warning).
|
||
- Runtime: local Uvicorn plus multipart curl — unauthorized `401` with `UNAUTHORIZED`, accepted `202` with queued identity, conflicting same key `409` with `IDEMPOTENCY_CONFLICT`.
|
||
- Gates: `npm test` 45/45, `npm run check`, Python `compileall`, tracked and untracked whitespace checks — all exit 0/equivalent clean.
|
||
- Cleanup: Uvicorn PID 268425 terminated and absent; temporary PDF, SQLite DB, and log removed; `.venv` retained.
|
||
- Rollback: remove `.gitignore` Unit 5 entries and `ocr-service/`, then revert tasks 3.1/3.2 and this Unit 5 progress/history block; Units 1–4 remain intact.
|
||
- Fresh evidence revision: `sha256:0d8b2c57bbab967473eaf99a7ca900168554e64254fa80197b3626b4570b58d0`; corrected final authored change count: 397 additions plus deletions, within 400.
|
||
|
||
## Unit 6: OCR Rendering and Image
|
||
|
||
### Implementation Summary
|
||
|
||
- Added deterministic 200 DPI PDF page rendering with a pre-render 25-megapixel guard, a PaddleOCR adapter, stable line identities, quality metrics, and the exact result schema.
|
||
- Added a deterministic fake-driven test seam and preserved ambiguous OCR tokens without correction.
|
||
- Added the private CPU image with PaddleOCR `3.4.0`, PaddlePaddle CPU `3.2.2`, baked Latin/mobile models, one Uvicorn worker, a non-root user, private job storage, healthcheck, and documented 3 CPU/5 GiB deployment limits.
|
||
- Added a Dockerfile-specific deny-by-default build context so restricted and unrelated repository paths are never sent to the Docker daemon.
|
||
|
||
### TDD Cycle Evidence
|
||
|
||
| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 3.3 | `ocr-service/tests/test_render.py` | Unit/integration fixture | Existing OCR API suite passed 8/8 before production edits. | Focused collection failed because `app.engine` did not exist. | Focused render suite passed 5/5; complete OCR suite passed 13/13. | Selected pages 1/3, invalid page and pixel-limit paths, repeatable fake results, and Paddle payload normalization exercise distinct behavior. | Named constants and pure metric/result transforms retained; focused suite remained 5/5. |
|
||
| 3.4 | `ocr-service/tests/test_render.py` | Static/container runtime | Existing OCR API suite passed 8/8 before production edits. | Docker contract was absent; deny-by-default context triangulation then failed until `Dockerfile.dockerignore` existed. | Image built with baked models; offline constrained container rendered and validated PNG/result output. | Static pin/runtime assertions plus real model build and `--network none` execution cover configuration and runtime paths. | Added missing OpenCV runtime libraries after the first build exposed `libGL.so.1`; exact corrected image passed. |
|
||
|
||
### Work Unit Evidence
|
||
|
||
| Evidence | Result |
|
||
|---|---|
|
||
| Focused test command and exact result | `ocr-service/.venv/bin/python -m pytest ocr-service/tests -k render -q` — exit 0; 5 passed, 8 deselected, 1 dependency deprecation warning. |
|
||
| Runtime harness command/scenario and exact result | `docker build --file ocr-service/Dockerfile --tag rag-ocr-service:unit6 .` — corrected exact image exited 0 and baked both models. A `docker run --rm --network none --cpus 3 --memory 5g ...` harness loaded cached PaddleOCR 3.4.0/PaddlePaddle 3.2.2 models, rendered a 1700x2200 PNG (23,309 bytes), returned schema `1`, one page, and one OCR line; host Pillow reopened it as PNG 1700x2200. |
|
||
| Rollback boundary | Remove `ocr-service/{Dockerfile,Dockerfile.dockerignore,app/engine.py,app/models.py,app/render.py,tests/test_render.py}` and revert Unit 6 changes in `app/main.py`, `requirements.txt`, and `README.md`; Units 1–5 remain intact. |
|
||
|
||
### Validation and Boundary
|
||
|
||
- RED was followed by GREEN and post-refactor reruns. One expected GREEN iteration corrected a test-side non-whitespace count from 16 to the mathematically correct 15.
|
||
- The first image build failed on missing `libGL.so.1`; adding the required slim-image runtime libraries produced a successful build and runtime harness. A later optional connectivity-check refactor rebuild exhausted Docker storage; that unvalidated one-line refactor was reverted, the validated Dockerfile bytes were restored, and the image tag was removed during cleanup.
|
||
- `npm test` passed 45/45; `npm run check`, Python `compileall`, tracked/untracked whitespace checks, and the full OCR pytest suite (13/13) passed.
|
||
- Start: Unit 5 private queue/API is complete. End: Unit 6 render/engine/schema/container behavior is complete. Unit 7 client work was not started.
|
||
- Native Unit 6 implementation/tests/docs are 397 authored additions plus deletions, within the 400-line budget; required SDD progress and workspace history metadata are administrative evidence outside that native slice.
|
||
- No specification or design deviation. The Docker image is private by deployment contract; CPU/RAM labels document limits that the platform must enforce.
|
||
|
||
## Unit 7: OCR Client and Durable Artifacts
|
||
|
||
### Implementation Summary
|
||
|
||
- Added a private OCR HTTP client that preserves the exact idempotency key across one initial submission and at most two transient resubmissions, treats `429` as retryable queue pressure without immediate resubmission, and treats deterministic failures as terminal.
|
||
- Added 2–15 second polling backoff plus strict acknowledgement, status, engine, job, document, ordered-page, metrics, line, and schema validation before a result can be persisted.
|
||
- Added durable `0600` original and canonical manifest writes, UUIDv5 document artifact identities, lexical path containment, verifiable hashes, and age-gated orphan sweeping that preserves retained, recent, non-UUID, and symlink entries.
|
||
- Left task 4.1 pending because Unit 7 proves only its retries/integrity clause; Unit 8 still owns flag-off `201`, OCR `202`/status, OCR-down fail-closed, and catalog-down `503` HTTP coverage.
|
||
|
||
### TDD Cycle Evidence
|
||
|
||
| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR |
|
||
|---|---|---|---|---|---|---|---|
|
||
| 4.2 | `tests/ocr/client.test.ts` | Unit/runtime fetch stub | Existing OCR detection suite passed 5/5; production file was new. | Focused suite failed with `ERR_MODULE_NOT_FOUND` for `src/modules/ocr/client.js` before production code existed. | Final focused suite passed 6/6; runtime retry scenario passed 1/1. | Network then `503` produced exactly three same-key attempts with 2s/4s backoff; `429` and `422` each stopped after one attempt; malformed acknowledgement/schema/pages failed integrity; polling used 2s/4s. | Adapted multipart bytes to a type-safe `Uint8Array`; focused suite and type check remained green. |
|
||
| 4.3 | `tests/ocr/client.test.ts` | Filesystem integration | Existing OCR detection suite passed 5/5; production file was new. | The same missing-module RED preceded artifact production code. | Private originals, canonical manifest, safe resolution, and sweep passed in the final 6/6 suite. | Two out-of-order documents proved canonical ordering/hashes; traversal was rejected; sweep removed only an old orphan while preserving retained/recent directories and a symlink. | A test-side assertion was corrected to inspect the retained dangling symlink with `lstat`; no production behavior changed, and the suite passed 6/6. |
|
||
|
||
### Test Summary
|
||
|
||
- **Tests written and passing:** 6 focused tests; unit/fetch-stub and filesystem integration layers.
|
||
- **Approval tests:** None — both production modules are new.
|
||
- **RED:** exit 1, module-not-found before either production file existed.
|
||
- **GREEN/REFACTOR:** exit 0, 6 passed, 0 failed.
|
||
|
||
### Work Unit Evidence
|
||
|
||
| Evidence | Result |
|
||
|---|---|
|
||
| Focused test command and exact result | `NODE_ENV=test npx --no-install tsx --test tests/ocr/client.test.ts` — exit 0; 6 passed, 0 failed. |
|
||
| Runtime harness command/scenario and exact result | `NODE_ENV=test npx --no-install tsx --test --test-name-pattern "runtime retry stub" tests/ocr/client.test.ts` — exit 0; 1 passed, 0 failed. Deterministic stub observed network failure, `503`, then `202`: exactly 3 submission attempts, one unchanged idempotency key, 2,000/4,000 ms backoffs, followed by a strictly validated two-page result. |
|
||
| Rollback boundary | Remove `src/modules/ocr/client.ts`, `src/modules/ocr/artifacts.ts`, and `tests/ocr/client.test.ts`; revert tasks 4.2/4.3 and this Unit 7 progress/history metadata. Units 1–6 remain intact. |
|
||
|
||
### Validation and Boundary
|
||
|
||
- `npm test` passed 51/51; `npm run check`, `npm run build`, tracked whitespace, and no-index whitespace checks for all three new files passed.
|
||
- Test-created `rag-ocr-artifacts-*` and `rag-ocr-sweep-*` directories were removed; no server, external OCR call, or persistent process was started; `ocr-service/.venv` was preserved.
|
||
- Start: Unit 6 OCR rendering and container runtime are complete. End: Unit 7 client retries/integrity and durable artifacts/orphan sweep are implemented and green.
|
||
- Out of scope and untouched: dispatcher, reconciler, ingest branch, app upload/status routes, Unit 8 tests, commits, pushes, PRs, native review, deployment, and restricted paths.
|
||
- The implementation/test slice is 506 authored additions (client 174, artifacts 129, tests 203). This exceeds the 400-line default after one honest cohesive assessment; no code-golf or second slicing pass was attempted. The maintainer explicitly approved `size:exception`, including 585 total native-accounting lines with required metadata.
|
||
- No specification or design deviation.
|
||
|
||
### Approved Size-Exception Corrective Context
|
||
|
||
- The native attempt was reset after failed evidence revision `sha256:dd224c3864adc8380dc7d9fd147cb448f5da8138277f3f346a35d7eeb3aca158`; the failure was budget-only, not a functional defect.
|
||
- Corrective work unit `unit-7-size-exception-validation` uses attempt token `sha256:0109cfd8d3b37a5672c65a37700269b89cb80e4e357a39f68a920779a1715c39` and must settle against the failed revision through the parent.
|
||
- Production and test files remain unchanged for the corrective rerun; only approval/reset evidence and fresh validation metadata may change.
|
||
- Task 4.1 remains pending, while tasks 4.2 and 4.3 remain complete. Unit 8 was not started.
|