feat(ocr): add private service queue

.gitignore

@@ -9,6 +9,9 @@ dist/
 llaves
 backups/
 npm-debug.log*
+ocr-service/.venv/
+ocr-service/*.db
+ocr-service/**/__pycache__/
 
 # Local agent caches and tooling artifacts
 .atl/

docs/HISTORIAL_SESIONES.md

@@ -3,7 +3,7 @@
 **Proyecto:** Workspace de tools IA para empresas  
 **Modulo:** RAG  
 **Ultima actualizacion:** 2026-09-14
-**Ultima modificacion por:** Subagente OCR Unit 4 Detection and Composition
+**Ultima modificacion por:** Subagente Correccion OCR Unit 5
 **Estado:** Activo
 
 ---
@@ -586,3 +586,24 @@ Continuidad operativa y evolutiva del modulo RAG.
 - `src/modules/ocr/{detection,composition}.ts`
 - `tests/ocr/detection.test.ts`
 - `openspec/changes/ocr-ingest-integration/{tasks.md,apply-progress.md}`, `docs/HISTORIAL_SESIONES.md`
+
+---
+
+### 2026-09-14 - Subagente OCR Unit 5 Reintento y Correccion - Servicio OCR HTTP validado
+
+**Agente:** **Subagente OCR Unit 5 Reintento y Correccion**
+**Rol/responsabilidad:** Implementar y corregir exclusivamente Unit 5, tareas 3.1 y 3.2, mediante TDD estricto y entorno virtual local autorizado.
+**Modelo:** openai/gpt-5.6-sol
+**Session IDs OpenCode:** retry `ses_f5fa0eea0ffegi8n6KEhZDWYzU`; correction `ses_f5edabfa9ffeDm6JTyPnX7Bh5M`
+**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
+
+**Trabajo realizado:**
+- Creado el manifiesto Python fijado, el entorno ignorado y retenido `ocr-service/.venv`, las pruebas HTTP RED-first y la API FastAPI con cola SQLite, autenticacion, idempotencia, allowlist, limites y health.
+- Diagnosticado el harness fallido: `&` envio la lista shell previa a un subshell y dejo vacios `KEY`, `REQUEST` y `CONFLICT` en los curls; se reprodujo `400 INVALID_REQUEST` y se corrigio solo el limite de backgrounding, sin cambiar la implementacion.
+
+**Validacion y estado final:**
+- RED valido por ausencia de `app`; GREEN focalizado: 8/8 pruebas aprobadas.
+- Se conserva la evidencia fallida `sha256:4a5c8ed8d8b302f7ff654d4f40fe0f13049bfdd29db4c76576f3847c0a9be15a`; el harness corregido devolvio exactamente `401/202/409` con cuerpos contractuales.
+- `npm test` 45/45, `npm run check`, compileall y espacios correctos; servidor y temporales eliminados, `.venv` retenido; tareas 3.1/3.2 completadas. Revision: `sha256:0d8b2c57bbab967473eaf99a7ca900168554e64254fa80197b3626b4570b58d0`.
+
+**Archivos modificados:** `.gitignore`, `ocr-service/{README.md,requirements.txt,app/,tests/}`, `openspec/changes/ocr-ingest-integration/apply-progress.md`, `docs/HISTORIAL_SESIONES.md`.

ocr-service/README.md

@@ -0,0 +1,23 @@
+# OCR Service Development Environment
+
+Unit 5 uses the repository-local virtual environment `ocr-service/.venv`. It is intentionally retained between runs and ignored by Git. Do not install these dependencies globally.
+
+## Create or recreate
+
+From the repository root:
+
+```bash
+rm -rf ocr-service/.venv
+python3 -m venv ocr-service/.venv
+ocr-service/.venv/bin/python -m pip install -r ocr-service/requirements.txt
+```
+
+Activate it with `source ocr-service/.venv/bin/activate`.
+
+## Test
+
+```bash
+ocr-service/.venv/bin/python -m pytest ocr-service/tests -k auth
+```
+
+PaddleOCR, rendering, container images, and production model readiness belong to Unit 6 and are not installed by this manifest.

ocr-service/app/main.py

@@ -0,0 +1,175 @@
+import hashlib
+import hmac
+import json
+import os
+import sqlite3
+import threading
+import uuid
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Annotated, Any
+
+from fastapi import Depends, FastAPI, File, Form, Header, HTTPException, Response, UploadFile
+
+
+MAX_UPLOAD_BYTES = 50 * 1024 * 1024
+MAX_PAGES = 100
+QUEUE_CAPACITY = 3
+ALLOWED_CONFIG = {
+    "languages": ["es", "en"],
+    "dpi": 200,
+    "engine": "paddleocr",
+    "engineVersion": "3.4.0",
+    "runtimeVersion": "3.2.2",
+    "configVersion": "ocr-v1",
+    "returnLayout": True,
+}
+REQUEST_FIELDS = {"documentSha256", "pages", *ALLOWED_CONFIG}
+
+
+def fail(status: int, code: str, message: str, retryable: bool = False, headers: dict[str, str] | None = None) -> None:
+    raise HTTPException(status, {"code": code, "message": message, "retryable": retryable}, headers)
+
+
+def page_hash(pages: list[int]) -> str:
+    value = json.dumps(pages, separators=(",", ":")).encode()
+    return hashlib.sha256(value).hexdigest()
+
+
+class JobQueue:
+    def __init__(self, path: str | Path):
+        self.connection = sqlite3.connect(str(path), check_same_thread=False)
+        self.connection.row_factory = sqlite3.Row
+        self.lock = threading.Lock()
+        self.connection.execute(
+            "CREATE TABLE IF NOT EXISTS jobs (job_id TEXT PRIMARY KEY, idempotency_key TEXT UNIQUE, "
+            "payload_hash TEXT, document_sha256 TEXT, pages TEXT, status TEXT, created_at TEXT)"
+        )
+        self.connection.commit()
+
+    def depth(self) -> int:
+        row = self.connection.execute("SELECT count(*) AS count FROM jobs WHERE status IN ('queued','running')").fetchone()
+        return int(row["count"])
+
+    def contains(self, key: str) -> bool:
+        return self.connection.execute("SELECT 1 FROM jobs WHERE idempotency_key=?", (key,)).fetchone() is not None
+
+    @staticmethod
+    def ack(row: sqlite3.Row) -> dict[str, Any]:
+        return {
+            "jobId": row["job_id"],
+            "status": "queued",
+            "documentSha256": row["document_sha256"],
+            "requestedPages": json.loads(row["pages"]),
+            "configVersion": "ocr-v1",
+            "createdAt": row["created_at"],
+        }
+
+    def submit(self, key: str, request: dict[str, Any]) -> dict[str, Any]:
+        payload_hash = hashlib.sha256(json.dumps(request, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
+        with self.lock:
+            row = self.connection.execute("SELECT * FROM jobs WHERE idempotency_key=?", (key,)).fetchone()
+            if row:
+                if row["payload_hash"] != payload_hash:
+                    fail(409, "IDEMPOTENCY_CONFLICT", "The idempotency key is already bound to another request")
+                return self.ack(row)
+            if self.depth() >= QUEUE_CAPACITY:
+                fail(429, "QUEUE_FULL", "The OCR queue is full", True, {"Retry-After": "2"})
+            values = (
+                f"ocr_{uuid.uuid4()}", key, payload_hash, request["documentSha256"],
+                json.dumps(request["pages"]), "queued", datetime.now(timezone.utc).isoformat(),
+            )
+            self.connection.execute("INSERT INTO jobs VALUES (?,?,?,?,?,?,?)", values)
+            self.connection.commit()
+            return self.ack(self.connection.execute("SELECT * FROM jobs WHERE job_id=?", (values[0],)).fetchone())
+
+    def status(self, job_id: str) -> dict[str, Any] | None:
+        row = self.connection.execute("SELECT * FROM jobs WHERE job_id=?", (job_id,)).fetchone()
+        if not row:
+            return None
+        return {"jobId": job_id, "status": row["status"], "completedPages": 0,
+                "totalPages": len(json.loads(row["pages"])), "error": None}
+
+    def delete(self, job_id: str) -> None:
+        with self.lock:
+            self.connection.execute("DELETE FROM jobs WHERE job_id=?", (job_id,))
+            self.connection.commit()
+
+
+def create_app(
+    token: str,
+    db_path: str | Path = ":memory:",
+    max_upload_bytes: int = MAX_UPLOAD_BYTES,
+    engine_ready: bool = False,
+) -> FastAPI:
+    application = FastAPI(title="Private OCR Service", docs_url=None, redoc_url=None)
+    queue = JobQueue(db_path)
+
+    def authorize(authorization: Annotated[str | None, Header()] = None) -> None:
+        scheme, _, supplied = (authorization or "").partition(" ")
+        if not token or scheme != "Bearer" or not hmac.compare_digest(supplied, token):
+            fail(401, "UNAUTHORIZED", "Valid bearer authorization is required", headers={"WWW-Authenticate": "Bearer"})
+
+    @application.get("/health/live")
+    def live() -> dict[str, str]:
+        return {"status": "ok"}
+
+    @application.get("/health/ready")
+    def ready(response: Response) -> dict[str, int | bool]:
+        if not engine_ready:
+            response.status_code = 503
+        return {"ready": engine_ready, "queueDepth": queue.depth(), "queueCapacity": QUEUE_CAPACITY, "concurrency": 1}
+
+    @application.post("/v1/jobs", status_code=202, dependencies=[Depends(authorize)])
+    async def create_job(
+        file: Annotated[UploadFile, File()], request: Annotated[str, Form()],
+        idempotency_key: Annotated[str | None, Header(alias="Idempotency-Key")] = None,
+    ) -> dict[str, Any]:
+        content = await file.read(max_upload_bytes + 1)
+        if len(content) > max_upload_bytes:
+            fail(413, "UPLOAD_LIMIT_EXCEEDED", "PDF exceeds the upload limit")
+        try:
+            payload = json.loads(request)
+        except (json.JSONDecodeError, TypeError):
+            fail(400, "INVALID_REQUEST", "Request must be valid JSON")
+        if not isinstance(payload, dict) or set(payload) != REQUEST_FIELDS:
+            fail(400, "INVALID_REQUEST", "Request fields do not match the contract")
+        pages = payload["pages"]
+        if not isinstance(pages, list) or not pages or any(type(page) is not int or page < 1 for page in pages):
+            fail(422, "INVALID_PAGES", "Pages must be positive one-based integers")
+        if len(pages) > MAX_PAGES:
+            fail(413, "PAGE_LIMIT_EXCEEDED", "OCR jobs accept at most 100 pages")
+        if pages != sorted(set(pages)):
+            fail(422, "INVALID_PAGES", "Pages must be unique and ordered")
+        if any(payload[name] != value for name, value in ALLOWED_CONFIG.items()):
+            fail(400, "CONFIG_NOT_ALLOWED", "OCR configuration is not allowlisted")
+        digest = hashlib.sha256(content).hexdigest()
+        if payload["documentSha256"] != digest:
+            fail(422, "INTEGRITY_MISMATCH", "PDF bytes do not match documentSha256")
+        if file.content_type != "application/pdf" or not content.startswith(b"%PDF-") or b"/Encrypt" in content:
+            fail(422, "UNSUPPORTED_PDF", "PDF is corrupt, encrypted, or unsupported")
+        expected_key = f'{digest}:ocr-v1:{page_hash(pages)}'
+        if idempotency_key != expected_key and not queue.contains(idempotency_key or ""):
+            fail(400, "INVALID_IDEMPOTENCY_KEY", "Idempotency-Key does not match request identity")
+        return queue.submit(idempotency_key or "", payload)
+
+    @application.get("/v1/jobs/{job_id}", dependencies=[Depends(authorize)])
+    def get_job(job_id: str) -> dict[str, Any]:
+        status = queue.status(job_id)
+        if status is None:
+            fail(404, "JOB_NOT_FOUND", "OCR job does not exist")
+        return status
+
+    @application.delete("/v1/jobs/{job_id}", status_code=204, dependencies=[Depends(authorize)])
+    def delete_job(job_id: str) -> Response:
+        queue.delete(job_id)
+        return Response(status_code=204)
+
+    return application
+
+
+app = create_app(
+    os.getenv("OCR_INTERNAL_TOKEN", ""),
+    os.getenv("OCR_JOBS_DB", ":memory:"),
+    engine_ready=os.getenv("OCR_ENGINE_READY") == "1",
+)

ocr-service/requirements.txt

@@ -0,0 +1,5 @@
+fastapi==0.116.1
+httpx==0.28.1
+pytest==8.4.1
+python-multipart==0.0.20
+uvicorn==0.35.0

ocr-service/tests/test_api.py

@@ -0,0 +1,133 @@
+import hashlib
+import json
+import sys
+from pathlib import Path
+
+import pytest
+from fastapi.testclient import TestClient
+
+sys.path.insert(0, str(Path(__file__).parents[1]))
+
+from app.main import create_app
+
+
+TOKEN = "unit-5-test-token"
+PDF = b"%PDF-1.4\nunit five\n%%EOF"
+
+
+def request_for(pdf: bytes = PDF, pages: list[int] | None = None) -> dict:
+    return {
+        "documentSha256": hashlib.sha256(pdf).hexdigest(),
+        "pages": pages or [1, 2],
+        "languages": ["es", "en"],
+        "dpi": 200,
+        "engine": "paddleocr",
+        "engineVersion": "3.4.0",
+        "runtimeVersion": "3.2.2",
+        "configVersion": "ocr-v1",
+        "returnLayout": True,
+    }
+
+
+def key_for(request: dict) -> str:
+    pages = json.dumps(request["pages"], separators=(",", ":")).encode()
+    return f'{request["documentSha256"]}:ocr-v1:{hashlib.sha256(pages).hexdigest()}'
+
+
+def submit(client: TestClient, request: dict, pdf: bytes = PDF, key: str | None = None):
+    return client.post(
+        "/v1/jobs",
+        headers={"Authorization": f"Bearer {TOKEN}", "Idempotency-Key": key or key_for(request)},
+        files={
+            "file": ("input.pdf", pdf, "application/pdf"),
+            "request": (None, json.dumps(request), "application/json"),
+        },
+    )
+
+
+@pytest.fixture
+def client(tmp_path: Path) -> TestClient:
+    return TestClient(create_app(TOKEN, tmp_path / "jobs.db", engine_ready=True))
+
+
+def test_auth_rejects_missing_and_wrong_bearer_and_health_exposes_no_secret(client: TestClient):
+    live = client.get("/health/live")
+    ready = client.get("/health/ready")
+    assert live.json() == {"status": "ok"}
+    assert ready.json() == {"ready": True, "queueDepth": 0, "queueCapacity": 3, "concurrency": 1}
+    for authorization in (None, "Bearer wrong-token"):
+        headers = {"Idempotency-Key": key_for(request_for())}
+        if authorization:
+            headers["Authorization"] = authorization
+        response = client.post(
+            "/v1/jobs",
+            headers=headers,
+            files={"file": ("input.pdf", PDF, "application/pdf"), "request": (None, json.dumps(request_for()))},
+        )
+        assert response.status_code == 401
+        assert response.json()["detail"]["retryable"] is False
+        assert TOKEN not in response.text
+
+    unavailable = TestClient(create_app(TOKEN, ":memory:", engine_ready=False)).get("/health/ready")
+    assert unavailable.status_code == 503
+    assert unavailable.json()["ready"] is False
+
+
+def test_auth_submission_is_idempotent_and_conflicting_payload_is_terminal(client: TestClient):
+    request = request_for()
+    first = submit(client, request)
+    repeated = submit(client, request)
+    conflict = request_for(pages=[1])
+    conflicting = submit(client, conflict, key=key_for(request))
+    assert first.status_code == repeated.status_code == 202
+    assert first.json() == repeated.json()
+    assert first.json()["requestedPages"] == [1, 2]
+    assert conflicting.status_code == 409
+    assert conflicting.json()["detail"] == {
+        "code": "IDEMPOTENCY_CONFLICT",
+        "message": "The idempotency key is already bound to another request",
+        "retryable": False,
+    }
+
+
+@pytest.mark.parametrize(
+    ("field", "value"),
+    [("languages", ["en"]), ("dpi", 300), ("engine", "tesseract"), ("returnLayout", False)],
+)
+def test_auth_allowlist_rejects_client_selected_configuration(client: TestClient, field: str, value: object):
+    request = request_for()
+    request[field] = value
+    response = submit(client, request, key="different-key")
+    assert response.status_code == 400
+    assert response.json()["detail"]["retryable"] is False
+
+
+def test_auth_limits_corrupt_pdf_and_integrity_mismatch_are_terminal(tmp_path: Path):
+    client = TestClient(create_app(TOKEN, tmp_path / "limited.db", max_upload_bytes=8, engine_ready=True))
+    regular = TestClient(create_app(TOKEN, tmp_path / "regular.db", engine_ready=True))
+    oversized = submit(client, request_for(PDF), PDF)
+    too_many = submit(regular, request_for(pages=list(range(1, 102))))
+    corrupt = submit(regular, request_for(b"not-pdf"), b"not-pdf")
+    wrong_hash = request_for()
+    wrong_hash["documentSha256"] = "0" * 64
+    mismatch = submit(regular, wrong_hash, key=key_for(wrong_hash))
+    assert oversized.status_code == too_many.status_code == 413
+    assert corrupt.status_code == mismatch.status_code == 422
+    assert mismatch.json()["detail"]["code"] == "INTEGRITY_MISMATCH"
+    for response in (oversized, too_many, corrupt, mismatch):
+        assert response.json()["detail"]["retryable"] is False
+        assert "Retry-After" not in response.headers
+
+
+def test_auth_queue_pressure_is_retryable_and_status_and_delete_are_authenticated(client: TestClient):
+    accepted = [submit(client, request_for(pdf), pdf) for pdf in (PDF, PDF + b"1", PDF + b"2")]
+    pressure = submit(client, request_for(PDF + b"3"), PDF + b"3")
+    assert [response.status_code for response in accepted] == [202, 202, 202]
+    assert pressure.status_code == 429
+    assert pressure.json()["detail"]["retryable"] is True
+    assert pressure.headers["Retry-After"] == "2"
+    job_id = accepted[0].json()["jobId"]
+    status = client.get(f"/v1/jobs/{job_id}", headers={"Authorization": f"Bearer {TOKEN}"})
+    assert status.json() == {"jobId": job_id, "status": "queued", "completedPages": 0, "totalPages": 2, "error": None}
+    assert client.delete(f"/v1/jobs/{job_id}", headers={"Authorization": f"Bearer {TOKEN}"}).status_code == 204
+    assert client.delete(f"/v1/jobs/{job_id}", headers={"Authorization": f"Bearer {TOKEN}"}).status_code == 204

openspec/changes/ocr-ingest-integration/apply-progress.md

@@ -3,9 +3,9 @@
 ## Current State
 
 - **Mode:** Strict TDD
-- **Delivery:** Feature-branch-chain, Units 1–4 complete; maintainer-approved `size:exception` for Unit 2 only
+- **Delivery:** Feature-branch-chain, Units 1–5 complete; maintainer-approved `size:exception` for Unit 2 only
-- **Completed tasks:** 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 2.5
+- **Completed tasks:** 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 2.3, 2.4, 2.5, 3.1, 3.2
-- **Overall task progress:** 9/30 complete
+- **Overall task progress:** 11/30 complete
 
 ## Unit 1: Migration
 
@@ -166,3 +166,28 @@ None — the migration follows the proposal, specifications, design, and closed
 - Start: Unit 3 page extraction and parser threat routing are complete. End: detection, blank/quality gates, deterministic composition, hashes, and risk tokens are green.
 - Out of scope: OCR service, client, dispatcher, ingest routing, review APIs, commits, pushes, PRs, deployment, and restricted paths.
 - No specification or design deviation.
+
+## Unit 5: OCR Service
+
+### Implementation and Correction Summary
+
+- Added pinned Unit 5 FastAPI test dependencies and retained the ignored `ocr-service/.venv` environment.
+- Added RED-first HTTP tests and a SQLite-backed private API for bearer auth, idempotency, allowlisting, upload/page limits, queue pressure, status, deletion, and health.
+- Preserved failed evidence `sha256:4a5c8ed8d8b302f7ff654d4f40fe0f13049bfdd29db4c76576f3847c0a9be15a`: its harness returned `401/400/400`; inspection showed `&` backgrounded the preceding shell AND-list, so request variables existed only in the child shell and authenticated curls sent an empty `request` form, reproduced as `400 INVALID_REQUEST`.
+- Corrected only the harness command boundary; the 381-line candidate implementation remained unchanged before progress persistence.
+
+### TDD Cycle Evidence
+
+| Task | Test File | Layer | Safety Net | RED | GREEN | TRIANGULATE | REFACTOR |
+|---|---|---|---|---|---|---|---|
+| 3.1 | `ocr-service/tests/test_api.py` | HTTP integration | N/A (new service) | Collection failed with `ModuleNotFoundError: No module named 'app'`; failed localhost runtime evidence then supplied correction RED. | 8/8 focused tests and corrected curl `401/202/409` passed. | Auth variants, idempotency/conflict, allowlist, limits, pressure, integrity, and real multipart requests exercised distinct paths. | No production refactor: the defect was shell variable scope in the harness. |
+| 3.2 | `ocr-service/tests/test_api.py` | HTTP integration/SQLite | N/A (new service) | Same missing-service RED covered the API; failed harness blocked completion. | 8/8 focused tests and real Uvicorn harness passed. | Ready/not-ready health, queue capacity, status, deletion, and process cleanup exercised distinct paths. | Pure page hashing and centralized terminal errors retained; no correction needed. |
+
+### Work Unit Evidence
+
+- Focused: `ocr-service/.venv/bin/python -m pytest ocr-service/tests -k auth` — exit 0; 8 passed, 0 failed (one dependency deprecation warning).
+- Runtime: local Uvicorn plus multipart curl — unauthorized `401` with `UNAUTHORIZED`, accepted `202` with queued identity, conflicting same key `409` with `IDEMPOTENCY_CONFLICT`.
+- Gates: `npm test` 45/45, `npm run check`, Python `compileall`, tracked and untracked whitespace checks — all exit 0/equivalent clean.
+- Cleanup: Uvicorn PID 268425 terminated and absent; temporary PDF, SQLite DB, and log removed; `.venv` retained.
+- Rollback: remove `.gitignore` Unit 5 entries and `ocr-service/`, then revert tasks 3.1/3.2 and this Unit 5 progress/history block; Units 1–4 remain intact.
+- Fresh evidence revision: `sha256:0d8b2c57bbab967473eaf99a7ca900168554e64254fa80197b3626b4570b58d0`; corrected final authored change count: 397 additions plus deletions, within 400.

openspec/changes/ocr-ingest-integration/tasks.md

@@ -44,8 +44,8 @@ Tracker feature/ocr-ingest-integration is draft/no-merge and sole main target. P
 
 ## 3 OCR Service (Units 5–6; P5/P6)
 
-- [ ] 3.1 RED HTTP: bearer 401; idempotency/conflict 409; allowlist; limits 413/422; pressure 429; no-retry; integrity mismatch
+- [x] 3.1 RED HTTP: bearer 401; idempotency/conflict 409; allowlist; limits 413/422; pressure 429; no-retry; integrity mismatch
-- [ ] 3.2 GREEN: `ocr-service/` API, queue, allowlist, limits, health
+- [x] 3.2 GREEN: `ocr-service/` API, queue, allowlist, limits, health
 - [ ] 3.3 GREEN: PaddleOCR 3.4.0, 200 DPI, schema, stub
 - [ ] 3.4 GREEN: `ocr-service/Dockerfile` pinned wheels/models/resources