feat(rag): add knowledge lifecycle management

This commit is contained in:
Paco POR-CORREO 2026-09-11 19:31:45 +02:00
parent e5f739eb6c
commit 551cfe8816
33 changed files with 5308 additions and 261 deletions

View file

@ -11,3 +11,10 @@ ANSWER_PROVIDER=openrouter
ANSWER_MODEL=openai/gpt-4.1-mini
ANSWER_BASE_URL=https://openrouter.ai/api/v1
ANSWER_API_KEY=
POSTGRES_URL=postgresql://rag:rag@postgres:5432/rag
POSTGRES_SSL=false
LIFECYCLE_ADMIN_TOKEN=
KNOWLEDGE_LIFECYCLE_ENFORCED=false
INGEST_WRITES_ENABLED=true
LIFECYCLE_RECONCILE_INTERVAL_MS=300000
LIFECYCLE_INDEXING_STALE_TIMEOUT_MS=1800000

View file

@ -3,6 +3,7 @@ WORKDIR /app
COPY package.json package-lock.json tsconfig.json ./
RUN npm ci
COPY src ./src
COPY migrations ./migrations
COPY public ./public
RUN npm run build
@ -12,6 +13,7 @@ ENV NODE_ENV=production
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist
COPY --from=build /app/migrations ./migrations
COPY public ./public
EXPOSE 3000
CMD ["node", "dist/server.js"]
CMD ["sh", "-c", "node dist/modules/catalog/migrations.js && node dist/server.js"]

View file

@ -2,9 +2,9 @@
**Proyecto:** Workspace de tools IA para empresas
**Modulo:** RAG
**Ultima actualizacion:** 2026-09-08
**Ultima modificacion por:** Agente RAG 2
**Estado:** Operativa y validada en produccion
**Ultima actualizacion:** 2026-09-11
**Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
**Estado:** Punto 2 implementado localmente; pendiente de migracion y validacion en produccion
Esta guia explica como descubrir y consumir la API. El contrato tecnico completo y canonico se publica en OpenAPI 3.1.1; este documento prioriza el camino rapido, las decisiones de uso y los ejemplos habituales.
@ -24,7 +24,7 @@ https://rag.por-correo.com
`GET /help` indica expresamente que debe consultarse `/openapi.json` para obtener los parametros, cuerpos, respuestas, errores y ejemplos completos de cada endpoint.
La API no requiere autenticacion actualmente. No debe asumirse que esta situacion es definitiva: la autenticacion y la proteccion de operaciones sensibles forman parte del pendiente 5.
La API general sigue sin autenticacion completa. Las operaciones administrativas del ciclo de vida usan una proteccion acotada con `Authorization: Bearer <LIFECYCLE_ADMIN_TOKEN>`; esto no sustituye la autenticacion general pendiente.
## Elegir la operacion correcta
@ -32,18 +32,20 @@ La API no requiere autenticacion actualmente. No debe asumirse que esta situacio
|---|---|
| Comprobar servicio y capacidades | `GET /health` |
| Ver scopes ingeridos | `GET /sources` |
| Ver versiones de una fuente | `GET /sources/{sourceId}/versions` |
| Activar, rollback o purgar versiones | `/sources/*` con token administrativo |
| Recuperar contexto para otro agente | `POST /retrieve` |
| Obtener una respuesta final con citas | `POST /answer` |
| Comparar una respuesta sin RAG | `POST /answer/direct` |
| Conversar con bootstrap e historial | `POST /chat` |
| Ingerir una ruta accesible por el servidor | `POST /ingest` |
| Subir un archivo o carpeta ZIP | `POST /ingest/upload` |
| Eliminar una fuente antes de reingestar | `POST /cleanup` |
| Eliminar una fuente antes de reingestar | `POST /cleanup` legacy, o purga de version no activa |
| Consultar o actualizar evaluaciones | `/logs/*` |
## Catalogo completo
La API queda formada por 16 operaciones tras incorporar el descubrimiento:
La API queda formada por las operaciones de descubrimiento, ingesta, retrieval, evaluacion y ciclo de vida:
| Metodo | Ruta | Descripcion |
|---|---|---|
@ -52,6 +54,13 @@ La API queda formada por 16 operaciones tras incorporar el descubrimiento:
| `GET` | `/playground` | Interfaz web de prueba |
| `GET` | `/health` | Estado, proveedores, parsers y chunking |
| `GET` | `/sources` | Scopes disponibles en Qdrant |
| `GET` | `/sources/{sourceId}` | Fuente registrada en el catalogo |
| `GET` | `/sources/{sourceId}/versions` | Versiones de una fuente |
| `GET` | `/sources/{sourceId}/versions/{versionId}` | Detalle de una version |
| `POST` | `/sources/{sourceId}/versions/{versionId}/activate` | Activacion administrativa con precondicion |
| `POST` | `/sources/{sourceId}/rollback` | Rollback administrativo sin recalcular embeddings |
| `POST` | `/sources/{sourceId}/versions/{versionId}/retry` | Reintento explicito de version fallida o huerfana con fuente durable |
| `DELETE` | `/sources/{sourceId}/versions/{versionId}` | Purga administrativa de una version no activa |
| `GET` | `/models/answer` | Modelo por defecto y modelos seleccionables |
| `POST` | `/ingest` | Ingesta desde una ruta del servidor |
| `POST` | `/ingest/upload` | Upload `multipart/form-data` |
@ -102,6 +111,15 @@ Los errores JSON siguen esta forma:
Actualmente la mayoria de fallos de validacion o de proveedores se devuelven como `500`. `POST /cleanup` y `POST /ingest/upload` tambien pueden devolver `400`, y `PATCH /logs/{id}` devuelve `404` si no encuentra el log. Mejorar la validacion queda fuera de este bloque para no cambiar silenciosamente el comportamiento existente.
Con `KNOWLEDGE_LIFECYCLE_ENFORCED=true`, la API devuelve errores mas especificos para el catalogo:
| Codigo HTTP | Caso |
|---|---|
| `400` | Falta `expectedActiveVersionId` cuando `activate=true` |
| `409` | Cambio concurrente de version activa, version en curso o intento de purgar una activa |
| `422` | Fuente vacia, sin documentos soportados o sin contenido util |
| `503` | PostgreSQL, Qdrant, embeddings o version activa inconsistente |
## Recuperar contexto
`POST /retrieve` es la operacion recomendada cuando otro agente o flujo generara la respuesta final.
@ -248,12 +266,22 @@ Respuesta:
"sourceType": "folder",
"sourceRef": "/data/project-docs",
"mode": "mechanical",
"tags": ["project", "documentation"]
"tags": ["project", "documentation"],
"activate": true,
"expectedActiveVersionId": null
}
```
`readPath` es un campo interno para separar la ruta fisica de lectura de la referencia logica. Los consumidores normales no deben enviarlo.
Con ciclo de vida activo:
- `activate` por defecto es `true`.
- si `activate=true`, `expectedActiveVersionId` es obligatorio: `null` para primera activacion y UUID para reemplazar una version existente.
- una ingesta identica a la version activa responde `noOp: true` y no escribe nuevos puntos.
- `activate=false` deja la version en `ready` para activarla despues con una operacion administrativa.
- la respuesta exitosa cambia a `201 Created` e incluye `sourceId`, `versionId`, `versionNumber`, `state`, `previousVersionId`, `activated` y `noOp`.
### Upload
`POST /ingest/upload` usa `multipart/form-data`:
@ -265,6 +293,8 @@ Respuesta:
| `mode` | No | `mechanical` o `interactive` |
| `tags` | No | Lista separada por comas |
| `isZipFolder` | No | Texto `true` o `false` |
| `activate` | No | Texto `true` o `false` |
| `expectedActiveVersionId` | Si activa | UUID o texto `null` |
```bash
curl -sS -X POST "https://rag.por-correo.com/ingest/upload" \
@ -274,11 +304,52 @@ curl -sS -X POST "https://rag.por-correo.com/ingest/upload" \
-F "tags=customer-a,support"
```
Ambas variantes responden `202` con el origen aceptado, archivos descubiertos, documentos procesados, chunks almacenados y coleccion. El upload añade `uploadedResource`.
Con enforcement desactivado, ambas variantes conservan `202`. Con enforcement activado, responden `201` y usan el catalogo PostgreSQL como autoridad de version activa. El upload añade `uploadedResource`.
## Ciclo de vida de fuentes
El catalogo PostgreSQL es la autoridad para saber que fuentes existen, que versiones tienen y cual es la version activa. Qdrant conserva los puntos fisicos versionados, pero retrieval solo consulta versiones activas resueltas desde PostgreSQL.
### Listado y detalle
```text
GET /sources
GET /sources/{sourceId}
GET /sources/{sourceId}/versions
GET /sources/{sourceId}/versions/{versionId}
```
`GET /sources` mantiene los campos usados por el playground (`sourceId`, `sourceRef`, `chunkModes`, `tags`) y añade `activeVersionId`, `activeVersionNumber`, `state`, `needsReingest` y `updatedAt`.
### Activacion y rollback
```bash
curl -sS -X POST "https://rag.por-correo.com/sources/src:default:folder:docs/versions/<version-id>/activate" \
-H "Authorization: Bearer $LIFECYCLE_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"expectedActiveVersionId":"<active-version-id>"}'
```
Rollback usa la misma activacion atomica del puntero PostgreSQL:
```json
{
"targetVersionId": "uuid-version-anterior",
"expectedActiveVersionId": "uuid-version-activa-actual"
}
```
### Purga
`DELETE /sources/{sourceId}/versions/{versionId}` exige token administrativo y solo acepta versiones no activas. La purga marca la version como `purging`, borra puntos Qdrant con `wait=true`, verifica que no quedan puntos y finalmente deja la version como `purged`.
### Reintento explicito
`POST /sources/{sourceId}/versions/{versionId}/retry` acepta versiones `failed`, `pending` o `indexing`. Si recibe una fuente durable (`sourceType`, `sourceRef` y opcionalmente `readPath`) cierra versiones huerfanas como fallidas y crea una version nueva mediante `/ingest`. Si no recibe fuente durable, devuelve `202` con instrucciones: los uploads temporales no se reintentan en sitio porque el original ya no esta garantizado.
## Limpieza controlada
`POST /cleanup` elimina los chunks que coinciden con un scope:
`POST /cleanup` elimina los chunks que coinciden con un scope solo en modo legacy y con `INGEST_WRITES_ENABLED=true`:
```json
{

File diff suppressed because it is too large Load diff

View file

@ -2,9 +2,9 @@
**Proyecto:** Workspace de tools IA para empresas
**Modulo:** RAG
**Ultima actualizacion:** 2026-04-05
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales
**Estado:** V1 desplegada en VPS2
**Ultima actualizacion:** 2026-09-11
**Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
**Estado:** V1 desplegada en VPS2; ciclo de vida implementado localmente pendiente de migracion productiva
---
@ -200,6 +200,12 @@ Dato importante para `RAG`:
- `ANSWER_MODEL=openai/gpt-4.1-mini`
- `ANSWER_BASE_URL=https://openrouter.ai/api/v1`
- `ANSWER_API_KEY=<valor real>`
- `POSTGRES_URL=<conexion interna a PostgreSQL>`
- `POSTGRES_SSL=false` salvo que la conexion lo exija
- `LIFECYCLE_ADMIN_TOKEN=<secreto administrativo>`
- `KNOWLEDGE_LIFECYCLE_ENFORCED=false` durante despliegue inicial y migracion
- `INGEST_WRITES_ENABLED=false` durante la ventana de migracion legacy
- `LIFECYCLE_RECONCILE_INTERVAL_MS=300000`
### Variables de entorno con las que quedo operativo en VPS2
@ -219,6 +225,15 @@ Valores funcionales confirmados en EasyPanel:
- `ANSWER_BASE_URL=https://openrouter.ai/api/v1`
- `ANSWER_API_KEY=<secreto>`
Variables pendientes de completar para activar el punto 2 en produccion:
- `POSTGRES_URL=<secreto>`
- `POSTGRES_SSL=false`
- `LIFECYCLE_ADMIN_TOKEN=<secreto>`
- `KNOWLEDGE_LIFECYCLE_ENFORCED=false` hasta completar la migracion legacy
- `INGEST_WRITES_ENABLED=true` en operacion normal; `false` durante migracion
- `LIFECYCLE_RECONCILE_INTERVAL_MS=300000`
Para respaldo operativo local fuera de EasyPanel se deja un archivo ignorado por Git:
- `RAG/.env.easypanel.local`
@ -228,7 +243,27 @@ Ese archivo no debe versionarse porque contiene secretos reales.
### Volumen persistente
- no es obligatorio para la app RAG en esta fase
- el estado persistente principal vive en `qdrant`
- el estado persistente vectorial vive en `qdrant`
- desde el punto 2, el catalogo de fuentes y versiones vive en PostgreSQL
### Arranque con migraciones
La imagen arranca ejecutando primero:
```bash
node dist/modules/catalog/migrations.js && node dist/server.js
```
Si `POSTGRES_URL` no esta configurada, el migrador informa que omite migraciones. Si PostgreSQL esta configurado y una migracion falla o cambia su checksum, el servicio no debe completar el arranque.
### Secuencia operativa del punto 2
1. Desplegar el codigo con `KNOWLEDGE_LIFECYCLE_ENFORCED=false`.
2. Bloquear escrituras con `INGEST_WRITES_ENABLED=false` durante la migracion.
3. Ejecutar `npm run migrate:legacy:lifecycle` en modo dry-run y revisar colisiones.
4. Tomar snapshot Qdrant antes de cualquier migracion destructiva.
5. Catalogar el corpus legacy y verificar recuentos PostgreSQL/Qdrant. Si el proceso cae, reanudar con `npm run migrate:legacy:lifecycle -- --apply --snapshot-ref=<snapshot> --batch-id=<batch-id>`.
6. Activar `KNOWLEDGE_LIFECYCLE_ENFORCED=true` solo despues de validar retrieval y playground.
---

View file

@ -93,3 +93,123 @@ Dar continuidad al RAG en `RAG/` a partir del estado actual documentado.
- Pendiente 1 completado y validado definitivamente en `https://rag.por-correo.com`.
- La API dispone de descubrimiento rapido y contrato OpenAPI para sus 16 operaciones.
- Siguiente prioridad: ciclo de vida del conocimiento.
---
### 2026-09-08 - Agente RAG 2
**Modelo:** gpt-5.6-sol
**Session ID OpenCode:** `ses_29bdbd003ffeLrLjUlFgnp08Y7`
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
**Subagentes de analisis:**
- `Subagente Extraccion de versiones PaddleOCR`: contraste de versiones PyPI compatibles para el benchmark CPU.
- `Subagente Validacion del ciclo de vida`: revision adversarial del catalogo PostgreSQL y versionado Qdrant.
- `Subagente Validacion OCR`: revision adversarial de deteccion, servicio privado, quality gates y revision humana.
- `Subagente Auditoria de claridad del contrato`: detecto contradicciones de hashes, idempotencia, migracion y revision.
- `Subagente Revalidacion de ejecutabilidad`: detecto bloqueos restantes en metadata, concurrencia y artefactos.
- `Subagente Revalidacion de resiliencia`: reviso carreras, barreras de migracion y recuperacion de jobs.
- `Subagente Revision final del contrato`: verifico y cerro la secuencia segura de rollback legacy.
**Trabajo realizado:**
- Auditoria del pipeline actual de ingesta, IDs, Qdrant, parser PDF, API e infraestructura EasyPanel.
- Confirmacion de PostgreSQL 17 como catalogo transaccional y Qdrant como almacenamiento vectorial versionado.
- Benchmark aislado en VPS2 con el PDF mixto real de FacturaTech.
- Descarte de Tesseract como motor unico por omitir identificadores criticos.
- Seleccion de PaddleOCR como motor principal y deteccion de errores alfanumericos de alta confianza que obligan a revision humana.
- Creacion de `docs/CONTRATO_CICLO_VIDA_Y_OCR.md` con esquema, estados, APIs, migracion, OCR, tareas, pruebas, despliegue y rollback.
- Enlace del contrato desde el backlog canonico.
- Correccion del contrato mediante revisiones adversariales hasta eliminar sus bloqueos criticos conocidos.
- Limpieza de los artefactos e imagenes temporales usados en el benchmark local y de VPS2.
**Estado final:**
- Diseño de los pendientes 2 y 3 cerrado y listo para implementacion secuencial.
- El punto 2 debe completarse y validarse en produccion antes de iniciar el punto 3.
- No se modifico codigo ni produccion durante esta fase de diseño.
**Archivos modificados:**
- `RAG/docs/CONTRATO_CICLO_VIDA_Y_OCR.md`
- `RAG/docs/PENDIENTES_RAG.md`
- `RAG/docs/HISTORIAL_SESIONES.md`
- `docs/INDICE_DOCUMENTACION.md`
- `docs/HISTORIAL_SESIONES.md`
---
### 2026-09-11 - Subagente Implementacion ciclo de vida del conocimiento
**Modelo:** openai/gpt-5.5
**Session ID OpenCode:** `ses_f6fe0ad30ffeiottYsATsRemCo`
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
**Rol asumido:**
Ejecutar la fase apply del punto 2 del RAG y corregir los bloqueadores finales detectados antes de commit, push o despliegue.
**Trabajo realizado:**
- Ajuste del migrador legacy para permitir reanudacion por `--batch-id`, reutilizando versiones ya creadas y completando activacion tras actualizaciones de payload ya aplicadas.
- Endurecimiento de rollback legacy con lock global de migracion, comprobacion de snapshot y operaciones serializadas sobre fuentes/versiones.
- Reconciliador protegido con advisory lock no bloqueante, timeout configurable basado en `indexing_started_at` y validacion de coleccion/dimensiones.
- Protecciones adicionales: `POST /cleanup` respeta `INGEST_WRITES_ENABLED`, `markPurged` valida `rowCount`, rollback usa locks y OpenAPI refleja `503` en health con `ok` booleano.
- Script `migrate:legacy:lifecycle` ajustado para ejecutarse desde `dist/` con Node tras build.
- Añadidos tests unitarios para reanudacion legacy, activacion idempotente, cleanup con escrituras deshabilitadas, reconciliador, mismatch de coleccion y `markPurged`.
**Validacion local:**
- `npm run check` correcto.
- `npm test` correcto: 18 tests pasan.
- `npm run build` correcto.
- `git diff --check` correcto.
**Estado final:**
- Punto 2 implementado y validado localmente.
- No se hizo commit, push, deploy ni lectura de `.env`.
- Sigue pendiente la verificacion independiente y la validacion/migracion en produccion antes de marcar el punto 2 como cerrado.
**Archivos modificados en esta correccion:**
- `.env.example`
- `package.json`
- `package-lock.json`
- `src/config/env.ts`
- `src/modules/catalog/client.ts`
- `src/modules/catalog/repository.ts`
- `src/modules/catalog/reconciler.ts`
- `src/modules/vectorstore/client.ts`
- `src/modules/retrieve/service.ts`
- `src/modules/ingest/service.ts`
- `src/scripts/migrate-legacy-lifecycle.ts`
- `src/api/openapi.ts`
- `tests/lifecycle-services.test.ts`
- `docs/API_RAG.md`
- `docs/DESPLIEGUE_EASYPANEL.md`
- `docs/SISTEMA_RAG_BASE.md`
- `RAG/docs/HISTORIAL_SESIONES.md`
---
### 2026-09-11 (reanudacion final) - Subagente Implementacion ciclo de vida del conocimiento
**Modelo:** openai/gpt-5.5
**Session ID OpenCode:** `ses_f6fe0ad30ffeiottYsATsRemCo`
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
**Rol asumido:**
Completar los 4 bloqueos del sdd-verify y cerrar la validacion local del punto 2.
**Trabajo realizado:**
- Reconciliador: recupera versiones huerfanas `pending/indexing` a `ready` contando puntos Qdrant y validando documentos, coleccion y dimensiones; marca `failed` solo si quedan incompletas; try-locks evitan matar ingestas vivas.
- Ingesta: rechaza embeddings con dimensiones distintas de las esperadas antes del upsert y antes de alcanzar `ready`.
- Rollback legacy: transaccion diferida del lote, confirmacion de cero punteros activos, restauracion de snapshot con verificacion de recuentos y limpieza posterior del catalogo.
- OpenAPI: `/sources` documenta `503`.
- Retrieval: locks compartidos adquiridos antes de validar recuento/dimensiones y mantenidos durante toda la consulta Qdrant.
- Interrupcion por limite de uso a mitad de la validacion; reanudada tras cambio de cuenta y completada hasta el final.
**Validacion local:**
- `npm run check` correcto.
- `npm test` correcto: 25/25 tests pasan.
- `npm run build` correcto.
- `git diff --check` correcto.
- Verificacion independiente (sdd-verify): PASS local.
**Estado final:**
- Implementacion local del punto 2 verificada de forma independiente: PASS local.
- No se hizo commit, push, deploy ni lectura de `.env`.
- Pendiente: commit/push, deploy con enforcement desactivado, migracion legacy y validacion en produccion antes de cerrar el punto 2.

View file

@ -2,9 +2,9 @@
**Proyecto:** Workspace de tools IA para empresas
**Modulo:** RAG
**Ultima actualizacion:** 2026-04-02
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales
**Estado:** En definicion
**Ultima actualizacion:** 2026-09-11
**Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
**Estado:** Punto 2 implementado localmente; pendiente de migracion y validacion en produccion
---
@ -531,3 +531,48 @@ El enriquecimiento ligero debe ser:
2. Como construiremos exactamente `source_id`, `document_id` y `chunk_id`.
3. Que nombre tendran las fuentes logicas por defecto para documentos sueltos.
4. Como distinguiremos entre fuente nueva, fuente actualizada y fuente obsoleta.
---
## Ciclo de vida del conocimiento implementado localmente
El punto 2 introduce un catalogo PostgreSQL como autoridad de fuentes y versiones. Qdrant deja de decidir que conocimiento esta activo: cada punto fisico queda asociado a `source_version_id`, y retrieval filtra por las versiones activas resueltas desde PostgreSQL cuando `KNOWLEDGE_LIFECYCLE_ENFORCED=true`.
### Flujo de ingesta versionada
1. Resolver `source_id` y registrar un intento en `rag_ingestion_attempts`.
2. Descubrir documentos soportados y calcular hashes de originales.
3. Normalizar contenido y calcular hashes de contenido, fingerprint de procesamiento y hash de metadatos.
4. Detectar si ya existe una version reutilizable.
5. Crear una version `pending`, pasarla a `indexing`, escribir puntos Qdrant versionados y verificar recuentos.
6. Pasar la version a `ready`.
7. Si `activate=true`, activar con precondicion `expectedActiveVersionId`.
Una escritura parcial en Qdrant no queda visible para retrieval hasta que PostgreSQL mueve atomica y correctamente el puntero `active_version_id`.
### Nuevos campos de ingesta
```json
{
"activate": true,
"expectedActiveVersionId": null
}
```
- `activate=true` exige `expectedActiveVersionId`.
- Para primera activacion se envia `null`.
- Para reemplazo se envia el UUID de la version activa que el cliente espera reemplazar.
- `activate=false` permite dejar una version preparada en estado `ready`.
### Identidad y hashes
- `document_key` de uploads se deriva de `source_ref`, no de rutas temporales.
- `original_hash` usa SHA-256 de bytes originales.
- `content_hash` usa SHA-256 del contenido normalizado.
- `source_content_hash` y `original_manifest_hash` usan pares ordenados `document_key + NUL + hash`.
- `processing_fingerprint` cambia cuando cambian parser, normalizacion, chunking, proveedor, modelo o dimensiones de embeddings.
- El ID fisico de Qdrant usa UUIDv5 con `source_version_id + chunk_id`.
### Compatibilidad temporal
Con `KNOWLEDGE_LIFECYCLE_ENFORCED=false`, el camino legacy se conserva para permitir despliegue, dry-run y migracion controlada. La validacion final del punto 2 exige activar enforcement despues de catalogar el corpus legacy.

View file

@ -17,6 +17,8 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
## 2. Ciclo de vida del conocimiento
**Diseño:** Contrato de implementacion cerrado en [`CONTRATO_CICLO_VIDA_Y_OCR.md`](./CONTRATO_CICLO_VIDA_Y_OCR.md). Pendiente de implementar y validar en produccion antes de iniciar el punto 3.
- Crear un catalogo de fuentes y versiones ingeridas.
- Saber que documento esta vigente, obsoleto, reemplazado o pendiente de reingesta.
- Evitar duplicados y permitir actualizaciones incrementales controladas.
@ -25,6 +27,8 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
## 3. OCR integrado en la ingesta
**Diseño:** Contrato de implementacion cerrado en [`CONTRATO_CICLO_VIDA_Y_OCR.md`](./CONTRATO_CICLO_VIDA_Y_OCR.md). Su ejecucion queda bloqueada hasta completar el punto 2.
- Detectar PDFs con capturas, imagenes o una capa textual insuficiente.
- Ejecutar OCR automaticamente o bloquear la ingesta para revision cuando no pueda garantizarse la cobertura.
- Generar un documento intermedio auditable cuando sea necesario.
@ -46,6 +50,7 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
- Aplicar rate limiting y limites de tamano o tipo de archivo.
- Revisar CORS, exposicion publica, registros sensibles y permisos por operacion.
- Sustituir la configuracion sin credenciales de n8n cuando exista el mecanismo de autenticacion definitivo.
- Actualizar Express/`qs` para cerrar los avisos moderados pendientes de `npm audit`.
## 6. Pruebas automatizadas

View file

@ -2,9 +2,9 @@
**Proyecto:** Workspace de tools IA para empresas
**Modulo:** RAG
**Ultima actualizacion:** 2026-04-05
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales
**Estado:** V1 operativa desplegada
**Ultima actualizacion:** 2026-09-11
**Ultima modificacion por:** Agente RAG 2
**Estado:** V1 operativa; ciclo de vida implementado localmente y pendiente de despliegue
---
@ -105,6 +105,14 @@ Pruebas funcionales ya superadas:
- consulta conceptual sobre caracteristicas del RAG
- consulta tecnica en modo codigo sobre la construccion de `source_id`
### Ciclo de vida del conocimiento
Cada fuente dispone de versiones controladas en PostgreSQL, mientras Qdrant conserva sus chunks y embeddings. Solo la version activa participa en las consultas.
Esto permite actualizar una fuente sin mezclar contenido antiguo y nuevo, detectar ingestas identicas, conservar versiones anteriores y hacer rollback sin recalcular embeddings. Los parametros de consulta actuales no cambian.
Esta mejora esta implementada y validada localmente. Falta desplegarla y registrar el corpus existente como version inicial antes de activarla en produccion.
Los pendientes vigentes, su prioridad y su estado se mantienen en [`PENDIENTES_RAG.md`](./PENDIENTES_RAG.md).
---

View file

@ -0,0 +1,206 @@
CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE TABLE IF NOT EXISTS rag_sources (
source_id text PRIMARY KEY,
tenant text NOT NULL DEFAULT 'default',
source_type text NOT NULL CHECK (source_type IN ('file', 'folder')),
source_ref text NOT NULL,
active_version_id uuid NULL,
needs_reingest boolean NOT NULL DEFAULT false,
disabled_at timestamptz NULL,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS rag_source_versions (
version_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
source_id text NOT NULL REFERENCES rag_sources(source_id) ON DELETE RESTRICT,
version_number bigint NOT NULL CHECK (version_number > 0),
previous_version_id uuid NULL,
state text NOT NULL CHECK (state IN (
'pending', 'indexing', 'review_required', 'ready', 'active',
'superseded', 'failed', 'rejected', 'purging', 'purged'
)),
original_manifest_hash char(64) NOT NULL,
source_content_hash char(64) NULL,
processing_fingerprint char(64) NOT NULL,
metadata_hash char(64) NOT NULL,
tags text[] NOT NULL DEFAULT '{}',
activate_requested boolean NOT NULL DEFAULT true,
base_active_version_id uuid NULL,
embedding_provider text NOT NULL,
embedding_model text NOT NULL,
embedding_dimensions integer NOT NULL CHECK (embedding_dimensions > 0),
qdrant_collection text NOT NULL,
expected_document_count integer NOT NULL DEFAULT 0 CHECK (expected_document_count >= 0),
expected_point_count integer NOT NULL DEFAULT 0 CHECK (expected_point_count >= 0),
verified_point_count integer NOT NULL DEFAULT 0 CHECK (verified_point_count >= 0),
error_code text NULL,
error_detail text NULL,
created_at timestamptz NOT NULL DEFAULT now(),
indexing_started_at timestamptz NULL,
ready_at timestamptz NULL,
activated_at timestamptz NULL,
superseded_at timestamptz NULL,
reviewed_at timestamptz NULL,
reviewed_by text NULL,
artifact_state text NOT NULL DEFAULT 'none'
CHECK (artifact_state IN ('none', 'present', 'retention_deleting', 'retention_deleted')),
retention_due_at timestamptz NULL,
purged_at timestamptz NULL,
UNIQUE (source_id, version_number),
UNIQUE (version_id, source_id),
FOREIGN KEY (previous_version_id, source_id)
REFERENCES rag_source_versions(version_id, source_id)
DEFERRABLE INITIALLY DEFERRED,
FOREIGN KEY (base_active_version_id, source_id)
REFERENCES rag_source_versions(version_id, source_id)
DEFERRABLE INITIALLY DEFERRED
);
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_constraint WHERE conname = 'rag_sources_active_version_fk'
) THEN
ALTER TABLE rag_sources
ADD CONSTRAINT rag_sources_active_version_fk
FOREIGN KEY (active_version_id, source_id)
REFERENCES rag_source_versions(version_id, source_id)
DEFERRABLE INITIALLY DEFERRED;
END IF;
END $$;
CREATE UNIQUE INDEX IF NOT EXISTS rag_one_active_version_per_source
ON rag_source_versions(source_id)
WHERE state = 'active';
CREATE UNIQUE INDEX IF NOT EXISTS rag_one_reusable_content_version
ON rag_source_versions(source_id, source_content_hash, processing_fingerprint, metadata_hash)
WHERE source_content_hash IS NOT NULL
AND state NOT IN ('failed', 'rejected', 'purged');
CREATE INDEX IF NOT EXISTS rag_source_versions_source_state_idx
ON rag_source_versions(source_id, state);
CREATE TABLE IF NOT EXISTS rag_version_documents (
version_id uuid NOT NULL REFERENCES rag_source_versions(version_id) ON DELETE RESTRICT,
document_id text NOT NULL,
document_key text NOT NULL,
original_hash char(64) NOT NULL,
original_hash_kind text NOT NULL CHECK (original_hash_kind IN ('bytes', 'legacy-derived')),
content_hash char(64) NULL,
mime_type text NOT NULL,
title text NOT NULL,
extraction_method text NOT NULL DEFAULT 'native',
index_state text NOT NULL CHECK (index_state IN ('pending', 'indexing', 'ready', 'failed')),
chunk_count integer NOT NULL DEFAULT 0 CHECK (chunk_count >= 0),
artifact_manifest_path text NULL,
error_detail text NULL,
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (version_id, document_id),
UNIQUE (version_id, document_key)
);
CREATE INDEX IF NOT EXISTS rag_version_documents_document_idx
ON rag_version_documents(document_id);
CREATE TABLE IF NOT EXISTS rag_ingestion_attempts (
attempt_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
source_id text NOT NULL REFERENCES rag_sources(source_id) ON DELETE RESTRICT,
version_id uuid NULL REFERENCES rag_source_versions(version_id) ON DELETE SET NULL,
state text NOT NULL CHECK (state IN ('received', 'processing', 'completed', 'failed')),
input_locator text NOT NULL,
input_hash char(64) NULL,
error_code text NULL,
error_detail text NULL,
created_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz NULL
);
CREATE TABLE IF NOT EXISTS rag_system_state (
singleton boolean PRIMARY KEY DEFAULT true CHECK (singleton),
maintenance boolean NOT NULL DEFAULT false,
maintenance_reason text NULL,
updated_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO rag_system_state(singleton) VALUES (true)
ON CONFLICT (singleton) DO NOTHING;
CREATE TABLE IF NOT EXISTS rag_legacy_migration_batches (
batch_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
state text NOT NULL CHECK (state IN ('planned', 'running', 'verified', 'rolled_back', 'failed')),
snapshot_ref text NOT NULL,
report jsonb NOT NULL DEFAULT '{}',
created_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz NULL
);
CREATE TABLE IF NOT EXISTS rag_legacy_migration_items (
batch_id uuid NOT NULL REFERENCES rag_legacy_migration_batches(batch_id) ON DELETE RESTRICT,
source_id text NOT NULL,
version_id uuid NULL REFERENCES rag_source_versions(version_id) ON DELETE SET NULL,
expected_point_count integer NOT NULL CHECK (expected_point_count >= 0),
state text NOT NULL CHECK (state IN ('planned', 'migrated', 'verified', 'rolled_back', 'failed')),
error_detail text NULL,
PRIMARY KEY (batch_id, source_id)
);
CREATE OR REPLACE FUNCTION rag_touch_updated_at()
RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
NEW.updated_at = now();
RETURN NEW;
END;
$$;
DROP TRIGGER IF EXISTS rag_sources_touch_updated_at ON rag_sources;
CREATE TRIGGER rag_sources_touch_updated_at
BEFORE UPDATE ON rag_sources
FOR EACH ROW EXECUTE FUNCTION rag_touch_updated_at();
CREATE OR REPLACE FUNCTION rag_validate_active_source_pointer()
RETURNS trigger LANGUAGE plpgsql AS $$
DECLARE
invalid_pointer_count integer;
orphan_active_count integer;
BEGIN
SELECT count(*) INTO invalid_pointer_count
FROM rag_sources s
LEFT JOIN rag_source_versions v
ON v.version_id = s.active_version_id
AND v.source_id = s.source_id
AND v.state = 'active'
WHERE s.active_version_id IS NOT NULL
AND v.version_id IS NULL;
IF invalid_pointer_count > 0 THEN
RAISE EXCEPTION 'rag_sources.active_version_id must point to an active version in the same source';
END IF;
SELECT count(*) INTO orphan_active_count
FROM rag_source_versions v
JOIN rag_sources s ON s.source_id = v.source_id
WHERE v.state = 'active'
AND s.active_version_id IS DISTINCT FROM v.version_id;
IF orphan_active_count > 0 THEN
RAISE EXCEPTION 'every active rag_source_versions row must be the source active_version_id';
END IF;
RETURN NULL;
END;
$$;
DROP TRIGGER IF EXISTS rag_sources_validate_active_pointer ON rag_sources;
CREATE CONSTRAINT TRIGGER rag_sources_validate_active_pointer
AFTER INSERT OR UPDATE ON rag_sources
DEFERRABLE INITIALLY DEFERRED
FOR EACH ROW EXECUTE FUNCTION rag_validate_active_source_pointer();
DROP TRIGGER IF EXISTS rag_versions_validate_active_pointer ON rag_source_versions;
CREATE CONSTRAINT TRIGGER rag_versions_validate_active_pointer
AFTER INSERT OR UPDATE ON rag_source_versions
DEFERRABLE INITIALLY DEFERRED
FOR EACH ROW EXECUTE FUNCTION rag_validate_active_source_pointer();

509
package-lock.json generated
View file

@ -9,12 +9,13 @@
"version": "0.1.0",
"dependencies": {
"@qdrant/js-client-rest": "^1.15.0",
"adm-zip": "^0.5.17",
"adm-zip": "^0.6.1",
"dotenv": "^16.4.5",
"express": "^4.21.2",
"multer": "^2.0.0",
"multer": "^2.3.0",
"openai": "^4.104.0",
"pdf-parse": "^1.1.1"
"pdf-parse": "^1.1.1",
"pg": "^8.23.0"
},
"devDependencies": {
"@types/adm-zip": "^0.5.8",
@ -22,14 +23,15 @@
"@types/multer": "^1.4.12",
"@types/node": "^22.15.3",
"@types/pdf-parse": "^1.1.4",
"@types/pg": "^8.23.1",
"tsx": "^4.19.3",
"typescript": "^5.8.3"
}
},
"node_modules/@esbuild/aix-ppc64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz",
"integrity": "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.2.tgz",
"integrity": "sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==",
"cpu": [
"ppc64"
],
@ -44,9 +46,9 @@
}
},
"node_modules/@esbuild/android-arm": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.27.7.tgz",
"integrity": "sha512-jbPXvB4Yj2yBV7HUfE2KHe4GJX51QplCN1pGbYjvsyCZbQmies29EoJbkEc+vYuU5o45AfQn37vZlyXy4YJ8RQ==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.28.2.tgz",
"integrity": "sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==",
"cpu": [
"arm"
],
@ -61,9 +63,9 @@
}
},
"node_modules/@esbuild/android-arm64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.27.7.tgz",
"integrity": "sha512-62dPZHpIXzvChfvfLJow3q5dDtiNMkwiRzPylSCfriLvZeq0a1bWChrGx/BbUbPwOrsWKMn8idSllklzBy+dgQ==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.28.2.tgz",
"integrity": "sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==",
"cpu": [
"arm64"
],
@ -78,9 +80,9 @@
}
},
"node_modules/@esbuild/android-x64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.27.7.tgz",
"integrity": "sha512-x5VpMODneVDb70PYV2VQOmIUUiBtY3D3mPBG8NxVk5CogneYhkR7MmM3yR/uMdITLrC1ml/NV1rj4bMJuy9MCg==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.28.2.tgz",
"integrity": "sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==",
"cpu": [
"x64"
],
@ -95,9 +97,9 @@
}
},
"node_modules/@esbuild/darwin-arm64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.27.7.tgz",
"integrity": "sha512-5lckdqeuBPlKUwvoCXIgI2D9/ABmPq3Rdp7IfL70393YgaASt7tbju3Ac+ePVi3KDH6N2RqePfHnXkaDtY9fkw==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.28.2.tgz",
"integrity": "sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==",
"cpu": [
"arm64"
],
@ -112,9 +114,9 @@
}
},
"node_modules/@esbuild/darwin-x64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.27.7.tgz",
"integrity": "sha512-rYnXrKcXuT7Z+WL5K980jVFdvVKhCHhUwid+dDYQpH+qu+TefcomiMAJpIiC2EM3Rjtq0sO3StMV/+3w3MyyqQ==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.28.2.tgz",
"integrity": "sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==",
"cpu": [
"x64"
],
@ -129,9 +131,9 @@
}
},
"node_modules/@esbuild/freebsd-arm64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.27.7.tgz",
"integrity": "sha512-B48PqeCsEgOtzME2GbNM2roU29AMTuOIN91dsMO30t+Ydis3z/3Ngoj5hhnsOSSwNzS+6JppqWsuhTp6E82l2w==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.28.2.tgz",
"integrity": "sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==",
"cpu": [
"arm64"
],
@ -146,9 +148,9 @@
}
},
"node_modules/@esbuild/freebsd-x64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.27.7.tgz",
"integrity": "sha512-jOBDK5XEjA4m5IJK3bpAQF9/Lelu/Z9ZcdhTRLf4cajlB+8VEhFFRjWgfy3M1O4rO2GQ/b2dLwCUGpiF/eATNQ==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.28.2.tgz",
"integrity": "sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==",
"cpu": [
"x64"
],
@ -163,9 +165,9 @@
}
},
"node_modules/@esbuild/linux-arm": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.27.7.tgz",
"integrity": "sha512-RkT/YXYBTSULo3+af8Ib0ykH8u2MBh57o7q/DAs3lTJlyVQkgQvlrPTnjIzzRPQyavxtPtfg0EopvDyIt0j1rA==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.28.2.tgz",
"integrity": "sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==",
"cpu": [
"arm"
],
@ -180,9 +182,9 @@
}
},
"node_modules/@esbuild/linux-arm64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.27.7.tgz",
"integrity": "sha512-RZPHBoxXuNnPQO9rvjh5jdkRmVizktkT7TCDkDmQ0W2SwHInKCAV95GRuvdSvA7w4VMwfCjUiPwDi0ZO6Nfe9A==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.28.2.tgz",
"integrity": "sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==",
"cpu": [
"arm64"
],
@ -197,9 +199,9 @@
}
},
"node_modules/@esbuild/linux-ia32": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.27.7.tgz",
"integrity": "sha512-GA48aKNkyQDbd3KtkplYWT102C5sn/EZTY4XROkxONgruHPU72l+gW+FfF8tf2cFjeHaRbWpOYa/uRBz/Xq1Pg==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.28.2.tgz",
"integrity": "sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==",
"cpu": [
"ia32"
],
@ -214,9 +216,9 @@
}
},
"node_modules/@esbuild/linux-loong64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.27.7.tgz",
"integrity": "sha512-a4POruNM2oWsD4WKvBSEKGIiWQF8fZOAsycHOt6JBpZ+JN2n2JH9WAv56SOyu9X5IqAjqSIPTaJkqN8F7XOQ5Q==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.28.2.tgz",
"integrity": "sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==",
"cpu": [
"loong64"
],
@ -231,9 +233,9 @@
}
},
"node_modules/@esbuild/linux-mips64el": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.27.7.tgz",
"integrity": "sha512-KabT5I6StirGfIz0FMgl1I+R1H73Gp0ofL9A3nG3i/cYFJzKHhouBV5VWK1CSgKvVaG4q1RNpCTR2LuTVB3fIw==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.28.2.tgz",
"integrity": "sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==",
"cpu": [
"mips64el"
],
@ -248,9 +250,9 @@
}
},
"node_modules/@esbuild/linux-ppc64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.27.7.tgz",
"integrity": "sha512-gRsL4x6wsGHGRqhtI+ifpN/vpOFTQtnbsupUF5R5YTAg+y/lKelYR1hXbnBdzDjGbMYjVJLJTd2OFmMewAgwlQ==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.28.2.tgz",
"integrity": "sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==",
"cpu": [
"ppc64"
],
@ -265,9 +267,9 @@
}
},
"node_modules/@esbuild/linux-riscv64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.27.7.tgz",
"integrity": "sha512-hL25LbxO1QOngGzu2U5xeXtxXcW+/GvMN3ejANqXkxZ/opySAZMrc+9LY/WyjAan41unrR3YrmtTsUpwT66InQ==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.28.2.tgz",
"integrity": "sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==",
"cpu": [
"riscv64"
],
@ -282,9 +284,9 @@
}
},
"node_modules/@esbuild/linux-s390x": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.27.7.tgz",
"integrity": "sha512-2k8go8Ycu1Kb46vEelhu1vqEP+UeRVj2zY1pSuPdgvbd5ykAw82Lrro28vXUrRmzEsUV0NzCf54yARIK8r0fdw==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.28.2.tgz",
"integrity": "sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==",
"cpu": [
"s390x"
],
@ -299,9 +301,9 @@
}
},
"node_modules/@esbuild/linux-x64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.27.7.tgz",
"integrity": "sha512-hzznmADPt+OmsYzw1EE33ccA+HPdIqiCRq7cQeL1Jlq2gb1+OyWBkMCrYGBJ+sxVzve2ZJEVeePbLM2iEIZSxA==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.28.2.tgz",
"integrity": "sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==",
"cpu": [
"x64"
],
@ -316,9 +318,9 @@
}
},
"node_modules/@esbuild/netbsd-arm64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.27.7.tgz",
"integrity": "sha512-b6pqtrQdigZBwZxAn1UpazEisvwaIDvdbMbmrly7cDTMFnw/+3lVxxCTGOrkPVnsYIosJJXAsILG9XcQS+Yu6w==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.28.2.tgz",
"integrity": "sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==",
"cpu": [
"arm64"
],
@ -333,9 +335,9 @@
}
},
"node_modules/@esbuild/netbsd-x64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.27.7.tgz",
"integrity": "sha512-OfatkLojr6U+WN5EDYuoQhtM+1xco+/6FSzJJnuWiUw5eVcicbyK3dq5EeV/QHT1uy6GoDhGbFpprUiHUYggrw==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.28.2.tgz",
"integrity": "sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==",
"cpu": [
"x64"
],
@ -350,9 +352,9 @@
}
},
"node_modules/@esbuild/openbsd-arm64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.27.7.tgz",
"integrity": "sha512-AFuojMQTxAz75Fo8idVcqoQWEHIXFRbOc1TrVcFSgCZtQfSdc1RXgB3tjOn/krRHENUB4j00bfGjyl2mJrU37A==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.28.2.tgz",
"integrity": "sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==",
"cpu": [
"arm64"
],
@ -367,9 +369,9 @@
}
},
"node_modules/@esbuild/openbsd-x64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.27.7.tgz",
"integrity": "sha512-+A1NJmfM8WNDv5CLVQYJ5PshuRm/4cI6WMZRg1by1GwPIQPCTs1GLEUHwiiQGT5zDdyLiRM/l1G0Pv54gvtKIg==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.28.2.tgz",
"integrity": "sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==",
"cpu": [
"x64"
],
@ -384,9 +386,9 @@
}
},
"node_modules/@esbuild/openharmony-arm64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.27.7.tgz",
"integrity": "sha512-+KrvYb/C8zA9CU/g0sR6w2RBw7IGc5J2BPnc3dYc5VJxHCSF1yNMxTV5LQ7GuKteQXZtspjFbiuW5/dOj7H4Yw==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.28.2.tgz",
"integrity": "sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==",
"cpu": [
"arm64"
],
@ -401,9 +403,9 @@
}
},
"node_modules/@esbuild/sunos-x64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.27.7.tgz",
"integrity": "sha512-ikktIhFBzQNt/QDyOL580ti9+5mL/YZeUPKU2ivGtGjdTYoqz6jObj6nOMfhASpS4GU4Q/Clh1QtxWAvcYKamA==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.28.2.tgz",
"integrity": "sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==",
"cpu": [
"x64"
],
@ -418,9 +420,9 @@
}
},
"node_modules/@esbuild/win32-arm64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.27.7.tgz",
"integrity": "sha512-7yRhbHvPqSpRUV7Q20VuDwbjW5kIMwTHpptuUzV+AA46kiPze5Z7qgt6CLCK3pWFrHeNfDd1VKgyP4O+ng17CA==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.28.2.tgz",
"integrity": "sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==",
"cpu": [
"arm64"
],
@ -435,9 +437,9 @@
}
},
"node_modules/@esbuild/win32-ia32": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.27.7.tgz",
"integrity": "sha512-SmwKXe6VHIyZYbBLJrhOoCJRB/Z1tckzmgTLfFYOfpMAx63BJEaL9ExI8x7v0oAO3Zh6D/Oi1gVxEYr5oUCFhw==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.28.2.tgz",
"integrity": "sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==",
"cpu": [
"ia32"
],
@ -452,9 +454,9 @@
}
},
"node_modules/@esbuild/win32-x64": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.27.7.tgz",
"integrity": "sha512-56hiAJPhwQ1R4i+21FVF7V8kSD5zZTdHcVuRFMW0hn753vVfQN8xlx4uOPT4xoGH0Z/oVATuR82AiqSTDIpaHg==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.28.2.tgz",
"integrity": "sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==",
"cpu": [
"x64"
],
@ -597,6 +599,18 @@
"@types/node": "*"
}
},
"node_modules/@types/pg": {
"version": "8.23.1",
"resolved": "https://registry.npmjs.org/@types/pg/-/pg-8.23.1.tgz",
"integrity": "sha512-fKVHpikPdg4GKks3JuLEhvwSyvwzF23hnabPy6DD8ljVbC7+6J5dQzdv4arV6jqq57djnMgs1HKBxX4P8aBI3A==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/node": "*",
"pg-protocol": "*",
"pg-types": "^2.2.0"
}
},
"node_modules/@types/qs": {
"version": "6.15.0",
"resolved": "https://registry.npmjs.org/@types/qs/-/qs-6.15.0.tgz",
@ -658,12 +672,12 @@
}
},
"node_modules/adm-zip": {
"version": "0.5.17",
"resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.5.17.tgz",
"integrity": "sha512-+Ut8d9LLqwEvHHJl1+PIHqoyDxFgVN847JTVM3Izi3xHDWPE4UtzzXysMZQs64DMcrJfBeS/uoEP4AD3HQHnQQ==",
"version": "0.6.1",
"resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.1.tgz",
"integrity": "sha512-Xwrja8nx9e5o2N1my4DsKCeKpdrnACyr1wtbPxBDgGzKzKyE9kRtBFA8mWldI+RVlD7CBZNWY/wQ2+ydwOR6kQ==",
"license": "MIT",
"engines": {
"node": ">=12.0"
"node": ">=14.0"
}
},
"node_modules/agentkeepalive": {
@ -697,9 +711,9 @@
"license": "MIT"
},
"node_modules/body-parser": {
"version": "1.20.4",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.4.tgz",
"integrity": "sha512-ZTgYYLMOXY9qKU/57FAo8F+HA2dGX7bqGc71txDRC1rS4frdFI5R7NhluHxH6M0YItAP0sHB4uqAOcYKxO6uGA==",
"version": "1.20.8",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.8.tgz",
"integrity": "sha512-JNcyFQ64OiijEkPzUBTCe+hyPXUD/3LEldGQ6iF5LR1w00mx9o7xtDWHXBY2iItjdCFGoilOLNQbH943ut7pHA==",
"license": "MIT",
"dependencies": {
"bytes": "~3.1.2",
@ -710,7 +724,7 @@
"http-errors": "~2.0.1",
"iconv-lite": "~0.4.24",
"on-finished": "~2.4.1",
"qs": "~6.14.0",
"qs": "~6.16.0",
"raw-body": "~2.5.3",
"type-is": "~1.6.18",
"unpipe": "~1.0.0"
@ -720,6 +734,22 @@
"npm": "1.2.8000 || >= 1.4.16"
}
},
"node_modules/body-parser/node_modules/qs": {
"version": "6.16.0",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",
"integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==",
"license": "BSD-3-Clause",
"dependencies": {
"es-define-property": "^1.0.1",
"side-channel": "^1.1.1"
},
"engines": {
"node": ">=0.6"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/buffer-from": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
@ -962,9 +992,9 @@
}
},
"node_modules/esbuild": {
"version": "0.27.7",
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz",
"integrity": "sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w==",
"version": "0.28.2",
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.2.tgz",
"integrity": "sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==",
"dev": true,
"hasInstallScript": true,
"license": "MIT",
@ -975,32 +1005,32 @@
"node": ">=18"
},
"optionalDependencies": {
"@esbuild/aix-ppc64": "0.27.7",
"@esbuild/android-arm": "0.27.7",
"@esbuild/android-arm64": "0.27.7",
"@esbuild/android-x64": "0.27.7",
"@esbuild/darwin-arm64": "0.27.7",
"@esbuild/darwin-x64": "0.27.7",
"@esbuild/freebsd-arm64": "0.27.7",
"@esbuild/freebsd-x64": "0.27.7",
"@esbuild/linux-arm": "0.27.7",
"@esbuild/linux-arm64": "0.27.7",
"@esbuild/linux-ia32": "0.27.7",
"@esbuild/linux-loong64": "0.27.7",
"@esbuild/linux-mips64el": "0.27.7",
"@esbuild/linux-ppc64": "0.27.7",
"@esbuild/linux-riscv64": "0.27.7",
"@esbuild/linux-s390x": "0.27.7",
"@esbuild/linux-x64": "0.27.7",
"@esbuild/netbsd-arm64": "0.27.7",
"@esbuild/netbsd-x64": "0.27.7",
"@esbuild/openbsd-arm64": "0.27.7",
"@esbuild/openbsd-x64": "0.27.7",
"@esbuild/openharmony-arm64": "0.27.7",
"@esbuild/sunos-x64": "0.27.7",
"@esbuild/win32-arm64": "0.27.7",
"@esbuild/win32-ia32": "0.27.7",
"@esbuild/win32-x64": "0.27.7"
"@esbuild/aix-ppc64": "0.28.2",
"@esbuild/android-arm": "0.28.2",
"@esbuild/android-arm64": "0.28.2",
"@esbuild/android-x64": "0.28.2",
"@esbuild/darwin-arm64": "0.28.2",
"@esbuild/darwin-x64": "0.28.2",
"@esbuild/freebsd-arm64": "0.28.2",
"@esbuild/freebsd-x64": "0.28.2",
"@esbuild/linux-arm": "0.28.2",
"@esbuild/linux-arm64": "0.28.2",
"@esbuild/linux-ia32": "0.28.2",
"@esbuild/linux-loong64": "0.28.2",
"@esbuild/linux-mips64el": "0.28.2",
"@esbuild/linux-ppc64": "0.28.2",
"@esbuild/linux-riscv64": "0.28.2",
"@esbuild/linux-s390x": "0.28.2",
"@esbuild/linux-x64": "0.28.2",
"@esbuild/netbsd-arm64": "0.28.2",
"@esbuild/netbsd-x64": "0.28.2",
"@esbuild/openbsd-arm64": "0.28.2",
"@esbuild/openbsd-x64": "0.28.2",
"@esbuild/openharmony-arm64": "0.28.2",
"@esbuild/sunos-x64": "0.28.2",
"@esbuild/win32-arm64": "0.28.2",
"@esbuild/win32-ia32": "0.28.2",
"@esbuild/win32-x64": "0.28.2"
}
},
"node_modules/escape-html": {
@ -1028,14 +1058,14 @@
}
},
"node_modules/express": {
"version": "4.22.1",
"resolved": "https://registry.npmjs.org/express/-/express-4.22.1.tgz",
"integrity": "sha512-F2X8g9P1X7uCPZMA3MVf9wcTqlyNp7IhH5qPCI0izhaOIYXaW9L535tGA3qmjRzpH+bZczqq7hVKxTR4NWnu+g==",
"version": "4.22.2",
"resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz",
"integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==",
"license": "MIT",
"dependencies": {
"accepts": "~1.3.8",
"array-flatten": "1.1.1",
"body-parser": "~1.20.3",
"body-parser": "~1.20.5",
"content-disposition": "~0.5.4",
"content-type": "~1.0.4",
"cookie": "~0.7.1",
@ -1054,7 +1084,7 @@
"parseurl": "~1.3.3",
"path-to-regexp": "~0.1.12",
"proxy-addr": "~2.0.7",
"qs": "~6.14.0",
"qs": "~6.15.1",
"range-parser": "~1.2.1",
"safe-buffer": "5.2.1",
"send": "~0.19.0",
@ -1092,16 +1122,16 @@
}
},
"node_modules/form-data": {
"version": "4.0.5",
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.5.tgz",
"integrity": "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==",
"version": "4.0.6",
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.6.tgz",
"integrity": "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==",
"license": "MIT",
"dependencies": {
"asynckit": "^0.4.0",
"combined-stream": "^1.0.8",
"es-set-tostringtag": "^2.1.0",
"hasown": "^2.0.2",
"mime-types": "^2.1.12"
"hasown": "^2.0.4",
"mime-types": "^2.1.35"
},
"engines": {
"node": ">= 6"
@ -1205,19 +1235,6 @@
"node": ">= 0.4"
}
},
"node_modules/get-tsconfig": {
"version": "4.13.7",
"resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-4.13.7.tgz",
"integrity": "sha512-7tN6rFgBlMgpBML5j8typ92BKFi2sFQvIdpAqLA2beia5avZDrMs0FLZiM5etShWq5irVyGcGMEA1jcDaK7A/Q==",
"dev": true,
"license": "MIT",
"dependencies": {
"resolve-pkg-maps": "^1.0.0"
},
"funding": {
"url": "https://github.com/privatenumber/get-tsconfig?sponsor=1"
}
},
"node_modules/gopd": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
@ -1258,9 +1275,9 @@
}
},
"node_modules/hasown": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
"integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
"version": "2.0.4",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz",
"integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==",
"license": "MIT",
"dependencies": {
"function-bind": "^1.1.2"
@ -1401,9 +1418,9 @@
"license": "MIT"
},
"node_modules/multer": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/multer/-/multer-2.1.1.tgz",
"integrity": "sha512-mo+QTzKlx8R7E5ylSXxWzGoXoZbOsRMpyitcht8By2KHvMbf3tjwosZ/Mu/XYU6UuJ3VZnODIrak5ZrPiPyB6A==",
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/multer/-/multer-2.3.0.tgz",
"integrity": "sha512-cjNbm3sttszgZeGfJR124D+jFEfkXCVAsoPBmFn9X7UxmDSFHWqE2CoEj0vrmSpuAFnqWR1Szcm9QTsiHr60Xw==",
"license": "MIT",
"dependencies": {
"append-field": "^1.0.0",
@ -1574,6 +1591,134 @@
"url": "https://github.com/sponsors/mehmet-kozan"
}
},
"node_modules/pg": {
"version": "8.23.0",
"resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
"integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",
"license": "MIT",
"dependencies": {
"pg-connection-string": "^2.14.0",
"pg-pool": "^3.14.0",
"pg-protocol": "^1.16.0",
"pg-types": "2.2.0",
"pgpass": "1.0.5"
},
"engines": {
"node": ">= 16.0.0"
},
"optionalDependencies": {
"pg-cloudflare": "^1.4.0"
},
"peerDependencies": {
"pg-native": ">=3.0.1"
},
"peerDependenciesMeta": {
"pg-native": {
"optional": true
}
}
},
"node_modules/pg-cloudflare": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
"integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
"license": "MIT",
"optional": true
},
"node_modules/pg-connection-string": {
"version": "2.14.0",
"resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",
"integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
"license": "MIT"
},
"node_modules/pg-int8": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
"integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
"license": "ISC",
"engines": {
"node": ">=4.0.0"
}
},
"node_modules/pg-pool": {
"version": "3.14.0",
"resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
"integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
"license": "MIT",
"peerDependencies": {
"pg": ">=8.0"
}
},
"node_modules/pg-protocol": {
"version": "1.16.0",
"resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",
"integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",
"license": "MIT"
},
"node_modules/pg-types": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
"integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
"license": "MIT",
"dependencies": {
"pg-int8": "1.0.1",
"postgres-array": "~2.0.0",
"postgres-bytea": "~1.0.0",
"postgres-date": "~1.0.4",
"postgres-interval": "^1.1.0"
},
"engines": {
"node": ">=4"
}
},
"node_modules/pgpass": {
"version": "1.0.5",
"resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
"integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
"license": "MIT",
"dependencies": {
"split2": "^4.1.0"
}
},
"node_modules/postgres-array": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
"integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
"license": "MIT",
"engines": {
"node": ">=4"
}
},
"node_modules/postgres-bytea": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
"integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/postgres-date": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
"integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/postgres-interval": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
"integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
"license": "MIT",
"dependencies": {
"xtend": "^4.0.0"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/proxy-addr": {
"version": "2.0.7",
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
@ -1588,12 +1733,13 @@
}
},
"node_modules/qs": {
"version": "6.14.2",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.14.2.tgz",
"integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==",
"version": "6.15.3",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz",
"integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==",
"license": "BSD-3-Clause",
"dependencies": {
"side-channel": "^1.1.0"
"es-define-property": "^1.0.1",
"side-channel": "^1.1.1"
},
"engines": {
"node": ">=0.6"
@ -1640,16 +1786,6 @@
"node": ">= 6"
}
},
"node_modules/resolve-pkg-maps": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/resolve-pkg-maps/-/resolve-pkg-maps-1.0.0.tgz",
"integrity": "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==",
"dev": true,
"license": "MIT",
"funding": {
"url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1"
}
},
"node_modules/safe-buffer": {
"version": "5.2.1",
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
@ -1728,14 +1864,14 @@
"license": "ISC"
},
"node_modules/side-channel": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz",
"integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==",
"version": "1.1.1",
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz",
"integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"object-inspect": "^1.13.3",
"side-channel-list": "^1.0.0",
"object-inspect": "^1.13.4",
"side-channel-list": "^1.0.1",
"side-channel-map": "^1.0.1",
"side-channel-weakmap": "^1.0.2"
},
@ -1747,13 +1883,13 @@
}
},
"node_modules/side-channel-list": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.0.tgz",
"integrity": "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",
"integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",
"license": "MIT",
"dependencies": {
"es-errors": "^1.3.0",
"object-inspect": "^1.13.3"
"object-inspect": "^1.13.4"
},
"engines": {
"node": ">= 0.4"
@ -1799,6 +1935,15 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/split2": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
"integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
"license": "ISC",
"engines": {
"node": ">= 10.x"
}
},
"node_modules/statuses": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
@ -1841,14 +1986,13 @@
"license": "MIT"
},
"node_modules/tsx": {
"version": "4.21.0",
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.21.0.tgz",
"integrity": "sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw==",
"version": "4.23.13",
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.13.tgz",
"integrity": "sha512-BL5MGkRln6aDYhb0xbQlEAGw743BaZYWdbWtdJOBriYJboKgUUYCadFp2/FpBBZquBC/ezNBn7wMMPx7FDZUDw==",
"dev": true,
"license": "MIT",
"dependencies": {
"esbuild": "~0.27.0",
"get-tsconfig": "^4.7.5"
"esbuild": "~0.28.0"
},
"bin": {
"tsx": "dist/cli.mjs"
@ -1893,9 +2037,9 @@
}
},
"node_modules/undici": {
"version": "6.24.1",
"resolved": "https://registry.npmjs.org/undici/-/undici-6.24.1.tgz",
"integrity": "sha512-sC+b0tB1whOCzbtlx20fx3WgCXwkW627p4EA9uM+/tNNPkSS+eSEld6pAs9nDv7WbY1UUljBMYPtu9BCOrCWKA==",
"version": "6.28.1",
"resolved": "https://registry.npmjs.org/undici/-/undici-6.28.1.tgz",
"integrity": "sha512-zWpdTVD54H48CIybL0rWQ3ukpb9d23wM7eH5RtfdmeP70cWHNjtfo7P4vZX+5CoDcO53J4Pu5uXp7lNfjc6DRA==",
"license": "MIT",
"engines": {
"node": ">=18.17"
@ -1964,6 +2108,15 @@
"tr46": "~0.0.3",
"webidl-conversions": "^3.0.0"
}
},
"node_modules/xtend": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
"integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
"license": "MIT",
"engines": {
"node": ">=0.4"
}
}
}
}

View file

@ -6,17 +6,21 @@
"scripts": {
"dev": "tsx watch src/server.ts",
"build": "tsc -p tsconfig.json",
"start": "node dist/server.js",
"start": "node dist/modules/catalog/migrations.js && node dist/server.js",
"migrate:lifecycle": "node dist/modules/catalog/migrations.js",
"migrate:legacy:lifecycle": "node dist/scripts/migrate-legacy-lifecycle.js",
"test": "NODE_ENV=test tsx --test tests/**/*.test.ts",
"check": "tsc --noEmit -p tsconfig.json"
},
"dependencies": {
"@qdrant/js-client-rest": "^1.15.0",
"adm-zip": "^0.5.17",
"adm-zip": "^0.6.1",
"dotenv": "^16.4.5",
"express": "^4.21.2",
"multer": "^2.0.0",
"multer": "^2.3.0",
"openai": "^4.104.0",
"pdf-parse": "^1.1.1"
"pdf-parse": "^1.1.1",
"pg": "^8.23.0"
},
"devDependencies": {
"@types/adm-zip": "^0.5.8",
@ -24,6 +28,7 @@
"@types/multer": "^1.4.12",
"@types/node": "^22.15.3",
"@types/pdf-parse": "^1.1.4",
"@types/pg": "^8.23.1",
"tsx": "^4.19.3",
"typescript": "^5.8.3"
}

View file

@ -38,6 +38,8 @@ const btnClearUpload = document.getElementById("btnClearUpload");
const uploadStatusText = document.getElementById("uploadStatusText");
const ingestMode = document.getElementById("ingestMode");
const ingestTags = document.getElementById("ingestTags");
const ingestActivate = document.getElementById("ingestActivate");
const ingestExpectedActiveVersionId = document.getElementById("ingestExpectedActiveVersionId");
const ingestModeHint = document.getElementById("ingestModeHint");
const cleanupScopeSelect = document.getElementById("cleanupScopeSelect");
@ -80,7 +82,17 @@ function splitTags(value) {
function buildScopeLabel(scope) {
const modes = scope.chunkModes.join(", ") || "sin modo";
return `${scope.sourceRef} [${modes}]`;
const version = scope.activeVersionNumber ? ` v${scope.activeVersionNumber}` : "";
const state = scope.state ? ` ${scope.state}` : "";
return `${scope.sourceRef}${version}${state} [${modes}]`;
}
function normalizeExpectedActiveVersionInput() {
const value = ingestExpectedActiveVersionId.value.trim();
if (!value) {
return undefined;
}
return value;
}
let previousSourceRef = ""; // Para guardar el valor original de SourceRef
@ -363,6 +375,11 @@ ingestButton.addEventListener("click", async () => {
formData.append("file", ingestUploadFile.files[0]);
formData.append("mode", ingestMode.value);
formData.append("tags", splitTags(ingestTags.value).join(","));
formData.append("activate", ingestActivate.value);
const expectedActiveVersionId = normalizeExpectedActiveVersionInput();
if (expectedActiveVersionId !== undefined) {
formData.append("expectedActiveVersionId", expectedActiveVersionId);
}
if (ingestScopeMode.value === "custom" && ingestSourceId.value.trim()) {
formData.append("sourceId", ingestSourceId.value.trim());
}
@ -414,6 +431,11 @@ ingestButton.addEventListener("click", async () => {
formData.append("isZipFolder", "true");
formData.append("mode", ingestMode.value);
formData.append("tags", splitTags(ingestTags.value).join(","));
formData.append("activate", ingestActivate.value);
const expectedActiveVersionId = normalizeExpectedActiveVersionInput();
if (expectedActiveVersionId !== undefined) {
formData.append("expectedActiveVersionId", expectedActiveVersionId);
}
if (ingestScopeMode.value === "custom" && ingestSourceId.value.trim()) {
formData.append("sourceId", ingestSourceId.value.trim());
}
@ -431,7 +453,9 @@ ingestButton.addEventListener("click", async () => {
sourceType: ingestSourceType.value,
sourceRef: ingestSourceRef.value,
mode: ingestMode.value,
tags: splitTags(ingestTags.value)
tags: splitTags(ingestTags.value),
activate: ingestActivate.value === "true",
expectedActiveVersionId: normalizeExpectedActiveVersionInput() === "null" ? null : normalizeExpectedActiveVersionInput()
});
}

View file

@ -75,8 +75,17 @@
<label>Tags (coma separada)
<input id="ingestTags" value="workspace,global-docs" />
</label>
<label>Activar al terminar
<select id="ingestActivate">
<option value="true">si, activar version</option>
<option value="false">no, dejar en ready</option>
</select>
</label>
<label>Version activa esperada
<input id="ingestExpectedActiveVersionId" placeholder="uuid actual o null para primera activacion" />
</label>
</div>
<p class="helper" id="ingestModeHint">Si seleccionas un archivo local, el playground usara upload directo y podras aislarlo con un `sourceId` propio para no mezclarlo con otros scopes.</p>
<p class="helper" id="ingestModeHint">Si el ciclo de vida esta activo, una ingesta con activacion exige indicar la version activa esperada; usa <code>null</code> para la primera activacion de una fuente.</p>
<div class="actions">
<button id="ingestButton">Lanzar ingesta</button>
</div>
@ -87,7 +96,7 @@
<section id="tab-cleanup" class="tab-panel">
<article class="panel">
<h2>Limpieza controlada</h2>
<p class="helper">Permite eliminar contexto ya ingerido del RAG antes de reingestar una fuente, para evitar duplicados y versiones mezcladas.</p>
<p class="helper">Esta operacion queda como compatibilidad legacy. Con ciclo de vida, usa la purga administrativa de versiones no activas.</p>
<div class="grid single-grid">
<label>Scope disponible a limpiar
<select id="cleanupScopeSelect">

View file

@ -30,6 +30,7 @@ export const openApiDocument = {
{ name: "Discovery", description: "API contract and browser playground." },
{ name: "Status", description: "Service capabilities and available resources." },
{ name: "Ingestion", description: "Knowledge ingestion and cleanup." },
{ name: "Lifecycle", description: "Knowledge source versions, activation, rollback, and purge." },
{ name: "Retrieval", description: "Context retrieval and model-backed answers." },
{ name: "Evaluation", description: "Evaluation log capture and review." }
],
@ -70,7 +71,8 @@ export const openApiDocument = {
tags: ["Status"],
summary: "Check service health and capabilities",
responses: {
"200": jsonResponse("Current service status.", ref("HealthResponse"))
"200": jsonResponse("Current service status.", ref("HealthResponse")),
"503": jsonResponse("One or more required services are unavailable.", ref("HealthResponse"))
}
}
},
@ -83,10 +85,117 @@ export const openApiDocument = {
type: "array",
items: ref("AvailableScope")
}),
"503": serverError,
"500": serverError
}
}
},
"/sources/{sourceId}": {
get: {
tags: ["Lifecycle"],
summary: "Get a catalog source",
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
responses: {
"200": jsonResponse("Catalog source.", ref("AvailableScope")),
"404": jsonResponse("Source not found.", ref("Error")),
"503": serverError
}
}
},
"/sources/{sourceId}/versions": {
get: {
tags: ["Lifecycle"],
summary: "List source versions",
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
responses: {
"200": jsonResponse("Source versions.", { type: "array", items: ref("SourceVersion") }),
"503": serverError
}
}
},
"/sources/{sourceId}/versions/{versionId}": {
get: {
tags: ["Lifecycle"],
summary: "Get one source version",
parameters: [
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
],
responses: {
"200": jsonResponse("Source version.", ref("SourceVersion")),
"404": jsonResponse("Version not found.", ref("Error")),
"503": serverError
}
},
delete: {
tags: ["Lifecycle"],
summary: "Purge a non-active source version",
security: [{ bearerAuth: [] }],
parameters: [
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
],
responses: {
"200": jsonResponse("Version purged.", ref("PurgeResponse")),
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
"409": jsonResponse("Active versions cannot be purged.", ref("Error")),
"503": serverError
}
}
},
"/sources/{sourceId}/versions/{versionId}/activate": {
post: {
tags: ["Lifecycle"],
summary: "Activate a ready or superseded version",
security: [{ bearerAuth: [] }],
parameters: [
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
],
requestBody: { required: true, content: jsonContent(ref("ActivateVersionRequest")) },
responses: {
"200": jsonResponse("Activated source version.", ref("SourceVersion")),
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
"409": jsonResponse("Active version precondition failed.", ref("Error")),
"503": serverError
}
}
},
"/sources/{sourceId}/rollback": {
post: {
tags: ["Lifecycle"],
summary: "Rollback a source to a previous version",
security: [{ bearerAuth: [] }],
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
requestBody: { required: true, content: jsonContent(ref("RollbackRequest")) },
responses: {
"200": jsonResponse("Rollback activated source version.", ref("SourceVersion")),
"400": jsonResponse("Required rollback fields are missing.", ref("Error")),
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
"409": jsonResponse("Active version precondition failed.", ref("Error")),
"503": serverError
}
}
},
"/sources/{sourceId}/versions/{versionId}/retry": {
post: {
tags: ["Lifecycle"],
summary: "Explicitly retry a failed or orphaned version by re-submitting a durable source",
security: [{ bearerAuth: [] }],
parameters: [
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
],
requestBody: { required: false, content: jsonContent(ref("RetryVersionRequest")) },
responses: {
"201": jsonResponse("Replacement version created by explicit retry.", ref("IngestResponse")),
"202": jsonResponse("Retry instructions returned because no durable source payload was supplied.", ref("RetryVersionResponse")),
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
"409": jsonResponse("Version state is not retryable.", ref("Error")),
"503": serverError
}
}
},
"/models/answer": {
get: {
tags: ["Status"],
@ -112,7 +221,12 @@ export const openApiDocument = {
})
},
responses: {
"202": jsonResponse("Ingestion completed and accepted.", ref("IngestResponse")),
"201": jsonResponse("Lifecycle ingestion completed and committed.", ref("IngestResponse")),
"202": jsonResponse("Legacy ingestion completed and accepted while lifecycle enforcement is disabled.", ref("IngestResponse")),
"400": jsonResponse("Invalid lifecycle activation request.", ref("Error")),
"409": jsonResponse("Concurrent active version change or reusable version in progress.", ref("Error")),
"422": jsonResponse("Empty or unsupported source.", ref("Error")),
"503": serverError,
"500": serverError
}
}
@ -130,8 +244,12 @@ export const openApiDocument = {
}
},
responses: {
"202": jsonResponse("Upload ingested successfully.", ref("UploadIngestResponse")),
"201": jsonResponse("Lifecycle upload ingested successfully.", ref("UploadIngestResponse")),
"202": jsonResponse("Legacy upload ingested successfully while lifecycle enforcement is disabled.", ref("UploadIngestResponse")),
"400": jsonResponse("The file field is missing.", ref("Error"), { ok: false, error: "Missing file upload" }),
"409": jsonResponse("Concurrent active version change or reusable version in progress.", ref("Error")),
"422": jsonResponse("Empty or unsupported source.", ref("Error")),
"503": serverError,
"500": serverError
}
}
@ -150,6 +268,9 @@ export const openApiDocument = {
responses: {
"200": jsonResponse("Matching chunks deleted.", ref("CleanupResponse")),
"400": jsonResponse("Scope is missing or unsafe for cleanup.", ref("Error")),
"401": jsonResponse("Missing or invalid lifecycle admin token when lifecycle enforcement is active.", ref("Error")),
"410": jsonResponse("Legacy cleanup is disabled while lifecycle enforcement is active.", ref("Error")),
"503": serverError,
"500": serverError
}
}
@ -272,6 +393,9 @@ export const openApiDocument = {
}
},
components: {
securitySchemes: {
bearerAuth: { type: "http", scheme: "bearer" }
},
schemas: {
Error: {
type: "object",
@ -289,6 +413,10 @@ export const openApiDocument = {
tags: { type: "array", items: { type: "string" } }
}
},
SourceVersionState: {
type: "string",
enum: ["pending", "indexing", "review_required", "ready", "active", "superseded", "failed", "rejected", "purging", "purged"]
},
AvailableScope: {
type: "object",
required: ["sourceId", "sourceRef", "chunkModes", "tags"],
@ -296,7 +424,12 @@ export const openApiDocument = {
sourceId: { type: "string" },
sourceRef: { type: "string" },
chunkModes: { type: "array", items: { $ref: "#/components/schemas/ChunkMode" } },
tags: { type: "array", items: { type: "string" } }
tags: { type: "array", items: { type: "string" } },
activeVersionId: { type: ["string", "null"], format: "uuid" },
activeVersionNumber: { type: ["integer", "null"] },
state: { ...ref("SourceVersionState"), nullable: true },
needsReingest: { type: "boolean" },
updatedAt: { type: "string", format: "date-time" }
}
},
ChunkMode: { type: "string", enum: ["documental", "codigo", "auto"] },
@ -310,18 +443,27 @@ export const openApiDocument = {
sourceRef: { type: "string" },
readPath: { type: "string", description: "Internal filesystem path used when the logical sourceRef differs from the read location." },
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
tags: { type: "array", items: { type: "string" } }
tags: { type: "array", items: { type: "string" } },
activate: { type: "boolean", default: true },
expectedActiveVersionId: { type: ["string", "null"], format: "uuid", description: "Required when activate is true. Use null for first activation." }
}
},
IngestResponse: {
type: "object",
required: ["accepted", "source", "filesDiscovered", "documentsProcessed", "chunksStored", "collectionName"],
required: ["accepted", "sourceId", "filesDiscovered", "documentsProcessed", "chunksStored", "collectionName"],
properties: {
accepted: { type: "boolean", const: true },
source: ref("IngestRequest"),
sourceId: { type: "string" },
versionId: { type: "string", format: "uuid" },
versionNumber: { type: "integer" },
state: ref("SourceVersionState"),
previousVersionId: { type: ["string", "null"], format: "uuid" },
filesDiscovered: { type: "integer" },
documentsProcessed: { type: "integer" },
chunksStored: { type: "integer" },
activated: { type: "boolean" },
noOp: { type: "boolean" },
collectionName: { type: "string" }
}
},
@ -333,7 +475,9 @@ export const openApiDocument = {
sourceId: { type: "string" },
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
tags: { type: "string", description: "Comma-separated tags." },
isZipFolder: { type: "string", enum: ["true", "false"], default: "false" }
isZipFolder: { type: "string", enum: ["true", "false"], default: "false" },
activate: { type: "string", enum: ["true", "false"], default: "true" },
expectedActiveVersionId: { type: "string", description: "UUID or literal null when activate is true." }
}
},
UploadIngestResponse: {
@ -355,7 +499,7 @@ export const openApiDocument = {
type: "object",
required: ["ok", "deleted"],
properties: {
ok: { type: "boolean", const: true },
ok: { type: "boolean" },
deleted: { type: "integer" }
}
},
@ -389,6 +533,8 @@ export const openApiDocument = {
chunkId: { type: "string" },
documentId: { type: "string" },
sourceId: { type: "string" },
sourceVersionId: { type: "string", format: "uuid" },
sourceVersionNumber: { type: "integer" },
title: { type: "string" },
sectionTitle: { type: "string" },
content: { type: "string" },
@ -433,6 +579,8 @@ export const openApiDocument = {
properties: {
chunkId: { type: "string" },
documentId: { type: "string" },
sourceVersionId: { type: "string", format: "uuid" },
sourceVersionNumber: { type: "integer" },
title: { type: "string" },
sectionTitle: { type: "string" },
startLine: { type: "integer" },
@ -571,18 +719,87 @@ export const openApiDocument = {
},
HealthResponse: {
type: "object",
required: ["ok", "service", "environment", "embeddings", "answer", "vectorStore", "parsers", "chunking"],
required: ["ok", "service", "environment", "embeddings", "answer", "vectorStore", "postgres", "knowledgeLifecycle", "parsers", "chunking"],
properties: {
ok: { type: "boolean", const: true },
ok: { type: "boolean" },
service: { type: "string", const: "rag" },
environment: { type: "string" },
embeddings: ref("ProviderModel"),
answer: ref("ProviderModel"),
vectorStore: { type: "object", additionalProperties: true },
postgres: { type: "object", additionalProperties: true },
knowledgeLifecycle: { type: "object", additionalProperties: true },
parsers: { type: "array", items: { type: "string" } },
chunking: { type: "object", additionalProperties: true }
}
},
SourceVersion: {
type: "object",
required: ["versionId", "sourceId", "versionNumber", "state", "tags", "embeddingProvider", "embeddingModel", "embeddingDimensions", "expectedDocumentCount", "expectedPointCount", "verifiedPointCount", "qdrantCollection"],
properties: {
versionId: { type: "string", format: "uuid" },
sourceId: { type: "string" },
versionNumber: { type: "integer" },
previousVersionId: { type: ["string", "null"], format: "uuid" },
state: ref("SourceVersionState"),
tags: { type: "array", items: { type: "string" } },
sourceContentHash: { type: ["string", "null"] },
processingFingerprint: { type: "string" },
metadataHash: { type: "string" },
embeddingProvider: { type: "string" },
embeddingModel: { type: "string" },
embeddingDimensions: { type: "integer" },
expectedDocumentCount: { type: "integer" },
expectedPointCount: { type: "integer" },
verifiedPointCount: { type: "integer" },
qdrantCollection: { type: "string" }
}
},
ActivateVersionRequest: {
type: "object",
required: ["expectedActiveVersionId"],
properties: { expectedActiveVersionId: { type: ["string", "null"], format: "uuid" } }
},
RollbackRequest: {
type: "object",
required: ["targetVersionId", "expectedActiveVersionId"],
properties: {
targetVersionId: { type: "string", format: "uuid" },
expectedActiveVersionId: { type: "string", format: "uuid" }
}
},
RetryVersionRequest: {
type: "object",
properties: {
sourceType: { type: "string", enum: ["file", "folder"] },
sourceRef: { type: "string" },
readPath: { type: "string" },
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
tags: { type: "array", items: { type: "string" } },
activate: { type: "boolean", default: true },
expectedActiveVersionId: { type: ["string", "null"], format: "uuid" }
}
},
RetryVersionResponse: {
type: "object",
required: ["ok", "retryRequired", "sourceId", "versionId", "message"],
properties: {
ok: { type: "boolean", const: true },
retryRequired: { type: "boolean", const: true },
sourceId: { type: "string" },
versionId: { type: "string", format: "uuid" },
message: { type: "string" }
}
},
PurgeResponse: {
type: "object",
required: ["ok", "purged", "versionId"],
properties: {
ok: { type: "boolean", const: true },
purged: { type: "boolean", const: true },
versionId: { type: "string", format: "uuid" }
}
},
ProviderModel: {
type: "object",
required: ["provider", "model"],

View file

@ -4,6 +4,7 @@ import type { Request } from "express";
import { writeFile, unlink, mkdtemp, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { timingSafeEqual } from "node:crypto";
import { fileURLToPath } from "node:url";
import AdmZip from "adm-zip";
import { buildApiHelp, openApiDocument } from "./api/openapi.js";
@ -16,6 +17,9 @@ import { documentalChunkingPolicy } from "./modules/process/chunking.js";
import { RetrieveService } from "./modules/retrieve/service.js";
import { supportedParserExtensions } from "./modules/parsers/parser-registry.js";
import { QdrantVectorStoreClient } from "./modules/vectorstore/client.js";
import { getCatalogPool } from "./modules/catalog/client.js";
import { KnowledgeLifecycleReconciler } from "./modules/catalog/reconciler.js";
import { CatalogError, CatalogRepository } from "./modules/catalog/repository.js";
import type { ChatMessage, ChunkMode, RetrieveIntent, RetrieveScope } from "./shared/types/rag.js";
type UploadRequest = Request & {
@ -30,10 +34,50 @@ export function createApp() {
const upload = multer({ storage: multer.memoryStorage() });
const embeddingProvider = new OpenRouterEmbeddingProvider();
const vectorStore = new QdrantVectorStoreClient();
const catalogPool = getCatalogPool();
const catalog = catalogPool ? new CatalogRepository(catalogPool) : undefined;
const reconciler = new KnowledgeLifecycleReconciler(catalog, vectorStore);
const evaluationLogs = new EvaluationLogService(embeddingProvider);
const ingestService = new IngestService(embeddingProvider, vectorStore);
const retrieveService = new RetrieveService(embeddingProvider, vectorStore);
const ingestService = new IngestService(embeddingProvider, vectorStore, catalog);
const retrieveService = new RetrieveService(embeddingProvider, vectorStore, catalog);
const answerService = new AnswerService(retrieveService);
reconciler.start();
function sendError(res: express.Response, error: unknown, fallback: string) {
const upstreamUnavailable = error instanceof Error && /(ECONNREFUSED|ETIMEDOUT|timeout|connection|database|postgres|qdrant)/i.test(error.message);
const statusCode = error instanceof CatalogError ? error.statusCode : upstreamUnavailable ? 503 : 500;
const body = {
ok: false,
error: error instanceof Error ? error.message : fallback,
code: error instanceof CatalogError ? error.code : undefined
};
res.status(statusCode).json(body);
}
function requireLifecycleAdmin(req: express.Request, res: express.Response): boolean {
const header = req.header("authorization") ?? "";
const token = header.startsWith("Bearer ") ? header.slice("Bearer ".length) : "";
if (!env.lifecycleAdminToken || !token) {
res.status(401).json({ ok: false, error: "Lifecycle admin token is required" });
return false;
}
const expected = Buffer.from(env.lifecycleAdminToken);
const actual = Buffer.from(token);
const valid = expected.length === actual.length && timingSafeEqual(expected, actual);
if (!valid) {
res.status(401).json({ ok: false, error: "Lifecycle admin token is invalid" });
return false;
}
return true;
}
function requireCatalog(res: express.Response): CatalogRepository | undefined {
if (!catalog) {
res.status(503).json({ ok: false, error: "Knowledge catalog is not configured", code: "CATALOG_UNAVAILABLE" });
return undefined;
}
return catalog;
}
function needsContextLog(summary?: string, itemsCount = 0, answer?: string) {
if (itemsCount === 0) {
@ -76,8 +120,11 @@ export function createApp() {
app.get("/health", async (_req, res) => {
const vectorHealth = await vectorStore.healthcheck();
res.json({
ok: true,
const postgresHealth = catalog ? await catalog.healthcheck() : { ok: false, kind: "postgres" as const };
const reconcilerStatus = reconciler.getStatus();
const ok = vectorHealth.ok && (!env.knowledgeLifecycleEnforced || (postgresHealth.ok && reconcilerStatus.ok));
res.status(ok ? 200 : 503).json({
ok,
service: "rag",
environment: env.nodeEnv,
embeddings: {
@ -89,6 +136,12 @@ export function createApp() {
model: env.answerModel
},
vectorStore: vectorHealth,
postgres: postgresHealth,
knowledgeLifecycle: {
enforced: env.knowledgeLifecycleEnforced,
ingestWritesEnabled: env.ingestWritesEnabled,
reconciler: reconcilerStatus
},
parsers: supportedParserExtensions(),
chunking: documentalChunkingPolicy
});
@ -96,10 +149,55 @@ export function createApp() {
app.get("/sources", async (_req, res) => {
try {
const scopes = await vectorStore.listScopes();
if (env.knowledgeLifecycleEnforced && !catalog) {
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
}
const scopes = env.knowledgeLifecycleEnforced && catalog
? await catalog.listSources()
: await vectorStore.listScopes();
res.json(scopes);
} catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown sources error" });
sendError(res, error, "Unknown sources error");
}
});
app.get("/sources/:sourceId", async (req, res) => {
const repository = requireCatalog(res);
if (!repository) return;
try {
const source = await repository.getSource(String(req.params.sourceId));
if (!source) {
res.status(404).json({ ok: false, error: "Source not found" });
return;
}
res.json(source);
} catch (error) {
sendError(res, error, "Unknown source error");
}
});
app.get("/sources/:sourceId/versions", async (req, res) => {
const repository = requireCatalog(res);
if (!repository) return;
try {
res.json(await repository.listVersions(String(req.params.sourceId)));
} catch (error) {
sendError(res, error, "Unknown versions error");
}
});
app.get("/sources/:sourceId/versions/:versionId", async (req, res) => {
const repository = requireCatalog(res);
if (!repository) return;
try {
const version = await repository.getVersion(String(req.params.sourceId), String(req.params.versionId));
if (!version) {
res.status(404).json({ ok: false, error: "Version not found" });
return;
}
res.json(version);
} catch (error) {
sendError(res, error, "Unknown version error");
}
});
@ -128,14 +226,23 @@ export function createApp() {
app.post("/ingest", async (req, res) => {
try {
const result = await ingestService.ingest(req.body);
res.status(202).json(result);
res.status(env.knowledgeLifecycleEnforced ? 201 : 202).json(result);
} catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown ingest error" });
sendError(res, error, "Unknown ingest error");
}
});
app.post("/cleanup", async (req, res) => {
try {
if (env.knowledgeLifecycleEnforced) {
if (!requireLifecycleAdmin(req, res)) return;
res.status(410).json({
ok: false,
error: "POST /cleanup is deprecated while knowledge lifecycle is enforced; purge non-active versions with DELETE /sources/:sourceId/versions/:versionId.",
code: "CLEANUP_DEPRECATED_UNDER_LIFECYCLE"
});
return;
}
if (!req.body.scope) {
res.status(400).json({ ok: false, error: "Missing scope for cleanup" });
return;
@ -155,7 +262,100 @@ export function createApp() {
const result = await ingestService.cleanup(scope);
res.status(200).json({ ok: true, ...result });
} catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown cleanup error" });
sendError(res, error, "Unknown cleanup error");
}
});
app.post("/sources/:sourceId/versions/:versionId/activate", async (req, res) => {
if (!requireLifecycleAdmin(req, res)) return;
const repository = requireCatalog(res);
if (!repository) return;
try {
const version = await repository.activateVersion(String(req.params.sourceId), String(req.params.versionId), req.body.expectedActiveVersionId);
res.json(version);
} catch (error) {
sendError(res, error, "Unknown activation error");
}
});
app.post("/sources/:sourceId/rollback", async (req, res) => {
if (!requireLifecycleAdmin(req, res)) return;
const repository = requireCatalog(res);
if (!repository) return;
try {
if (!req.body.targetVersionId || !req.body.expectedActiveVersionId) {
res.status(400).json({ ok: false, error: "targetVersionId and expectedActiveVersionId are required" });
return;
}
const version = await repository.rollback(String(req.params.sourceId), String(req.body.targetVersionId), String(req.body.expectedActiveVersionId));
res.json(version);
} catch (error) {
sendError(res, error, "Unknown rollback error");
}
});
app.post("/sources/:sourceId/versions/:versionId/retry", async (req, res) => {
if (!requireLifecycleAdmin(req, res)) return;
const repository = requireCatalog(res);
if (!repository) return;
try {
const version = await repository.getVersion(String(req.params.sourceId), String(req.params.versionId));
if (!version) {
res.status(404).json({ ok: false, error: "Version not found" });
return;
}
if (!["failed", "pending", "indexing"].includes(version.state)) {
res.status(409).json({ ok: false, error: "Only failed or orphaned pending/indexing versions can be retried" });
return;
}
if (req.body.sourceType && req.body.sourceRef) {
if (["pending", "indexing"].includes(version.state)) {
await repository.markFailed(version.versionId, "EXPLICIT_RETRY_REPLACED", "Explicit retry requested; the orphaned version was closed before creating a replacement.");
}
const result = await ingestService.ingest({
sourceId: version.sourceId,
sourceType: req.body.sourceType,
sourceRef: String(req.body.sourceRef),
readPath: req.body.readPath ? String(req.body.readPath) : undefined,
mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
tags: Array.isArray(req.body.tags) ? req.body.tags.map(String) : version.tags,
activate: req.body.activate === undefined ? true : Boolean(req.body.activate),
expectedActiveVersionId: req.body.expectedActiveVersionId === undefined ? null : req.body.expectedActiveVersionId
});
res.status(201).json(result);
return;
}
res.status(202).json({
ok: true,
retryRequired: true,
sourceId: version.sourceId,
versionId: version.versionId,
message: "Explicit retry is safe only by re-submitting the original source to /ingest with the same sourceId and expectedActiveVersionId. Temporary uploads without durable originals cannot be retried in place."
});
} catch (error) {
sendError(res, error, "Unknown retry error");
}
});
app.delete("/sources/:sourceId/versions/:versionId", async (req, res) => {
if (!requireLifecycleAdmin(req, res)) return;
const repository = requireCatalog(res);
if (!repository) return;
try {
const sourceId = String(req.params.sourceId);
const versionId = String(req.params.versionId);
await repository.withSourceLock(sourceId, async () => repository.withVersionExclusiveLock(versionId, async () => {
await repository.markPurging(sourceId, versionId);
await vectorStore.deleteVersionPoints(versionId);
const remaining = await vectorStore.countVersionPoints(versionId);
if (remaining !== 0) {
throw new CatalogError("Version purge did not delete all points", 503, "VERSION_PURGE_INCOMPLETE");
}
await repository.markPurged(versionId);
}));
res.json({ ok: true, purged: true, versionId });
} catch (error) {
sendError(res, error, "Unknown purge error");
}
});
@ -181,7 +381,6 @@ export function createApp() {
let result;
if (isZipFolder) {
// Logica para carpetas subidas como ZIP
extractDirPath = await mkdtemp(path.join(tempDirBase, "rag-upload-"));
const zip = new AdmZip(tempFilePath);
zip.extractAllTo(extractDirPath, true);
@ -189,29 +388,32 @@ export function createApp() {
result = await ingestService.ingest({
sourceId: req.body.sourceId ? String(req.body.sourceId) : undefined,
sourceType: "folder",
sourceRef: req.file.originalname.replace(/\.zip$/i, ""), // Usamos el nombre original sin .zip como ref
sourceRef: req.file.originalname.replace(/\.zip$/i, ""),
readPath: extractDirPath,
mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
tags
tags,
activate: req.body.activate === undefined ? undefined : req.body.activate === "true",
expectedActiveVersionId: req.body.expectedActiveVersionId === "null" ? null : req.body.expectedActiveVersionId ? String(req.body.expectedActiveVersionId) : undefined
});
} else {
// Logica para archivos sueltos
result = await ingestService.ingest({
sourceId: req.body.sourceId ? String(req.body.sourceId) : undefined,
sourceType: "file",
sourceRef: req.file.originalname,
readPath: tempFilePath,
mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
tags
tags,
activate: req.body.activate === undefined ? undefined : req.body.activate === "true",
expectedActiveVersionId: req.body.expectedActiveVersionId === "null" ? null : req.body.expectedActiveVersionId ? String(req.body.expectedActiveVersionId) : undefined
});
}
res.status(202).json({
res.status(env.knowledgeLifecycleEnforced ? 201 : 202).json({
...result,
uploadedResource: req.file.originalname
});
} catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown upload ingest error" });
sendError(res, error, "Unknown upload ingest error");
} finally {
if (tempFilePath) {
await unlink(tempFilePath).catch(() => undefined);
@ -284,7 +486,7 @@ export function createApp() {
res.json(result);
} catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown retrieve error" });
sendError(res, error, "Unknown retrieve error");
}
});
@ -335,7 +537,7 @@ export function createApp() {
res.json(result);
} catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown answer error" });
sendError(res, error, "Unknown answer error");
}
});

View file

@ -1,7 +1,9 @@
import { config as loadEnv } from "dotenv";
loadEnv();
loadEnv({ path: ".env.local", override: true });
if (process.env.NODE_ENV !== "test") {
loadEnv();
loadEnv({ path: ".env.local", override: true });
}
function requireEnv(name: string, fallback?: string): string {
const value = process.env[name] ?? fallback;
@ -11,6 +13,24 @@ function requireEnv(name: string, fallback?: string): string {
return value;
}
function optionalEnv(...names: string[]): string {
for (const name of names) {
const value = process.env[name];
if (value) {
return value;
}
}
return "";
}
function booleanEnv(name: string, fallback: boolean): boolean {
const value = process.env[name];
if (value === undefined) {
return fallback;
}
return ["1", "true", "yes", "on"].includes(value.toLowerCase());
}
export const env = {
nodeEnv: process.env.NODE_ENV ?? "development",
port: Number(process.env.PORT ?? 3000),
@ -25,5 +45,12 @@ export const env = {
answerProvider: requireEnv("ANSWER_PROVIDER", "openrouter"),
answerModel: requireEnv("ANSWER_MODEL", "openai/gpt-4.1-mini"),
answerBaseUrl: requireEnv("ANSWER_BASE_URL", "https://openrouter.ai/api/v1"),
answerApiKey: process.env.ANSWER_API_KEY ?? process.env.EMBEDDING_API_KEY ?? ""
answerApiKey: process.env.ANSWER_API_KEY ?? process.env.EMBEDDING_API_KEY ?? "",
postgresUrl: optionalEnv("POSTGRES_URL", "DATABASE_URL"),
postgresSsl: booleanEnv("POSTGRES_SSL", false),
lifecycleAdminToken: process.env.LIFECYCLE_ADMIN_TOKEN ?? "",
knowledgeLifecycleEnforced: booleanEnv("KNOWLEDGE_LIFECYCLE_ENFORCED", false),
ingestWritesEnabled: booleanEnv("INGEST_WRITES_ENABLED", true),
lifecycleReconcileIntervalMs: Number(process.env.LIFECYCLE_RECONCILE_INTERVAL_MS ?? 300000),
lifecycleIndexingStaleTimeoutMs: Number(process.env.LIFECYCLE_INDEXING_STALE_TIMEOUT_MS ?? 1800000)
} as const;

View file

@ -116,6 +116,8 @@ export class AnswerService {
citations: retrieved.items.map((item) => ({
chunkId: item.chunkId,
documentId: item.documentId,
sourceVersionId: item.sourceVersionId,
sourceVersionNumber: item.sourceVersionNumber,
title: item.title,
sectionTitle: item.sectionTitle,
startLine: item.startLine,

View file

@ -0,0 +1,94 @@
import pg from "pg";
import { env } from "../../config/env.js";
const { Pool } = pg;
export type PgPool = pg.Pool;
export type PgPoolClient = pg.PoolClient;
let sharedPool: PgPool | undefined;
export function hasCatalogConnection(): boolean {
return Boolean(env.postgresUrl);
}
export function buildCatalogPool(): PgPool {
if (!env.postgresUrl) {
throw new Error("Missing POSTGRES_URL or DATABASE_URL for knowledge catalog");
}
return new Pool({
connectionString: env.postgresUrl,
ssl: env.postgresSsl ? { rejectUnauthorized: false } : undefined
});
}
export function getCatalogPool(): PgPool | undefined {
if (!hasCatalogConnection()) {
return undefined;
}
sharedPool ??= buildCatalogPool();
return sharedPool;
}
export async function withTransaction<T>(pool: PgPool, handler: (client: PgPoolClient) => Promise<T>): Promise<T> {
const client = await pool.connect();
try {
await client.query("BEGIN");
await client.query("SET CONSTRAINTS ALL DEFERRED");
const result = await handler(client);
await client.query("COMMIT");
return result;
} catch (error) {
await client.query("ROLLBACK").catch(() => undefined);
throw error;
} finally {
client.release();
}
}
export async function withCatalogAdvisoryLock<T>(pool: PgPool, lockName: string, handler: () => Promise<T>): Promise<T> {
const client = await pool.connect();
try {
await client.query("SELECT pg_advisory_lock(hashtext($1))", [lockName]);
return await handler();
} finally {
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [lockName]).catch(() => undefined);
client.release();
}
}
export async function withCatalogTryAdvisoryLock<T>(pool: PgPool, lockName: string, handler: () => Promise<T>): Promise<T | undefined> {
const client = await pool.connect();
let locked = false;
try {
const result = await client.query<{ locked: boolean }>("SELECT pg_try_advisory_lock(hashtext($1)) AS locked", [lockName]);
locked = Boolean(result.rows[0]?.locked);
if (!locked) {
return undefined;
}
return await handler();
} finally {
if (locked) {
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [lockName]).catch(() => undefined);
}
client.release();
}
}
export async function withCatalogAdvisorySharedLocks<T>(pool: PgPool, lockNames: string[], handler: () => Promise<T>): Promise<T> {
const names = [...new Set(lockNames)].sort((left, right) => Buffer.compare(Buffer.from(left), Buffer.from(right)));
const client = await pool.connect();
try {
for (const name of names) {
await client.query("SELECT pg_advisory_lock_shared(hashtext($1))", [name]);
}
return await handler();
} finally {
for (const name of names.reverse()) {
await client.query("SELECT pg_advisory_unlock_shared(hashtext($1))", [name]).catch(() => undefined);
}
client.release();
}
}

View file

@ -0,0 +1,9 @@
export class CatalogError extends Error {
constructor(
message: string,
readonly statusCode = 500,
readonly code = "CATALOG_ERROR"
) {
super(message);
}
}

View file

@ -0,0 +1,150 @@
import { buildDocumentId, buildMetadataHash, hashOrderedPairs, sha256Hex } from "../../shared/utils/ids.js";
import type { SourceType } from "../../shared/types/rag.js";
export type LegacyPointId = string | number;
export interface LegacyPointSnapshot {
id: LegacyPointId;
payload: Record<string, unknown>;
}
export interface LegacyDocumentPlan {
documentId: string;
documentKey: string;
originalHash: string;
contentHash: string;
mimeType: string;
title: string;
chunkCount: number;
}
export interface LegacySourcePlan {
sourceId: string;
sourceType: SourceType;
sourceRef: string;
pointIds: LegacyPointId[];
tags: string[];
documents: LegacyDocumentPlan[];
originalManifestHash: string;
sourceContentHash: string;
metadataHash: string;
embeddingProvider: string;
embeddingModel: string;
embeddingDimensions: number;
blockedReasons: string[];
}
export interface LegacyMigrationPlan {
ok: boolean;
sources: LegacySourcePlan[];
blockedReasons: string[];
}
function stringValue(value: unknown): string {
return typeof value === "string" ? value.trim() : "";
}
function stringArray(value: unknown): string[] {
return Array.isArray(value) ? value.map(String).filter(Boolean).sort() : [];
}
function unique(values: string[]): string[] {
return [...new Set(values.filter(Boolean))];
}
function sortedByUtf8<T>(values: T[], getKey: (value: T) => string): T[] {
return [...values].sort((left, right) => Buffer.compare(Buffer.from(getKey(left), "utf8"), Buffer.from(getKey(right), "utf8")));
}
export function buildLegacyMigrationPlan(points: LegacyPointSnapshot[]): LegacyMigrationPlan {
const groups = new Map<string, LegacyPointSnapshot[]>();
const globalBlocked: string[] = [];
for (const point of points) {
const sourceId = stringValue(point.payload.source_id);
if (!sourceId) {
globalBlocked.push(`Point ${String(point.id)} is missing source_id.`);
continue;
}
const entries = groups.get(sourceId) ?? [];
entries.push(point);
groups.set(sourceId, entries);
}
const sources = sortedByUtf8([...groups.entries()], ([sourceId]) => sourceId).map(([sourceId, sourcePoints]) => {
const blockedReasons: string[] = [];
const sourceRefs = unique(sourcePoints.map((point) => stringValue(point.payload.source_ref)));
const sourceTypes = unique(sourcePoints.map((point) => stringValue(point.payload.source_type)));
const providers = unique(sourcePoints.map((point) => stringValue(point.payload.embedding_provider)));
const models = unique(sourcePoints.map((point) => stringValue(point.payload.embedding_model)));
const dimensions = unique(sourcePoints.map((point) => String(Number(point.payload.embedding_dimensions ?? 0))).filter((value) => value !== "0"));
const tagSnapshot = unique(sourcePoints.flatMap((point) => stringArray(point.payload.tags)));
if (sourceRefs.length !== 1) blockedReasons.push("Source has zero or multiple source_ref values.");
if (sourceTypes.length !== 1 || !["file", "folder"].includes(sourceTypes[0])) blockedReasons.push("Source has no unique file/folder source_type.");
if (providers.length !== 1) blockedReasons.push("Source has no unique embedding provider.");
if (models.length !== 1) blockedReasons.push("Source has no unique embedding model.");
if (dimensions.length !== 1) blockedReasons.push("Source has no unique non-zero embedding dimensions.");
const documentGroups = new Map<string, LegacyPointSnapshot[]>();
for (const point of sourcePoints) {
const documentId = stringValue(point.payload.document_id);
const documentKey = stringValue(point.payload.document_key);
const chunkId = stringValue(point.payload.chunk_id);
const content = stringValue(point.payload.content);
if (!documentId || !documentKey || !chunkId || !content) {
blockedReasons.push(`Point ${String(point.id)} has weak document identity or empty content.`);
continue;
}
if (documentId !== buildDocumentId(sourceId, documentKey)) {
blockedReasons.push(`Point ${String(point.id)} document_id does not match source_id and document_key.`);
}
const existing = documentGroups.get(documentId) ?? [];
existing.push(point);
documentGroups.set(documentId, existing);
}
const documents: LegacyDocumentPlan[] = sortedByUtf8([...documentGroups.entries()], ([documentId]) => documentId).map(([documentId, documentPoints]) => {
const documentKeys = unique(documentPoints.map((point) => stringValue(point.payload.document_key)));
const mimeTypes = unique(documentPoints.map((point) => stringValue(point.payload.mime_type) || "text/plain"));
const titles = unique(documentPoints.map((point) => stringValue(point.payload.title) || documentKeys[0]));
if (documentKeys.length !== 1) blockedReasons.push(`Document ${documentId} has no unique document_key.`);
if (mimeTypes.length !== 1) blockedReasons.push(`Document ${documentId} has no unique mime_type.`);
const orderedContents = sortedByUtf8(documentPoints, (point) => stringValue(point.payload.chunk_id))
.map((point) => stringValue(point.payload.content))
.join("\n");
const contentHash = sha256Hex(orderedContents);
return {
documentId,
documentKey: documentKeys[0] ?? "",
originalHash: contentHash,
contentHash,
mimeType: mimeTypes[0] ?? "text/plain",
title: titles[0] ?? documentKeys[0] ?? documentId,
chunkCount: documentPoints.length
};
});
const originalManifestHash = hashOrderedPairs(documents.map((document) => [document.documentKey, document.originalHash])) ?? "";
const sourceContentHash = hashOrderedPairs(documents.map((document) => [document.documentKey, document.contentHash])) ?? "";
return {
sourceId,
sourceType: sourceTypes[0] as SourceType,
sourceRef: sourceRefs[0] ?? "",
pointIds: sourcePoints.map((point) => point.id),
tags: tagSnapshot,
documents,
originalManifestHash,
sourceContentHash,
metadataHash: buildMetadataHash(tagSnapshot, { sourceType: sourceTypes[0], sourceRef: sourceRefs[0] }),
embeddingProvider: providers[0] ?? "",
embeddingModel: models[0] ?? "",
embeddingDimensions: Number(dimensions[0] ?? 0),
blockedReasons: unique(blockedReasons)
};
});
const blockedReasons = [...globalBlocked, ...sources.flatMap((source) => source.blockedReasons.map((reason) => `${source.sourceId}: ${reason}`))];
return { ok: blockedReasons.length === 0, sources, blockedReasons };
}

View file

@ -0,0 +1,11 @@
import { CatalogError } from "./errors.js";
export function normalizeExpectedActiveVersion(value: string | null | undefined, activate: boolean): string | null | undefined {
if (!activate) {
return undefined;
}
if (value === undefined) {
throw new CatalogError("expectedActiveVersionId is required when activate is true", 400, "EXPECTED_ACTIVE_VERSION_REQUIRED");
}
return value;
}

View file

@ -0,0 +1,90 @@
import { createHash } from "node:crypto";
import { readdir, readFile } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { buildCatalogPool, hasCatalogConnection, type PgPool } from "./client.js";
const GLOBAL_MIGRATION_LOCK = "rag:knowledge-lifecycle:migrations";
function sha256(value: string): string {
return createHash("sha256").update(value).digest("hex");
}
function resolveMigrationsDir(): string {
const currentFile = fileURLToPath(import.meta.url);
return path.resolve(path.dirname(currentFile), "../../../migrations");
}
export async function runKnowledgeLifecycleMigrations(pool: PgPool = buildCatalogPool()): Promise<void> {
const migrationsDir = resolveMigrationsDir();
const files = (await readdir(migrationsDir))
.filter((file) => /^\d+_.*\.sql$/.test(file))
.sort();
const client = await pool.connect();
try {
await client.query("SELECT pg_advisory_lock(hashtext($1))", [GLOBAL_MIGRATION_LOCK]);
await client.query(`
CREATE TABLE IF NOT EXISTS rag_schema_migrations (
name text PRIMARY KEY,
checksum char(64) NOT NULL,
applied_at timestamptz NOT NULL DEFAULT now()
)
`);
for (const file of files) {
const sql = await readFile(path.join(migrationsDir, file), "utf8");
const checksum = sha256(sql);
const existing = await client.query<{ checksum: string }>(
"SELECT checksum FROM rag_schema_migrations WHERE name = $1",
[file]
);
if (existing.rowCount && existing.rows[0].checksum !== checksum) {
throw new Error(`Migration checksum mismatch: ${file}`);
}
if (existing.rowCount) {
continue;
}
await client.query("BEGIN");
try {
await client.query(sql);
await client.query(
"INSERT INTO rag_schema_migrations(name, checksum) VALUES ($1, $2)",
[file, checksum]
);
await client.query("COMMIT");
} catch (error) {
await client.query("ROLLBACK").catch(() => undefined);
throw error;
}
}
} finally {
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [GLOBAL_MIGRATION_LOCK]).catch(() => undefined);
client.release();
}
}
async function main() {
if (!hasCatalogConnection()) {
console.warn("Knowledge lifecycle migrations skipped: POSTGRES_URL or DATABASE_URL is not configured.");
return;
}
const pool = buildCatalogPool();
try {
await runKnowledgeLifecycleMigrations(pool);
console.log("Knowledge lifecycle migrations completed.");
} finally {
await pool.end();
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
main().catch((error) => {
console.error(error instanceof Error ? error.message : error);
process.exit(1);
});
}

View file

@ -0,0 +1,123 @@
import { env } from "../../config/env.js";
import type { VectorStoreClient } from "../vectorstore/client.js";
import type { CatalogRepository } from "./repository.js";
export interface ReconcilerStatus {
ok: boolean;
lastRunAt?: string;
lastError?: string;
inconsistentSources: string[];
orphanedVersionsRecovered?: number;
orphanedVersionsFailed?: number;
invariantViolations?: string[];
}
export class KnowledgeLifecycleReconciler {
private status: ReconcilerStatus = { ok: true, inconsistentSources: [] };
private timer: NodeJS.Timeout | undefined;
constructor(
private readonly catalog: CatalogRepository | undefined,
private readonly vectorStore: VectorStoreClient
) {}
getStatus(): ReconcilerStatus {
return this.status;
}
start(): void {
if (!env.knowledgeLifecycleEnforced || !this.catalog || this.timer) {
return;
}
void this.runOnce();
this.timer = setInterval(() => {
void this.runOnce();
}, env.lifecycleReconcileIntervalMs);
this.timer.unref();
}
async runOnce(): Promise<ReconcilerStatus> {
if (!this.catalog) {
this.status = { ok: false, lastRunAt: new Date().toISOString(), lastError: "Knowledge catalog is not configured", inconsistentSources: [] };
return this.status;
}
const catalog = this.catalog;
try {
const lockedStatus = await catalog.withGlobalTryLock("rag:knowledge-lifecycle:reconciler", async () => {
const activeVersions = await catalog.resolveActiveVersions();
const inconsistentSources: string[] = [];
const invariantViolations = await catalog.validateActiveInvariant();
const orphanRecovery = await this.recoverOrphanedVersions(catalog);
for (const version of activeVersions) {
if (version.qdrantCollection !== env.qdrantCollection) {
inconsistentSources.push(version.sourceId);
await catalog.markNeedsReingest(version.sourceId);
continue;
}
await this.vectorStore.validateCollectionDimensions(version.embeddingDimensions);
const count = await this.vectorStore.countVersionPoints(version.sourceVersionId);
if (count !== version.expectedPointCount) {
inconsistentSources.push(version.sourceId);
await catalog.markNeedsReingest(version.sourceId);
}
}
return {
ok: inconsistentSources.length === 0 && invariantViolations.length === 0,
lastRunAt: new Date().toISOString(),
inconsistentSources,
orphanedVersionsRecovered: orphanRecovery.ready,
orphanedVersionsFailed: orphanRecovery.failed,
invariantViolations
};
});
this.status = lockedStatus ?? { ...this.status, lastRunAt: new Date().toISOString() };
return this.status;
} catch (error) {
this.status = {
ok: false,
lastRunAt: new Date().toISOString(),
lastError: error instanceof Error ? error.message : "Unknown reconciler error",
inconsistentSources: []
};
return this.status;
}
}
private async recoverOrphanedVersions(catalog: CatalogRepository): Promise<{ ready: number; failed: number }> {
const candidates = await catalog.listOrphanedIndexingCandidates(env.lifecycleIndexingStaleTimeoutMs);
let ready = 0;
let failed = 0;
for (const candidate of candidates) {
const recovered = await catalog.withSourceTryLock(candidate.sourceId, async () => catalog.withVersionTryLock(candidate.versionId, async () => {
const documentsComplete = await catalog.hasCompleteVersionDocuments(candidate.versionId, candidate.expectedDocumentCount, candidate.expectedPointCount);
const collectionMatches = candidate.qdrantCollection === env.qdrantCollection;
if (collectionMatches) {
await this.vectorStore.validateCollectionDimensions(candidate.embeddingDimensions);
}
const pointCount = collectionMatches ? await this.vectorStore.countVersionPoints(candidate.versionId) : 0;
if (collectionMatches && documentsComplete && pointCount === candidate.expectedPointCount) {
const changed = await catalog.recoverVersionReady(candidate.versionId, pointCount);
if (changed) {
return "ready" as const;
}
}
await catalog.markFailed(candidate.versionId, "ORPHANED_INDEXING_INCOMPLETE", "Stale pending/indexing version has incomplete documents, dimensions, collection, or point count.");
return "failed" as const;
}));
if (recovered === "ready") {
ready += 1;
} else if (recovered === "failed") {
failed += 1;
}
}
return { ready, failed };
}
}

View file

@ -0,0 +1,641 @@
import type { AvailableScope, ChunkMode, IngestSourceInput, SourceType, SourceVersionState, RetrieveScope } from "../../shared/types/rag.js";
import { withCatalogAdvisoryLock, withCatalogAdvisorySharedLocks, withCatalogTryAdvisoryLock, withTransaction, type PgPool, type PgPoolClient } from "./client.js";
import { CatalogError } from "./errors.js";
import { normalizeExpectedActiveVersion } from "./lifecycle.js";
export interface CatalogDocumentInput {
documentId: string;
documentKey: string;
originalHash: string;
originalHashKind: "bytes" | "legacy-derived";
contentHash: string | null;
mimeType: string;
title: string;
chunkCount: number;
}
export interface CreateVersionInput {
sourceId: string;
sourceType: SourceType;
sourceRef: string;
originalManifestHash: string;
sourceContentHash: string | null;
processingFingerprint: string;
metadataHash: string;
tags: string[];
activateRequested: boolean;
embeddingProvider: string;
embeddingModel: string;
embeddingDimensions: number;
qdrantCollection: string;
expectedDocumentCount: number;
expectedPointCount: number;
documents: CatalogDocumentInput[];
}
export interface CatalogVersionRow {
versionId: string;
sourceId: string;
versionNumber: number;
previousVersionId: string | null;
state: SourceVersionState;
tags: string[];
sourceContentHash: string | null;
processingFingerprint: string;
metadataHash: string;
embeddingProvider: string;
embeddingModel: string;
embeddingDimensions: number;
expectedDocumentCount: number;
expectedPointCount: number;
verifiedPointCount: number;
qdrantCollection: string;
}
export interface ActiveVersionScope {
sourceId: string;
sourceVersionId: string;
sourceVersionNumber: number;
expectedPointCount: number;
embeddingDimensions: number;
qdrantCollection: string;
}
export interface OrphanedVersionCandidate {
sourceId: string;
versionId: string;
expectedDocumentCount: number;
expectedPointCount: number;
embeddingDimensions: number;
qdrantCollection: string;
}
type VersionDbRow = {
version_id: string;
source_id: string;
version_number: string | number;
previous_version_id: string | null;
state: SourceVersionState;
tags: string[];
source_content_hash: string | null;
processing_fingerprint: string;
metadata_hash: string;
embedding_provider: string;
embedding_model: string;
embedding_dimensions: string | number;
expected_document_count: string | number;
expected_point_count: string | number;
verified_point_count: string | number;
qdrant_collection: string;
};
function toVersion(row: VersionDbRow): CatalogVersionRow {
return {
versionId: row.version_id,
sourceId: row.source_id,
versionNumber: Number(row.version_number),
previousVersionId: row.previous_version_id,
state: row.state,
tags: row.tags ?? [],
sourceContentHash: row.source_content_hash,
processingFingerprint: row.processing_fingerprint,
metadataHash: row.metadata_hash,
embeddingProvider: row.embedding_provider,
embeddingModel: row.embedding_model,
embeddingDimensions: Number(row.embedding_dimensions),
expectedDocumentCount: Number(row.expected_document_count),
expectedPointCount: Number(row.expected_point_count),
verifiedPointCount: Number(row.verified_point_count),
qdrantCollection: row.qdrant_collection
};
}
export class CatalogRepository {
constructor(private readonly pool: PgPool) {}
async healthcheck(): Promise<{ ok: boolean; kind: "postgres" }> {
try {
await this.pool.query("SELECT 1");
return { ok: true, kind: "postgres" };
} catch {
return { ok: false, kind: "postgres" };
}
}
async withSourceLock<T>(sourceId: string, handler: () => Promise<T>): Promise<T> {
return withCatalogAdvisoryLock(this.pool, `rag:source:${sourceId}`, handler);
}
async withSourceTryLock<T>(sourceId: string, handler: () => Promise<T>): Promise<T | undefined> {
return withCatalogTryAdvisoryLock(this.pool, `rag:source:${sourceId}`, handler);
}
async withVersionSharedLocks<T>(versionIds: string[], handler: () => Promise<T>): Promise<T> {
return withCatalogAdvisorySharedLocks(this.pool, versionIds.map((versionId) => `rag:version:${versionId}`), handler);
}
async withVersionExclusiveLock<T>(versionId: string, handler: () => Promise<T>): Promise<T> {
return withCatalogAdvisoryLock(this.pool, `rag:version:${versionId}`, handler);
}
async withVersionTryLock<T>(versionId: string, handler: () => Promise<T>): Promise<T | undefined> {
return withCatalogTryAdvisoryLock(this.pool, `rag:version:${versionId}`, handler);
}
async withGlobalTryLock<T>(lockName: string, handler: () => Promise<T>): Promise<T | undefined> {
return withCatalogTryAdvisoryLock(this.pool, lockName, handler);
}
async isMaintenanceEnabled(client?: PgPoolClient, lockRow = false): Promise<boolean> {
const executor = client ?? this.pool;
const result = await executor.query<{ maintenance: boolean }>(
`SELECT maintenance FROM rag_system_state WHERE singleton = true${lockRow ? " FOR UPDATE" : ""}`
);
return Boolean(result.rows[0]?.maintenance);
}
async beginAttempt(source: IngestSourceInput & { sourceId: string }): Promise<string> {
return withTransaction(this.pool, async (client) => {
if (await this.isMaintenanceEnabled(client, true)) {
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
}
const sourceResult = await client.query<{ source_type: SourceType; source_ref: string }>(
`INSERT INTO rag_sources(source_id, source_type, source_ref)
VALUES ($1, $2, $3)
ON CONFLICT (source_id) DO NOTHING
RETURNING source_type, source_ref`,
[source.sourceId, source.sourceType, source.sourceRef]
);
const existing = sourceResult.rowCount
? sourceResult.rows[0]
: (await client.query<{ source_type: SourceType; source_ref: string }>(
"SELECT source_type, source_ref FROM rag_sources WHERE source_id = $1",
[source.sourceId]
)).rows[0];
if (!existing || existing.source_type !== source.sourceType || existing.source_ref !== source.sourceRef) {
throw new CatalogError("sourceId already exists with a different sourceType or sourceRef", 409, "SOURCE_IDENTITY_CONFLICT");
}
const result = await client.query<{ attempt_id: string }>(
`INSERT INTO rag_ingestion_attempts(source_id, state, input_locator)
VALUES ($1, 'received', $2)
RETURNING attempt_id`,
[source.sourceId, source.readPath ?? source.sourceRef]
);
return result.rows[0].attempt_id;
});
}
async updateAttempt(attemptId: string, values: { state: "processing" | "completed" | "failed"; versionId?: string | null; inputHash?: string | null; errorCode?: string; errorDetail?: string }): Promise<void> {
await this.pool.query(
`UPDATE rag_ingestion_attempts
SET state = $2,
version_id = COALESCE($3, version_id),
input_hash = COALESCE($4, input_hash),
error_code = $5,
error_detail = $6,
completed_at = CASE WHEN $2 IN ('completed', 'failed') THEN now() ELSE completed_at END
WHERE attempt_id = $1`,
[attemptId, values.state, values.versionId ?? null, values.inputHash ?? null, values.errorCode ?? null, values.errorDetail ?? null]
);
}
async findReusableVersion(input: Pick<CreateVersionInput, "sourceId" | "sourceContentHash" | "processingFingerprint" | "metadataHash">): Promise<CatalogVersionRow | undefined> {
if (!input.sourceContentHash) {
return undefined;
}
const result = await this.pool.query<VersionDbRow>(
`SELECT * FROM rag_source_versions
WHERE source_id = $1
AND source_content_hash = $2
AND processing_fingerprint = $3
AND metadata_hash = $4
AND state NOT IN ('failed', 'rejected', 'purged')
ORDER BY version_number DESC
LIMIT 1`,
[input.sourceId, input.sourceContentHash, input.processingFingerprint, input.metadataHash]
);
return result.rows[0] ? toVersion(result.rows[0]) : undefined;
}
async createPendingVersion(input: CreateVersionInput): Promise<CatalogVersionRow> {
return withTransaction(this.pool, async (client) => {
if (await this.isMaintenanceEnabled(client, true)) {
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
}
await client.query(
`INSERT INTO rag_sources(source_id, source_type, source_ref)
VALUES ($1, $2, $3)
ON CONFLICT (source_id) DO NOTHING`,
[input.sourceId, input.sourceType, input.sourceRef]
);
const source = await client.query<{ active_version_id: string | null; source_type: SourceType; source_ref: string }>(
"SELECT active_version_id, source_type, source_ref FROM rag_sources WHERE source_id = $1 FOR UPDATE",
[input.sourceId]
);
if (!source.rowCount) {
throw new CatalogError("Source not found after creation", 500, "SOURCE_CREATE_FAILED");
}
if (source.rows[0].source_type !== input.sourceType || source.rows[0].source_ref !== input.sourceRef) {
throw new CatalogError("sourceId already exists with a different sourceType or sourceRef", 409, "SOURCE_IDENTITY_CONFLICT");
}
const baseActiveVersionId = source.rows[0].active_version_id;
const versionNumberResult = await client.query<{ next_version_number: string }>(
"SELECT COALESCE(MAX(version_number), 0) + 1 AS next_version_number FROM rag_source_versions WHERE source_id = $1",
[input.sourceId]
);
const versionNumber = Number(versionNumberResult.rows[0].next_version_number);
const versionResult = await client.query<VersionDbRow>(
`INSERT INTO rag_source_versions(
source_id, version_number, previous_version_id, state, original_manifest_hash,
source_content_hash, processing_fingerprint, metadata_hash, tags, activate_requested,
base_active_version_id, embedding_provider, embedding_model, embedding_dimensions,
qdrant_collection, expected_document_count, expected_point_count
) VALUES ($1, $2, $3, 'pending', $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16)
RETURNING *`,
[
input.sourceId,
versionNumber,
baseActiveVersionId,
input.originalManifestHash,
input.sourceContentHash,
input.processingFingerprint,
input.metadataHash,
input.tags,
input.activateRequested,
baseActiveVersionId,
input.embeddingProvider,
input.embeddingModel,
input.embeddingDimensions,
input.qdrantCollection,
input.expectedDocumentCount,
input.expectedPointCount
]
);
const version = toVersion(versionResult.rows[0]);
for (const document of input.documents) {
await client.query(
`INSERT INTO rag_version_documents(
version_id, document_id, document_key, original_hash, original_hash_kind,
content_hash, mime_type, title, index_state, chunk_count
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'pending', $9)`,
[
version.versionId,
document.documentId,
document.documentKey,
document.originalHash,
document.originalHashKind,
document.contentHash,
document.mimeType,
document.title,
document.chunkCount
]
);
}
return version;
});
}
async markIndexing(versionId: string): Promise<void> {
const result = await this.pool.query(
`UPDATE rag_source_versions SET state = 'indexing', indexing_started_at = now() WHERE version_id = $1 AND state = 'pending'`,
[versionId]
);
if (result.rowCount !== 1) {
throw new CatalogError("Version could not transition from pending to indexing", 409, "INVALID_VERSION_STATE");
}
await this.pool.query("UPDATE rag_version_documents SET index_state = 'indexing' WHERE version_id = $1", [versionId]);
}
async markReady(versionId: string, verifiedPointCount: number): Promise<CatalogVersionRow> {
return withTransaction(this.pool, async (client) => {
await client.query("UPDATE rag_version_documents SET index_state = 'ready' WHERE version_id = $1", [versionId]);
const result = await client.query<VersionDbRow>(
`UPDATE rag_source_versions
SET state = 'ready', verified_point_count = $2, ready_at = now()
WHERE version_id = $1 AND state = 'indexing' AND expected_point_count = $2
RETURNING *`,
[versionId, verifiedPointCount]
);
if (result.rowCount !== 1) {
throw new CatalogError("Version could not transition from indexing to ready", 409, "INVALID_VERSION_STATE");
}
return toVersion(result.rows[0]);
});
}
async markFailed(versionId: string | undefined, code: string, detail: string): Promise<void> {
if (!versionId) {
return;
}
const result = await this.pool.query(
`UPDATE rag_source_versions
SET state = 'failed', error_code = $2, error_detail = $3
WHERE version_id = $1 AND state IN ('pending', 'indexing')`,
[versionId, code, detail.slice(0, 2000)]
);
if (result.rowCount === 1) {
await this.pool.query("UPDATE rag_version_documents SET index_state = 'failed', error_detail = $2 WHERE version_id = $1", [versionId, detail.slice(0, 2000)]);
}
}
async activateVersion(sourceId: string, versionId: string, expectedActiveVersionId: string | null | undefined): Promise<CatalogVersionRow> {
const expected = normalizeExpectedActiveVersion(expectedActiveVersionId, true);
return withTransaction(this.pool, async (client) => {
if (await this.isMaintenanceEnabled(client, true)) {
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
}
const source = await client.query<{ active_version_id: string | null }>(
"SELECT active_version_id FROM rag_sources WHERE source_id = $1 FOR UPDATE",
[sourceId]
);
if (!source.rowCount) {
throw new CatalogError("Source not found", 404, "SOURCE_NOT_FOUND");
}
const currentActive = source.rows[0].active_version_id;
if (currentActive !== expected) {
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
}
const version = await client.query<VersionDbRow>(
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2 FOR UPDATE",
[sourceId, versionId]
);
if (!version.rowCount) {
throw new CatalogError("Version not found", 404, "VERSION_NOT_FOUND");
}
if (!["ready", "superseded", "active"].includes(version.rows[0].state)) {
throw new CatalogError("Only ready or superseded versions can be activated", 409, "VERSION_NOT_ACTIVATABLE");
}
if (version.rows[0].state === "active") {
if (currentActive !== versionId) {
throw new CatalogError("Active version invariant is inconsistent", 503, "ACTIVE_VERSION_INVARIANT_FAILED");
}
return toVersion(version.rows[0]);
}
if (currentActive) {
await client.query(
"UPDATE rag_source_versions SET state = 'superseded', superseded_at = now() WHERE source_id = $1 AND version_id = $2",
[sourceId, currentActive]
);
}
const updated = await client.query<VersionDbRow>(
`UPDATE rag_source_versions
SET state = 'active', activated_at = now(), error_code = NULL, error_detail = NULL
WHERE source_id = $1 AND version_id = $2
RETURNING *`,
[sourceId, versionId]
);
if (updated.rowCount !== 1) {
throw new CatalogError("Version activation failed", 500, "VERSION_ACTIVATION_FAILED");
}
await client.query("UPDATE rag_sources SET active_version_id = $2, needs_reingest = false WHERE source_id = $1", [sourceId, versionId]);
return toVersion(updated.rows[0]);
});
}
async assertActiveVersionPrecondition(sourceId: string, expectedActiveVersionId: string | null | undefined): Promise<void> {
const expected = normalizeExpectedActiveVersion(expectedActiveVersionId, true);
const result = await this.pool.query<{ active_version_id: string | null }>(
"SELECT active_version_id FROM rag_sources WHERE source_id = $1",
[sourceId]
);
if (!result.rowCount) {
throw new CatalogError("Source not found", 404, "SOURCE_NOT_FOUND");
}
if (result.rows[0].active_version_id !== expected) {
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
}
}
async listSources(): Promise<AvailableScope[]> {
const result = await this.pool.query<{
source_id: string;
source_ref: string;
tags: string[] | null;
active_version_id: string | null;
version_number: string | number | null;
state: SourceVersionState | null;
needs_reingest: boolean;
updated_at: Date;
}>(
`SELECT s.source_id, s.source_ref, v.tags, s.active_version_id, v.version_number, v.state, s.needs_reingest, s.updated_at
FROM rag_sources s
LEFT JOIN rag_source_versions v ON v.version_id = s.active_version_id
WHERE s.disabled_at IS NULL
ORDER BY s.source_ref ASC`
);
return result.rows.map((row) => ({
sourceId: row.source_id,
sourceRef: row.source_ref,
chunkModes: ["documental", "codigo"] as ChunkMode[],
tags: row.tags ?? [],
activeVersionId: row.active_version_id,
activeVersionNumber: row.version_number === null ? null : Number(row.version_number),
state: row.state,
needsReingest: row.needs_reingest,
updatedAt: row.updated_at.toISOString()
}));
}
async getSource(sourceId: string): Promise<AvailableScope | undefined> {
return (await this.listSources()).find((source) => source.sourceId === sourceId);
}
async listVersions(sourceId: string): Promise<CatalogVersionRow[]> {
const result = await this.pool.query<VersionDbRow>(
"SELECT * FROM rag_source_versions WHERE source_id = $1 ORDER BY version_number DESC",
[sourceId]
);
return result.rows.map(toVersion);
}
async getVersion(sourceId: string, versionId: string): Promise<CatalogVersionRow | undefined> {
const result = await this.pool.query<VersionDbRow>(
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2",
[sourceId, versionId]
);
return result.rows[0] ? toVersion(result.rows[0]) : undefined;
}
async resolveActiveVersions(scope?: RetrieveScope): Promise<ActiveVersionScope[]> {
const mustTags = scope?.tags ?? [];
const result = await this.pool.query<{
source_id: string;
version_id: string;
version_number: string | number;
expected_point_count: string | number;
embedding_dimensions: string | number;
qdrant_collection: string;
}>(
`SELECT s.source_id, v.version_id, v.version_number, v.expected_point_count, v.embedding_dimensions, v.qdrant_collection
FROM rag_sources s
JOIN rag_source_versions v ON v.version_id = s.active_version_id AND v.state = 'active'
WHERE s.disabled_at IS NULL
AND ($1::text IS NULL OR s.source_id = $1)
AND ($2::text IS NULL OR s.source_ref = $2)
AND ($3::text[] IS NULL OR v.tags @> $3::text[])`,
[scope?.sourceId ?? null, scope?.sourceRef ?? null, mustTags.length ? mustTags : null]
);
return result.rows.map((row) => ({
sourceId: row.source_id,
sourceVersionId: row.version_id,
sourceVersionNumber: Number(row.version_number),
expectedPointCount: Number(row.expected_point_count),
embeddingDimensions: Number(row.embedding_dimensions),
qdrantCollection: row.qdrant_collection
}));
}
async markNeedsReingest(sourceId: string): Promise<void> {
await this.pool.query("UPDATE rag_sources SET needs_reingest = true WHERE source_id = $1", [sourceId]);
}
async rollback(sourceId: string, targetVersionId: string, expectedActiveVersionId: string): Promise<CatalogVersionRow> {
return this.withSourceLock(sourceId, async () => this.withVersionExclusiveLock(targetVersionId, async () => {
const version = await this.activateVersion(sourceId, targetVersionId, expectedActiveVersionId);
const source = await this.getSource(sourceId);
if (source?.activeVersionId !== targetVersionId) {
throw new CatalogError("Rollback verification failed", 503, "ROLLBACK_VERIFICATION_FAILED");
}
return version;
}));
}
async markPurging(sourceId: string, versionId: string): Promise<CatalogVersionRow> {
return withTransaction(this.pool, async (client) => {
const version = await client.query<VersionDbRow>(
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2 FOR UPDATE",
[sourceId, versionId]
);
if (!version.rowCount) {
throw new CatalogError("Version not found", 404, "VERSION_NOT_FOUND");
}
if (version.rows[0].state === "active") {
throw new CatalogError("Active versions cannot be purged", 409, "ACTIVE_VERSION_PURGE_FORBIDDEN");
}
if (!["ready", "superseded", "failed", "rejected", "purging"].includes(version.rows[0].state)) {
throw new CatalogError("Version state cannot be purged", 409, "VERSION_NOT_PURGEABLE");
}
const updated = await client.query<VersionDbRow>(
"UPDATE rag_source_versions SET state = 'purging' WHERE source_id = $1 AND version_id = $2 RETURNING *",
[sourceId, versionId]
);
return toVersion(updated.rows[0]);
});
}
async markPurged(versionId: string): Promise<void> {
await withTransaction(this.pool, async (client) => {
await client.query("DELETE FROM rag_version_documents WHERE version_id = $1", [versionId]);
const result = await client.query(
`UPDATE rag_source_versions
SET state = 'purged', verified_point_count = 0, artifact_state = 'retention_deleted', purged_at = now()
WHERE version_id = $1 AND state = 'purging'`,
[versionId]
);
if (result.rowCount !== 1) {
throw new CatalogError("Version could not transition from purging to purged", 409, "INVALID_VERSION_STATE");
}
});
}
async validateActiveInvariant(): Promise<string[]> {
const result = await this.pool.query<{ source_id: string }>(
`SELECT s.source_id
FROM rag_sources s
LEFT JOIN rag_source_versions v
ON v.version_id = s.active_version_id
AND v.source_id = s.source_id
AND v.state = 'active'
WHERE (s.active_version_id IS NOT NULL AND v.version_id IS NULL)
UNION
SELECT v.source_id
FROM rag_source_versions v
JOIN rag_sources s ON s.source_id = v.source_id
WHERE v.state = 'active'
AND s.active_version_id IS DISTINCT FROM v.version_id`
);
return result.rows.map((row) => row.source_id);
}
async listOrphanedIndexingCandidates(maxAgeMs = 1800000): Promise<OrphanedVersionCandidate[]> {
const result = await this.pool.query<{
version_id: string;
source_id: string;
expected_document_count: string | number;
expected_point_count: string | number;
embedding_dimensions: string | number;
qdrant_collection: string;
}>(
`SELECT version_id, source_id, expected_document_count, expected_point_count, embedding_dimensions, qdrant_collection
FROM rag_source_versions
WHERE (
state = 'pending'
AND created_at < now() - ($1::text || ' milliseconds')::interval
) OR (
state = 'indexing'
AND indexing_started_at IS NOT NULL
AND indexing_started_at < now() - ($1::text || ' milliseconds')::interval
)`,
[String(maxAgeMs)]
);
return result.rows.map((row) => ({
sourceId: row.source_id,
versionId: row.version_id,
expectedDocumentCount: Number(row.expected_document_count),
expectedPointCount: Number(row.expected_point_count),
embeddingDimensions: Number(row.embedding_dimensions),
qdrantCollection: row.qdrant_collection
}));
}
async hasCompleteVersionDocuments(versionId: string, expectedDocumentCount: number, expectedPointCount: number): Promise<boolean> {
const result = await this.pool.query<{
document_count: string | number;
point_count: string | number;
missing_content_count: string | number;
}>(
`SELECT COUNT(*) AS document_count,
COALESCE(SUM(chunk_count), 0) AS point_count,
COUNT(*) FILTER (WHERE content_hash IS NULL) AS missing_content_count
FROM rag_version_documents
WHERE version_id = $1`,
[versionId]
);
const row = result.rows[0];
return Number(row?.document_count ?? 0) === expectedDocumentCount
&& Number(row?.point_count ?? 0) === expectedPointCount
&& Number(row?.missing_content_count ?? 0) === 0;
}
async recoverVersionReady(versionId: string, verifiedPointCount: number): Promise<boolean> {
return withTransaction(this.pool, async (client) => {
await client.query("UPDATE rag_version_documents SET index_state = 'ready' WHERE version_id = $1", [versionId]);
const result = await client.query(
`UPDATE rag_source_versions
SET state = 'ready', verified_point_count = $2, ready_at = now(), error_code = NULL, error_detail = NULL
WHERE version_id = $1 AND state IN ('pending', 'indexing') AND expected_point_count = $2`,
[versionId, verifiedPointCount]
);
return result.rowCount === 1;
});
}
async recoverOrphanedIndexingVersions(maxAgeMs = 1800000): Promise<Array<{ versionId: string; sourceId: string; action: "ready" | "failed" }>> {
const candidates = await this.listOrphanedIndexingCandidates(maxAgeMs);
const recovered: Array<{ versionId: string; sourceId: string; action: "ready" | "failed" }> = [];
for (const candidate of candidates) {
await this.markFailed(candidate.versionId, "ORPHANED_INDEXING", "Recovered by reconciler after stale pending/indexing state.");
recovered.push({ versionId: candidate.versionId, sourceId: candidate.sourceId, action: "failed" });
}
return recovered;
}
}
export { CatalogError };

View file

@ -1,25 +1,72 @@
import path from "node:path";
import { readFile } from "node:fs/promises";
import { parseDocument, isSupportedDocument } from "../parsers/parser-registry.js";
import { chunkDocument, codeChunkingPolicy, documentalChunkingPolicy } from "../process/chunking.js";
import type { EmbeddingProvider } from "../embeddings/provider.js";
import type { VectorStoreClient } from "../vectorstore/client.js";
import type { IngestResult, IngestSourceInput, IngestedChunk, RetrieveScope } from "../../shared/types/rag.js";
import { buildChunkId, buildDocumentId, buildQdrantPointId, buildSourceId, normalizeDocumentKey } from "../../shared/utils/ids.js";
import {
buildChunkId,
buildDocumentId,
buildMetadataHash,
buildProcessingFingerprint,
buildQdrantPointId,
buildSourceId,
buildVersionedQdrantPointId,
hashOrderedPairs,
normalizeContentForHash,
normalizeDocumentKey,
sha256Hex
} from "../../shared/utils/ids.js";
import { listFilesRecursively } from "../../shared/utils/files.js";
import { env } from "../../config/env.js";
import { CatalogError, type CatalogDocumentInput, type CatalogRepository } from "../catalog/repository.js";
type PreparedDocument = CatalogDocumentInput & {
chunks: ReturnType<typeof chunkDocument>;
chunkMode: "documental" | "codigo";
content: string;
};
type OriginalDocument = {
filePath: string;
documentKey: string;
documentId: string;
bytes: Buffer;
originalHash: string;
};
export class IngestService {
constructor(
private readonly embeddingProvider: EmbeddingProvider,
private readonly vectorStore: VectorStoreClient
private readonly vectorStore: VectorStoreClient,
private readonly catalog?: CatalogRepository
) {}
async cleanup(scope: RetrieveScope): Promise<{ deleted: number }> {
if (!env.ingestWritesEnabled) {
throw new CatalogError("Ingest writes are disabled during knowledge catalog maintenance", 503, "INGEST_WRITES_DISABLED");
}
const count = await this.vectorStore.deleteScope(scope);
return { deleted: count };
}
async ingest(source: IngestSourceInput): Promise<IngestResult> {
if (!env.ingestWritesEnabled) {
throw new CatalogError("Ingest writes are disabled during knowledge catalog maintenance", 503, "INGEST_WRITES_DISABLED");
}
if (env.knowledgeLifecycleEnforced) {
if (!this.catalog) {
throw new CatalogError("Knowledge lifecycle is enforced but PostgreSQL is not configured", 503, "CATALOG_UNAVAILABLE");
}
return this.ingestWithLifecycle(source, this.catalog);
}
return this.ingestLegacy(source);
}
private async ingestLegacy(source: IngestSourceInput): Promise<IngestResult> {
const sourceId = source.sourceId ?? buildSourceId(source.sourceType, source.sourceRef);
const discoveredFiles = await this.resolveInputFiles(source);
const supportedFiles = discoveredFiles.filter(isSupportedDocument);
@ -79,6 +126,7 @@ export class IngestService {
return {
accepted: true,
sourceId,
source,
filesDiscovered: supportedFiles.length,
documentsProcessed,
@ -87,6 +135,292 @@ export class IngestService {
};
}
private async ingestWithLifecycle(source: IngestSourceInput, catalog: CatalogRepository): Promise<IngestResult> {
const activate = source.activate ?? true;
if (activate && source.expectedActiveVersionId === undefined) {
throw new CatalogError("expectedActiveVersionId is required when activate is true", 400, "EXPECTED_ACTIVE_VERSION_REQUIRED");
}
const sourceId = source.sourceId ?? buildSourceId(source.sourceType, source.sourceRef);
const attemptId = await catalog.beginAttempt({ ...source, sourceId });
let versionId: string | undefined;
let canMarkVersionFailed = false;
try {
await catalog.updateAttempt(attemptId, { state: "processing" });
const discoveredFiles = await this.resolveInputFiles(source);
const supportedFiles = discoveredFiles.filter(isSupportedDocument);
const originalDocuments = await this.readOriginalDocuments(source, sourceId, supportedFiles);
if (originalDocuments.length === 0) {
throw new CatalogError("The source has no supported documents", 422, "EMPTY_SOURCE");
}
const originalManifestHash = hashOrderedPairs(originalDocuments.map((document) => [document.documentKey, document.originalHash]));
if (!originalManifestHash) {
throw new CatalogError("Original manifest hash could not be calculated", 422, "ORIGINAL_MANIFEST_HASH_UNAVAILABLE");
}
const preparedDocuments = await this.prepareDocuments(originalDocuments);
if (preparedDocuments.length === 0) {
throw new CatalogError("The source has no supported documents with useful content", 422, "EMPTY_SOURCE");
}
const tags = source.tags ?? [];
const sourceContentHash = hashOrderedPairs(preparedDocuments.map((document) => [document.documentKey, document.contentHash]));
const embeddingDimensions = this.embeddingProvider.dimensions ?? 4096;
const processingFingerprint = buildProcessingFingerprint({
parserVersion: "native-v1",
normalizationPolicy: "bom-crlf-trim-final-lf-v1",
chunking: {
code: codeChunkingPolicy,
documental: documentalChunkingPolicy
},
embeddingProvider: this.embeddingProvider.providerName,
embeddingModel: this.embeddingProvider.modelName,
embeddingDimensions
});
const metadataHash = buildMetadataHash(tags, { sourceType: source.sourceType, sourceRef: source.sourceRef });
return await catalog.withSourceLock(sourceId, async () => {
const reusable = await catalog.findReusableVersion({ sourceId, sourceContentHash, processingFingerprint, metadataHash });
if (reusable?.state === "active") {
if (activate) {
await catalog.assertActiveVersionPrecondition(sourceId, source.expectedActiveVersionId);
}
await catalog.updateAttempt(attemptId, { state: "completed", versionId: reusable.versionId, inputHash: originalManifestHash });
return {
accepted: true,
sourceId,
versionId: reusable.versionId,
versionNumber: reusable.versionNumber,
state: reusable.state,
previousVersionId: reusable.previousVersionId,
filesDiscovered: supportedFiles.length,
documentsProcessed: reusable.expectedDocumentCount,
chunksStored: 0,
activated: true,
noOp: true,
collectionName: env.qdrantCollection
};
}
if (reusable && ["pending", "indexing", "review_required"].includes(reusable.state)) {
throw new CatalogError("A reusable version is already being indexed", 409, "REUSABLE_VERSION_IN_PROGRESS");
}
if (reusable && ["ready", "superseded"].includes(reusable.state)) {
if (!activate) {
await catalog.updateAttempt(attemptId, { state: "completed", versionId: reusable.versionId, inputHash: originalManifestHash });
return {
accepted: true,
sourceId,
versionId: reusable.versionId,
versionNumber: reusable.versionNumber,
state: reusable.state,
previousVersionId: reusable.previousVersionId,
filesDiscovered: supportedFiles.length,
documentsProcessed: reusable.expectedDocumentCount,
chunksStored: 0,
activated: false,
noOp: false,
collectionName: env.qdrantCollection
};
}
const activated = await catalog.activateVersion(sourceId, reusable.versionId, source.expectedActiveVersionId);
await catalog.updateAttempt(attemptId, { state: "completed", versionId: activated.versionId, inputHash: originalManifestHash });
return {
accepted: true,
sourceId,
versionId: activated.versionId,
versionNumber: activated.versionNumber,
state: activated.state,
previousVersionId: activated.previousVersionId,
filesDiscovered: supportedFiles.length,
documentsProcessed: activated.expectedDocumentCount,
chunksStored: 0,
activated: true,
noOp: false,
collectionName: env.qdrantCollection
};
}
const expectedPointCount = preparedDocuments.reduce((total, document) => total + document.chunkCount, 0);
const created = await catalog.createPendingVersion({
sourceId,
sourceType: source.sourceType,
sourceRef: source.sourceRef,
originalManifestHash,
sourceContentHash,
processingFingerprint,
metadataHash,
tags,
activateRequested: activate,
embeddingProvider: this.embeddingProvider.providerName,
embeddingModel: this.embeddingProvider.modelName,
embeddingDimensions,
qdrantCollection: env.qdrantCollection,
expectedDocumentCount: preparedDocuments.length,
expectedPointCount,
documents: preparedDocuments
});
versionId = created.versionId;
canMarkVersionFailed = true;
await catalog.updateAttempt(attemptId, { state: "processing", versionId, inputHash: originalManifestHash });
await catalog.markIndexing(versionId);
const chunksStored = await this.writeVersionedChunks(source, sourceId, created.versionId, created.versionNumber, preparedDocuments, processingFingerprint, tags, embeddingDimensions);
const verifiedPointCount = await this.vectorStore.countVersionPoints(created.versionId);
if (verifiedPointCount !== expectedPointCount || chunksStored !== expectedPointCount) {
throw new CatalogError("Indexed point count does not match the expected catalog count", 503, "SOURCE_VERSION_INCONSISTENT");
}
const ready = await catalog.markReady(created.versionId, verifiedPointCount);
canMarkVersionFailed = false;
const finalVersion = activate
? await catalog.activateVersion(sourceId, ready.versionId, source.expectedActiveVersionId)
: ready;
await catalog.updateAttempt(attemptId, { state: "completed", versionId: finalVersion.versionId, inputHash: originalManifestHash });
return {
accepted: true,
sourceId,
versionId: finalVersion.versionId,
versionNumber: finalVersion.versionNumber,
state: finalVersion.state,
previousVersionId: finalVersion.previousVersionId,
filesDiscovered: supportedFiles.length,
documentsProcessed: preparedDocuments.length,
chunksStored,
activated: finalVersion.state === "active",
noOp: false,
collectionName: env.qdrantCollection
};
});
} catch (error) {
const detail = error instanceof Error ? error.message : "Unknown lifecycle ingest error";
if (canMarkVersionFailed) {
await catalog.markFailed(versionId, error instanceof CatalogError ? error.code : "INGEST_FAILED", detail).catch(() => undefined);
}
await catalog.updateAttempt(attemptId, {
state: "failed",
versionId,
errorCode: error instanceof CatalogError ? error.code : "INGEST_FAILED",
errorDetail: detail
}).catch(() => undefined);
throw error;
}
}
private async readOriginalDocuments(source: IngestSourceInput, sourceId: string, files: string[]): Promise<OriginalDocument[]> {
const originals: OriginalDocument[] = [];
for (const filePath of files) {
const bytes = await readFile(filePath);
const documentKey = this.buildDocumentKey(source, filePath);
originals.push({
filePath,
documentKey,
documentId: buildDocumentId(sourceId, documentKey),
bytes,
originalHash: sha256Hex(bytes)
});
}
return originals.sort((left, right) => Buffer.compare(Buffer.from(left.documentKey, "utf8"), Buffer.from(right.documentKey, "utf8")));
}
private async prepareDocuments(originals: OriginalDocument[]): Promise<PreparedDocument[]> {
const documents: PreparedDocument[] = [];
for (const original of originals) {
const parsed = await parseDocument(original.filePath);
const normalizedContent = normalizeContentForHash(parsed.content);
if (!normalizedContent.trim()) {
continue;
}
const chunkingPolicy = parsed.chunkMode === "codigo" ? codeChunkingPolicy : documentalChunkingPolicy;
const chunks = chunkDocument(parsed.title, normalizedContent, chunkingPolicy);
if (chunks.length === 0) {
continue;
}
documents.push({
documentId: original.documentId,
documentKey: original.documentKey,
originalHash: original.originalHash,
originalHashKind: "bytes",
contentHash: sha256Hex(normalizedContent),
mimeType: parsed.mimeType,
title: parsed.title,
chunkCount: chunks.length,
chunks,
chunkMode: chunkingPolicy.mode,
content: normalizedContent
});
}
return documents.sort((left, right) => Buffer.compare(Buffer.from(left.documentKey, "utf8"), Buffer.from(right.documentKey, "utf8")));
}
private buildDocumentKey(source: IngestSourceInput, filePath: string): string {
const logicalRef = source.sourceType === "folder"
? path.relative(path.resolve(source.readPath ?? source.sourceRef), path.resolve(filePath))
: source.sourceRef;
return normalizeDocumentKey(logicalRef || path.basename(filePath));
}
private async writeVersionedChunks(
source: IngestSourceInput,
sourceId: string,
versionId: string,
versionNumber: number,
documents: PreparedDocument[],
processingFingerprint: string,
tags: string[],
expectedEmbeddingDimensions: number
): Promise<number> {
let chunksStored = 0;
for (const document of documents) {
const embeddings = await this.embeddingProvider.embed(document.chunks.map((chunk) => chunk.content));
if (embeddings.length !== document.chunks.length || embeddings.some((embedding) => embedding.length !== expectedEmbeddingDimensions)) {
throw new CatalogError("Embedding provider returned invalid dimensions", 503, "EMBEDDING_DIMENSIONS_INVALID");
}
const qdrantChunks: IngestedChunk[] = document.chunks.map((chunk, index) => {
const chunkId = buildChunkId(document.documentId, document.chunkMode, chunk.index);
return {
id: buildVersionedQdrantPointId(versionId, chunkId),
vector: embeddings[index],
payload: {
chunk_id: chunkId,
source_id: sourceId,
source_version_id: versionId,
source_version_number: versionNumber,
source_type: source.sourceType,
source_ref: source.sourceRef,
document_id: document.documentId,
document_key: document.documentKey,
document_content_hash: document.contentHash,
title: document.title,
mime_type: document.mimeType,
section_title: chunk.sectionTitle,
chunk_mode: document.chunkMode,
chunk_index: chunk.index,
start_line: chunk.startLine,
end_line: chunk.endLine,
content: chunk.content,
tags,
embedding_provider: this.embeddingProvider.providerName,
embedding_model: this.embeddingProvider.modelName,
embedding_dimensions: embeddings[index]?.length ?? 0,
processing_fingerprint: processingFingerprint,
indexed_at: new Date().toISOString(),
write_state: "staged"
}
};
});
await this.vectorStore.upsert(qdrantChunks);
chunksStored += qdrantChunks.length;
}
return chunksStored;
}
private async resolveInputFiles(source: IngestSourceInput): Promise<string[]> {
const targetPath = source.readPath ?? source.sourceRef;
if (source.sourceType === "file") {

View file

@ -1,6 +1,8 @@
import type { EmbeddingProvider } from "../embeddings/provider.js";
import type { VectorStoreClient } from "../vectorstore/client.js";
import type { ChunkMode, RetrieveIntent, RetrieveResponse, RetrieveScope } from "../../shared/types/rag.js";
import { env } from "../../config/env.js";
import { CatalogError, type ActiveVersionScope, type CatalogRepository } from "../catalog/repository.js";
function unique(values: string[]): string[] {
return [...new Set(values.filter(Boolean))];
@ -176,13 +178,18 @@ async function embedQuery(provider: EmbeddingProvider, query: string): Promise<n
export class RetrieveService {
constructor(
private readonly embeddingProvider: EmbeddingProvider,
private readonly vectorStore: VectorStoreClient
private readonly vectorStore: VectorStoreClient,
private readonly catalog?: CatalogRepository
) {}
async retrieve(mode: ChunkMode, intent: RetrieveIntent, query: string, scope?: RetrieveScope): Promise<RetrieveResponse> {
const items = intent === "bootstrap"
? await this.retrieveBootstrap(mode, query, scope)
: await this.retrieveSpecific(mode, query, scope);
const activeVersions = await this.resolveActiveVersions(scope);
const items = await this.withActiveVersionReadLocks(activeVersions, async () => {
await this.validateActiveVersions(activeVersions);
return intent === "bootstrap"
? this.retrieveBootstrap(mode, query, scope, activeVersions)
: this.retrieveSpecific(mode, query, scope, activeVersions);
});
const topics = unique(items.map((item) => item.title)).slice(0, 8);
const criticalPoints = items
@ -207,13 +214,58 @@ export class RetrieveService {
};
}
private async retrieveSpecific(mode: ChunkMode, query: string, scope?: RetrieveScope) {
private async withActiveVersionReadLocks<T>(activeVersions: ActiveVersionScope[] | undefined, handler: () => Promise<T>): Promise<T> {
if (!env.knowledgeLifecycleEnforced || !this.catalog || !activeVersions) {
return handler();
}
return this.catalog.withVersionSharedLocks(activeVersions.map((version) => version.sourceVersionId), handler);
}
private async resolveActiveVersions(scope?: RetrieveScope): Promise<ActiveVersionScope[] | undefined> {
if (!env.knowledgeLifecycleEnforced) {
return undefined;
}
if (!this.catalog) {
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
}
try {
return await this.catalog.resolveActiveVersions(scope);
} catch (error) {
if (error instanceof CatalogError) {
throw error;
}
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
}
}
private async validateActiveVersions(activeVersions: ActiveVersionScope[] | undefined): Promise<void> {
if (!env.knowledgeLifecycleEnforced || !this.catalog || !activeVersions) {
return;
}
for (const version of activeVersions) {
if (version.qdrantCollection !== env.qdrantCollection) {
await this.catalog.markNeedsReingest(version.sourceId);
throw new CatalogError("Active source version targets a different Qdrant collection", 503, "SOURCE_VERSION_INCONSISTENT");
}
await this.vectorStore.validateCollectionDimensions(version.embeddingDimensions);
const count = await this.vectorStore.countVersionPoints(version.sourceVersionId);
if (count !== version.expectedPointCount) {
await this.catalog.markNeedsReingest(version.sourceId);
throw new CatalogError("Active source version is inconsistent", 503, "SOURCE_VERSION_INCONSISTENT");
}
}
}
private async retrieveSpecific(mode: ChunkMode, query: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
const queries = buildSpecificQueries(query);
const merged = new Map<string, RetrieveResponse["items"][number]>();
for (const subquery of queries) {
const queryVector = await embedQuery(this.embeddingProvider, subquery);
const results = await this.vectorStore.search(queryVector, 6, mode, scope);
const results = await this.vectorStore.search(queryVector, 6, mode, scope, activeVersions);
for (const item of results) {
const existing = merged.get(item.chunkId);
@ -226,13 +278,13 @@ export class RetrieveService {
return rankItems(query, [...merged.values()]).slice(0, 8);
}
private async retrieveBootstrap(mode: ChunkMode, query: string, scope?: RetrieveScope) {
private async retrieveBootstrap(mode: ChunkMode, query: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
const queries = buildBootstrapQueries(query);
const merged = new Map<string, RetrieveResponse["items"][number]>();
for (const subquery of queries) {
const queryVector = await embedQuery(this.embeddingProvider, subquery);
const results = await this.vectorStore.search(queryVector, 6, mode, scope);
const results = await this.vectorStore.search(queryVector, 6, mode, scope, activeVersions);
for (const item of results) {
const existing = merged.get(item.chunkId);
@ -243,7 +295,7 @@ export class RetrieveService {
}
if (merged.size === 0 && scope && (scope.sourceId || scope.sourceRef || (scope.tags && scope.tags.length > 0))) {
const explored = await this.vectorStore.browseScope(12, mode, scope);
const explored = await this.vectorStore.browseScope(12, mode, scope, activeVersions);
for (const item of explored) {
merged.set(item.chunkId, item);
}

View file

@ -1,6 +1,7 @@
import { QdrantClient } from "@qdrant/js-client-rest";
import { env } from "../../config/env.js";
import type { AvailableScope, IngestedChunk, RetrieveScope, RetrievedItem } from "../../shared/types/rag.js";
import type { ActiveVersionScope } from "../catalog/repository.js";
export function buildQdrantClient(): QdrantClient {
const url = new URL(env.qdrantUrl);
@ -17,14 +18,32 @@ export interface VectorStoreClient {
readonly kind: string;
healthcheck(): Promise<{ ok: boolean; kind: string }>;
ensureCollection(vectorSize: number): Promise<void>;
ensurePayloadIndexes(): Promise<void>;
upsert(chunks: IngestedChunk[]): Promise<void>;
search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]>;
search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]>;
listScopes(): Promise<AvailableScope[]>;
browseScope(limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]>;
browseScope(limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]>;
deleteScope(scope: RetrieveScope): Promise<number>;
validateCollectionDimensions(expectedDimensions: number): Promise<void>;
countVersionPoints(sourceVersionId: string): Promise<number>;
deleteVersionPoints(sourceVersionId: string): Promise<number>;
}
function buildSearchFilter(mode?: string, scope?: RetrieveScope) {
function extractVectorSize(collectionInfo: unknown): number | undefined {
const info = collectionInfo as { config?: { params?: { vectors?: unknown } } };
const vectors = info.config?.params?.vectors;
if (!vectors || typeof vectors !== "object") {
return undefined;
}
if ("size" in vectors && typeof vectors.size === "number") {
return vectors.size;
}
const namedVectors = Object.values(vectors as Record<string, unknown>);
const firstNamed = namedVectors[0] as { size?: unknown } | undefined;
return typeof firstNamed?.size === "number" ? firstNamed.size : undefined;
}
function buildSearchFilter(mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
const must: Array<Record<string, unknown>> = [];
if (mode && mode !== "auto") {
@ -57,6 +76,13 @@ function buildSearchFilter(mode?: string, scope?: RetrieveScope) {
}
}
if (activeVersions) {
must.push({
key: "source_version_id",
match: { any: activeVersions.length > 0 ? activeVersions.map((version) => version.sourceVersionId) : ["__no_active_versions__"] }
});
}
return must.length > 0 ? { must } : undefined;
}
@ -93,18 +119,47 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
distance: "Cosine"
}
});
} else {
await this.validateCollectionDimensions(vectorSize);
}
await this.ensurePayloadIndexes();
this.collectionReady = true;
}
async ensurePayloadIndexes(): Promise<void> {
const collections = await this.client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
if (!exists) {
return;
}
const collectionInfo = await this.client.getCollection(env.qdrantCollection) as { payload_schema?: Record<string, unknown> };
const payloadSchema = collectionInfo.payload_schema ?? {};
for (const fieldName of ["source_id", "source_version_id", "document_id", "chunk_mode", "tags"]) {
if (payloadSchema[fieldName]) {
continue;
}
await this.client.createPayloadIndex(env.qdrantCollection, {
field_name: fieldName,
field_schema: "keyword",
wait: true
});
}
}
async upsert(chunks: IngestedChunk[]): Promise<void> {
if (chunks.length === 0) {
return;
}
await this.ensureCollection(chunks[0].vector.length);
await this.client.upsert(env.qdrantCollection, {
const vectorSize = chunks[0].vector.length;
if (vectorSize === 0 || chunks.some((chunk) => chunk.vector.length !== vectorSize)) {
throw new Error("Qdrant upsert rejected: vectors must be non-empty and have consistent dimensions");
}
await this.ensureCollection(vectorSize);
const result = await this.client.upsert(env.qdrantCollection, {
wait: true,
points: chunks.map((chunk) => ({
id: chunk.id,
@ -112,21 +167,26 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
payload: chunk.payload
}))
});
if (result.status !== "completed") {
throw new Error(`Qdrant upsert did not complete: ${result.status}`);
}
}
async search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]> {
async search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]> {
await this.ensureCollection(queryVector.length);
const result = await this.client.search(env.qdrantCollection, {
vector: queryVector,
limit,
with_payload: true,
filter: buildSearchFilter(mode, scope)
filter: buildSearchFilter(mode, scope, activeVersions)
});
return result.map((point) => ({
chunkId: String(point.payload?.chunk_id ?? point.id),
documentId: String(point.payload?.document_id ?? ""),
sourceId: String(point.payload?.source_id ?? ""),
sourceVersionId: point.payload?.source_version_id ? String(point.payload.source_version_id) : undefined,
sourceVersionNumber: point.payload?.source_version_number ? Number(point.payload.source_version_number) : undefined,
title: String(point.payload?.title ?? ""),
sectionTitle: point.payload?.section_title ? String(point.payload.section_title) : undefined,
content: String(point.payload?.content ?? ""),
@ -192,7 +252,7 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
return [...scopeMap.values()].sort((left, right) => left.sourceRef.localeCompare(right.sourceRef));
}
async browseScope(limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]> {
async browseScope(limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]> {
const collections = await this.client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
@ -203,13 +263,15 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
const response = await this.client.scroll(env.qdrantCollection, {
limit,
with_payload: true,
filter: buildSearchFilter(mode, scope)
filter: buildSearchFilter(mode, scope, activeVersions)
});
return response.points.map((point) => ({
chunkId: String(point.payload?.chunk_id ?? point.id),
documentId: String(point.payload?.document_id ?? ""),
sourceId: String(point.payload?.source_id ?? ""),
sourceVersionId: point.payload?.source_version_id ? String(point.payload.source_version_id) : undefined,
sourceVersionNumber: point.payload?.source_version_number ? Number(point.payload.source_version_number) : undefined,
title: String(point.payload?.title ?? ""),
sectionTitle: point.payload?.section_title ? String(point.payload.section_title) : undefined,
content: String(point.payload?.content ?? ""),
@ -233,15 +295,64 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
return 0;
}
// Contamos antes de borrar para devolver la cantidad
const countResult = await this.client.count(env.qdrantCollection, {
filter
});
await this.client.delete(env.qdrantCollection, {
filter
const deleteResult = await this.client.delete(env.qdrantCollection, {
filter,
wait: true
});
if (deleteResult.status !== "completed") {
throw new Error(`Qdrant delete did not complete: ${deleteResult.status}`);
}
return countResult.count;
}
async validateCollectionDimensions(expectedDimensions: number): Promise<void> {
const collections = await this.client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
if (!exists) {
return;
}
const collectionInfo = await this.client.getCollection(env.qdrantCollection);
const actualDimensions = extractVectorSize(collectionInfo);
if (actualDimensions !== undefined && actualDimensions !== expectedDimensions) {
throw new Error(`Qdrant collection dimension mismatch: expected ${expectedDimensions}, got ${actualDimensions}`);
}
}
async countVersionPoints(sourceVersionId: string): Promise<number> {
const collections = await this.client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
if (!exists) {
return 0;
}
const result = await this.client.count(env.qdrantCollection, {
filter: {
must: [{ key: "source_version_id", match: { value: sourceVersionId } }]
},
exact: true
});
return result.count;
}
async deleteVersionPoints(sourceVersionId: string): Promise<number> {
const count = await this.countVersionPoints(sourceVersionId);
if (count === 0) {
return 0;
}
const deleteResult = await this.client.delete(env.qdrantCollection, {
filter: {
must: [{ key: "source_version_id", match: { value: sourceVersionId } }]
},
wait: true
});
if (deleteResult.status !== "completed") {
throw new Error(`Qdrant version delete did not complete: ${deleteResult.status}`);
}
return count;
}
}

View file

@ -0,0 +1,448 @@
import { buildCatalogPool, withCatalogAdvisoryLock, withTransaction, type PgPool } from "../modules/catalog/client.js";
import { pathToFileURL } from "node:url";
import { buildLegacyMigrationPlan, type LegacyPointSnapshot, type LegacySourcePlan } from "../modules/catalog/legacy-migration.js";
import { CatalogError } from "../modules/catalog/errors.js";
import { env } from "../config/env.js";
import { buildQdrantClient } from "../modules/vectorstore/client.js";
import { sha256Hex } from "../shared/utils/ids.js";
type QdrantClientInstance = ReturnType<typeof buildQdrantClient>;
function argValue(name: string): string | undefined {
const prefix = `${name}=`;
const entry = process.argv.find((arg) => arg.startsWith(prefix));
return entry?.slice(prefix.length);
}
function requireArg(name: string): string {
const value = argValue(name);
if (!value) {
throw new Error(`Missing required argument ${name}=...`);
}
return value;
}
async function readLegacyPoints(client: QdrantClientInstance, includeVersionIds = new Set<string>()): Promise<LegacyPointSnapshot[]> {
const collections = await client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
if (!exists) {
return [];
}
const points: LegacyPointSnapshot[] = [];
let nextOffset = undefined;
do {
const response = await client.scroll(env.qdrantCollection, {
limit: 256,
offset: nextOffset,
with_payload: true,
with_vector: false
});
for (const point of response.points) {
const sourceVersionId = point.payload?.source_version_id ? String(point.payload.source_version_id) : "";
if (sourceVersionId && !includeVersionIds.has(sourceVersionId)) {
continue;
}
points.push({ id: point.id, payload: point.payload ?? {} });
}
nextOffset = response.next_page_offset;
} while (nextOffset);
return points;
}
async function assertMaintenance(pool: PgPool): Promise<void> {
if (env.ingestWritesEnabled) {
throw new CatalogError("INGEST_WRITES_ENABLED must be false during legacy migration", 503, "WRITES_NOT_DISABLED");
}
await withTransaction(pool, async (client) => {
const result = await client.query<{ maintenance: boolean }>("SELECT maintenance FROM rag_system_state WHERE singleton = true FOR UPDATE");
if (!result.rows[0]?.maintenance) {
throw new CatalogError("rag_system_state.maintenance must be true during legacy migration", 503, "MAINTENANCE_REQUIRED");
}
});
}
async function readBatchVersionIds(pool: PgPool, batchId: string): Promise<Set<string>> {
const result = await pool.query<{ version_id: string }>(
"SELECT version_id FROM rag_legacy_migration_items WHERE batch_id = $1 AND version_id IS NOT NULL",
[batchId]
);
return new Set(result.rows.map((row) => row.version_id));
}
async function createOrReuseBatch(pool: PgPool, snapshotRef: string, plan: { sources: LegacySourcePlan[] }, batchId?: string): Promise<string> {
if (batchId) {
const existing = await pool.query<{ state: string; snapshot_ref: string }>(
"SELECT state, snapshot_ref FROM rag_legacy_migration_batches WHERE batch_id = $1",
[batchId]
);
if (!existing.rowCount) {
throw new CatalogError(`Legacy migration batch not found: ${batchId}`, 404, "LEGACY_BATCH_NOT_FOUND");
}
if (existing.rows[0].snapshot_ref !== snapshotRef) {
throw new CatalogError("Legacy migration batch snapshot_ref does not match the requested snapshot", 409, "LEGACY_BATCH_SNAPSHOT_MISMATCH");
}
if (existing.rows[0].state === "rolled_back") {
throw new CatalogError("Rolled back legacy migration batches cannot be resumed", 409, "LEGACY_BATCH_ROLLED_BACK");
}
return batchId;
}
const batch = await pool.query<{ batch_id: string }>(
"INSERT INTO rag_legacy_migration_batches(state, snapshot_ref, report) VALUES ('planned', $1, $2) RETURNING batch_id",
[snapshotRef, { sourceCount: plan.sources.length, pointCount: plan.sources.reduce((total, source) => total + source.pointIds.length, 0) }]
);
return batch.rows[0].batch_id;
}
export async function createCatalogVersion(pool: PgPool, batchId: string, source: LegacySourcePlan, collection: string): Promise<{ versionId: string; versionNumber: number }> {
return withTransaction(pool, async (client) => {
const item = await client.query<{ version_id: string | null; expected_point_count: number }>(
"SELECT version_id, expected_point_count FROM rag_legacy_migration_items WHERE batch_id = $1 AND source_id = $2 FOR UPDATE",
[batchId, source.sourceId]
);
if (item.rows[0]?.version_id) {
const reusable = await client.query<{ version_id: string; version_number: string; expected_point_count: string; source_type: string; source_ref: string; qdrant_collection: string }>(
`SELECT v.version_id, v.version_number, v.expected_point_count, s.source_type, s.source_ref, v.qdrant_collection
FROM rag_source_versions v
JOIN rag_sources s ON s.source_id = v.source_id
WHERE v.version_id = $1 AND v.source_id = $2
FOR UPDATE OF v, s`,
[item.rows[0].version_id, source.sourceId]
);
if (!reusable.rowCount) {
throw new CatalogError(`Legacy migration item references a missing version for ${source.sourceId}`, 409, "LEGACY_VERSION_MISSING");
}
const row = reusable.rows[0];
if (row.source_type !== source.sourceType || row.source_ref !== source.sourceRef || row.qdrant_collection !== collection || Number(row.expected_point_count) !== source.pointIds.length) {
throw new CatalogError(`Legacy migration resume is not compatible for ${source.sourceId}`, 409, "LEGACY_RESUME_INCOMPATIBLE");
}
return { versionId: row.version_id, versionNumber: Number(row.version_number) };
}
const existing = await client.query("SELECT 1 FROM rag_sources WHERE source_id = $1", [source.sourceId]);
if (existing.rowCount) {
throw new CatalogError(`Source already exists in catalog: ${source.sourceId}`, 409, "LEGACY_SOURCE_ALREADY_CATALOGED");
}
await client.query(
"INSERT INTO rag_sources(source_id, source_type, source_ref) VALUES ($1, $2, $3)",
[source.sourceId, source.sourceType, source.sourceRef]
);
const version = await client.query<{ version_id: string; version_number: string }>(
`INSERT INTO rag_source_versions(
source_id, version_number, state, original_manifest_hash, source_content_hash,
processing_fingerprint, metadata_hash, tags, activate_requested, embedding_provider,
embedding_model, embedding_dimensions, qdrant_collection, expected_document_count,
expected_point_count, verified_point_count
) VALUES ($1, 1, 'ready', $2, $3, $4, $5, $6, true, $7, $8, $9, $10, $11, $12, 0)
RETURNING version_id, version_number`,
[
source.sourceId,
source.originalManifestHash,
source.sourceContentHash,
sha256Hex(`legacy:${batchId}`),
source.metadataHash,
source.tags,
source.embeddingProvider,
source.embeddingModel,
source.embeddingDimensions,
collection,
source.documents.length,
source.pointIds.length
]
);
const versionId = version.rows[0].version_id;
for (const document of source.documents) {
await client.query(
`INSERT INTO rag_version_documents(
version_id, document_id, document_key, original_hash, original_hash_kind,
content_hash, mime_type, title, index_state, chunk_count
) VALUES ($1, $2, $3, $4, 'legacy-derived', $5, $6, $7, 'ready', $8)`,
[versionId, document.documentId, document.documentKey, document.originalHash, document.contentHash, document.mimeType, document.title, document.chunkCount]
);
}
if (item.rowCount) {
await client.query(
`UPDATE rag_legacy_migration_items
SET version_id = $3, expected_point_count = $4, state = 'migrated', error_detail = NULL
WHERE batch_id = $1 AND source_id = $2`,
[batchId, source.sourceId, versionId, source.pointIds.length]
);
} else {
await client.query(
`INSERT INTO rag_legacy_migration_items(batch_id, source_id, version_id, expected_point_count, state)
VALUES ($1, $2, $3, $4, 'migrated')`,
[batchId, source.sourceId, versionId, source.pointIds.length]
);
}
return { versionId, versionNumber: Number(version.rows[0].version_number) };
});
}
async function setLifecyclePayload(client: QdrantClientInstance, source: LegacySourcePlan, versionId: string, versionNumber: number, batchId: string): Promise<void> {
const result = await client.setPayload(env.qdrantCollection, {
points: source.pointIds,
payload: {
source_version_id: versionId,
source_version_number: versionNumber,
processing_fingerprint: sha256Hex(`legacy:${batchId}`),
write_state: "legacy-migrated"
},
wait: true
});
if (result.status !== "completed") {
throw new Error(`Qdrant payload update did not complete for ${source.sourceId}: ${result.status}`);
}
}
async function countVersionPoints(client: QdrantClientInstance, versionId: string): Promise<number> {
const result = await client.count(env.qdrantCollection, {
filter: { must: [{ key: "source_version_id", match: { value: versionId } }] },
exact: true
});
return result.count;
}
async function countLegacySourcePoints(client: QdrantClientInstance, sourceId: string): Promise<number> {
let count = 0;
let nextOffset = undefined;
do {
const response = await client.scroll(env.qdrantCollection, {
limit: 256,
offset: nextOffset,
with_payload: ["source_id", "source_version_id"],
with_vector: false,
filter: { must: [{ key: "source_id", match: { value: sourceId } }] }
});
for (const point of response.points) {
if (!point.payload?.source_version_id) {
count += 1;
}
}
nextOffset = response.next_page_offset;
} while (nextOffset);
return count;
}
export async function activateMigratedVersion(pool: PgPool, sourceId: string, versionId: string, verifiedPointCount: number): Promise<void> {
await withTransaction(pool, async (client) => {
const existing = await client.query<{ state: string; active_version_id: string | null }>(
`SELECT v.state, s.active_version_id
FROM rag_source_versions v
JOIN rag_sources s ON s.source_id = v.source_id
WHERE v.source_id = $1 AND v.version_id = $2
FOR UPDATE OF v, s`,
[sourceId, versionId]
);
if (existing.rows[0]?.state === "active" && existing.rows[0].active_version_id === versionId) {
await client.query("UPDATE rag_source_versions SET verified_point_count = $2 WHERE source_id = $1 AND version_id = $3", [sourceId, verifiedPointCount, versionId]);
return;
}
const version = await client.query(
`UPDATE rag_source_versions
SET state = 'active', verified_point_count = $2, ready_at = now(), activated_at = now()
WHERE source_id = $1 AND version_id = $3 AND state = 'ready'
RETURNING version_id`,
[sourceId, verifiedPointCount, versionId]
);
if (version.rowCount !== 1) {
throw new CatalogError(`Could not activate migrated version for ${sourceId}`, 409, "LEGACY_ACTIVATION_FAILED");
}
await client.query("UPDATE rag_sources SET active_version_id = $2 WHERE source_id = $1", [sourceId, versionId]);
});
}
export async function rollbackMigratedBatch(
pool: PgPool,
qdrant: QdrantClientInstance,
batchId: string,
snapshotRef: string,
snapshotChecksum?: string
): Promise<{ ok: true; mode: "rollback"; batchId: string; sources: string[] }> {
const itemsWithVersions = await withTransaction(pool, async (client) => {
const batch = await client.query<{ state: string; snapshot_ref: string }>(
"SELECT state, snapshot_ref FROM rag_legacy_migration_batches WHERE batch_id = $1 FOR UPDATE",
[batchId]
);
if (!batch.rowCount) {
throw new CatalogError(`Legacy migration batch not found: ${batchId}`, 404, "LEGACY_BATCH_NOT_FOUND");
}
if (batch.rows[0].snapshot_ref !== snapshotRef) {
throw new CatalogError("Legacy migration batch snapshot_ref does not match the requested snapshot", 409, "LEGACY_BATCH_SNAPSHOT_MISMATCH");
}
const items = await client.query<{ source_id: string; version_id: string | null; expected_point_count: string | number }>(
"SELECT source_id, version_id, expected_point_count FROM rag_legacy_migration_items WHERE batch_id = $1 FOR UPDATE",
[batchId]
);
const rollbackItems = items.rows.filter((item): item is { source_id: string; version_id: string; expected_point_count: string | number } => Boolean(item.version_id));
if (rollbackItems.length === 0 && batch.rows[0].state === "rolled_back") {
return [];
}
await client.query(
`SELECT s.source_id
FROM rag_sources s
JOIN rag_legacy_migration_items i ON i.source_id = s.source_id
WHERE i.batch_id = $1 AND i.version_id IS NOT NULL
FOR UPDATE OF s`,
[batchId]
);
await client.query(
`SELECT v.version_id
FROM rag_source_versions v
JOIN rag_legacy_migration_items i ON i.version_id = v.version_id
WHERE i.batch_id = $1 AND i.version_id IS NOT NULL
FOR UPDATE OF v`,
[batchId]
);
await client.query(
`UPDATE rag_sources s
SET active_version_id = NULL
FROM rag_legacy_migration_items i
WHERE i.batch_id = $1
AND i.source_id = s.source_id
AND i.version_id IS NOT NULL
AND s.active_version_id = i.version_id`,
[batchId]
);
await client.query(
`UPDATE rag_source_versions v
SET state = 'purging'
FROM rag_legacy_migration_items i
WHERE i.batch_id = $1
AND i.version_id = v.version_id`,
[batchId]
);
const activePointers = await client.query<{ count: string }>(
`SELECT COUNT(*) AS count
FROM rag_sources s
JOIN rag_legacy_migration_items i ON i.source_id = s.source_id
WHERE i.batch_id = $1
AND i.version_id IS NOT NULL
AND s.active_version_id = i.version_id`,
[batchId]
);
if (Number(activePointers.rows[0]?.count ?? 0) !== 0) {
throw new CatalogError("Legacy rollback preparation left active source pointers", 503, "LEGACY_ROLLBACK_ACTIVE_POINTERS_REMAIN");
}
return rollbackItems.map((item) => ({
sourceId: item.source_id,
versionId: item.version_id,
expectedPointCount: Number(item.expected_point_count)
}));
});
if (itemsWithVersions.length === 0) {
return { ok: true, mode: "rollback", batchId, sources: [] };
}
const recovered = await qdrant.recoverSnapshot(env.qdrantCollection, {
location: snapshotRef,
priority: "snapshot",
checksum: snapshotChecksum
});
if (!recovered) {
throw new Error("Qdrant snapshot recovery returned false");
}
for (const item of itemsWithVersions) {
const legacyCount = await countLegacySourcePoints(qdrant, item.sourceId);
if (legacyCount !== item.expectedPointCount) {
throw new CatalogError(`Restored legacy point count mismatch for ${item.sourceId}`, 503, "LEGACY_RESTORE_POINT_COUNT_MISMATCH");
}
}
await withTransaction(pool, async (client) => {
for (const item of itemsWithVersions) {
await client.query("DELETE FROM rag_version_documents WHERE version_id = $1", [item.versionId]);
await client.query("DELETE FROM rag_source_versions WHERE version_id = $1", [item.versionId]);
await client.query("DELETE FROM rag_sources WHERE source_id = $1", [item.sourceId]);
await client.query("UPDATE rag_legacy_migration_items SET version_id = NULL, state = 'rolled_back' WHERE batch_id = $1 AND source_id = $2", [batchId, item.sourceId]);
}
await client.query("UPDATE rag_legacy_migration_batches SET state = 'rolled_back', completed_at = now() WHERE batch_id = $1", [batchId]);
});
return { ok: true, mode: "rollback", batchId, sources: itemsWithVersions.map((item) => item.sourceId) };
}
async function dryRun() {
const qdrant = buildQdrantClient();
const points = await readLegacyPoints(qdrant);
const plan = buildLegacyMigrationPlan(points);
return { mode: "dry-run", collection: env.qdrantCollection, legacyPoints: points.length, ...plan };
}
async function applyMigration() {
const snapshotRef = requireArg("--snapshot-ref");
const requestedBatchId = argValue("--batch-id");
const pool = buildCatalogPool();
const qdrant = buildQdrantClient();
let batchId = requestedBatchId;
try {
return await withCatalogAdvisoryLock(pool, "rag:knowledge-lifecycle:legacy-migration", async () => {
await assertMaintenance(pool);
const includeVersionIds = requestedBatchId ? await readBatchVersionIds(pool, requestedBatchId) : new Set<string>();
const plan = buildLegacyMigrationPlan(await readLegacyPoints(qdrant, includeVersionIds));
if (!plan.ok) {
return { ok: false, mode: "apply", batchId: requestedBatchId, blockedReasons: plan.blockedReasons };
}
batchId = await createOrReuseBatch(pool, snapshotRef, plan, requestedBatchId);
const migrated: Array<{ sourceId: string; versionId: string; pointCount: number }> = [];
for (const source of plan.sources) {
const { versionId, versionNumber } = await createCatalogVersion(pool, batchId, source, env.qdrantCollection);
await setLifecyclePayload(qdrant, source, versionId, versionNumber, batchId);
const verifiedPointCount = await countVersionPoints(qdrant, versionId);
if (verifiedPointCount !== source.pointIds.length) {
throw new CatalogError(`Point count mismatch for ${source.sourceId}`, 503, "LEGACY_POINT_COUNT_MISMATCH");
}
await activateMigratedVersion(pool, source.sourceId, versionId, verifiedPointCount);
await pool.query("UPDATE rag_legacy_migration_items SET state = 'verified', error_detail = NULL WHERE batch_id = $1 AND source_id = $2", [batchId, source.sourceId]);
migrated.push({ sourceId: source.sourceId, versionId, pointCount: verifiedPointCount });
}
await pool.query("UPDATE rag_legacy_migration_batches SET state = 'verified', completed_at = now(), report = report || $2::jsonb WHERE batch_id = $1", [batchId, JSON.stringify({ migrated })]);
return { ok: true, mode: "apply", batchId, migrated };
});
} catch (error) {
if (batchId) {
await pool.query("UPDATE rag_legacy_migration_batches SET state = 'failed', completed_at = now(), report = report || $2::jsonb WHERE batch_id = $1 AND state <> 'verified'", [batchId, JSON.stringify({ error: error instanceof Error ? error.message : "Unknown legacy migration error" })]).catch(() => undefined);
}
throw error;
} finally {
await pool.end();
}
}
async function rollbackMigration() {
const batchId = requireArg("--batch-id");
const snapshotRef = requireArg("--snapshot-ref");
const snapshotChecksum = argValue("--snapshot-checksum");
const pool = buildCatalogPool();
const qdrant = buildQdrantClient();
try {
return await withCatalogAdvisoryLock(pool, "rag:knowledge-lifecycle:legacy-migration", async () => {
await assertMaintenance(pool);
return rollbackMigratedBatch(pool, qdrant, batchId, snapshotRef, snapshotChecksum);
});
} finally {
await pool.end();
}
}
async function main() {
const report = process.argv.includes("--apply")
? await applyMigration()
: process.argv.includes("--rollback")
? await rollbackMigration()
: await dryRun();
console.log(JSON.stringify(report, null, 2));
if (!report.ok) process.exitCode = 1;
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
main().catch((error) => {
console.error(error instanceof Error ? error.message : error);
process.exit(1);
});
}

View file

@ -6,6 +6,18 @@ export type IngestMode = "mechanical" | "interactive";
export type SourceType = "file" | "folder";
export type SourceVersionState =
| "pending"
| "indexing"
| "review_required"
| "ready"
| "active"
| "superseded"
| "failed"
| "rejected"
| "purging"
| "purged";
export interface IngestSourceInput {
sourceId?: string;
sourceType: SourceType;
@ -13,6 +25,8 @@ export interface IngestSourceInput {
readPath?: string;
mode?: IngestMode;
tags?: string[];
activate?: boolean;
expectedActiveVersionId?: string | null;
}
export interface IngestedChunk {
@ -23,10 +37,17 @@ export interface IngestedChunk {
export interface IngestResult {
accepted: true;
source: IngestSourceInput;
source?: IngestSourceInput;
sourceId: string;
versionId?: string;
versionNumber?: number;
state?: SourceVersionState;
previousVersionId?: string | null;
filesDiscovered: number;
documentsProcessed: number;
chunksStored: number;
activated?: boolean;
noOp?: boolean;
collectionName: string;
}
@ -34,6 +55,8 @@ export interface RetrievedItem {
chunkId: string;
documentId: string;
sourceId: string;
sourceVersionId?: string;
sourceVersionNumber?: number;
title: string;
sectionTitle?: string;
content: string;
@ -54,6 +77,11 @@ export interface AvailableScope {
sourceRef: string;
chunkModes: ChunkMode[];
tags: string[];
activeVersionId?: string | null;
activeVersionNumber?: number | null;
state?: SourceVersionState | null;
needsReingest?: boolean;
updatedAt?: string;
}
export interface RetrieveResponse {
@ -80,6 +108,8 @@ export interface AnswerResponse {
citations: Array<{
chunkId: string;
documentId: string;
sourceVersionId?: string;
sourceVersionNumber?: number;
title: string;
sectionTitle?: string;
startLine?: number;

View file

@ -1,6 +1,17 @@
import path from "node:path";
import { createHash } from "node:crypto";
const UUID_NAMESPACE_URL = "6ba7b811-9dad-11d1-80b4-00c04fd430c8";
function uuidToBytes(uuid: string): Buffer {
return Buffer.from(uuid.replace(/-/g, ""), "hex");
}
function bytesToUuid(bytes: Buffer): string {
const hex = bytes.toString("hex");
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
}
function asciiNormalize(value: string): string {
return value
.normalize("NFD")
@ -17,6 +28,58 @@ export function normalizeDocumentKey(value: string): string {
.toLowerCase();
}
export function sha256Hex(value: string | Buffer): string {
return createHash("sha256").update(value).digest("hex");
}
export function normalizeContentForHash(content: string): string {
const normalized = content
.replace(/^\uFEFF/, "")
.replace(/\r\n/g, "\n")
.replace(/\r/g, "\n")
.split("\n")
.map((line) => line.replace(/[ \t]+$/g, ""))
.join("\n")
.replace(/^\n+|\n+$/g, "");
return `${normalized}\n`;
}
export function canonicalJson(value: unknown): string {
if (Array.isArray(value)) {
return `[${value.map(canonicalJson).join(",")}]`;
}
if (value && typeof value === "object") {
const entries = Object.entries(value as Record<string, unknown>)
.filter(([, entryValue]) => entryValue !== undefined)
.sort(([left], [right]) => Buffer.compare(Buffer.from(left, "utf8"), Buffer.from(right, "utf8")));
return `{${entries.map(([key, entryValue]) => `${JSON.stringify(key)}:${canonicalJson(entryValue)}`).join(",")}}`;
}
return JSON.stringify(value);
}
export function hashOrderedPairs(pairs: Array<[string, string | null]>): string | null {
if (pairs.some(([, value]) => value === null)) {
return null;
}
const serialized = [...pairs]
.sort(([left], [right]) => Buffer.compare(Buffer.from(left, "utf8"), Buffer.from(right, "utf8")))
.map(([key, value]) => `${key}\0${value ?? ""}`)
.join("\n");
return sha256Hex(serialized);
}
export function buildProcessingFingerprint(input: Record<string, unknown>): string {
return sha256Hex(canonicalJson(input));
}
export function buildMetadataHash(tags: string[], metadata: Record<string, unknown> = {}): string {
return sha256Hex(canonicalJson({ metadata, tags: [...tags].sort() }));
}
export function buildSourceId(sourceType: string, sourceRef: string): string {
const readable = normalizeDocumentKey(path.basename(sourceRef) || sourceRef).replace(/\.[^.]+$/, "") || "source";
const suffix = createHash("sha1").update(sourceRef).digest("hex").slice(0, 8);
@ -36,3 +99,12 @@ export function buildQdrantPointId(value: string): string {
const hex = createHash("sha1").update(value).digest("hex").slice(0, 32);
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
}
export function buildVersionedQdrantPointId(sourceVersionId: string, chunkId: string): string {
const namespace = uuidToBytes(UUID_NAMESPACE_URL);
const name = Buffer.from(`rag-qdrant-point:${sourceVersionId}:${chunkId}`, "utf8");
const hash = createHash("sha1").update(Buffer.concat([namespace, name])).digest().subarray(0, 16);
hash[6] = (hash[6] & 0x0f) | 0x50;
hash[8] = (hash[8] & 0x3f) | 0x80;
return bytesToUuid(hash);
}

View file

@ -0,0 +1,514 @@
import test from "node:test";
import assert from "node:assert/strict";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { createApp } from "../src/app.js";
import { openApiDocument } from "../src/api/openapi.js";
import { env } from "../src/config/env.js";
import { CatalogError } from "../src/modules/catalog/errors.js";
import { buildLegacyMigrationPlan } from "../src/modules/catalog/legacy-migration.js";
import { KnowledgeLifecycleReconciler } from "../src/modules/catalog/reconciler.js";
import { CatalogRepository } from "../src/modules/catalog/repository.js";
import { activateMigratedVersion, createCatalogVersion, rollbackMigratedBatch } from "../src/scripts/migrate-legacy-lifecycle.js";
import { IngestService } from "../src/modules/ingest/service.js";
import { RetrieveService } from "../src/modules/retrieve/service.js";
import type { EmbeddingProvider } from "../src/modules/embeddings/provider.js";
import type { VectorStoreClient } from "../src/modules/vectorstore/client.js";
function setEnvFlag(name: keyof typeof env, value: unknown) {
(env as unknown as Record<string, unknown>)[name] = value;
}
async function withEnvFlags(updates: Partial<Record<keyof typeof env, unknown>>, handler: () => Promise<void>) {
const previous = new Map<keyof typeof env, unknown>();
for (const [name, value] of Object.entries(updates) as Array<[keyof typeof env, unknown]>) {
previous.set(name, env[name]);
setEnvFlag(name, value);
}
try {
await handler();
} finally {
for (const [name, value] of previous) {
setEnvFlag(name, value);
}
}
}
async function withTempMarkdown(content: string, handler: (filePath: string) => Promise<void>) {
const dir = await mkdtemp(path.join(os.tmpdir(), "rag-lifecycle-test-"));
try {
const filePath = path.join(dir, "source.md");
await writeFile(filePath, content, "utf8");
await handler(filePath);
} finally {
await rm(dir, { recursive: true, force: true });
}
}
function fakeEmbeddingProvider(): EmbeddingProvider {
return {
providerName: "test-provider",
modelName: "test-model",
dimensions: 3,
async embed(input: string[]) {
return input.map(() => [0.1, 0.2, 0.3]);
}
};
}
function fakeVectorStore() {
const chunks: unknown[] = [];
const vectorStore: Partial<VectorStoreClient> = {
kind: "fake",
async upsert(input) {
chunks.push(...input);
},
async countVersionPoints() {
return chunks.length;
},
async validateCollectionDimensions() {
return undefined;
}
};
return { vectorStore: vectorStore as VectorStoreClient, chunks };
}
function fakeTransactionPool(handler: (sql: string, params?: unknown[]) => Promise<{ rowCount: number; rows: unknown[] }>) {
const client = {
async query(sql: string, params?: unknown[]) {
if (/^(BEGIN|COMMIT|ROLLBACK|SET CONSTRAINTS)/.test(sql.trim())) {
return { rowCount: 0, rows: [] };
}
return handler(sql, params);
},
release() {
return undefined;
}
};
return {
async connect() {
return client;
},
async query(sql: string, params?: unknown[]) {
return handler(sql, params);
}
};
}
test("activation failure leaves a newly indexed version ready instead of failed", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
const { vectorStore } = fakeVectorStore();
const calls: string[] = [];
const catalog = {
async beginAttempt() { return "attempt-1"; },
async updateAttempt() { return undefined; },
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
async findReusableVersion() { return undefined; },
async createPendingVersion() { return { versionId: "version-1", versionNumber: 1, state: "pending", previousVersionId: null }; },
async markIndexing() { calls.push("indexing"); },
async markReady() { calls.push("ready"); return { versionId: "version-1", versionNumber: 1, state: "ready", previousVersionId: null }; },
async activateVersion() { throw new CatalogError("precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED"); },
async markFailed() { calls.push("failed"); }
};
const service = new IngestService(fakeEmbeddingProvider(), vectorStore, catalog as never);
await assert.rejects(
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: "other" }),
/precondition failed/
);
assert.deepEqual(calls, ["indexing", "ready"]);
}));
});
test("active no-op validates the expected active version precondition", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
const { vectorStore, chunks } = fakeVectorStore();
let preconditionChecked = false;
const catalog = {
async beginAttempt() { return "attempt-1"; },
async updateAttempt() { return undefined; },
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
async findReusableVersion() { return { versionId: "active-1", versionNumber: 1, state: "active", previousVersionId: null, expectedDocumentCount: 1 }; },
async assertActiveVersionPrecondition() {
preconditionChecked = true;
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
},
async markFailed() { throw new Error("markFailed must not run for no-op precondition errors"); }
};
const service = new IngestService(fakeEmbeddingProvider(), vectorStore, catalog as never);
await assert.rejects(
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: "wrong" }),
/precondition failed/i
);
assert.equal(preconditionChecked, true);
assert.equal(chunks.length, 0);
}));
});
test("/sources fails closed and /cleanup refuses deletion when lifecycle enforcement is enabled", async () => {
setEnvFlag("knowledgeLifecycleEnforced", true);
setEnvFlag("lifecycleAdminToken", "test-token");
const server = createApp().listen(0);
try {
const address = server.address();
assert.ok(address && typeof address === "object");
const baseUrl = `http://127.0.0.1:${address.port}`;
const sources = await fetch(`${baseUrl}/sources`);
assert.equal(sources.status, 503);
const cleanupWithoutAdmin = await fetch(`${baseUrl}/cleanup`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ scope: { sourceId: "src:default:folder:docs" } })
});
assert.equal(cleanupWithoutAdmin.status, 401);
const cleanup = await fetch(`${baseUrl}/cleanup`, {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: "Bearer test-token" },
body: JSON.stringify({ scope: { sourceId: "src:default:folder:docs" } })
});
assert.equal(cleanup.status, 410);
} finally {
server.close();
setEnvFlag("knowledgeLifecycleEnforced", false);
setEnvFlag("lifecycleAdminToken", "");
}
});
test("retrieval fails closed when lifecycle is enforced without a catalog", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
const service = new RetrieveService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore);
await assert.rejects(
service.retrieve("documental", "specific", "hello"),
(error) => error instanceof CatalogError && error.statusCode === 503 && error.code === "CATALOG_UNAVAILABLE"
);
});
});
test("repository state guards do not mark non-pending/indexing versions as failed", async () => {
const sqlCalls: string[] = [];
const pool = {
async query(sql: string) {
sqlCalls.push(sql);
return sql.includes("UPDATE rag_source_versions") ? { rowCount: 0, rows: [] } : { rowCount: 1, rows: [] };
}
};
const repository = new CatalogRepository(pool as never);
await repository.markFailed("ready-version", "TEST", "must not change ready");
assert.equal(sqlCalls.some((sql) => sql.includes("UPDATE rag_version_documents")), false);
assert.match(sqlCalls[0], /state IN \('pending', 'indexing'\)/);
});
test("reconciler reports invariant violations and recovers orphaned versions", async () => {
const calls: string[] = [];
const catalog = {
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
async resolveActiveVersions() { return [{ sourceId: "src:one", sourceVersionId: "v1", expectedPointCount: 2, sourceVersionNumber: 1, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }]; },
async validateActiveInvariant() { return ["src:bad"]; },
async listOrphanedIndexingCandidates() { return []; },
async markNeedsReingest(sourceId: string) { calls.push(sourceId); }
};
const vectorStore = { async validateCollectionDimensions() { return undefined; }, async countVersionPoints() { return 1; } };
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
const status = await reconciler.runOnce();
assert.equal(status.ok, false);
assert.deepEqual(status.inconsistentSources, ["src:one"]);
assert.deepEqual(status.invariantViolations, ["src:bad"]);
assert.equal(status.orphanedVersionsRecovered, 0);
assert.deepEqual(calls, ["src:one"]);
});
test("reconciler recovers stale complete indexing versions to ready", async () => {
const calls: string[] = [];
const catalog = {
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
async resolveActiveVersions() { return []; },
async validateActiveInvariant() { return []; },
async listOrphanedIndexingCandidates() {
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
},
async withSourceTryLock(_sourceId: string, handler: () => Promise<unknown>) { calls.push("source-lock"); return handler(); },
async withVersionTryLock(_versionId: string, handler: () => Promise<unknown>) { calls.push("version-lock"); return handler(); },
async hasCompleteVersionDocuments() { return true; },
async recoverVersionReady() { calls.push("ready"); return true; },
async markFailed() { calls.push("failed"); }
};
const vectorStore = { async validateCollectionDimensions() { calls.push("dimensions"); }, async countVersionPoints() { calls.push("count"); return 2; } };
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
const status = await reconciler.runOnce();
assert.equal(status.ok, true);
assert.equal(status.orphanedVersionsRecovered, 1);
assert.equal(status.orphanedVersionsFailed, 0);
assert.deepEqual(calls, ["source-lock", "version-lock", "dimensions", "count", "ready"]);
});
test("reconciler marks stale partial indexing versions as failed", async () => {
const calls: string[] = [];
const catalog = {
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
async resolveActiveVersions() { return []; },
async validateActiveInvariant() { return []; },
async listOrphanedIndexingCandidates() {
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
},
async withSourceTryLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
async withVersionTryLock(_versionId: string, handler: () => Promise<unknown>) { return handler(); },
async hasCompleteVersionDocuments() { return true; },
async recoverVersionReady() { calls.push("ready"); return true; },
async markFailed() { calls.push("failed"); }
};
const vectorStore = { async validateCollectionDimensions() { return undefined; }, async countVersionPoints() { return 1; } };
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
const status = await reconciler.runOnce();
assert.equal(status.orphanedVersionsRecovered, 0);
assert.equal(status.orphanedVersionsFailed, 1);
assert.deepEqual(calls, ["failed"]);
});
test("reconciler does not touch stale versions while a source lock is held", async () => {
const calls: string[] = [];
const catalog = {
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
async resolveActiveVersions() { return []; },
async validateActiveInvariant() { return []; },
async listOrphanedIndexingCandidates() {
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
},
async withSourceTryLock() { return undefined; },
async markFailed() { calls.push("failed"); }
};
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, fakeVectorStore().vectorStore);
const status = await reconciler.runOnce();
assert.equal(status.orphanedVersionsRecovered, 0);
assert.equal(status.orphanedVersionsFailed, 0);
assert.deepEqual(calls, []);
});
test("reconciler skips when another reconciler holds the advisory lock", async () => {
const catalog = {
async withGlobalTryLock() { return undefined; },
async resolveActiveVersions() { throw new Error("must not run without the lock"); }
};
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, fakeVectorStore().vectorStore);
const status = await reconciler.runOnce();
assert.equal(status.ok, true);
assert.equal(status.inconsistentSources.length, 0);
});
test("retrieval rejects active versions from a different Qdrant collection", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
const catalog = {
async resolveActiveVersions() {
return [{ sourceId: "src:one", sourceVersionId: "v1", sourceVersionNumber: 1, expectedPointCount: 1, embeddingDimensions: 3, qdrantCollection: "other_collection" }];
},
async markNeedsReingest() { return undefined; },
async withVersionSharedLocks(_versionIds: string[], handler: () => Promise<unknown>) { return handler(); }
};
const service = new RetrieveService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore, catalog as never);
await assert.rejects(
service.retrieve("documental", "specific", "hello"),
(error) => error instanceof CatalogError && error.code === "SOURCE_VERSION_INCONSISTENT"
);
});
});
test("retrieval holds shared locks while validating counts and searching Qdrant", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
const calls: string[] = [];
const catalog = {
async resolveActiveVersions() {
calls.push("resolve");
return [{ sourceId: "src:one", sourceVersionId: "v1", sourceVersionNumber: 1, expectedPointCount: 1, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
},
async withVersionSharedLocks(_versionIds: string[], handler: () => Promise<unknown>) {
calls.push("lock:start");
const result = await handler();
calls.push("lock:end");
return result;
},
async markNeedsReingest() { calls.push("needs-reingest"); }
};
const vectorStore = {
kind: "fake",
async validateCollectionDimensions() { calls.push("dimensions"); },
async countVersionPoints() { calls.push("count"); return 1; },
async search() { calls.push("search"); return []; },
async browseScope() { return []; }
};
const service = new RetrieveService(fakeEmbeddingProvider(), vectorStore as never, catalog as never);
await service.retrieve("documental", "specific", "hello");
assert.deepEqual(calls, ["resolve", "lock:start", "dimensions", "count", "search", "lock:end"]);
});
});
test("lifecycle ingest rejects vectors that do not match configured embedding dimensions before upsert", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
const calls: string[] = [];
const provider: EmbeddingProvider = {
providerName: "test-provider",
modelName: "test-model",
dimensions: 3,
async embed(input: string[]) {
return input.map(() => [0.1, 0.2]);
}
};
const vectorStore = {
kind: "fake",
async upsert() { calls.push("upsert"); },
async countVersionPoints() { return 0; }
};
const catalog = {
async beginAttempt() { return "attempt-1"; },
async updateAttempt() { return undefined; },
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
async findReusableVersion() { return undefined; },
async createPendingVersion() { return { versionId: "version-1", versionNumber: 1, state: "pending", previousVersionId: null }; },
async markIndexing() { calls.push("indexing"); },
async markReady() { calls.push("ready"); return { versionId: "version-1", versionNumber: 1, state: "ready", previousVersionId: null }; },
async markFailed() { calls.push("failed"); }
};
const service = new IngestService(provider, vectorStore as never, catalog as never);
await assert.rejects(
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: null }),
(error) => error instanceof CatalogError && error.code === "EMBEDDING_DIMENSIONS_INVALID"
);
assert.deepEqual(calls, ["indexing", "failed"]);
}));
});
test("cleanup respects disabled ingest writes even on the legacy path", async () => {
await withEnvFlags({ ingestWritesEnabled: false, knowledgeLifecycleEnforced: false }, async () => {
const service = new IngestService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore);
await assert.rejects(
service.cleanup({ sourceId: "src:default:folder:docs" }),
(error) => error instanceof CatalogError && error.code === "INGEST_WRITES_DISABLED"
);
});
});
test("markPurged fails when the version is no longer in purging state", async () => {
const pool = fakeTransactionPool(async (sql) => {
if (sql.includes("UPDATE rag_source_versions")) {
return { rowCount: 0, rows: [] };
}
return { rowCount: 1, rows: [] };
});
const repository = new CatalogRepository(pool as never);
await assert.rejects(
repository.markPurged("version-1"),
(error) => error instanceof CatalogError && error.code === "INVALID_VERSION_STATE"
);
});
test("legacy migration apply reuses a catalog version created before payload update", async () => {
const source = buildLegacyMigrationPlan([{ id: "p1", payload: {
source_id: "src:default:file:source.md",
source_type: "file",
source_ref: "source.md",
document_id: "doc:src:default:file:source.md:source.md",
document_key: "source.md",
chunk_id: "chk:doc:src:default:file:source.md:source.md:documental:0001",
content: "content",
embedding_provider: "openrouter",
embedding_model: "qwen",
embedding_dimensions: 3
} }]).sources[0];
const sqlCalls: string[] = [];
const pool = fakeTransactionPool(async (sql) => {
sqlCalls.push(sql);
if (sql.includes("FROM rag_legacy_migration_items")) {
return { rowCount: 1, rows: [{ version_id: "version-1", expected_point_count: 1 }] };
}
if (sql.includes("FROM rag_source_versions v")) {
return { rowCount: 1, rows: [{ version_id: "version-1", version_number: "1", expected_point_count: "1", source_type: "file", source_ref: "source.md", qdrant_collection: env.qdrantCollection }] };
}
return { rowCount: 1, rows: [] };
});
const result = await createCatalogVersion(pool as never, "batch-1", source, env.qdrantCollection);
assert.deepEqual(result, { versionId: "version-1", versionNumber: 1 });
assert.equal(sqlCalls.some((sql) => sql.includes("INSERT INTO rag_sources")), false);
});
test("legacy migration activation is idempotent after payload update and prior activation", async () => {
const sqlCalls: string[] = [];
const pool = fakeTransactionPool(async (sql) => {
sqlCalls.push(sql);
if (sql.includes("SELECT v.state")) {
return { rowCount: 1, rows: [{ state: "active", active_version_id: "version-1" }] };
}
return { rowCount: 1, rows: [] };
});
await activateMigratedVersion(pool as never, "src:default:file:source.md", "version-1", 1);
assert.equal(sqlCalls.some((sql) => sql.includes("SET state = 'active'")), false);
});
test("legacy rollback prepares the whole batch before restoring and validates restored legacy counts before catalog cleanup", async () => {
const calls: string[] = [];
const pool = fakeTransactionPool(async (sql) => {
if (sql.includes("SELECT state, snapshot_ref")) {
calls.push("batch-lock");
return { rowCount: 1, rows: [{ state: "verified", snapshot_ref: "snapshot-url" }] };
}
if (sql.includes("SELECT source_id, version_id, expected_point_count")) {
calls.push("items-lock");
return { rowCount: 1, rows: [{ source_id: "src:default:file:source.md", version_id: "version-1", expected_point_count: 1 }] };
}
if (sql.includes("SET active_version_id = NULL")) calls.push("active-null");
if (sql.includes("SET state = 'purging'")) calls.push("purging");
if (sql.includes("COUNT(*) AS count")) {
calls.push("zero-active-check");
return { rowCount: 1, rows: [{ count: "0" }] };
}
if (sql.includes("DELETE FROM rag_version_documents")) calls.push("delete-docs");
if (sql.includes("UPDATE rag_legacy_migration_items SET version_id = NULL")) calls.push("items-rolled-back");
return { rowCount: 1, rows: [] };
});
const qdrant = {
async recoverSnapshot() { calls.push("restore-snapshot"); return true; },
async scroll() {
calls.push("verify-legacy-count");
return { points: [{ id: "p1", payload: { source_id: "src:default:file:source.md" } }], next_page_offset: undefined };
}
};
const result = await rollbackMigratedBatch(pool as never, qdrant as never, "batch-1", "snapshot-url");
assert.equal(result.ok, true);
assert.ok(calls.indexOf("zero-active-check") < calls.indexOf("restore-snapshot"));
assert.ok(calls.indexOf("verify-legacy-count") < calls.indexOf("delete-docs"));
assert.ok(calls.includes("items-rolled-back"));
});
test("OpenAPI documents /sources 503 responses", () => {
const sources = openApiDocument.paths["/sources"].get.responses;
assert.ok("503" in sources);
});
test("legacy migration dry-run blocks weak document identity", () => {
const plan = buildLegacyMigrationPlan([
{
id: "p1",
payload: {
source_id: "src:default:folder:docs",
source_ref: "docs",
source_type: "folder",
document_id: "doc:src:default:folder:docs:readme.md",
chunk_id: "chk:doc:src:default:folder:docs:readme.md:documental:0001",
content: "content",
embedding_provider: "openrouter",
embedding_model: "qwen",
embedding_dimensions: 3
}
}
]);
assert.equal(plan.ok, false);
assert.match(plan.blockedReasons.join("\n"), /weak document identity/);
});

View file

@ -0,0 +1,61 @@
import test from "node:test";
import assert from "node:assert/strict";
import {
buildMetadataHash,
buildProcessingFingerprint,
buildVersionedQdrantPointId,
hashOrderedPairs,
normalizeContentForHash,
sha256Hex
} from "../src/shared/utils/ids.js";
import { CatalogError } from "../src/modules/catalog/errors.js";
import { normalizeExpectedActiveVersion } from "../src/modules/catalog/lifecycle.js";
test("versioned Qdrant point IDs are deterministic UUIDv5 values", () => {
const versionId = "123e4567-e89b-12d3-a456-426614174000";
const chunkId = "chk:doc:src:default:folder:docs:readme.md:documental:0001";
const first = buildVersionedQdrantPointId(versionId, chunkId);
const second = buildVersionedQdrantPointId(versionId, chunkId);
const differentVersion = buildVersionedQdrantPointId("123e4567-e89b-12d3-a456-426614174001", chunkId);
assert.equal(first, second);
assert.notEqual(first, differentVersion);
assert.match(first, /^[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/);
});
test("content hashing normalization is canonical", () => {
const left = normalizeContentForHash("\uFEFFTitle\r\nBody \r\n\r\n");
const right = normalizeContentForHash("Title\nBody\n");
assert.equal(left, "Title\nBody\n");
assert.equal(sha256Hex(left), sha256Hex(right));
});
test("ordered pair hashes are independent from input order", () => {
const first = hashOrderedPairs([["b.md", "hash-b"], ["a.md", "hash-a"]]);
const second = hashOrderedPairs([["a.md", "hash-a"], ["b.md", "hash-b"]]);
assert.equal(first, second);
assert.equal(hashOrderedPairs([["a.md", null]]), null);
});
test("processing and metadata fingerprints are stable with sorted keys", () => {
assert.equal(
buildProcessingFingerprint({ model: "qwen", provider: "openrouter" }),
buildProcessingFingerprint({ provider: "openrouter", model: "qwen" })
);
assert.equal(
buildMetadataHash(["b", "a"], { sourceRef: "docs" }),
buildMetadataHash(["a", "b"], { sourceRef: "docs" })
);
});
test("activation precondition is mandatory when activation is requested", () => {
assert.throws(
() => normalizeExpectedActiveVersion(undefined, true),
(error) => error instanceof CatalogError && error.statusCode === 400
);
assert.equal(normalizeExpectedActiveVersion(null, true), null);
assert.equal(normalizeExpectedActiveVersion(undefined, false), undefined);
});