paco/rag-service
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(rag): add knowledge lifecycle management

Browse source
This commit is contained in:
Paco POR-CORREO 2026-09-11 19:31:45 +02:00
parent e5f739eb6c
commit 551cfe8816
33 changed files with 5308 additions and 261 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
.env.example
View file

@ -11,3 +11,10 @@ ANSWER_PROVIDER=openrouter
ANSWER_MODEL=openai/gpt-4.1-mini ANSWER_MODEL=openai/gpt-4.1-mini
ANSWER_BASE_URL=https://openrouter.ai/api/v1 ANSWER_BASE_URL=https://openrouter.ai/api/v1
ANSWER_API_KEY= ANSWER_API_KEY=
POSTGRES_URL=postgresql://rag:rag@postgres:5432/rag
POSTGRES_SSL=false
LIFECYCLE_ADMIN_TOKEN=
KNOWLEDGE_LIFECYCLE_ENFORCED=false
INGEST_WRITES_ENABLED=true
LIFECYCLE_RECONCILE_INTERVAL_MS=300000
LIFECYCLE_INDEXING_STALE_TIMEOUT_MS=1800000

4
Dockerfile
View file

@ -3,6 +3,7 @@ WORKDIR /app
COPY package.json package-lock.json tsconfig.json ./ COPY package.json package-lock.json tsconfig.json ./
RUN npm ci RUN npm ci
COPY src ./src COPY src ./src
COPY migrations ./migrations
COPY public ./public COPY public ./public
RUN npm run build RUN npm run build
@ -12,6 +13,7 @@ ENV NODE_ENV=production
COPY package.json package-lock.json ./ COPY package.json package-lock.json ./
RUN npm ci --omit=dev RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist COPY --from=build /app/dist ./dist
COPY --from=build /app/migrations ./migrations
COPY public ./public COPY public ./public
EXPOSE 3000 EXPOSE 3000
CMD ["node", "dist/server.js"] CMD ["sh", "-c", "node dist/modules/catalog/migrations.js && node dist/server.js"]

89
docs/API_RAG.md
View file

@ -2,9 +2,9 @@
**Proyecto:** Workspace de tools IA para empresas **Proyecto:** Workspace de tools IA para empresas
**Modulo:** RAG **Modulo:** RAG
**Ultima actualizacion:** 2026-09-08 **Ultima actualizacion:** 2026-09-11
**Ultima modificacion por:** Agente RAG 2 **Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
**Estado:** Operativa y validada en produccion **Estado:** Punto 2 implementado localmente; pendiente de migracion y validacion en produccion
Esta guia explica como descubrir y consumir la API. El contrato tecnico completo y canonico se publica en OpenAPI 3.1.1; este documento prioriza el camino rapido, las decisiones de uso y los ejemplos habituales. Esta guia explica como descubrir y consumir la API. El contrato tecnico completo y canonico se publica en OpenAPI 3.1.1; este documento prioriza el camino rapido, las decisiones de uso y los ejemplos habituales.
@ -24,7 +24,7 @@ https://rag.por-correo.com
`GET /help` indica expresamente que debe consultarse `/openapi.json` para obtener los parametros, cuerpos, respuestas, errores y ejemplos completos de cada endpoint. `GET /help` indica expresamente que debe consultarse `/openapi.json` para obtener los parametros, cuerpos, respuestas, errores y ejemplos completos de cada endpoint.
La API no requiere autenticacion actualmente. No debe asumirse que esta situacion es definitiva: la autenticacion y la proteccion de operaciones sensibles forman parte del pendiente 5. La API general sigue sin autenticacion completa. Las operaciones administrativas del ciclo de vida usan una proteccion acotada con `Authorization: Bearer <LIFECYCLE_ADMIN_TOKEN>`; esto no sustituye la autenticacion general pendiente.
## Elegir la operacion correcta ## Elegir la operacion correcta
@ -32,18 +32,20 @@ La API no requiere autenticacion actualmente. No debe asumirse que esta situacio
|---|---| |---|---|
| Comprobar servicio y capacidades | `GET /health` | | Comprobar servicio y capacidades | `GET /health` |
| Ver scopes ingeridos | `GET /sources` | | Ver scopes ingeridos | `GET /sources` |
| Ver versiones de una fuente | `GET /sources/{sourceId}/versions` |
| Activar, rollback o purgar versiones | `/sources/*` con token administrativo |
| Recuperar contexto para otro agente | `POST /retrieve` | | Recuperar contexto para otro agente | `POST /retrieve` |
| Obtener una respuesta final con citas | `POST /answer` | | Obtener una respuesta final con citas | `POST /answer` |
| Comparar una respuesta sin RAG | `POST /answer/direct` | | Comparar una respuesta sin RAG | `POST /answer/direct` |
| Conversar con bootstrap e historial | `POST /chat` | | Conversar con bootstrap e historial | `POST /chat` |
| Ingerir una ruta accesible por el servidor | `POST /ingest` | | Ingerir una ruta accesible por el servidor | `POST /ingest` |
| Subir un archivo o carpeta ZIP | `POST /ingest/upload` | | Subir un archivo o carpeta ZIP | `POST /ingest/upload` |
| Eliminar una fuente antes de reingestar | `POST /cleanup` | | Eliminar una fuente antes de reingestar | `POST /cleanup` legacy, o purga de version no activa |
| Consultar o actualizar evaluaciones | `/logs/*` | | Consultar o actualizar evaluaciones | `/logs/*` |
## Catalogo completo ## Catalogo completo
La API queda formada por 16 operaciones tras incorporar el descubrimiento: La API queda formada por las operaciones de descubrimiento, ingesta, retrieval, evaluacion y ciclo de vida:
| Metodo | Ruta | Descripcion | | Metodo | Ruta | Descripcion |
|---|---|---| |---|---|---|
@ -52,6 +54,13 @@ La API queda formada por 16 operaciones tras incorporar el descubrimiento:
| `GET` | `/playground` | Interfaz web de prueba | | `GET` | `/playground` | Interfaz web de prueba |
| `GET` | `/health` | Estado, proveedores, parsers y chunking | | `GET` | `/health` | Estado, proveedores, parsers y chunking |
| `GET` | `/sources` | Scopes disponibles en Qdrant | | `GET` | `/sources` | Scopes disponibles en Qdrant |
| `GET` | `/sources/{sourceId}` | Fuente registrada en el catalogo |
| `GET` | `/sources/{sourceId}/versions` | Versiones de una fuente |
| `GET` | `/sources/{sourceId}/versions/{versionId}` | Detalle de una version |
| `POST` | `/sources/{sourceId}/versions/{versionId}/activate` | Activacion administrativa con precondicion |
| `POST` | `/sources/{sourceId}/rollback` | Rollback administrativo sin recalcular embeddings |
| `POST` | `/sources/{sourceId}/versions/{versionId}/retry` | Reintento explicito de version fallida o huerfana con fuente durable |
| `DELETE` | `/sources/{sourceId}/versions/{versionId}` | Purga administrativa de una version no activa |
| `GET` | `/models/answer` | Modelo por defecto y modelos seleccionables | | `GET` | `/models/answer` | Modelo por defecto y modelos seleccionables |
| `POST` | `/ingest` | Ingesta desde una ruta del servidor | | `POST` | `/ingest` | Ingesta desde una ruta del servidor |
| `POST` | `/ingest/upload` | Upload `multipart/form-data` | | `POST` | `/ingest/upload` | Upload `multipart/form-data` |
@ -102,6 +111,15 @@ Los errores JSON siguen esta forma:
Actualmente la mayoria de fallos de validacion o de proveedores se devuelven como `500`. `POST /cleanup` y `POST /ingest/upload` tambien pueden devolver `400`, y `PATCH /logs/{id}` devuelve `404` si no encuentra el log. Mejorar la validacion queda fuera de este bloque para no cambiar silenciosamente el comportamiento existente. Actualmente la mayoria de fallos de validacion o de proveedores se devuelven como `500`. `POST /cleanup` y `POST /ingest/upload` tambien pueden devolver `400`, y `PATCH /logs/{id}` devuelve `404` si no encuentra el log. Mejorar la validacion queda fuera de este bloque para no cambiar silenciosamente el comportamiento existente.
Con `KNOWLEDGE_LIFECYCLE_ENFORCED=true`, la API devuelve errores mas especificos para el catalogo:
| Codigo HTTP | Caso |
|---|---|
| `400` | Falta `expectedActiveVersionId` cuando `activate=true` |
| `409` | Cambio concurrente de version activa, version en curso o intento de purgar una activa |
| `422` | Fuente vacia, sin documentos soportados o sin contenido util |
| `503` | PostgreSQL, Qdrant, embeddings o version activa inconsistente |
## Recuperar contexto ## Recuperar contexto
`POST /retrieve` es la operacion recomendada cuando otro agente o flujo generara la respuesta final. `POST /retrieve` es la operacion recomendada cuando otro agente o flujo generara la respuesta final.
@ -248,12 +266,22 @@ Respuesta:
"sourceType": "folder", "sourceType": "folder",
"sourceRef": "/data/project-docs", "sourceRef": "/data/project-docs",
"mode": "mechanical", "mode": "mechanical",
"tags": ["project", "documentation"] "tags": ["project", "documentation"],
"activate": true,
"expectedActiveVersionId": null
} }
``` ```
`readPath` es un campo interno para separar la ruta fisica de lectura de la referencia logica. Los consumidores normales no deben enviarlo. `readPath` es un campo interno para separar la ruta fisica de lectura de la referencia logica. Los consumidores normales no deben enviarlo.
Con ciclo de vida activo:
- `activate` por defecto es `true`.
- si `activate=true`, `expectedActiveVersionId` es obligatorio: `null` para primera activacion y UUID para reemplazar una version existente.
- una ingesta identica a la version activa responde `noOp: true` y no escribe nuevos puntos.
- `activate=false` deja la version en `ready` para activarla despues con una operacion administrativa.
- la respuesta exitosa cambia a `201 Created` e incluye `sourceId`, `versionId`, `versionNumber`, `state`, `previousVersionId`, `activated` y `noOp`.
### Upload ### Upload
`POST /ingest/upload` usa `multipart/form-data`: `POST /ingest/upload` usa `multipart/form-data`:
@ -265,6 +293,8 @@ Respuesta:
| `mode` | No | `mechanical` o `interactive` | | `mode` | No | `mechanical` o `interactive` |
| `tags` | No | Lista separada por comas | | `tags` | No | Lista separada por comas |
| `isZipFolder` | No | Texto `true` o `false` | | `isZipFolder` | No | Texto `true` o `false` |
| `activate` | No | Texto `true` o `false` |
| `expectedActiveVersionId` | Si activa | UUID o texto `null` |
```bash ```bash
curl -sS -X POST "https://rag.por-correo.com/ingest/upload" \ curl -sS -X POST "https://rag.por-correo.com/ingest/upload" \
@ -274,11 +304,52 @@ curl -sS -X POST "https://rag.por-correo.com/ingest/upload" \
-F "tags=customer-a,support" -F "tags=customer-a,support"
``` ```
Ambas variantes responden `202` con el origen aceptado, archivos descubiertos, documentos procesados, chunks almacenados y coleccion. El upload añade `uploadedResource`. Con enforcement desactivado, ambas variantes conservan `202`. Con enforcement activado, responden `201` y usan el catalogo PostgreSQL como autoridad de version activa. El upload añade `uploadedResource`.
## Ciclo de vida de fuentes
El catalogo PostgreSQL es la autoridad para saber que fuentes existen, que versiones tienen y cual es la version activa. Qdrant conserva los puntos fisicos versionados, pero retrieval solo consulta versiones activas resueltas desde PostgreSQL.
### Listado y detalle
```text
GET /sources
GET /sources/{sourceId}
GET /sources/{sourceId}/versions
GET /sources/{sourceId}/versions/{versionId}
```
`GET /sources` mantiene los campos usados por el playground (`sourceId`, `sourceRef`, `chunkModes`, `tags`) y añade `activeVersionId`, `activeVersionNumber`, `state`, `needsReingest` y `updatedAt`.
### Activacion y rollback
```bash
curl -sS -X POST "https://rag.por-correo.com/sources/src:default:folder:docs/versions/<version-id>/activate" \
-H "Authorization: Bearer $LIFECYCLE_ADMIN_TOKEN" \
-H "Content-Type: application/json" \
-d '{"expectedActiveVersionId":"<active-version-id>"}'
```
Rollback usa la misma activacion atomica del puntero PostgreSQL:
```json
{
"targetVersionId": "uuid-version-anterior",
"expectedActiveVersionId": "uuid-version-activa-actual"
}
```
### Purga
`DELETE /sources/{sourceId}/versions/{versionId}` exige token administrativo y solo acepta versiones no activas. La purga marca la version como `purging`, borra puntos Qdrant con `wait=true`, verifica que no quedan puntos y finalmente deja la version como `purged`.
### Reintento explicito
`POST /sources/{sourceId}/versions/{versionId}/retry` acepta versiones `failed`, `pending` o `indexing`. Si recibe una fuente durable (`sourceType`, `sourceRef` y opcionalmente `readPath`) cierra versiones huerfanas como fallidas y crea una version nueva mediante `/ingest`. Si no recibe fuente durable, devuelve `202` con instrucciones: los uploads temporales no se reintentan en sitio porque el original ya no esta garantizado.
## Limpieza controlada ## Limpieza controlada
`POST /cleanup` elimina los chunks que coinciden con un scope: `POST /cleanup` elimina los chunks que coinciden con un scope solo en modo legacy y con `INGEST_WRITES_ENABLED=true`:
```json ```json
{ {

1169
docs/CONTRATO_CICLO_VIDA_Y_OCR.md Normal file
View file

File diff suppressed because it is too large Load diff

43
docs/DESPLIEGUE_EASYPANEL.md
View file

@ -2,9 +2,9 @@
**Proyecto:** Workspace de tools IA para empresas **Proyecto:** Workspace de tools IA para empresas
**Modulo:** RAG **Modulo:** RAG
**Ultima actualizacion:** 2026-04-05 **Ultima actualizacion:** 2026-09-11
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales **Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
**Estado:** V1 desplegada en VPS2 **Estado:** V1 desplegada en VPS2; ciclo de vida implementado localmente pendiente de migracion productiva
--- ---
@ -200,6 +200,12 @@ Dato importante para `RAG`:
- `ANSWER_MODEL=openai/gpt-4.1-mini` - `ANSWER_MODEL=openai/gpt-4.1-mini`
- `ANSWER_BASE_URL=https://openrouter.ai/api/v1` - `ANSWER_BASE_URL=https://openrouter.ai/api/v1`
- `ANSWER_API_KEY=<valor real>` - `ANSWER_API_KEY=<valor real>`
- `POSTGRES_URL=<conexion interna a PostgreSQL>`
- `POSTGRES_SSL=false` salvo que la conexion lo exija
- `LIFECYCLE_ADMIN_TOKEN=<secreto administrativo>`
- `KNOWLEDGE_LIFECYCLE_ENFORCED=false` durante despliegue inicial y migracion
- `INGEST_WRITES_ENABLED=false` durante la ventana de migracion legacy
- `LIFECYCLE_RECONCILE_INTERVAL_MS=300000`
### Variables de entorno con las que quedo operativo en VPS2 ### Variables de entorno con las que quedo operativo en VPS2
@ -219,6 +225,15 @@ Valores funcionales confirmados en EasyPanel:
- `ANSWER_BASE_URL=https://openrouter.ai/api/v1` - `ANSWER_BASE_URL=https://openrouter.ai/api/v1`
- `ANSWER_API_KEY=<secreto>` - `ANSWER_API_KEY=<secreto>`
Variables pendientes de completar para activar el punto 2 en produccion:
- `POSTGRES_URL=<secreto>`
- `POSTGRES_SSL=false`
- `LIFECYCLE_ADMIN_TOKEN=<secreto>`
- `KNOWLEDGE_LIFECYCLE_ENFORCED=false` hasta completar la migracion legacy
- `INGEST_WRITES_ENABLED=true` en operacion normal; `false` durante migracion
- `LIFECYCLE_RECONCILE_INTERVAL_MS=300000`
Para respaldo operativo local fuera de EasyPanel se deja un archivo ignorado por Git: Para respaldo operativo local fuera de EasyPanel se deja un archivo ignorado por Git:
- `RAG/.env.easypanel.local` - `RAG/.env.easypanel.local`
@ -228,7 +243,27 @@ Ese archivo no debe versionarse porque contiene secretos reales.
### Volumen persistente ### Volumen persistente
- no es obligatorio para la app RAG en esta fase - no es obligatorio para la app RAG en esta fase
- el estado persistente principal vive en `qdrant` - el estado persistente vectorial vive en `qdrant`
- desde el punto 2, el catalogo de fuentes y versiones vive en PostgreSQL
### Arranque con migraciones
La imagen arranca ejecutando primero:
```bash
node dist/modules/catalog/migrations.js && node dist/server.js
```
Si `POSTGRES_URL` no esta configurada, el migrador informa que omite migraciones. Si PostgreSQL esta configurado y una migracion falla o cambia su checksum, el servicio no debe completar el arranque.
### Secuencia operativa del punto 2
1. Desplegar el codigo con `KNOWLEDGE_LIFECYCLE_ENFORCED=false`.
2. Bloquear escrituras con `INGEST_WRITES_ENABLED=false` durante la migracion.
3. Ejecutar `npm run migrate:legacy:lifecycle` en modo dry-run y revisar colisiones.
4. Tomar snapshot Qdrant antes de cualquier migracion destructiva.
5. Catalogar el corpus legacy y verificar recuentos PostgreSQL/Qdrant. Si el proceso cae, reanudar con `npm run migrate:legacy:lifecycle -- --apply --snapshot-ref=<snapshot> --batch-id=<batch-id>`.
6. Activar `KNOWLEDGE_LIFECYCLE_ENFORCED=true` solo despues de validar retrieval y playground.
--- ---

120
docs/HISTORIAL_SESIONES.md
View file

@ -93,3 +93,123 @@ Dar continuidad al RAG en `RAG/` a partir del estado actual documentado.
- Pendiente 1 completado y validado definitivamente en `https://rag.por-correo.com`. - Pendiente 1 completado y validado definitivamente en `https://rag.por-correo.com`.
- La API dispone de descubrimiento rapido y contrato OpenAPI para sus 16 operaciones. - La API dispone de descubrimiento rapido y contrato OpenAPI para sus 16 operaciones.
- Siguiente prioridad: ciclo de vida del conocimiento. - Siguiente prioridad: ciclo de vida del conocimiento.
---
### 2026-09-08 - Agente RAG 2
**Modelo:** gpt-5.6-sol
**Session ID OpenCode:** `ses_29bdbd003ffeLrLjUlFgnp08Y7`
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
**Subagentes de analisis:**
- `Subagente Extraccion de versiones PaddleOCR`: contraste de versiones PyPI compatibles para el benchmark CPU.
- `Subagente Validacion del ciclo de vida`: revision adversarial del catalogo PostgreSQL y versionado Qdrant.
- `Subagente Validacion OCR`: revision adversarial de deteccion, servicio privado, quality gates y revision humana.
- `Subagente Auditoria de claridad del contrato`: detecto contradicciones de hashes, idempotencia, migracion y revision.
- `Subagente Revalidacion de ejecutabilidad`: detecto bloqueos restantes en metadata, concurrencia y artefactos.
- `Subagente Revalidacion de resiliencia`: reviso carreras, barreras de migracion y recuperacion de jobs.
- `Subagente Revision final del contrato`: verifico y cerro la secuencia segura de rollback legacy.
**Trabajo realizado:**
- Auditoria del pipeline actual de ingesta, IDs, Qdrant, parser PDF, API e infraestructura EasyPanel.
- Confirmacion de PostgreSQL 17 como catalogo transaccional y Qdrant como almacenamiento vectorial versionado.
- Benchmark aislado en VPS2 con el PDF mixto real de FacturaTech.
- Descarte de Tesseract como motor unico por omitir identificadores criticos.
- Seleccion de PaddleOCR como motor principal y deteccion de errores alfanumericos de alta confianza que obligan a revision humana.
- Creacion de `docs/CONTRATO_CICLO_VIDA_Y_OCR.md` con esquema, estados, APIs, migracion, OCR, tareas, pruebas, despliegue y rollback.
- Enlace del contrato desde el backlog canonico.
- Correccion del contrato mediante revisiones adversariales hasta eliminar sus bloqueos criticos conocidos.
- Limpieza de los artefactos e imagenes temporales usados en el benchmark local y de VPS2.
**Estado final:**
- Diseño de los pendientes 2 y 3 cerrado y listo para implementacion secuencial.
- El punto 2 debe completarse y validarse en produccion antes de iniciar el punto 3.
- No se modifico codigo ni produccion durante esta fase de diseño.
**Archivos modificados:**
- `RAG/docs/CONTRATO_CICLO_VIDA_Y_OCR.md`
- `RAG/docs/PENDIENTES_RAG.md`
- `RAG/docs/HISTORIAL_SESIONES.md`
- `docs/INDICE_DOCUMENTACION.md`
- `docs/HISTORIAL_SESIONES.md`
---
### 2026-09-11 - Subagente Implementacion ciclo de vida del conocimiento
**Modelo:** openai/gpt-5.5
**Session ID OpenCode:** `ses_f6fe0ad30ffeiottYsATsRemCo`
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
**Rol asumido:**
Ejecutar la fase apply del punto 2 del RAG y corregir los bloqueadores finales detectados antes de commit, push o despliegue.
**Trabajo realizado:**
- Ajuste del migrador legacy para permitir reanudacion por `--batch-id`, reutilizando versiones ya creadas y completando activacion tras actualizaciones de payload ya aplicadas.
- Endurecimiento de rollback legacy con lock global de migracion, comprobacion de snapshot y operaciones serializadas sobre fuentes/versiones.
- Reconciliador protegido con advisory lock no bloqueante, timeout configurable basado en `indexing_started_at` y validacion de coleccion/dimensiones.
- Protecciones adicionales: `POST /cleanup` respeta `INGEST_WRITES_ENABLED`, `markPurged` valida `rowCount`, rollback usa locks y OpenAPI refleja `503` en health con `ok` booleano.
- Script `migrate:legacy:lifecycle` ajustado para ejecutarse desde `dist/` con Node tras build.
- Añadidos tests unitarios para reanudacion legacy, activacion idempotente, cleanup con escrituras deshabilitadas, reconciliador, mismatch de coleccion y `markPurged`.
**Validacion local:**
- `npm run check` correcto.
- `npm test` correcto: 18 tests pasan.
- `npm run build` correcto.
- `git diff --check` correcto.
**Estado final:**
- Punto 2 implementado y validado localmente.
- No se hizo commit, push, deploy ni lectura de `.env`.
- Sigue pendiente la verificacion independiente y la validacion/migracion en produccion antes de marcar el punto 2 como cerrado.
**Archivos modificados en esta correccion:**
- `.env.example`
- `package.json`
- `package-lock.json`
- `src/config/env.ts`
- `src/modules/catalog/client.ts`
- `src/modules/catalog/repository.ts`
- `src/modules/catalog/reconciler.ts`
- `src/modules/vectorstore/client.ts`
- `src/modules/retrieve/service.ts`
- `src/modules/ingest/service.ts`
- `src/scripts/migrate-legacy-lifecycle.ts`
- `src/api/openapi.ts`
- `tests/lifecycle-services.test.ts`
- `docs/API_RAG.md`
- `docs/DESPLIEGUE_EASYPANEL.md`
- `docs/SISTEMA_RAG_BASE.md`
- `RAG/docs/HISTORIAL_SESIONES.md`
---
### 2026-09-11 (reanudacion final) - Subagente Implementacion ciclo de vida del conocimiento
**Modelo:** openai/gpt-5.5
**Session ID OpenCode:** `ses_f6fe0ad30ffeiottYsATsRemCo`
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
**Rol asumido:**
Completar los 4 bloqueos del sdd-verify y cerrar la validacion local del punto 2.
**Trabajo realizado:**
- Reconciliador: recupera versiones huerfanas `pending/indexing` a `ready` contando puntos Qdrant y validando documentos, coleccion y dimensiones; marca `failed` solo si quedan incompletas; try-locks evitan matar ingestas vivas.
- Ingesta: rechaza embeddings con dimensiones distintas de las esperadas antes del upsert y antes de alcanzar `ready`.
- Rollback legacy: transaccion diferida del lote, confirmacion de cero punteros activos, restauracion de snapshot con verificacion de recuentos y limpieza posterior del catalogo.
- OpenAPI: `/sources` documenta `503`.
- Retrieval: locks compartidos adquiridos antes de validar recuento/dimensiones y mantenidos durante toda la consulta Qdrant.
- Interrupcion por limite de uso a mitad de la validacion; reanudada tras cambio de cuenta y completada hasta el final.
**Validacion local:**
- `npm run check` correcto.
- `npm test` correcto: 25/25 tests pasan.
- `npm run build` correcto.
- `git diff --check` correcto.
- Verificacion independiente (sdd-verify): PASS local.
**Estado final:**
- Implementacion local del punto 2 verificada de forma independiente: PASS local.
- No se hizo commit, push, deploy ni lectura de `.env`.
- Pendiente: commit/push, deploy con enforcement desactivado, migracion legacy y validacion en produccion antes de cerrar el punto 2.

51
docs/INGESTA.md
View file

@ -2,9 +2,9 @@
**Proyecto:** Workspace de tools IA para empresas **Proyecto:** Workspace de tools IA para empresas
**Modulo:** RAG **Modulo:** RAG
**Ultima actualizacion:** 2026-04-02 **Ultima actualizacion:** 2026-09-11
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales **Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
**Estado:** En definicion **Estado:** Punto 2 implementado localmente; pendiente de migracion y validacion en produccion
--- ---
@ -531,3 +531,48 @@ El enriquecimiento ligero debe ser:
2. Como construiremos exactamente `source_id`, `document_id` y `chunk_id`. 2. Como construiremos exactamente `source_id`, `document_id` y `chunk_id`.
3. Que nombre tendran las fuentes logicas por defecto para documentos sueltos. 3. Que nombre tendran las fuentes logicas por defecto para documentos sueltos.
4. Como distinguiremos entre fuente nueva, fuente actualizada y fuente obsoleta. 4. Como distinguiremos entre fuente nueva, fuente actualizada y fuente obsoleta.
---
## Ciclo de vida del conocimiento implementado localmente
El punto 2 introduce un catalogo PostgreSQL como autoridad de fuentes y versiones. Qdrant deja de decidir que conocimiento esta activo: cada punto fisico queda asociado a `source_version_id`, y retrieval filtra por las versiones activas resueltas desde PostgreSQL cuando `KNOWLEDGE_LIFECYCLE_ENFORCED=true`.
### Flujo de ingesta versionada
1. Resolver `source_id` y registrar un intento en `rag_ingestion_attempts`.
2. Descubrir documentos soportados y calcular hashes de originales.
3. Normalizar contenido y calcular hashes de contenido, fingerprint de procesamiento y hash de metadatos.
4. Detectar si ya existe una version reutilizable.
5. Crear una version `pending`, pasarla a `indexing`, escribir puntos Qdrant versionados y verificar recuentos.
6. Pasar la version a `ready`.
7. Si `activate=true`, activar con precondicion `expectedActiveVersionId`.
Una escritura parcial en Qdrant no queda visible para retrieval hasta que PostgreSQL mueve atomica y correctamente el puntero `active_version_id`.
### Nuevos campos de ingesta
```json
{
"activate": true,
"expectedActiveVersionId": null
}
```
- `activate=true` exige `expectedActiveVersionId`.
- Para primera activacion se envia `null`.
- Para reemplazo se envia el UUID de la version activa que el cliente espera reemplazar.
- `activate=false` permite dejar una version preparada en estado `ready`.
### Identidad y hashes
- `document_key` de uploads se deriva de `source_ref`, no de rutas temporales.
- `original_hash` usa SHA-256 de bytes originales.
- `content_hash` usa SHA-256 del contenido normalizado.
- `source_content_hash` y `original_manifest_hash` usan pares ordenados `document_key + NUL + hash`.
- `processing_fingerprint` cambia cuando cambian parser, normalizacion, chunking, proveedor, modelo o dimensiones de embeddings.
- El ID fisico de Qdrant usa UUIDv5 con `source_version_id + chunk_id`.
### Compatibilidad temporal
Con `KNOWLEDGE_LIFECYCLE_ENFORCED=false`, el camino legacy se conserva para permitir despliegue, dry-run y migracion controlada. La validacion final del punto 2 exige activar enforcement despues de catalogar el corpus legacy.

5
docs/PENDIENTES_RAG.md
View file

@ -17,6 +17,8 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
## 2. Ciclo de vida del conocimiento ## 2. Ciclo de vida del conocimiento
**Diseño:** Contrato de implementacion cerrado en [`CONTRATO_CICLO_VIDA_Y_OCR.md`](./CONTRATO_CICLO_VIDA_Y_OCR.md). Pendiente de implementar y validar en produccion antes de iniciar el punto 3.
- Crear un catalogo de fuentes y versiones ingeridas. - Crear un catalogo de fuentes y versiones ingeridas.
- Saber que documento esta vigente, obsoleto, reemplazado o pendiente de reingesta. - Saber que documento esta vigente, obsoleto, reemplazado o pendiente de reingesta.
- Evitar duplicados y permitir actualizaciones incrementales controladas. - Evitar duplicados y permitir actualizaciones incrementales controladas.
@ -25,6 +27,8 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
## 3. OCR integrado en la ingesta ## 3. OCR integrado en la ingesta
**Diseño:** Contrato de implementacion cerrado en [`CONTRATO_CICLO_VIDA_Y_OCR.md`](./CONTRATO_CICLO_VIDA_Y_OCR.md). Su ejecucion queda bloqueada hasta completar el punto 2.
- Detectar PDFs con capturas, imagenes o una capa textual insuficiente. - Detectar PDFs con capturas, imagenes o una capa textual insuficiente.
- Ejecutar OCR automaticamente o bloquear la ingesta para revision cuando no pueda garantizarse la cobertura. - Ejecutar OCR automaticamente o bloquear la ingesta para revision cuando no pueda garantizarse la cobertura.
- Generar un documento intermedio auditable cuando sea necesario. - Generar un documento intermedio auditable cuando sea necesario.
@ -46,6 +50,7 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
- Aplicar rate limiting y limites de tamano o tipo de archivo. - Aplicar rate limiting y limites de tamano o tipo de archivo.
- Revisar CORS, exposicion publica, registros sensibles y permisos por operacion. - Revisar CORS, exposicion publica, registros sensibles y permisos por operacion.
- Sustituir la configuracion sin credenciales de n8n cuando exista el mecanismo de autenticacion definitivo. - Sustituir la configuracion sin credenciales de n8n cuando exista el mecanismo de autenticacion definitivo.
- Actualizar Express/`qs` para cerrar los avisos moderados pendientes de `npm audit`.
## 6. Pruebas automatizadas ## 6. Pruebas automatizadas

14
docs/SISTEMA_RAG_BASE.md
View file

@ -2,9 +2,9 @@
**Proyecto:** Workspace de tools IA para empresas **Proyecto:** Workspace de tools IA para empresas
**Modulo:** RAG **Modulo:** RAG
**Ultima actualizacion:** 2026-04-05 **Ultima actualizacion:** 2026-09-11
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales **Ultima modificacion por:** Agente RAG 2
**Estado:** V1 operativa desplegada **Estado:** V1 operativa; ciclo de vida implementado localmente y pendiente de despliegue
--- ---
@ -105,6 +105,14 @@ Pruebas funcionales ya superadas:
- consulta conceptual sobre caracteristicas del RAG - consulta conceptual sobre caracteristicas del RAG
- consulta tecnica en modo codigo sobre la construccion de `source_id` - consulta tecnica en modo codigo sobre la construccion de `source_id`
### Ciclo de vida del conocimiento
Cada fuente dispone de versiones controladas en PostgreSQL, mientras Qdrant conserva sus chunks y embeddings. Solo la version activa participa en las consultas.
Esto permite actualizar una fuente sin mezclar contenido antiguo y nuevo, detectar ingestas identicas, conservar versiones anteriores y hacer rollback sin recalcular embeddings. Los parametros de consulta actuales no cambian.
Esta mejora esta implementada y validada localmente. Falta desplegarla y registrar el corpus existente como version inicial antes de activarla en produccion.
Los pendientes vigentes, su prioridad y su estado se mantienen en [`PENDIENTES_RAG.md`](./PENDIENTES_RAG.md). Los pendientes vigentes, su prioridad y su estado se mantienen en [`PENDIENTES_RAG.md`](./PENDIENTES_RAG.md).
--- ---

206
migrations/001_knowledge_lifecycle.sql Normal file
View file

@ -0,0 +1,206 @@
CREATE EXTENSION IF NOT EXISTS pgcrypto;
CREATE TABLE IF NOT EXISTS rag_sources (
source_id text PRIMARY KEY,
tenant text NOT NULL DEFAULT 'default',
source_type text NOT NULL CHECK (source_type IN ('file', 'folder')),
source_ref text NOT NULL,
active_version_id uuid NULL,
needs_reingest boolean NOT NULL DEFAULT false,
disabled_at timestamptz NULL,
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS rag_source_versions (
version_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
source_id text NOT NULL REFERENCES rag_sources(source_id) ON DELETE RESTRICT,
version_number bigint NOT NULL CHECK (version_number > 0),
previous_version_id uuid NULL,
state text NOT NULL CHECK (state IN (
'pending', 'indexing', 'review_required', 'ready', 'active',
'superseded', 'failed', 'rejected', 'purging', 'purged'
)),
original_manifest_hash char(64) NOT NULL,
source_content_hash char(64) NULL,
processing_fingerprint char(64) NOT NULL,
metadata_hash char(64) NOT NULL,
tags text[] NOT NULL DEFAULT '{}',
activate_requested boolean NOT NULL DEFAULT true,
base_active_version_id uuid NULL,
embedding_provider text NOT NULL,
embedding_model text NOT NULL,
embedding_dimensions integer NOT NULL CHECK (embedding_dimensions > 0),
qdrant_collection text NOT NULL,
expected_document_count integer NOT NULL DEFAULT 0 CHECK (expected_document_count >= 0),
expected_point_count integer NOT NULL DEFAULT 0 CHECK (expected_point_count >= 0),
verified_point_count integer NOT NULL DEFAULT 0 CHECK (verified_point_count >= 0),
error_code text NULL,
error_detail text NULL,
created_at timestamptz NOT NULL DEFAULT now(),
indexing_started_at timestamptz NULL,
ready_at timestamptz NULL,
activated_at timestamptz NULL,
superseded_at timestamptz NULL,
reviewed_at timestamptz NULL,
reviewed_by text NULL,
artifact_state text NOT NULL DEFAULT 'none'
CHECK (artifact_state IN ('none', 'present', 'retention_deleting', 'retention_deleted')),
retention_due_at timestamptz NULL,
purged_at timestamptz NULL,
UNIQUE (source_id, version_number),
UNIQUE (version_id, source_id),
FOREIGN KEY (previous_version_id, source_id)
REFERENCES rag_source_versions(version_id, source_id)
DEFERRABLE INITIALLY DEFERRED,
FOREIGN KEY (base_active_version_id, source_id)
REFERENCES rag_source_versions(version_id, source_id)
DEFERRABLE INITIALLY DEFERRED
);
DO $$
BEGIN
IF NOT EXISTS (
SELECT 1 FROM pg_constraint WHERE conname = 'rag_sources_active_version_fk'
) THEN
ALTER TABLE rag_sources
ADD CONSTRAINT rag_sources_active_version_fk
FOREIGN KEY (active_version_id, source_id)
REFERENCES rag_source_versions(version_id, source_id)
DEFERRABLE INITIALLY DEFERRED;
END IF;
END $$;
CREATE UNIQUE INDEX IF NOT EXISTS rag_one_active_version_per_source
ON rag_source_versions(source_id)
WHERE state = 'active';
CREATE UNIQUE INDEX IF NOT EXISTS rag_one_reusable_content_version
ON rag_source_versions(source_id, source_content_hash, processing_fingerprint, metadata_hash)
WHERE source_content_hash IS NOT NULL
AND state NOT IN ('failed', 'rejected', 'purged');
CREATE INDEX IF NOT EXISTS rag_source_versions_source_state_idx
ON rag_source_versions(source_id, state);
CREATE TABLE IF NOT EXISTS rag_version_documents (
version_id uuid NOT NULL REFERENCES rag_source_versions(version_id) ON DELETE RESTRICT,
document_id text NOT NULL,
document_key text NOT NULL,
original_hash char(64) NOT NULL,
original_hash_kind text NOT NULL CHECK (original_hash_kind IN ('bytes', 'legacy-derived')),
content_hash char(64) NULL,
mime_type text NOT NULL,
title text NOT NULL,
extraction_method text NOT NULL DEFAULT 'native',
index_state text NOT NULL CHECK (index_state IN ('pending', 'indexing', 'ready', 'failed')),
chunk_count integer NOT NULL DEFAULT 0 CHECK (chunk_count >= 0),
artifact_manifest_path text NULL,
error_detail text NULL,
created_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (version_id, document_id),
UNIQUE (version_id, document_key)
);
CREATE INDEX IF NOT EXISTS rag_version_documents_document_idx
ON rag_version_documents(document_id);
CREATE TABLE IF NOT EXISTS rag_ingestion_attempts (
attempt_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
source_id text NOT NULL REFERENCES rag_sources(source_id) ON DELETE RESTRICT,
version_id uuid NULL REFERENCES rag_source_versions(version_id) ON DELETE SET NULL,
state text NOT NULL CHECK (state IN ('received', 'processing', 'completed', 'failed')),
input_locator text NOT NULL,
input_hash char(64) NULL,
error_code text NULL,
error_detail text NULL,
created_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz NULL
);
CREATE TABLE IF NOT EXISTS rag_system_state (
singleton boolean PRIMARY KEY DEFAULT true CHECK (singleton),
maintenance boolean NOT NULL DEFAULT false,
maintenance_reason text NULL,
updated_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO rag_system_state(singleton) VALUES (true)
ON CONFLICT (singleton) DO NOTHING;
CREATE TABLE IF NOT EXISTS rag_legacy_migration_batches (
batch_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
state text NOT NULL CHECK (state IN ('planned', 'running', 'verified', 'rolled_back', 'failed')),
snapshot_ref text NOT NULL,
report jsonb NOT NULL DEFAULT '{}',
created_at timestamptz NOT NULL DEFAULT now(),
completed_at timestamptz NULL
);
CREATE TABLE IF NOT EXISTS rag_legacy_migration_items (
batch_id uuid NOT NULL REFERENCES rag_legacy_migration_batches(batch_id) ON DELETE RESTRICT,
source_id text NOT NULL,
version_id uuid NULL REFERENCES rag_source_versions(version_id) ON DELETE SET NULL,
expected_point_count integer NOT NULL CHECK (expected_point_count >= 0),
state text NOT NULL CHECK (state IN ('planned', 'migrated', 'verified', 'rolled_back', 'failed')),
error_detail text NULL,
PRIMARY KEY (batch_id, source_id)
);
CREATE OR REPLACE FUNCTION rag_touch_updated_at()
RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
NEW.updated_at = now();
RETURN NEW;
END;
$$;
DROP TRIGGER IF EXISTS rag_sources_touch_updated_at ON rag_sources;
CREATE TRIGGER rag_sources_touch_updated_at
BEFORE UPDATE ON rag_sources
FOR EACH ROW EXECUTE FUNCTION rag_touch_updated_at();
CREATE OR REPLACE FUNCTION rag_validate_active_source_pointer()
RETURNS trigger LANGUAGE plpgsql AS $$
DECLARE
invalid_pointer_count integer;
orphan_active_count integer;
BEGIN
SELECT count(*) INTO invalid_pointer_count
FROM rag_sources s
LEFT JOIN rag_source_versions v
ON v.version_id = s.active_version_id
AND v.source_id = s.source_id
AND v.state = 'active'
WHERE s.active_version_id IS NOT NULL
AND v.version_id IS NULL;
IF invalid_pointer_count > 0 THEN
RAISE EXCEPTION 'rag_sources.active_version_id must point to an active version in the same source';
END IF;
SELECT count(*) INTO orphan_active_count
FROM rag_source_versions v
JOIN rag_sources s ON s.source_id = v.source_id
WHERE v.state = 'active'
AND s.active_version_id IS DISTINCT FROM v.version_id;
IF orphan_active_count > 0 THEN
RAISE EXCEPTION 'every active rag_source_versions row must be the source active_version_id';
END IF;
RETURN NULL;
END;
$$;
DROP TRIGGER IF EXISTS rag_sources_validate_active_pointer ON rag_sources;
CREATE CONSTRAINT TRIGGER rag_sources_validate_active_pointer
AFTER INSERT OR UPDATE ON rag_sources
DEFERRABLE INITIALLY DEFERRED
FOR EACH ROW EXECUTE FUNCTION rag_validate_active_source_pointer();
DROP TRIGGER IF EXISTS rag_versions_validate_active_pointer ON rag_source_versions;
CREATE CONSTRAINT TRIGGER rag_versions_validate_active_pointer
AFTER INSERT OR UPDATE ON rag_source_versions
DEFERRABLE INITIALLY DEFERRED
FOR EACH ROW EXECUTE FUNCTION rag_validate_active_source_pointer();

509
package-lock.json generated
View file

@ -9,12 +9,13 @@
"version": "0.1.0", "version": "0.1.0",
"dependencies": { "dependencies": {
"@qdrant/js-client-rest": "^1.15.0", "@qdrant/js-client-rest": "^1.15.0",
"adm-zip": "^0.5.17", "adm-zip": "^0.6.1",
"dotenv": "^16.4.5", "dotenv": "^16.4.5",
"express": "^4.21.2", "express": "^4.21.2",
"multer": "^2.0.0", "multer": "^2.3.0",
"openai": "^4.104.0", "openai": "^4.104.0",
"pdf-parse": "^1.1.1" "pdf-parse": "^1.1.1",
"pg": "^8.23.0"
}, },
"devDependencies": { "devDependencies": {
"@types/adm-zip": "^0.5.8", "@types/adm-zip": "^0.5.8",
@ -22,14 +23,15 @@
"@types/multer": "^1.4.12", "@types/multer": "^1.4.12",
"@types/node": "^22.15.3", "@types/node": "^22.15.3",
"@types/pdf-parse": "^1.1.4", "@types/pdf-parse": "^1.1.4",
"@types/pg": "^8.23.1",
"tsx": "^4.19.3", "tsx": "^4.19.3",
"typescript": "^5.8.3" "typescript": "^5.8.3"
} }
}, },
"node_modules/@esbuild/aix-ppc64": { "node_modules/@esbuild/aix-ppc64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.2.tgz",
"integrity": "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg==", "integrity": "sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==",
"cpu": [ "cpu": [
"ppc64" "ppc64"
], ],
@ -44,9 +46,9 @@
} }
}, },
"node_modules/@esbuild/android-arm": { "node_modules/@esbuild/android-arm": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.28.2.tgz",
"integrity": "sha512-jbPXvB4Yj2yBV7HUfE2KHe4GJX51QplCN1pGbYjvsyCZbQmies29EoJbkEc+vYuU5o45AfQn37vZlyXy4YJ8RQ==", "integrity": "sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==",
"cpu": [ "cpu": [
"arm" "arm"
], ],
@ -61,9 +63,9 @@
} }
}, },
"node_modules/@esbuild/android-arm64": { "node_modules/@esbuild/android-arm64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.28.2.tgz",
"integrity": "sha512-62dPZHpIXzvChfvfLJow3q5dDtiNMkwiRzPylSCfriLvZeq0a1bWChrGx/BbUbPwOrsWKMn8idSllklzBy+dgQ==", "integrity": "sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==",
"cpu": [ "cpu": [
"arm64" "arm64"
], ],
@ -78,9 +80,9 @@
} }
}, },
"node_modules/@esbuild/android-x64": { "node_modules/@esbuild/android-x64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.28.2.tgz",
"integrity": "sha512-x5VpMODneVDb70PYV2VQOmIUUiBtY3D3mPBG8NxVk5CogneYhkR7MmM3yR/uMdITLrC1ml/NV1rj4bMJuy9MCg==", "integrity": "sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==",
"cpu": [ "cpu": [
"x64" "x64"
], ],
@ -95,9 +97,9 @@
} }
}, },
"node_modules/@esbuild/darwin-arm64": { "node_modules/@esbuild/darwin-arm64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.28.2.tgz",
"integrity": "sha512-5lckdqeuBPlKUwvoCXIgI2D9/ABmPq3Rdp7IfL70393YgaASt7tbju3Ac+ePVi3KDH6N2RqePfHnXkaDtY9fkw==", "integrity": "sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==",
"cpu": [ "cpu": [
"arm64" "arm64"
], ],
@ -112,9 +114,9 @@
} }
}, },
"node_modules/@esbuild/darwin-x64": { "node_modules/@esbuild/darwin-x64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.28.2.tgz",
"integrity": "sha512-rYnXrKcXuT7Z+WL5K980jVFdvVKhCHhUwid+dDYQpH+qu+TefcomiMAJpIiC2EM3Rjtq0sO3StMV/+3w3MyyqQ==", "integrity": "sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==",
"cpu": [ "cpu": [
"x64" "x64"
], ],
@ -129,9 +131,9 @@
} }
}, },
"node_modules/@esbuild/freebsd-arm64": { "node_modules/@esbuild/freebsd-arm64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.28.2.tgz",
"integrity": "sha512-B48PqeCsEgOtzME2GbNM2roU29AMTuOIN91dsMO30t+Ydis3z/3Ngoj5hhnsOSSwNzS+6JppqWsuhTp6E82l2w==", "integrity": "sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==",
"cpu": [ "cpu": [
"arm64" "arm64"
], ],
@ -146,9 +148,9 @@
} }
}, },
"node_modules/@esbuild/freebsd-x64": { "node_modules/@esbuild/freebsd-x64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.28.2.tgz",
"integrity": "sha512-jOBDK5XEjA4m5IJK3bpAQF9/Lelu/Z9ZcdhTRLf4cajlB+8VEhFFRjWgfy3M1O4rO2GQ/b2dLwCUGpiF/eATNQ==", "integrity": "sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==",
"cpu": [ "cpu": [
"x64" "x64"
], ],
@ -163,9 +165,9 @@
} }
}, },
"node_modules/@esbuild/linux-arm": { "node_modules/@esbuild/linux-arm": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.28.2.tgz",
"integrity": "sha512-RkT/YXYBTSULo3+af8Ib0ykH8u2MBh57o7q/DAs3lTJlyVQkgQvlrPTnjIzzRPQyavxtPtfg0EopvDyIt0j1rA==", "integrity": "sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==",
"cpu": [ "cpu": [
"arm" "arm"
], ],
@ -180,9 +182,9 @@
} }
}, },
"node_modules/@esbuild/linux-arm64": { "node_modules/@esbuild/linux-arm64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.28.2.tgz",
"integrity": "sha512-RZPHBoxXuNnPQO9rvjh5jdkRmVizktkT7TCDkDmQ0W2SwHInKCAV95GRuvdSvA7w4VMwfCjUiPwDi0ZO6Nfe9A==", "integrity": "sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==",
"cpu": [ "cpu": [
"arm64" "arm64"
], ],
@ -197,9 +199,9 @@
} }
}, },
"node_modules/@esbuild/linux-ia32": { "node_modules/@esbuild/linux-ia32": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.28.2.tgz",
"integrity": "sha512-GA48aKNkyQDbd3KtkplYWT102C5sn/EZTY4XROkxONgruHPU72l+gW+FfF8tf2cFjeHaRbWpOYa/uRBz/Xq1Pg==", "integrity": "sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==",
"cpu": [ "cpu": [
"ia32" "ia32"
], ],
@ -214,9 +216,9 @@
} }
}, },
"node_modules/@esbuild/linux-loong64": { "node_modules/@esbuild/linux-loong64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.28.2.tgz",
"integrity": "sha512-a4POruNM2oWsD4WKvBSEKGIiWQF8fZOAsycHOt6JBpZ+JN2n2JH9WAv56SOyu9X5IqAjqSIPTaJkqN8F7XOQ5Q==", "integrity": "sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==",
"cpu": [ "cpu": [
"loong64" "loong64"
], ],
@ -231,9 +233,9 @@
} }
}, },
"node_modules/@esbuild/linux-mips64el": { "node_modules/@esbuild/linux-mips64el": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.28.2.tgz",
"integrity": "sha512-KabT5I6StirGfIz0FMgl1I+R1H73Gp0ofL9A3nG3i/cYFJzKHhouBV5VWK1CSgKvVaG4q1RNpCTR2LuTVB3fIw==", "integrity": "sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==",
"cpu": [ "cpu": [
"mips64el" "mips64el"
], ],
@ -248,9 +250,9 @@
} }
}, },
"node_modules/@esbuild/linux-ppc64": { "node_modules/@esbuild/linux-ppc64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.28.2.tgz",
"integrity": "sha512-gRsL4x6wsGHGRqhtI+ifpN/vpOFTQtnbsupUF5R5YTAg+y/lKelYR1hXbnBdzDjGbMYjVJLJTd2OFmMewAgwlQ==", "integrity": "sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==",
"cpu": [ "cpu": [
"ppc64" "ppc64"
], ],
@ -265,9 +267,9 @@
} }
}, },
"node_modules/@esbuild/linux-riscv64": { "node_modules/@esbuild/linux-riscv64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.28.2.tgz",
"integrity": "sha512-hL25LbxO1QOngGzu2U5xeXtxXcW+/GvMN3ejANqXkxZ/opySAZMrc+9LY/WyjAan41unrR3YrmtTsUpwT66InQ==", "integrity": "sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==",
"cpu": [ "cpu": [
"riscv64" "riscv64"
], ],
@ -282,9 +284,9 @@
} }
}, },
"node_modules/@esbuild/linux-s390x": { "node_modules/@esbuild/linux-s390x": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.28.2.tgz",
"integrity": "sha512-2k8go8Ycu1Kb46vEelhu1vqEP+UeRVj2zY1pSuPdgvbd5ykAw82Lrro28vXUrRmzEsUV0NzCf54yARIK8r0fdw==", "integrity": "sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==",
"cpu": [ "cpu": [
"s390x" "s390x"
], ],
@ -299,9 +301,9 @@
} }
}, },
"node_modules/@esbuild/linux-x64": { "node_modules/@esbuild/linux-x64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.28.2.tgz",
"integrity": "sha512-hzznmADPt+OmsYzw1EE33ccA+HPdIqiCRq7cQeL1Jlq2gb1+OyWBkMCrYGBJ+sxVzve2ZJEVeePbLM2iEIZSxA==", "integrity": "sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==",
"cpu": [ "cpu": [
"x64" "x64"
], ],
@ -316,9 +318,9 @@
} }
}, },
"node_modules/@esbuild/netbsd-arm64": { "node_modules/@esbuild/netbsd-arm64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.28.2.tgz",
"integrity": "sha512-b6pqtrQdigZBwZxAn1UpazEisvwaIDvdbMbmrly7cDTMFnw/+3lVxxCTGOrkPVnsYIosJJXAsILG9XcQS+Yu6w==", "integrity": "sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==",
"cpu": [ "cpu": [
"arm64" "arm64"
], ],
@ -333,9 +335,9 @@
} }
}, },
"node_modules/@esbuild/netbsd-x64": { "node_modules/@esbuild/netbsd-x64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.28.2.tgz",
"integrity": "sha512-OfatkLojr6U+WN5EDYuoQhtM+1xco+/6FSzJJnuWiUw5eVcicbyK3dq5EeV/QHT1uy6GoDhGbFpprUiHUYggrw==", "integrity": "sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==",
"cpu": [ "cpu": [
"x64" "x64"
], ],
@ -350,9 +352,9 @@
} }
}, },
"node_modules/@esbuild/openbsd-arm64": { "node_modules/@esbuild/openbsd-arm64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.28.2.tgz",
"integrity": "sha512-AFuojMQTxAz75Fo8idVcqoQWEHIXFRbOc1TrVcFSgCZtQfSdc1RXgB3tjOn/krRHENUB4j00bfGjyl2mJrU37A==", "integrity": "sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==",
"cpu": [ "cpu": [
"arm64" "arm64"
], ],
@ -367,9 +369,9 @@
} }
}, },
"node_modules/@esbuild/openbsd-x64": { "node_modules/@esbuild/openbsd-x64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.28.2.tgz",
"integrity": "sha512-+A1NJmfM8WNDv5CLVQYJ5PshuRm/4cI6WMZRg1by1GwPIQPCTs1GLEUHwiiQGT5zDdyLiRM/l1G0Pv54gvtKIg==", "integrity": "sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==",
"cpu": [ "cpu": [
"x64" "x64"
], ],
@ -384,9 +386,9 @@
} }
}, },
"node_modules/@esbuild/openharmony-arm64": { "node_modules/@esbuild/openharmony-arm64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.28.2.tgz",
"integrity": "sha512-+KrvYb/C8zA9CU/g0sR6w2RBw7IGc5J2BPnc3dYc5VJxHCSF1yNMxTV5LQ7GuKteQXZtspjFbiuW5/dOj7H4Yw==", "integrity": "sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==",
"cpu": [ "cpu": [
"arm64" "arm64"
], ],
@ -401,9 +403,9 @@
} }
}, },
"node_modules/@esbuild/sunos-x64": { "node_modules/@esbuild/sunos-x64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.28.2.tgz",
"integrity": "sha512-ikktIhFBzQNt/QDyOL580ti9+5mL/YZeUPKU2ivGtGjdTYoqz6jObj6nOMfhASpS4GU4Q/Clh1QtxWAvcYKamA==", "integrity": "sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==",
"cpu": [ "cpu": [
"x64" "x64"
], ],
@ -418,9 +420,9 @@
} }
}, },
"node_modules/@esbuild/win32-arm64": { "node_modules/@esbuild/win32-arm64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.28.2.tgz",
"integrity": "sha512-7yRhbHvPqSpRUV7Q20VuDwbjW5kIMwTHpptuUzV+AA46kiPze5Z7qgt6CLCK3pWFrHeNfDd1VKgyP4O+ng17CA==", "integrity": "sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==",
"cpu": [ "cpu": [
"arm64" "arm64"
], ],
@ -435,9 +437,9 @@
} }
}, },
"node_modules/@esbuild/win32-ia32": { "node_modules/@esbuild/win32-ia32": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.28.2.tgz",
"integrity": "sha512-SmwKXe6VHIyZYbBLJrhOoCJRB/Z1tckzmgTLfFYOfpMAx63BJEaL9ExI8x7v0oAO3Zh6D/Oi1gVxEYr5oUCFhw==", "integrity": "sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==",
"cpu": [ "cpu": [
"ia32" "ia32"
], ],
@ -452,9 +454,9 @@
} }
}, },
"node_modules/@esbuild/win32-x64": { "node_modules/@esbuild/win32-x64": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.27.7.tgz", "resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.28.2.tgz",
"integrity": "sha512-56hiAJPhwQ1R4i+21FVF7V8kSD5zZTdHcVuRFMW0hn753vVfQN8xlx4uOPT4xoGH0Z/oVATuR82AiqSTDIpaHg==", "integrity": "sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==",
"cpu": [ "cpu": [
"x64" "x64"
], ],
@ -597,6 +599,18 @@
"@types/node": "*" "@types/node": "*"
} }
}, },
"node_modules/@types/pg": {
"version": "8.23.1",
"resolved": "https://registry.npmjs.org/@types/pg/-/pg-8.23.1.tgz",
"integrity": "sha512-fKVHpikPdg4GKks3JuLEhvwSyvwzF23hnabPy6DD8ljVbC7+6J5dQzdv4arV6jqq57djnMgs1HKBxX4P8aBI3A==",
"dev": true,
"license": "MIT",
"dependencies": {
"@types/node": "*",
"pg-protocol": "*",
"pg-types": "^2.2.0"
}
},
"node_modules/@types/qs": { "node_modules/@types/qs": {
"version": "6.15.0", "version": "6.15.0",
"resolved": "https://registry.npmjs.org/@types/qs/-/qs-6.15.0.tgz", "resolved": "https://registry.npmjs.org/@types/qs/-/qs-6.15.0.tgz",
@ -658,12 +672,12 @@
} }
}, },
"node_modules/adm-zip": { "node_modules/adm-zip": {
"version": "0.5.17", "version": "0.6.1",
"resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.5.17.tgz", "resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.1.tgz",
"integrity": "sha512-+Ut8d9LLqwEvHHJl1+PIHqoyDxFgVN847JTVM3Izi3xHDWPE4UtzzXysMZQs64DMcrJfBeS/uoEP4AD3HQHnQQ==", "integrity": "sha512-Xwrja8nx9e5o2N1my4DsKCeKpdrnACyr1wtbPxBDgGzKzKyE9kRtBFA8mWldI+RVlD7CBZNWY/wQ2+ydwOR6kQ==",
"license": "MIT", "license": "MIT",
"engines": { "engines": {
"node": ">=12.0" "node": ">=14.0"
} }
}, },
"node_modules/agentkeepalive": { "node_modules/agentkeepalive": {
@ -697,9 +711,9 @@
"license": "MIT" "license": "MIT"
}, },
"node_modules/body-parser": { "node_modules/body-parser": {
"version": "1.20.4", "version": "1.20.8",
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.4.tgz", "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.8.tgz",
"integrity": "sha512-ZTgYYLMOXY9qKU/57FAo8F+HA2dGX7bqGc71txDRC1rS4frdFI5R7NhluHxH6M0YItAP0sHB4uqAOcYKxO6uGA==", "integrity": "sha512-JNcyFQ64OiijEkPzUBTCe+hyPXUD/3LEldGQ6iF5LR1w00mx9o7xtDWHXBY2iItjdCFGoilOLNQbH943ut7pHA==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"bytes": "~3.1.2", "bytes": "~3.1.2",
@ -710,7 +724,7 @@
"http-errors": "~2.0.1", "http-errors": "~2.0.1",
"iconv-lite": "~0.4.24", "iconv-lite": "~0.4.24",
"on-finished": "~2.4.1", "on-finished": "~2.4.1",
"qs": "~6.14.0", "qs": "~6.16.0",
"raw-body": "~2.5.3", "raw-body": "~2.5.3",
"type-is": "~1.6.18", "type-is": "~1.6.18",
"unpipe": "~1.0.0" "unpipe": "~1.0.0"
@ -720,6 +734,22 @@
"npm": "1.2.8000 || >= 1.4.16" "npm": "1.2.8000 || >= 1.4.16"
} }
}, },
"node_modules/body-parser/node_modules/qs": {
"version": "6.16.0",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",
"integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==",
"license": "BSD-3-Clause",
"dependencies": {
"es-define-property": "^1.0.1",
"side-channel": "^1.1.1"
},
"engines": {
"node": ">=0.6"
},
"funding": {
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/buffer-from": { "node_modules/buffer-from": {
"version": "1.1.2", "version": "1.1.2",
"resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz", "resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
@ -962,9 +992,9 @@
} }
}, },
"node_modules/esbuild": { "node_modules/esbuild": {
"version": "0.27.7", "version": "0.28.2",
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz", "resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.2.tgz",
"integrity": "sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w==", "integrity": "sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==",
"dev": true, "dev": true,
"hasInstallScript": true, "hasInstallScript": true,
"license": "MIT", "license": "MIT",
@ -975,32 +1005,32 @@
"node": ">=18" "node": ">=18"
}, },
"optionalDependencies": { "optionalDependencies": {
"@esbuild/aix-ppc64": "0.27.7", "@esbuild/aix-ppc64": "0.28.2",
"@esbuild/android-arm": "0.27.7", "@esbuild/android-arm": "0.28.2",
"@esbuild/android-arm64": "0.27.7", "@esbuild/android-arm64": "0.28.2",
"@esbuild/android-x64": "0.27.7", "@esbuild/android-x64": "0.28.2",
"@esbuild/darwin-arm64": "0.27.7", "@esbuild/darwin-arm64": "0.28.2",
"@esbuild/darwin-x64": "0.27.7", "@esbuild/darwin-x64": "0.28.2",
"@esbuild/freebsd-arm64": "0.27.7", "@esbuild/freebsd-arm64": "0.28.2",
"@esbuild/freebsd-x64": "0.27.7", "@esbuild/freebsd-x64": "0.28.2",
"@esbuild/linux-arm": "0.27.7", "@esbuild/linux-arm": "0.28.2",
"@esbuild/linux-arm64": "0.27.7", "@esbuild/linux-arm64": "0.28.2",
"@esbuild/linux-ia32": "0.27.7", "@esbuild/linux-ia32": "0.28.2",
"@esbuild/linux-loong64": "0.27.7", "@esbuild/linux-loong64": "0.28.2",
"@esbuild/linux-mips64el": "0.27.7", "@esbuild/linux-mips64el": "0.28.2",
"@esbuild/linux-ppc64": "0.27.7", "@esbuild/linux-ppc64": "0.28.2",
"@esbuild/linux-riscv64": "0.27.7", "@esbuild/linux-riscv64": "0.28.2",
"@esbuild/linux-s390x": "0.27.7", "@esbuild/linux-s390x": "0.28.2",
"@esbuild/linux-x64": "0.27.7", "@esbuild/linux-x64": "0.28.2",
"@esbuild/netbsd-arm64": "0.27.7", "@esbuild/netbsd-arm64": "0.28.2",
"@esbuild/netbsd-x64": "0.27.7", "@esbuild/netbsd-x64": "0.28.2",
"@esbuild/openbsd-arm64": "0.27.7", "@esbuild/openbsd-arm64": "0.28.2",
"@esbuild/openbsd-x64": "0.27.7", "@esbuild/openbsd-x64": "0.28.2",
"@esbuild/openharmony-arm64": "0.27.7", "@esbuild/openharmony-arm64": "0.28.2",
"@esbuild/sunos-x64": "0.27.7", "@esbuild/sunos-x64": "0.28.2",
"@esbuild/win32-arm64": "0.27.7", "@esbuild/win32-arm64": "0.28.2",
"@esbuild/win32-ia32": "0.27.7", "@esbuild/win32-ia32": "0.28.2",
"@esbuild/win32-x64": "0.27.7" "@esbuild/win32-x64": "0.28.2"
} }
}, },
"node_modules/escape-html": { "node_modules/escape-html": {
@ -1028,14 +1058,14 @@
} }
}, },
"node_modules/express": { "node_modules/express": {
"version": "4.22.1", "version": "4.22.2",
"resolved": "https://registry.npmjs.org/express/-/express-4.22.1.tgz", "resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz",
"integrity": "sha512-F2X8g9P1X7uCPZMA3MVf9wcTqlyNp7IhH5qPCI0izhaOIYXaW9L535tGA3qmjRzpH+bZczqq7hVKxTR4NWnu+g==", "integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"accepts": "~1.3.8", "accepts": "~1.3.8",
"array-flatten": "1.1.1", "array-flatten": "1.1.1",
"body-parser": "~1.20.3", "body-parser": "~1.20.5",
"content-disposition": "~0.5.4", "content-disposition": "~0.5.4",
"content-type": "~1.0.4", "content-type": "~1.0.4",
"cookie": "~0.7.1", "cookie": "~0.7.1",
@ -1054,7 +1084,7 @@
"parseurl": "~1.3.3", "parseurl": "~1.3.3",
"path-to-regexp": "~0.1.12", "path-to-regexp": "~0.1.12",
"proxy-addr": "~2.0.7", "proxy-addr": "~2.0.7",
"qs": "~6.14.0", "qs": "~6.15.1",
"range-parser": "~1.2.1", "range-parser": "~1.2.1",
"safe-buffer": "5.2.1", "safe-buffer": "5.2.1",
"send": "~0.19.0", "send": "~0.19.0",
@ -1092,16 +1122,16 @@
} }
}, },
"node_modules/form-data": { "node_modules/form-data": {
"version": "4.0.5", "version": "4.0.6",
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.5.tgz", "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.6.tgz",
"integrity": "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==", "integrity": "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"asynckit": "^0.4.0", "asynckit": "^0.4.0",
"combined-stream": "^1.0.8", "combined-stream": "^1.0.8",
"es-set-tostringtag": "^2.1.0", "es-set-tostringtag": "^2.1.0",
"hasown": "^2.0.2", "hasown": "^2.0.4",
"mime-types": "^2.1.12" "mime-types": "^2.1.35"
}, },
"engines": { "engines": {
"node": ">= 6" "node": ">= 6"
@ -1205,19 +1235,6 @@
"node": ">= 0.4" "node": ">= 0.4"
} }
}, },
"node_modules/get-tsconfig": {
"version": "4.13.7",
"resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-4.13.7.tgz",
"integrity": "sha512-7tN6rFgBlMgpBML5j8typ92BKFi2sFQvIdpAqLA2beia5avZDrMs0FLZiM5etShWq5irVyGcGMEA1jcDaK7A/Q==",
"dev": true,
"license": "MIT",
"dependencies": {
"resolve-pkg-maps": "^1.0.0"
},
"funding": {
"url": "https://github.com/privatenumber/get-tsconfig?sponsor=1"
}
},
"node_modules/gopd": { "node_modules/gopd": {
"version": "1.2.0", "version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
@ -1258,9 +1275,9 @@
} }
}, },
"node_modules/hasown": { "node_modules/hasown": {
"version": "2.0.2", "version": "2.0.4",
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz", "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz",
"integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==", "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"function-bind": "^1.1.2" "function-bind": "^1.1.2"
@ -1401,9 +1418,9 @@
"license": "MIT" "license": "MIT"
}, },
"node_modules/multer": { "node_modules/multer": {
"version": "2.1.1", "version": "2.3.0",
"resolved": "https://registry.npmjs.org/multer/-/multer-2.1.1.tgz", "resolved": "https://registry.npmjs.org/multer/-/multer-2.3.0.tgz",
"integrity": "sha512-mo+QTzKlx8R7E5ylSXxWzGoXoZbOsRMpyitcht8By2KHvMbf3tjwosZ/Mu/XYU6UuJ3VZnODIrak5ZrPiPyB6A==", "integrity": "sha512-cjNbm3sttszgZeGfJR124D+jFEfkXCVAsoPBmFn9X7UxmDSFHWqE2CoEj0vrmSpuAFnqWR1Szcm9QTsiHr60Xw==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"append-field": "^1.0.0", "append-field": "^1.0.0",
@ -1574,6 +1591,134 @@
"url": "https://github.com/sponsors/mehmet-kozan" "url": "https://github.com/sponsors/mehmet-kozan"
} }
}, },
"node_modules/pg": {
"version": "8.23.0",
"resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
"integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",
"license": "MIT",
"dependencies": {
"pg-connection-string": "^2.14.0",
"pg-pool": "^3.14.0",
"pg-protocol": "^1.16.0",
"pg-types": "2.2.0",
"pgpass": "1.0.5"
},
"engines": {
"node": ">= 16.0.0"
},
"optionalDependencies": {
"pg-cloudflare": "^1.4.0"
},
"peerDependencies": {
"pg-native": ">=3.0.1"
},
"peerDependenciesMeta": {
"pg-native": {
"optional": true
}
}
},
"node_modules/pg-cloudflare": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
"integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
"license": "MIT",
"optional": true
},
"node_modules/pg-connection-string": {
"version": "2.14.0",
"resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",
"integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
"license": "MIT"
},
"node_modules/pg-int8": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
"integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
"license": "ISC",
"engines": {
"node": ">=4.0.0"
}
},
"node_modules/pg-pool": {
"version": "3.14.0",
"resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
"integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
"license": "MIT",
"peerDependencies": {
"pg": ">=8.0"
}
},
"node_modules/pg-protocol": {
"version": "1.16.0",
"resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",
"integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",
"license": "MIT"
},
"node_modules/pg-types": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
"integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
"license": "MIT",
"dependencies": {
"pg-int8": "1.0.1",
"postgres-array": "~2.0.0",
"postgres-bytea": "~1.0.0",
"postgres-date": "~1.0.4",
"postgres-interval": "^1.1.0"
},
"engines": {
"node": ">=4"
}
},
"node_modules/pgpass": {
"version": "1.0.5",
"resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
"integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
"license": "MIT",
"dependencies": {
"split2": "^4.1.0"
}
},
"node_modules/postgres-array": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
"integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
"license": "MIT",
"engines": {
"node": ">=4"
}
},
"node_modules/postgres-bytea": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
"integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/postgres-date": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
"integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
"license": "MIT",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/postgres-interval": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
"integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
"license": "MIT",
"dependencies": {
"xtend": "^4.0.0"
},
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/proxy-addr": { "node_modules/proxy-addr": {
"version": "2.0.7", "version": "2.0.7",
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz", "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
@ -1588,12 +1733,13 @@
} }
}, },
"node_modules/qs": { "node_modules/qs": {
"version": "6.14.2", "version": "6.15.3",
"resolved": "https://registry.npmjs.org/qs/-/qs-6.14.2.tgz", "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz",
"integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==", "integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==",
"license": "BSD-3-Clause", "license": "BSD-3-Clause",
"dependencies": { "dependencies": {
"side-channel": "^1.1.0" "es-define-property": "^1.0.1",
"side-channel": "^1.1.1"
}, },
"engines": { "engines": {
"node": ">=0.6" "node": ">=0.6"
@ -1640,16 +1786,6 @@
"node": ">= 6" "node": ">= 6"
} }
}, },
"node_modules/resolve-pkg-maps": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/resolve-pkg-maps/-/resolve-pkg-maps-1.0.0.tgz",
"integrity": "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==",
"dev": true,
"license": "MIT",
"funding": {
"url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1"
}
},
"node_modules/safe-buffer": { "node_modules/safe-buffer": {
"version": "5.2.1", "version": "5.2.1",
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
@ -1728,14 +1864,14 @@
"license": "ISC" "license": "ISC"
}, },
"node_modules/side-channel": { "node_modules/side-channel": {
"version": "1.1.0", "version": "1.1.1",
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz", "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz",
"integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==", "integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"es-errors": "^1.3.0", "es-errors": "^1.3.0",
"object-inspect": "^1.13.3", "object-inspect": "^1.13.4",
"side-channel-list": "^1.0.0", "side-channel-list": "^1.0.1",
"side-channel-map": "^1.0.1", "side-channel-map": "^1.0.1",
"side-channel-weakmap": "^1.0.2" "side-channel-weakmap": "^1.0.2"
}, },
@ -1747,13 +1883,13 @@
} }
}, },
"node_modules/side-channel-list": { "node_modules/side-channel-list": {
"version": "1.0.0", "version": "1.0.1",
"resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.0.tgz", "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",
"integrity": "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA==", "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"es-errors": "^1.3.0", "es-errors": "^1.3.0",
"object-inspect": "^1.13.3" "object-inspect": "^1.13.4"
}, },
"engines": { "engines": {
"node": ">= 0.4" "node": ">= 0.4"
@ -1799,6 +1935,15 @@
"url": "https://github.com/sponsors/ljharb" "url": "https://github.com/sponsors/ljharb"
} }
}, },
"node_modules/split2": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
"integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
"license": "ISC",
"engines": {
"node": ">= 10.x"
}
},
"node_modules/statuses": { "node_modules/statuses": {
"version": "2.0.2", "version": "2.0.2",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
@ -1841,14 +1986,13 @@
"license": "MIT" "license": "MIT"
}, },
"node_modules/tsx": { "node_modules/tsx": {
"version": "4.21.0", "version": "4.23.13",
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.21.0.tgz", "resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.13.tgz",
"integrity": "sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw==", "integrity": "sha512-BL5MGkRln6aDYhb0xbQlEAGw743BaZYWdbWtdJOBriYJboKgUUYCadFp2/FpBBZquBC/ezNBn7wMMPx7FDZUDw==",
"dev": true, "dev": true,
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"esbuild": "~0.27.0", "esbuild": "~0.28.0"
"get-tsconfig": "^4.7.5"
}, },
"bin": { "bin": {
"tsx": "dist/cli.mjs" "tsx": "dist/cli.mjs"
@ -1893,9 +2037,9 @@
} }
}, },
"node_modules/undici": { "node_modules/undici": {
"version": "6.24.1", "version": "6.28.1",
"resolved": "https://registry.npmjs.org/undici/-/undici-6.24.1.tgz", "resolved": "https://registry.npmjs.org/undici/-/undici-6.28.1.tgz",
"integrity": "sha512-sC+b0tB1whOCzbtlx20fx3WgCXwkW627p4EA9uM+/tNNPkSS+eSEld6pAs9nDv7WbY1UUljBMYPtu9BCOrCWKA==", "integrity": "sha512-zWpdTVD54H48CIybL0rWQ3ukpb9d23wM7eH5RtfdmeP70cWHNjtfo7P4vZX+5CoDcO53J4Pu5uXp7lNfjc6DRA==",
"license": "MIT", "license": "MIT",
"engines": { "engines": {
"node": ">=18.17" "node": ">=18.17"
@ -1964,6 +2108,15 @@
"tr46": "~0.0.3", "tr46": "~0.0.3",
"webidl-conversions": "^3.0.0" "webidl-conversions": "^3.0.0"
} }
},
"node_modules/xtend": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
"integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
"license": "MIT",
"engines": {
"node": ">=0.4"
}
} }
} }
} }

13
package.json
View file

@ -6,17 +6,21 @@
"scripts": { "scripts": {
"dev": "tsx watch src/server.ts", "dev": "tsx watch src/server.ts",
"build": "tsc -p tsconfig.json", "build": "tsc -p tsconfig.json",
"start": "node dist/server.js", "start": "node dist/modules/catalog/migrations.js && node dist/server.js",
"migrate:lifecycle": "node dist/modules/catalog/migrations.js",
"migrate:legacy:lifecycle": "node dist/scripts/migrate-legacy-lifecycle.js",
"test": "NODE_ENV=test tsx --test tests/**/*.test.ts",
"check": "tsc --noEmit -p tsconfig.json" "check": "tsc --noEmit -p tsconfig.json"
}, },
"dependencies": { "dependencies": {
"@qdrant/js-client-rest": "^1.15.0", "@qdrant/js-client-rest": "^1.15.0",
"adm-zip": "^0.5.17", "adm-zip": "^0.6.1",
"dotenv": "^16.4.5", "dotenv": "^16.4.5",
"express": "^4.21.2", "express": "^4.21.2",
"multer": "^2.0.0", "multer": "^2.3.0",
"openai": "^4.104.0", "openai": "^4.104.0",
"pdf-parse": "^1.1.1" "pdf-parse": "^1.1.1",
"pg": "^8.23.0"
}, },
"devDependencies": { "devDependencies": {
"@types/adm-zip": "^0.5.8", "@types/adm-zip": "^0.5.8",
@ -24,6 +28,7 @@
"@types/multer": "^1.4.12", "@types/multer": "^1.4.12",
"@types/node": "^22.15.3", "@types/node": "^22.15.3",
"@types/pdf-parse": "^1.1.4", "@types/pdf-parse": "^1.1.4",
"@types/pg": "^8.23.1",
"tsx": "^4.19.3", "tsx": "^4.19.3",
"typescript": "^5.8.3" "typescript": "^5.8.3"
} }

28
public/playground/app.js
View file

@ -38,6 +38,8 @@ const btnClearUpload = document.getElementById("btnClearUpload");
const uploadStatusText = document.getElementById("uploadStatusText"); const uploadStatusText = document.getElementById("uploadStatusText");
const ingestMode = document.getElementById("ingestMode"); const ingestMode = document.getElementById("ingestMode");
const ingestTags = document.getElementById("ingestTags"); const ingestTags = document.getElementById("ingestTags");
const ingestActivate = document.getElementById("ingestActivate");
const ingestExpectedActiveVersionId = document.getElementById("ingestExpectedActiveVersionId");
const ingestModeHint = document.getElementById("ingestModeHint"); const ingestModeHint = document.getElementById("ingestModeHint");
const cleanupScopeSelect = document.getElementById("cleanupScopeSelect"); const cleanupScopeSelect = document.getElementById("cleanupScopeSelect");
@ -80,7 +82,17 @@ function splitTags(value) {
function buildScopeLabel(scope) { function buildScopeLabel(scope) {
const modes = scope.chunkModes.join(", ") || "sin modo"; const modes = scope.chunkModes.join(", ") || "sin modo";
return `${scope.sourceRef} [${modes}]`; const version = scope.activeVersionNumber ? ` v${scope.activeVersionNumber}` : "";
const state = scope.state ? ` ${scope.state}` : "";
return `${scope.sourceRef}${version}${state} [${modes}]`;
}
function normalizeExpectedActiveVersionInput() {
const value = ingestExpectedActiveVersionId.value.trim();
if (!value) {
return undefined;
}
return value;
} }
let previousSourceRef = ""; // Para guardar el valor original de SourceRef let previousSourceRef = ""; // Para guardar el valor original de SourceRef
@ -363,6 +375,11 @@ ingestButton.addEventListener("click", async () => {
formData.append("file", ingestUploadFile.files[0]); formData.append("file", ingestUploadFile.files[0]);
formData.append("mode", ingestMode.value); formData.append("mode", ingestMode.value);
formData.append("tags", splitTags(ingestTags.value).join(",")); formData.append("tags", splitTags(ingestTags.value).join(","));
formData.append("activate", ingestActivate.value);
const expectedActiveVersionId = normalizeExpectedActiveVersionInput();
if (expectedActiveVersionId !== undefined) {
formData.append("expectedActiveVersionId", expectedActiveVersionId);
}
if (ingestScopeMode.value === "custom" && ingestSourceId.value.trim()) { if (ingestScopeMode.value === "custom" && ingestSourceId.value.trim()) {
formData.append("sourceId", ingestSourceId.value.trim()); formData.append("sourceId", ingestSourceId.value.trim());
} }
@ -414,6 +431,11 @@ ingestButton.addEventListener("click", async () => {
formData.append("isZipFolder", "true"); formData.append("isZipFolder", "true");
formData.append("mode", ingestMode.value); formData.append("mode", ingestMode.value);
formData.append("tags", splitTags(ingestTags.value).join(",")); formData.append("tags", splitTags(ingestTags.value).join(","));
formData.append("activate", ingestActivate.value);
const expectedActiveVersionId = normalizeExpectedActiveVersionInput();
if (expectedActiveVersionId !== undefined) {
formData.append("expectedActiveVersionId", expectedActiveVersionId);
}
if (ingestScopeMode.value === "custom" && ingestSourceId.value.trim()) { if (ingestScopeMode.value === "custom" && ingestSourceId.value.trim()) {
formData.append("sourceId", ingestSourceId.value.trim()); formData.append("sourceId", ingestSourceId.value.trim());
} }
@ -431,7 +453,9 @@ ingestButton.addEventListener("click", async () => {
sourceType: ingestSourceType.value, sourceType: ingestSourceType.value,
sourceRef: ingestSourceRef.value, sourceRef: ingestSourceRef.value,
mode: ingestMode.value, mode: ingestMode.value,
tags: splitTags(ingestTags.value) tags: splitTags(ingestTags.value),
activate: ingestActivate.value === "true",
expectedActiveVersionId: normalizeExpectedActiveVersionInput() === "null" ? null : normalizeExpectedActiveVersionInput()
}); });
} }

13
public/playground/index.html
View file

@ -75,8 +75,17 @@
<label>Tags (coma separada) <label>Tags (coma separada)
<input id="ingestTags" value="workspace,global-docs" /> <input id="ingestTags" value="workspace,global-docs" />
</label> </label>
<label>Activar al terminar
<select id="ingestActivate">
<option value="true">si, activar version</option>
<option value="false">no, dejar en ready</option>
</select>
</label>
<label>Version activa esperada
<input id="ingestExpectedActiveVersionId" placeholder="uuid actual o null para primera activacion" />
</label>
</div> </div>
<p class="helper" id="ingestModeHint">Si seleccionas un archivo local, el playground usara upload directo y podras aislarlo con un `sourceId` propio para no mezclarlo con otros scopes.</p> <p class="helper" id="ingestModeHint">Si el ciclo de vida esta activo, una ingesta con activacion exige indicar la version activa esperada; usa <code>null</code> para la primera activacion de una fuente.</p>
<div class="actions"> <div class="actions">
<button id="ingestButton">Lanzar ingesta</button> <button id="ingestButton">Lanzar ingesta</button>
</div> </div>
@ -87,7 +96,7 @@
<section id="tab-cleanup" class="tab-panel"> <section id="tab-cleanup" class="tab-panel">
<article class="panel"> <article class="panel">
<h2>Limpieza controlada</h2> <h2>Limpieza controlada</h2>
<p class="helper">Permite eliminar contexto ya ingerido del RAG antes de reingestar una fuente, para evitar duplicados y versiones mezcladas.</p> <p class="helper">Esta operacion queda como compatibilidad legacy. Con ciclo de vida, usa la purga administrativa de versiones no activas.</p>
<div class="grid single-grid"> <div class="grid single-grid">
<label>Scope disponible a limpiar <label>Scope disponible a limpiar
<select id="cleanupScopeSelect"> <select id="cleanupScopeSelect">

237
src/api/openapi.ts
View file

@ -30,6 +30,7 @@ export const openApiDocument = {
{ name: "Discovery", description: "API contract and browser playground." }, { name: "Discovery", description: "API contract and browser playground." },
{ name: "Status", description: "Service capabilities and available resources." }, { name: "Status", description: "Service capabilities and available resources." },
{ name: "Ingestion", description: "Knowledge ingestion and cleanup." }, { name: "Ingestion", description: "Knowledge ingestion and cleanup." },
{ name: "Lifecycle", description: "Knowledge source versions, activation, rollback, and purge." },
{ name: "Retrieval", description: "Context retrieval and model-backed answers." }, { name: "Retrieval", description: "Context retrieval and model-backed answers." },
{ name: "Evaluation", description: "Evaluation log capture and review." } { name: "Evaluation", description: "Evaluation log capture and review." }
], ],
@ -70,7 +71,8 @@ export const openApiDocument = {
tags: ["Status"], tags: ["Status"],
summary: "Check service health and capabilities", summary: "Check service health and capabilities",
responses: { responses: {
"200": jsonResponse("Current service status.", ref("HealthResponse")) "200": jsonResponse("Current service status.", ref("HealthResponse")),
"503": jsonResponse("One or more required services are unavailable.", ref("HealthResponse"))
} }
} }
}, },
@ -83,10 +85,117 @@ export const openApiDocument = {
type: "array", type: "array",
items: ref("AvailableScope") items: ref("AvailableScope")
}), }),
"503": serverError,
"500": serverError "500": serverError
} }
} }
}, },
"/sources/{sourceId}": {
get: {
tags: ["Lifecycle"],
summary: "Get a catalog source",
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
responses: {
"200": jsonResponse("Catalog source.", ref("AvailableScope")),
"404": jsonResponse("Source not found.", ref("Error")),
"503": serverError
}
}
},
"/sources/{sourceId}/versions": {
get: {
tags: ["Lifecycle"],
summary: "List source versions",
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
responses: {
"200": jsonResponse("Source versions.", { type: "array", items: ref("SourceVersion") }),
"503": serverError
}
}
},
"/sources/{sourceId}/versions/{versionId}": {
get: {
tags: ["Lifecycle"],
summary: "Get one source version",
parameters: [
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
],
responses: {
"200": jsonResponse("Source version.", ref("SourceVersion")),
"404": jsonResponse("Version not found.", ref("Error")),
"503": serverError
}
},
delete: {
tags: ["Lifecycle"],
summary: "Purge a non-active source version",
security: [{ bearerAuth: [] }],
parameters: [
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
],
responses: {
"200": jsonResponse("Version purged.", ref("PurgeResponse")),
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
"409": jsonResponse("Active versions cannot be purged.", ref("Error")),
"503": serverError
}
}
},
"/sources/{sourceId}/versions/{versionId}/activate": {
post: {
tags: ["Lifecycle"],
summary: "Activate a ready or superseded version",
security: [{ bearerAuth: [] }],
parameters: [
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
],
requestBody: { required: true, content: jsonContent(ref("ActivateVersionRequest")) },
responses: {
"200": jsonResponse("Activated source version.", ref("SourceVersion")),
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
"409": jsonResponse("Active version precondition failed.", ref("Error")),
"503": serverError
}
}
},
"/sources/{sourceId}/rollback": {
post: {
tags: ["Lifecycle"],
summary: "Rollback a source to a previous version",
security: [{ bearerAuth: [] }],
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
requestBody: { required: true, content: jsonContent(ref("RollbackRequest")) },
responses: {
"200": jsonResponse("Rollback activated source version.", ref("SourceVersion")),
"400": jsonResponse("Required rollback fields are missing.", ref("Error")),
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
"409": jsonResponse("Active version precondition failed.", ref("Error")),
"503": serverError
}
}
},
"/sources/{sourceId}/versions/{versionId}/retry": {
post: {
tags: ["Lifecycle"],
summary: "Explicitly retry a failed or orphaned version by re-submitting a durable source",
security: [{ bearerAuth: [] }],
parameters: [
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
],
requestBody: { required: false, content: jsonContent(ref("RetryVersionRequest")) },
responses: {
"201": jsonResponse("Replacement version created by explicit retry.", ref("IngestResponse")),
"202": jsonResponse("Retry instructions returned because no durable source payload was supplied.", ref("RetryVersionResponse")),
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
"409": jsonResponse("Version state is not retryable.", ref("Error")),
"503": serverError
}
}
},
"/models/answer": { "/models/answer": {
get: { get: {
tags: ["Status"], tags: ["Status"],
@ -112,7 +221,12 @@ export const openApiDocument = {
}) })
}, },
responses: { responses: {
"202": jsonResponse("Ingestion completed and accepted.", ref("IngestResponse")), "201": jsonResponse("Lifecycle ingestion completed and committed.", ref("IngestResponse")),
"202": jsonResponse("Legacy ingestion completed and accepted while lifecycle enforcement is disabled.", ref("IngestResponse")),
"400": jsonResponse("Invalid lifecycle activation request.", ref("Error")),
"409": jsonResponse("Concurrent active version change or reusable version in progress.", ref("Error")),
"422": jsonResponse("Empty or unsupported source.", ref("Error")),
"503": serverError,
"500": serverError "500": serverError
} }
} }
@ -130,8 +244,12 @@ export const openApiDocument = {
} }
}, },
responses: { responses: {
"202": jsonResponse("Upload ingested successfully.", ref("UploadIngestResponse")), "201": jsonResponse("Lifecycle upload ingested successfully.", ref("UploadIngestResponse")),
"202": jsonResponse("Legacy upload ingested successfully while lifecycle enforcement is disabled.", ref("UploadIngestResponse")),
"400": jsonResponse("The file field is missing.", ref("Error"), { ok: false, error: "Missing file upload" }), "400": jsonResponse("The file field is missing.", ref("Error"), { ok: false, error: "Missing file upload" }),
"409": jsonResponse("Concurrent active version change or reusable version in progress.", ref("Error")),
"422": jsonResponse("Empty or unsupported source.", ref("Error")),
"503": serverError,
"500": serverError "500": serverError
} }
} }
@ -150,6 +268,9 @@ export const openApiDocument = {
responses: { responses: {
"200": jsonResponse("Matching chunks deleted.", ref("CleanupResponse")), "200": jsonResponse("Matching chunks deleted.", ref("CleanupResponse")),
"400": jsonResponse("Scope is missing or unsafe for cleanup.", ref("Error")), "400": jsonResponse("Scope is missing or unsafe for cleanup.", ref("Error")),
"401": jsonResponse("Missing or invalid lifecycle admin token when lifecycle enforcement is active.", ref("Error")),
"410": jsonResponse("Legacy cleanup is disabled while lifecycle enforcement is active.", ref("Error")),
"503": serverError,
"500": serverError "500": serverError
} }
} }
@ -272,6 +393,9 @@ export const openApiDocument = {
} }
}, },
components: { components: {
securitySchemes: {
bearerAuth: { type: "http", scheme: "bearer" }
},
schemas: { schemas: {
Error: { Error: {
type: "object", type: "object",
@ -289,6 +413,10 @@ export const openApiDocument = {
tags: { type: "array", items: { type: "string" } } tags: { type: "array", items: { type: "string" } }
} }
}, },
SourceVersionState: {
type: "string",
enum: ["pending", "indexing", "review_required", "ready", "active", "superseded", "failed", "rejected", "purging", "purged"]
},
AvailableScope: { AvailableScope: {
type: "object", type: "object",
required: ["sourceId", "sourceRef", "chunkModes", "tags"], required: ["sourceId", "sourceRef", "chunkModes", "tags"],
@ -296,7 +424,12 @@ export const openApiDocument = {
sourceId: { type: "string" }, sourceId: { type: "string" },
sourceRef: { type: "string" }, sourceRef: { type: "string" },
chunkModes: { type: "array", items: { $ref: "#/components/schemas/ChunkMode" } }, chunkModes: { type: "array", items: { $ref: "#/components/schemas/ChunkMode" } },
tags: { type: "array", items: { type: "string" } } tags: { type: "array", items: { type: "string" } },
activeVersionId: { type: ["string", "null"], format: "uuid" },
activeVersionNumber: { type: ["integer", "null"] },
state: { ...ref("SourceVersionState"), nullable: true },
needsReingest: { type: "boolean" },
updatedAt: { type: "string", format: "date-time" }
} }
}, },
ChunkMode: { type: "string", enum: ["documental", "codigo", "auto"] }, ChunkMode: { type: "string", enum: ["documental", "codigo", "auto"] },
@ -310,18 +443,27 @@ export const openApiDocument = {
sourceRef: { type: "string" }, sourceRef: { type: "string" },
readPath: { type: "string", description: "Internal filesystem path used when the logical sourceRef differs from the read location." }, readPath: { type: "string", description: "Internal filesystem path used when the logical sourceRef differs from the read location." },
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" }, mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
tags: { type: "array", items: { type: "string" } } tags: { type: "array", items: { type: "string" } },
activate: { type: "boolean", default: true },
expectedActiveVersionId: { type: ["string", "null"], format: "uuid", description: "Required when activate is true. Use null for first activation." }
} }
}, },
IngestResponse: { IngestResponse: {
type: "object", type: "object",
required: ["accepted", "source", "filesDiscovered", "documentsProcessed", "chunksStored", "collectionName"], required: ["accepted", "sourceId", "filesDiscovered", "documentsProcessed", "chunksStored", "collectionName"],
properties: { properties: {
accepted: { type: "boolean", const: true }, accepted: { type: "boolean", const: true },
source: ref("IngestRequest"), source: ref("IngestRequest"),
sourceId: { type: "string" },
versionId: { type: "string", format: "uuid" },
versionNumber: { type: "integer" },
state: ref("SourceVersionState"),
previousVersionId: { type: ["string", "null"], format: "uuid" },
filesDiscovered: { type: "integer" }, filesDiscovered: { type: "integer" },
documentsProcessed: { type: "integer" }, documentsProcessed: { type: "integer" },
chunksStored: { type: "integer" }, chunksStored: { type: "integer" },
activated: { type: "boolean" },
noOp: { type: "boolean" },
collectionName: { type: "string" } collectionName: { type: "string" }
} }
}, },
@ -333,7 +475,9 @@ export const openApiDocument = {
sourceId: { type: "string" }, sourceId: { type: "string" },
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" }, mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
tags: { type: "string", description: "Comma-separated tags." }, tags: { type: "string", description: "Comma-separated tags." },
isZipFolder: { type: "string", enum: ["true", "false"], default: "false" } isZipFolder: { type: "string", enum: ["true", "false"], default: "false" },
activate: { type: "string", enum: ["true", "false"], default: "true" },
expectedActiveVersionId: { type: "string", description: "UUID or literal null when activate is true." }
} }
}, },
UploadIngestResponse: { UploadIngestResponse: {
@ -355,7 +499,7 @@ export const openApiDocument = {
type: "object", type: "object",
required: ["ok", "deleted"], required: ["ok", "deleted"],
properties: { properties: {
ok: { type: "boolean", const: true }, ok: { type: "boolean" },
deleted: { type: "integer" } deleted: { type: "integer" }
} }
}, },
@ -389,6 +533,8 @@ export const openApiDocument = {
chunkId: { type: "string" }, chunkId: { type: "string" },
documentId: { type: "string" }, documentId: { type: "string" },
sourceId: { type: "string" }, sourceId: { type: "string" },
sourceVersionId: { type: "string", format: "uuid" },
sourceVersionNumber: { type: "integer" },
title: { type: "string" }, title: { type: "string" },
sectionTitle: { type: "string" }, sectionTitle: { type: "string" },
content: { type: "string" }, content: { type: "string" },
@ -433,6 +579,8 @@ export const openApiDocument = {
properties: { properties: {
chunkId: { type: "string" }, chunkId: { type: "string" },
documentId: { type: "string" }, documentId: { type: "string" },
sourceVersionId: { type: "string", format: "uuid" },
sourceVersionNumber: { type: "integer" },
title: { type: "string" }, title: { type: "string" },
sectionTitle: { type: "string" }, sectionTitle: { type: "string" },
startLine: { type: "integer" }, startLine: { type: "integer" },
@ -571,18 +719,87 @@ export const openApiDocument = {
}, },
HealthResponse: { HealthResponse: {
type: "object", type: "object",
required: ["ok", "service", "environment", "embeddings", "answer", "vectorStore", "parsers", "chunking"], required: ["ok", "service", "environment", "embeddings", "answer", "vectorStore", "postgres", "knowledgeLifecycle", "parsers", "chunking"],
properties: { properties: {
ok: { type: "boolean", const: true }, ok: { type: "boolean" },
service: { type: "string", const: "rag" }, service: { type: "string", const: "rag" },
environment: { type: "string" }, environment: { type: "string" },
embeddings: ref("ProviderModel"), embeddings: ref("ProviderModel"),
answer: ref("ProviderModel"), answer: ref("ProviderModel"),
vectorStore: { type: "object", additionalProperties: true }, vectorStore: { type: "object", additionalProperties: true },
postgres: { type: "object", additionalProperties: true },
knowledgeLifecycle: { type: "object", additionalProperties: true },
parsers: { type: "array", items: { type: "string" } }, parsers: { type: "array", items: { type: "string" } },
chunking: { type: "object", additionalProperties: true } chunking: { type: "object", additionalProperties: true }
} }
}, },
SourceVersion: {
type: "object",
required: ["versionId", "sourceId", "versionNumber", "state", "tags", "embeddingProvider", "embeddingModel", "embeddingDimensions", "expectedDocumentCount", "expectedPointCount", "verifiedPointCount", "qdrantCollection"],
properties: {
versionId: { type: "string", format: "uuid" },
sourceId: { type: "string" },
versionNumber: { type: "integer" },
previousVersionId: { type: ["string", "null"], format: "uuid" },
state: ref("SourceVersionState"),
tags: { type: "array", items: { type: "string" } },
sourceContentHash: { type: ["string", "null"] },
processingFingerprint: { type: "string" },
metadataHash: { type: "string" },
embeddingProvider: { type: "string" },
embeddingModel: { type: "string" },
embeddingDimensions: { type: "integer" },
expectedDocumentCount: { type: "integer" },
expectedPointCount: { type: "integer" },
verifiedPointCount: { type: "integer" },
qdrantCollection: { type: "string" }
}
},
ActivateVersionRequest: {
type: "object",
required: ["expectedActiveVersionId"],
properties: { expectedActiveVersionId: { type: ["string", "null"], format: "uuid" } }
},
RollbackRequest: {
type: "object",
required: ["targetVersionId", "expectedActiveVersionId"],
properties: {
targetVersionId: { type: "string", format: "uuid" },
expectedActiveVersionId: { type: "string", format: "uuid" }
}
},
RetryVersionRequest: {
type: "object",
properties: {
sourceType: { type: "string", enum: ["file", "folder"] },
sourceRef: { type: "string" },
readPath: { type: "string" },
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
tags: { type: "array", items: { type: "string" } },
activate: { type: "boolean", default: true },
expectedActiveVersionId: { type: ["string", "null"], format: "uuid" }
}
},
RetryVersionResponse: {
type: "object",
required: ["ok", "retryRequired", "sourceId", "versionId", "message"],
properties: {
ok: { type: "boolean", const: true },
retryRequired: { type: "boolean", const: true },
sourceId: { type: "string" },
versionId: { type: "string", format: "uuid" },
message: { type: "string" }
}
},
PurgeResponse: {
type: "object",
required: ["ok", "purged", "versionId"],
properties: {
ok: { type: "boolean", const: true },
purged: { type: "boolean", const: true },
versionId: { type: "string", format: "uuid" }
}
},
ProviderModel: { ProviderModel: {
type: "object", type: "object",
required: ["provider", "model"], required: ["provider", "model"],

238
src/app.ts
View file

@ -4,6 +4,7 @@ import type { Request } from "express";
import { writeFile, unlink, mkdtemp, rm } from "node:fs/promises"; import { writeFile, unlink, mkdtemp, rm } from "node:fs/promises";
import os from "node:os"; import os from "node:os";
import path from "node:path"; import path from "node:path";
import { timingSafeEqual } from "node:crypto";
import { fileURLToPath } from "node:url"; import { fileURLToPath } from "node:url";
import AdmZip from "adm-zip"; import AdmZip from "adm-zip";
import { buildApiHelp, openApiDocument } from "./api/openapi.js"; import { buildApiHelp, openApiDocument } from "./api/openapi.js";
@ -16,6 +17,9 @@ import { documentalChunkingPolicy } from "./modules/process/chunking.js";
import { RetrieveService } from "./modules/retrieve/service.js"; import { RetrieveService } from "./modules/retrieve/service.js";
import { supportedParserExtensions } from "./modules/parsers/parser-registry.js"; import { supportedParserExtensions } from "./modules/parsers/parser-registry.js";
import { QdrantVectorStoreClient } from "./modules/vectorstore/client.js"; import { QdrantVectorStoreClient } from "./modules/vectorstore/client.js";
import { getCatalogPool } from "./modules/catalog/client.js";
import { KnowledgeLifecycleReconciler } from "./modules/catalog/reconciler.js";
import { CatalogError, CatalogRepository } from "./modules/catalog/repository.js";
import type { ChatMessage, ChunkMode, RetrieveIntent, RetrieveScope } from "./shared/types/rag.js"; import type { ChatMessage, ChunkMode, RetrieveIntent, RetrieveScope } from "./shared/types/rag.js";
type UploadRequest = Request & { type UploadRequest = Request & {
@ -30,10 +34,50 @@ export function createApp() {
const upload = multer({ storage: multer.memoryStorage() }); const upload = multer({ storage: multer.memoryStorage() });
const embeddingProvider = new OpenRouterEmbeddingProvider(); const embeddingProvider = new OpenRouterEmbeddingProvider();
const vectorStore = new QdrantVectorStoreClient(); const vectorStore = new QdrantVectorStoreClient();
const catalogPool = getCatalogPool();
const catalog = catalogPool ? new CatalogRepository(catalogPool) : undefined;
const reconciler = new KnowledgeLifecycleReconciler(catalog, vectorStore);
const evaluationLogs = new EvaluationLogService(embeddingProvider); const evaluationLogs = new EvaluationLogService(embeddingProvider);
const ingestService = new IngestService(embeddingProvider, vectorStore); const ingestService = new IngestService(embeddingProvider, vectorStore, catalog);
const retrieveService = new RetrieveService(embeddingProvider, vectorStore); const retrieveService = new RetrieveService(embeddingProvider, vectorStore, catalog);
const answerService = new AnswerService(retrieveService); const answerService = new AnswerService(retrieveService);
reconciler.start();
function sendError(res: express.Response, error: unknown, fallback: string) {
const upstreamUnavailable = error instanceof Error && /(ECONNREFUSED|ETIMEDOUT|timeout|connection|database|postgres|qdrant)/i.test(error.message);
const statusCode = error instanceof CatalogError ? error.statusCode : upstreamUnavailable ? 503 : 500;
const body = {
ok: false,
error: error instanceof Error ? error.message : fallback,
code: error instanceof CatalogError ? error.code : undefined
};
res.status(statusCode).json(body);
}
function requireLifecycleAdmin(req: express.Request, res: express.Response): boolean {
const header = req.header("authorization") ?? "";
const token = header.startsWith("Bearer ") ? header.slice("Bearer ".length) : "";
if (!env.lifecycleAdminToken || !token) {
res.status(401).json({ ok: false, error: "Lifecycle admin token is required" });
return false;
}
const expected = Buffer.from(env.lifecycleAdminToken);
const actual = Buffer.from(token);
const valid = expected.length === actual.length && timingSafeEqual(expected, actual);
if (!valid) {
res.status(401).json({ ok: false, error: "Lifecycle admin token is invalid" });
return false;
}
return true;
}
function requireCatalog(res: express.Response): CatalogRepository | undefined {
if (!catalog) {
res.status(503).json({ ok: false, error: "Knowledge catalog is not configured", code: "CATALOG_UNAVAILABLE" });
return undefined;
}
return catalog;
}
function needsContextLog(summary?: string, itemsCount = 0, answer?: string) { function needsContextLog(summary?: string, itemsCount = 0, answer?: string) {
if (itemsCount === 0) { if (itemsCount === 0) {
@ -76,8 +120,11 @@ export function createApp() {
app.get("/health", async (_req, res) => { app.get("/health", async (_req, res) => {
const vectorHealth = await vectorStore.healthcheck(); const vectorHealth = await vectorStore.healthcheck();
res.json({ const postgresHealth = catalog ? await catalog.healthcheck() : { ok: false, kind: "postgres" as const };
ok: true, const reconcilerStatus = reconciler.getStatus();
const ok = vectorHealth.ok && (!env.knowledgeLifecycleEnforced || (postgresHealth.ok && reconcilerStatus.ok));
res.status(ok ? 200 : 503).json({
ok,
service: "rag", service: "rag",
environment: env.nodeEnv, environment: env.nodeEnv,
embeddings: { embeddings: {
@ -89,6 +136,12 @@ export function createApp() {
model: env.answerModel model: env.answerModel
}, },
vectorStore: vectorHealth, vectorStore: vectorHealth,
postgres: postgresHealth,
knowledgeLifecycle: {
enforced: env.knowledgeLifecycleEnforced,
ingestWritesEnabled: env.ingestWritesEnabled,
reconciler: reconcilerStatus
},
parsers: supportedParserExtensions(), parsers: supportedParserExtensions(),
chunking: documentalChunkingPolicy chunking: documentalChunkingPolicy
}); });
@ -96,10 +149,55 @@ export function createApp() {
app.get("/sources", async (_req, res) => { app.get("/sources", async (_req, res) => {
try { try {
const scopes = await vectorStore.listScopes(); if (env.knowledgeLifecycleEnforced && !catalog) {
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
}
const scopes = env.knowledgeLifecycleEnforced && catalog
? await catalog.listSources()
: await vectorStore.listScopes();
res.json(scopes); res.json(scopes);
} catch (error) { } catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown sources error" }); sendError(res, error, "Unknown sources error");
}
});
app.get("/sources/:sourceId", async (req, res) => {
const repository = requireCatalog(res);
if (!repository) return;
try {
const source = await repository.getSource(String(req.params.sourceId));
if (!source) {
res.status(404).json({ ok: false, error: "Source not found" });
return;
}
res.json(source);
} catch (error) {
sendError(res, error, "Unknown source error");
}
});
app.get("/sources/:sourceId/versions", async (req, res) => {
const repository = requireCatalog(res);
if (!repository) return;
try {
res.json(await repository.listVersions(String(req.params.sourceId)));
} catch (error) {
sendError(res, error, "Unknown versions error");
}
});
app.get("/sources/:sourceId/versions/:versionId", async (req, res) => {
const repository = requireCatalog(res);
if (!repository) return;
try {
const version = await repository.getVersion(String(req.params.sourceId), String(req.params.versionId));
if (!version) {
res.status(404).json({ ok: false, error: "Version not found" });
return;
}
res.json(version);
} catch (error) {
sendError(res, error, "Unknown version error");
} }
}); });
@ -128,14 +226,23 @@ export function createApp() {
app.post("/ingest", async (req, res) => { app.post("/ingest", async (req, res) => {
try { try {
const result = await ingestService.ingest(req.body); const result = await ingestService.ingest(req.body);
res.status(202).json(result); res.status(env.knowledgeLifecycleEnforced ? 201 : 202).json(result);
} catch (error) { } catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown ingest error" }); sendError(res, error, "Unknown ingest error");
} }
}); });
app.post("/cleanup", async (req, res) => { app.post("/cleanup", async (req, res) => {
try { try {
if (env.knowledgeLifecycleEnforced) {
if (!requireLifecycleAdmin(req, res)) return;
res.status(410).json({
ok: false,
error: "POST /cleanup is deprecated while knowledge lifecycle is enforced; purge non-active versions with DELETE /sources/:sourceId/versions/:versionId.",
code: "CLEANUP_DEPRECATED_UNDER_LIFECYCLE"
});
return;
}
if (!req.body.scope) { if (!req.body.scope) {
res.status(400).json({ ok: false, error: "Missing scope for cleanup" }); res.status(400).json({ ok: false, error: "Missing scope for cleanup" });
return; return;
@ -155,7 +262,100 @@ export function createApp() {
const result = await ingestService.cleanup(scope); const result = await ingestService.cleanup(scope);
res.status(200).json({ ok: true, ...result }); res.status(200).json({ ok: true, ...result });
} catch (error) { } catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown cleanup error" }); sendError(res, error, "Unknown cleanup error");
}
});
app.post("/sources/:sourceId/versions/:versionId/activate", async (req, res) => {
if (!requireLifecycleAdmin(req, res)) return;
const repository = requireCatalog(res);
if (!repository) return;
try {
const version = await repository.activateVersion(String(req.params.sourceId), String(req.params.versionId), req.body.expectedActiveVersionId);
res.json(version);
} catch (error) {
sendError(res, error, "Unknown activation error");
}
});
app.post("/sources/:sourceId/rollback", async (req, res) => {
if (!requireLifecycleAdmin(req, res)) return;
const repository = requireCatalog(res);
if (!repository) return;
try {
if (!req.body.targetVersionId || !req.body.expectedActiveVersionId) {
res.status(400).json({ ok: false, error: "targetVersionId and expectedActiveVersionId are required" });
return;
}
const version = await repository.rollback(String(req.params.sourceId), String(req.body.targetVersionId), String(req.body.expectedActiveVersionId));
res.json(version);
} catch (error) {
sendError(res, error, "Unknown rollback error");
}
});
app.post("/sources/:sourceId/versions/:versionId/retry", async (req, res) => {
if (!requireLifecycleAdmin(req, res)) return;
const repository = requireCatalog(res);
if (!repository) return;
try {
const version = await repository.getVersion(String(req.params.sourceId), String(req.params.versionId));
if (!version) {
res.status(404).json({ ok: false, error: "Version not found" });
return;
}
if (!["failed", "pending", "indexing"].includes(version.state)) {
res.status(409).json({ ok: false, error: "Only failed or orphaned pending/indexing versions can be retried" });
return;
}
if (req.body.sourceType && req.body.sourceRef) {
if (["pending", "indexing"].includes(version.state)) {
await repository.markFailed(version.versionId, "EXPLICIT_RETRY_REPLACED", "Explicit retry requested; the orphaned version was closed before creating a replacement.");
}
const result = await ingestService.ingest({
sourceId: version.sourceId,
sourceType: req.body.sourceType,
sourceRef: String(req.body.sourceRef),
readPath: req.body.readPath ? String(req.body.readPath) : undefined,
mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
tags: Array.isArray(req.body.tags) ? req.body.tags.map(String) : version.tags,
activate: req.body.activate === undefined ? true : Boolean(req.body.activate),
expectedActiveVersionId: req.body.expectedActiveVersionId === undefined ? null : req.body.expectedActiveVersionId
});
res.status(201).json(result);
return;
}
res.status(202).json({
ok: true,
retryRequired: true,
sourceId: version.sourceId,
versionId: version.versionId,
message: "Explicit retry is safe only by re-submitting the original source to /ingest with the same sourceId and expectedActiveVersionId. Temporary uploads without durable originals cannot be retried in place."
});
} catch (error) {
sendError(res, error, "Unknown retry error");
}
});
app.delete("/sources/:sourceId/versions/:versionId", async (req, res) => {
if (!requireLifecycleAdmin(req, res)) return;
const repository = requireCatalog(res);
if (!repository) return;
try {
const sourceId = String(req.params.sourceId);
const versionId = String(req.params.versionId);
await repository.withSourceLock(sourceId, async () => repository.withVersionExclusiveLock(versionId, async () => {
await repository.markPurging(sourceId, versionId);
await vectorStore.deleteVersionPoints(versionId);
const remaining = await vectorStore.countVersionPoints(versionId);
if (remaining !== 0) {
throw new CatalogError("Version purge did not delete all points", 503, "VERSION_PURGE_INCOMPLETE");
}
await repository.markPurged(versionId);
}));
res.json({ ok: true, purged: true, versionId });
} catch (error) {
sendError(res, error, "Unknown purge error");
} }
}); });
@ -181,7 +381,6 @@ export function createApp() {
let result; let result;
if (isZipFolder) { if (isZipFolder) {
// Logica para carpetas subidas como ZIP
extractDirPath = await mkdtemp(path.join(tempDirBase, "rag-upload-")); extractDirPath = await mkdtemp(path.join(tempDirBase, "rag-upload-"));
const zip = new AdmZip(tempFilePath); const zip = new AdmZip(tempFilePath);
zip.extractAllTo(extractDirPath, true); zip.extractAllTo(extractDirPath, true);
@ -189,29 +388,32 @@ export function createApp() {
result = await ingestService.ingest({ result = await ingestService.ingest({
sourceId: req.body.sourceId ? String(req.body.sourceId) : undefined, sourceId: req.body.sourceId ? String(req.body.sourceId) : undefined,
sourceType: "folder", sourceType: "folder",
sourceRef: req.file.originalname.replace(/\.zip$/i, ""), // Usamos el nombre original sin .zip como ref sourceRef: req.file.originalname.replace(/\.zip$/i, ""),
readPath: extractDirPath, readPath: extractDirPath,
mode: req.body.mode === "interactive" ? "interactive" : "mechanical", mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
tags tags,
activate: req.body.activate === undefined ? undefined : req.body.activate === "true",
expectedActiveVersionId: req.body.expectedActiveVersionId === "null" ? null : req.body.expectedActiveVersionId ? String(req.body.expectedActiveVersionId) : undefined
}); });
} else { } else {
// Logica para archivos sueltos
result = await ingestService.ingest({ result = await ingestService.ingest({
sourceId: req.body.sourceId ? String(req.body.sourceId) : undefined, sourceId: req.body.sourceId ? String(req.body.sourceId) : undefined,
sourceType: "file", sourceType: "file",
sourceRef: req.file.originalname, sourceRef: req.file.originalname,
readPath: tempFilePath, readPath: tempFilePath,
mode: req.body.mode === "interactive" ? "interactive" : "mechanical", mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
tags tags,
activate: req.body.activate === undefined ? undefined : req.body.activate === "true",
expectedActiveVersionId: req.body.expectedActiveVersionId === "null" ? null : req.body.expectedActiveVersionId ? String(req.body.expectedActiveVersionId) : undefined
}); });
} }
res.status(202).json({ res.status(env.knowledgeLifecycleEnforced ? 201 : 202).json({
...result, ...result,
uploadedResource: req.file.originalname uploadedResource: req.file.originalname
}); });
} catch (error) { } catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown upload ingest error" }); sendError(res, error, "Unknown upload ingest error");
} finally { } finally {
if (tempFilePath) { if (tempFilePath) {
await unlink(tempFilePath).catch(() => undefined); await unlink(tempFilePath).catch(() => undefined);
@ -284,7 +486,7 @@ export function createApp() {
res.json(result); res.json(result);
} catch (error) { } catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown retrieve error" }); sendError(res, error, "Unknown retrieve error");
} }
}); });
@ -335,7 +537,7 @@ export function createApp() {
res.json(result); res.json(result);
} catch (error) { } catch (error) {
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown answer error" }); sendError(res, error, "Unknown answer error");
} }
}); });

29
src/config/env.ts
View file

@ -1,7 +1,9 @@
import { config as loadEnv } from "dotenv"; import { config as loadEnv } from "dotenv";
if (process.env.NODE_ENV !== "test") {
loadEnv(); loadEnv();
loadEnv({ path: ".env.local", override: true }); loadEnv({ path: ".env.local", override: true });
}
function requireEnv(name: string, fallback?: string): string { function requireEnv(name: string, fallback?: string): string {
const value = process.env[name] ?? fallback; const value = process.env[name] ?? fallback;
@ -11,6 +13,24 @@ function requireEnv(name: string, fallback?: string): string {
return value; return value;
} }
function optionalEnv(...names: string[]): string {
for (const name of names) {
const value = process.env[name];
if (value) {
return value;
}
}
return "";
}
function booleanEnv(name: string, fallback: boolean): boolean {
const value = process.env[name];
if (value === undefined) {
return fallback;
}
return ["1", "true", "yes", "on"].includes(value.toLowerCase());
}
export const env = { export const env = {
nodeEnv: process.env.NODE_ENV ?? "development", nodeEnv: process.env.NODE_ENV ?? "development",
port: Number(process.env.PORT ?? 3000), port: Number(process.env.PORT ?? 3000),
@ -25,5 +45,12 @@ export const env = {
answerProvider: requireEnv("ANSWER_PROVIDER", "openrouter"), answerProvider: requireEnv("ANSWER_PROVIDER", "openrouter"),
answerModel: requireEnv("ANSWER_MODEL", "openai/gpt-4.1-mini"), answerModel: requireEnv("ANSWER_MODEL", "openai/gpt-4.1-mini"),
answerBaseUrl: requireEnv("ANSWER_BASE_URL", "https://openrouter.ai/api/v1"), answerBaseUrl: requireEnv("ANSWER_BASE_URL", "https://openrouter.ai/api/v1"),
answerApiKey: process.env.ANSWER_API_KEY ?? process.env.EMBEDDING_API_KEY ?? "" answerApiKey: process.env.ANSWER_API_KEY ?? process.env.EMBEDDING_API_KEY ?? "",
postgresUrl: optionalEnv("POSTGRES_URL", "DATABASE_URL"),
postgresSsl: booleanEnv("POSTGRES_SSL", false),
lifecycleAdminToken: process.env.LIFECYCLE_ADMIN_TOKEN ?? "",
knowledgeLifecycleEnforced: booleanEnv("KNOWLEDGE_LIFECYCLE_ENFORCED", false),
ingestWritesEnabled: booleanEnv("INGEST_WRITES_ENABLED", true),
lifecycleReconcileIntervalMs: Number(process.env.LIFECYCLE_RECONCILE_INTERVAL_MS ?? 300000),
lifecycleIndexingStaleTimeoutMs: Number(process.env.LIFECYCLE_INDEXING_STALE_TIMEOUT_MS ?? 1800000)
} as const; } as const;

2
src/modules/answer/service.ts
View file

@ -116,6 +116,8 @@ export class AnswerService {
citations: retrieved.items.map((item) => ({ citations: retrieved.items.map((item) => ({
chunkId: item.chunkId, chunkId: item.chunkId,
documentId: item.documentId, documentId: item.documentId,
sourceVersionId: item.sourceVersionId,
sourceVersionNumber: item.sourceVersionNumber,
title: item.title, title: item.title,
sectionTitle: item.sectionTitle, sectionTitle: item.sectionTitle,
startLine: item.startLine, startLine: item.startLine,

94
src/modules/catalog/client.ts Normal file
View file

@ -0,0 +1,94 @@
import pg from "pg";
import { env } from "../../config/env.js";
const { Pool } = pg;
export type PgPool = pg.Pool;
export type PgPoolClient = pg.PoolClient;
let sharedPool: PgPool | undefined;
export function hasCatalogConnection(): boolean {
return Boolean(env.postgresUrl);
}
export function buildCatalogPool(): PgPool {
if (!env.postgresUrl) {
throw new Error("Missing POSTGRES_URL or DATABASE_URL for knowledge catalog");
}
return new Pool({
connectionString: env.postgresUrl,
ssl: env.postgresSsl ? { rejectUnauthorized: false } : undefined
});
}
export function getCatalogPool(): PgPool | undefined {
if (!hasCatalogConnection()) {
return undefined;
}
sharedPool ??= buildCatalogPool();
return sharedPool;
}
export async function withTransaction<T>(pool: PgPool, handler: (client: PgPoolClient) => Promise<T>): Promise<T> {
const client = await pool.connect();
try {
await client.query("BEGIN");
await client.query("SET CONSTRAINTS ALL DEFERRED");
const result = await handler(client);
await client.query("COMMIT");
return result;
} catch (error) {
await client.query("ROLLBACK").catch(() => undefined);
throw error;
} finally {
client.release();
}
}
export async function withCatalogAdvisoryLock<T>(pool: PgPool, lockName: string, handler: () => Promise<T>): Promise<T> {
const client = await pool.connect();
try {
await client.query("SELECT pg_advisory_lock(hashtext($1))", [lockName]);
return await handler();
} finally {
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [lockName]).catch(() => undefined);
client.release();
}
}
export async function withCatalogTryAdvisoryLock<T>(pool: PgPool, lockName: string, handler: () => Promise<T>): Promise<T | undefined> {
const client = await pool.connect();
let locked = false;
try {
const result = await client.query<{ locked: boolean }>("SELECT pg_try_advisory_lock(hashtext($1)) AS locked", [lockName]);
locked = Boolean(result.rows[0]?.locked);
if (!locked) {
return undefined;
}
return await handler();
} finally {
if (locked) {
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [lockName]).catch(() => undefined);
}
client.release();
}
}
export async function withCatalogAdvisorySharedLocks<T>(pool: PgPool, lockNames: string[], handler: () => Promise<T>): Promise<T> {
const names = [...new Set(lockNames)].sort((left, right) => Buffer.compare(Buffer.from(left), Buffer.from(right)));
const client = await pool.connect();
try {
for (const name of names) {
await client.query("SELECT pg_advisory_lock_shared(hashtext($1))", [name]);
}
return await handler();
} finally {
for (const name of names.reverse()) {
await client.query("SELECT pg_advisory_unlock_shared(hashtext($1))", [name]).catch(() => undefined);
}
client.release();
}
}

9
src/modules/catalog/errors.ts Normal file
View file

@ -0,0 +1,9 @@
export class CatalogError extends Error {
constructor(
message: string,
readonly statusCode = 500,
readonly code = "CATALOG_ERROR"
) {
super(message);
}
}

150
src/modules/catalog/legacy-migration.ts Normal file
View file

@ -0,0 +1,150 @@
import { buildDocumentId, buildMetadataHash, hashOrderedPairs, sha256Hex } from "../../shared/utils/ids.js";
import type { SourceType } from "../../shared/types/rag.js";
export type LegacyPointId = string | number;
export interface LegacyPointSnapshot {
id: LegacyPointId;
payload: Record<string, unknown>;
}
export interface LegacyDocumentPlan {
documentId: string;
documentKey: string;
originalHash: string;
contentHash: string;
mimeType: string;
title: string;
chunkCount: number;
}
export interface LegacySourcePlan {
sourceId: string;
sourceType: SourceType;
sourceRef: string;
pointIds: LegacyPointId[];
tags: string[];
documents: LegacyDocumentPlan[];
originalManifestHash: string;
sourceContentHash: string;
metadataHash: string;
embeddingProvider: string;
embeddingModel: string;
embeddingDimensions: number;
blockedReasons: string[];
}
export interface LegacyMigrationPlan {
ok: boolean;
sources: LegacySourcePlan[];
blockedReasons: string[];
}
function stringValue(value: unknown): string {
return typeof value === "string" ? value.trim() : "";
}
function stringArray(value: unknown): string[] {
return Array.isArray(value) ? value.map(String).filter(Boolean).sort() : [];
}
function unique(values: string[]): string[] {
return [...new Set(values.filter(Boolean))];
}
function sortedByUtf8<T>(values: T[], getKey: (value: T) => string): T[] {
return [...values].sort((left, right) => Buffer.compare(Buffer.from(getKey(left), "utf8"), Buffer.from(getKey(right), "utf8")));
}
export function buildLegacyMigrationPlan(points: LegacyPointSnapshot[]): LegacyMigrationPlan {
const groups = new Map<string, LegacyPointSnapshot[]>();
const globalBlocked: string[] = [];
for (const point of points) {
const sourceId = stringValue(point.payload.source_id);
if (!sourceId) {
globalBlocked.push(`Point ${String(point.id)} is missing source_id.`);
continue;
}
const entries = groups.get(sourceId) ?? [];
entries.push(point);
groups.set(sourceId, entries);
}
const sources = sortedByUtf8([...groups.entries()], ([sourceId]) => sourceId).map(([sourceId, sourcePoints]) => {
const blockedReasons: string[] = [];
const sourceRefs = unique(sourcePoints.map((point) => stringValue(point.payload.source_ref)));
const sourceTypes = unique(sourcePoints.map((point) => stringValue(point.payload.source_type)));
const providers = unique(sourcePoints.map((point) => stringValue(point.payload.embedding_provider)));
const models = unique(sourcePoints.map((point) => stringValue(point.payload.embedding_model)));
const dimensions = unique(sourcePoints.map((point) => String(Number(point.payload.embedding_dimensions ?? 0))).filter((value) => value !== "0"));
const tagSnapshot = unique(sourcePoints.flatMap((point) => stringArray(point.payload.tags)));
if (sourceRefs.length !== 1) blockedReasons.push("Source has zero or multiple source_ref values.");
if (sourceTypes.length !== 1 || !["file", "folder"].includes(sourceTypes[0])) blockedReasons.push("Source has no unique file/folder source_type.");
if (providers.length !== 1) blockedReasons.push("Source has no unique embedding provider.");
if (models.length !== 1) blockedReasons.push("Source has no unique embedding model.");
if (dimensions.length !== 1) blockedReasons.push("Source has no unique non-zero embedding dimensions.");
const documentGroups = new Map<string, LegacyPointSnapshot[]>();
for (const point of sourcePoints) {
const documentId = stringValue(point.payload.document_id);
const documentKey = stringValue(point.payload.document_key);
const chunkId = stringValue(point.payload.chunk_id);
const content = stringValue(point.payload.content);
if (!documentId || !documentKey || !chunkId || !content) {
blockedReasons.push(`Point ${String(point.id)} has weak document identity or empty content.`);
continue;
}
if (documentId !== buildDocumentId(sourceId, documentKey)) {
blockedReasons.push(`Point ${String(point.id)} document_id does not match source_id and document_key.`);
}
const existing = documentGroups.get(documentId) ?? [];
existing.push(point);
documentGroups.set(documentId, existing);
}
const documents: LegacyDocumentPlan[] = sortedByUtf8([...documentGroups.entries()], ([documentId]) => documentId).map(([documentId, documentPoints]) => {
const documentKeys = unique(documentPoints.map((point) => stringValue(point.payload.document_key)));
const mimeTypes = unique(documentPoints.map((point) => stringValue(point.payload.mime_type) || "text/plain"));
const titles = unique(documentPoints.map((point) => stringValue(point.payload.title) || documentKeys[0]));
if (documentKeys.length !== 1) blockedReasons.push(`Document ${documentId} has no unique document_key.`);
if (mimeTypes.length !== 1) blockedReasons.push(`Document ${documentId} has no unique mime_type.`);
const orderedContents = sortedByUtf8(documentPoints, (point) => stringValue(point.payload.chunk_id))
.map((point) => stringValue(point.payload.content))
.join("\n");
const contentHash = sha256Hex(orderedContents);
return {
documentId,
documentKey: documentKeys[0] ?? "",
originalHash: contentHash,
contentHash,
mimeType: mimeTypes[0] ?? "text/plain",
title: titles[0] ?? documentKeys[0] ?? documentId,
chunkCount: documentPoints.length
};
});
const originalManifestHash = hashOrderedPairs(documents.map((document) => [document.documentKey, document.originalHash])) ?? "";
const sourceContentHash = hashOrderedPairs(documents.map((document) => [document.documentKey, document.contentHash])) ?? "";
return {
sourceId,
sourceType: sourceTypes[0] as SourceType,
sourceRef: sourceRefs[0] ?? "",
pointIds: sourcePoints.map((point) => point.id),
tags: tagSnapshot,
documents,
originalManifestHash,
sourceContentHash,
metadataHash: buildMetadataHash(tagSnapshot, { sourceType: sourceTypes[0], sourceRef: sourceRefs[0] }),
embeddingProvider: providers[0] ?? "",
embeddingModel: models[0] ?? "",
embeddingDimensions: Number(dimensions[0] ?? 0),
blockedReasons: unique(blockedReasons)
};
});
const blockedReasons = [...globalBlocked, ...sources.flatMap((source) => source.blockedReasons.map((reason) => `${source.sourceId}: ${reason}`))];
return { ok: blockedReasons.length === 0, sources, blockedReasons };
}

11
src/modules/catalog/lifecycle.ts Normal file
View file

@ -0,0 +1,11 @@
import { CatalogError } from "./errors.js";
export function normalizeExpectedActiveVersion(value: string | null | undefined, activate: boolean): string | null | undefined {
if (!activate) {
return undefined;
}
if (value === undefined) {
throw new CatalogError("expectedActiveVersionId is required when activate is true", 400, "EXPECTED_ACTIVE_VERSION_REQUIRED");
}
return value;
}

90
src/modules/catalog/migrations.ts Normal file
View file

@ -0,0 +1,90 @@
import { createHash } from "node:crypto";
import { readdir, readFile } from "node:fs/promises";
import path from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { buildCatalogPool, hasCatalogConnection, type PgPool } from "./client.js";
const GLOBAL_MIGRATION_LOCK = "rag:knowledge-lifecycle:migrations";
function sha256(value: string): string {
return createHash("sha256").update(value).digest("hex");
}
function resolveMigrationsDir(): string {
const currentFile = fileURLToPath(import.meta.url);
return path.resolve(path.dirname(currentFile), "../../../migrations");
}
export async function runKnowledgeLifecycleMigrations(pool: PgPool = buildCatalogPool()): Promise<void> {
const migrationsDir = resolveMigrationsDir();
const files = (await readdir(migrationsDir))
.filter((file) => /^\d+_.*\.sql$/.test(file))
.sort();
const client = await pool.connect();
try {
await client.query("SELECT pg_advisory_lock(hashtext($1))", [GLOBAL_MIGRATION_LOCK]);
await client.query(`
CREATE TABLE IF NOT EXISTS rag_schema_migrations (
name text PRIMARY KEY,
checksum char(64) NOT NULL,
applied_at timestamptz NOT NULL DEFAULT now()
)
`);
for (const file of files) {
const sql = await readFile(path.join(migrationsDir, file), "utf8");
const checksum = sha256(sql);
const existing = await client.query<{ checksum: string }>(
"SELECT checksum FROM rag_schema_migrations WHERE name = $1",
[file]
);
if (existing.rowCount && existing.rows[0].checksum !== checksum) {
throw new Error(`Migration checksum mismatch: ${file}`);
}
if (existing.rowCount) {
continue;
}
await client.query("BEGIN");
try {
await client.query(sql);
await client.query(
"INSERT INTO rag_schema_migrations(name, checksum) VALUES ($1, $2)",
[file, checksum]
);
await client.query("COMMIT");
} catch (error) {
await client.query("ROLLBACK").catch(() => undefined);
throw error;
}
}
} finally {
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [GLOBAL_MIGRATION_LOCK]).catch(() => undefined);
client.release();
}
}
async function main() {
if (!hasCatalogConnection()) {
console.warn("Knowledge lifecycle migrations skipped: POSTGRES_URL or DATABASE_URL is not configured.");
return;
}
const pool = buildCatalogPool();
try {
await runKnowledgeLifecycleMigrations(pool);
console.log("Knowledge lifecycle migrations completed.");
} finally {
await pool.end();
}
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
main().catch((error) => {
console.error(error instanceof Error ? error.message : error);
process.exit(1);
});
}

123
src/modules/catalog/reconciler.ts Normal file
View file

@ -0,0 +1,123 @@
import { env } from "../../config/env.js";
import type { VectorStoreClient } from "../vectorstore/client.js";
import type { CatalogRepository } from "./repository.js";
export interface ReconcilerStatus {
ok: boolean;
lastRunAt?: string;
lastError?: string;
inconsistentSources: string[];
orphanedVersionsRecovered?: number;
orphanedVersionsFailed?: number;
invariantViolations?: string[];
}
export class KnowledgeLifecycleReconciler {
private status: ReconcilerStatus = { ok: true, inconsistentSources: [] };
private timer: NodeJS.Timeout | undefined;
constructor(
private readonly catalog: CatalogRepository | undefined,
private readonly vectorStore: VectorStoreClient
) {}
getStatus(): ReconcilerStatus {
return this.status;
}
start(): void {
if (!env.knowledgeLifecycleEnforced || !this.catalog || this.timer) {
return;
}
void this.runOnce();
this.timer = setInterval(() => {
void this.runOnce();
}, env.lifecycleReconcileIntervalMs);
this.timer.unref();
}
async runOnce(): Promise<ReconcilerStatus> {
if (!this.catalog) {
this.status = { ok: false, lastRunAt: new Date().toISOString(), lastError: "Knowledge catalog is not configured", inconsistentSources: [] };
return this.status;
}
const catalog = this.catalog;
try {
const lockedStatus = await catalog.withGlobalTryLock("rag:knowledge-lifecycle:reconciler", async () => {
const activeVersions = await catalog.resolveActiveVersions();
const inconsistentSources: string[] = [];
const invariantViolations = await catalog.validateActiveInvariant();
const orphanRecovery = await this.recoverOrphanedVersions(catalog);
for (const version of activeVersions) {
if (version.qdrantCollection !== env.qdrantCollection) {
inconsistentSources.push(version.sourceId);
await catalog.markNeedsReingest(version.sourceId);
continue;
}
await this.vectorStore.validateCollectionDimensions(version.embeddingDimensions);
const count = await this.vectorStore.countVersionPoints(version.sourceVersionId);
if (count !== version.expectedPointCount) {
inconsistentSources.push(version.sourceId);
await catalog.markNeedsReingest(version.sourceId);
}
}
return {
ok: inconsistentSources.length === 0 && invariantViolations.length === 0,
lastRunAt: new Date().toISOString(),
inconsistentSources,
orphanedVersionsRecovered: orphanRecovery.ready,
orphanedVersionsFailed: orphanRecovery.failed,
invariantViolations
};
});
this.status = lockedStatus ?? { ...this.status, lastRunAt: new Date().toISOString() };
return this.status;
} catch (error) {
this.status = {
ok: false,
lastRunAt: new Date().toISOString(),
lastError: error instanceof Error ? error.message : "Unknown reconciler error",
inconsistentSources: []
};
return this.status;
}
}
private async recoverOrphanedVersions(catalog: CatalogRepository): Promise<{ ready: number; failed: number }> {
const candidates = await catalog.listOrphanedIndexingCandidates(env.lifecycleIndexingStaleTimeoutMs);
let ready = 0;
let failed = 0;
for (const candidate of candidates) {
const recovered = await catalog.withSourceTryLock(candidate.sourceId, async () => catalog.withVersionTryLock(candidate.versionId, async () => {
const documentsComplete = await catalog.hasCompleteVersionDocuments(candidate.versionId, candidate.expectedDocumentCount, candidate.expectedPointCount);
const collectionMatches = candidate.qdrantCollection === env.qdrantCollection;
if (collectionMatches) {
await this.vectorStore.validateCollectionDimensions(candidate.embeddingDimensions);
}
const pointCount = collectionMatches ? await this.vectorStore.countVersionPoints(candidate.versionId) : 0;
if (collectionMatches && documentsComplete && pointCount === candidate.expectedPointCount) {
const changed = await catalog.recoverVersionReady(candidate.versionId, pointCount);
if (changed) {
return "ready" as const;
}
}
await catalog.markFailed(candidate.versionId, "ORPHANED_INDEXING_INCOMPLETE", "Stale pending/indexing version has incomplete documents, dimensions, collection, or point count.");
return "failed" as const;
}));
if (recovered === "ready") {
ready += 1;
} else if (recovered === "failed") {
failed += 1;
}
}
return { ready, failed };
}
}

641
src/modules/catalog/repository.ts Normal file
View file

@ -0,0 +1,641 @@
import type { AvailableScope, ChunkMode, IngestSourceInput, SourceType, SourceVersionState, RetrieveScope } from "../../shared/types/rag.js";
import { withCatalogAdvisoryLock, withCatalogAdvisorySharedLocks, withCatalogTryAdvisoryLock, withTransaction, type PgPool, type PgPoolClient } from "./client.js";
import { CatalogError } from "./errors.js";
import { normalizeExpectedActiveVersion } from "./lifecycle.js";
export interface CatalogDocumentInput {
documentId: string;
documentKey: string;
originalHash: string;
originalHashKind: "bytes" | "legacy-derived";
contentHash: string | null;
mimeType: string;
title: string;
chunkCount: number;
}
export interface CreateVersionInput {
sourceId: string;
sourceType: SourceType;
sourceRef: string;
originalManifestHash: string;
sourceContentHash: string | null;
processingFingerprint: string;
metadataHash: string;
tags: string[];
activateRequested: boolean;
embeddingProvider: string;
embeddingModel: string;
embeddingDimensions: number;
qdrantCollection: string;
expectedDocumentCount: number;
expectedPointCount: number;
documents: CatalogDocumentInput[];
}
export interface CatalogVersionRow {
versionId: string;
sourceId: string;
versionNumber: number;
previousVersionId: string | null;
state: SourceVersionState;
tags: string[];
sourceContentHash: string | null;
processingFingerprint: string;
metadataHash: string;
embeddingProvider: string;
embeddingModel: string;
embeddingDimensions: number;
expectedDocumentCount: number;
expectedPointCount: number;
verifiedPointCount: number;
qdrantCollection: string;
}
export interface ActiveVersionScope {
sourceId: string;
sourceVersionId: string;
sourceVersionNumber: number;
expectedPointCount: number;
embeddingDimensions: number;
qdrantCollection: string;
}
export interface OrphanedVersionCandidate {
sourceId: string;
versionId: string;
expectedDocumentCount: number;
expectedPointCount: number;
embeddingDimensions: number;
qdrantCollection: string;
}
type VersionDbRow = {
version_id: string;
source_id: string;
version_number: string | number;
previous_version_id: string | null;
state: SourceVersionState;
tags: string[];
source_content_hash: string | null;
processing_fingerprint: string;
metadata_hash: string;
embedding_provider: string;
embedding_model: string;
embedding_dimensions: string | number;
expected_document_count: string | number;
expected_point_count: string | number;
verified_point_count: string | number;
qdrant_collection: string;
};
function toVersion(row: VersionDbRow): CatalogVersionRow {
return {
versionId: row.version_id,
sourceId: row.source_id,
versionNumber: Number(row.version_number),
previousVersionId: row.previous_version_id,
state: row.state,
tags: row.tags ?? [],
sourceContentHash: row.source_content_hash,
processingFingerprint: row.processing_fingerprint,
metadataHash: row.metadata_hash,
embeddingProvider: row.embedding_provider,
embeddingModel: row.embedding_model,
embeddingDimensions: Number(row.embedding_dimensions),
expectedDocumentCount: Number(row.expected_document_count),
expectedPointCount: Number(row.expected_point_count),
verifiedPointCount: Number(row.verified_point_count),
qdrantCollection: row.qdrant_collection
};
}
export class CatalogRepository {
constructor(private readonly pool: PgPool) {}
async healthcheck(): Promise<{ ok: boolean; kind: "postgres" }> {
try {
await this.pool.query("SELECT 1");
return { ok: true, kind: "postgres" };
} catch {
return { ok: false, kind: "postgres" };
}
}
async withSourceLock<T>(sourceId: string, handler: () => Promise<T>): Promise<T> {
return withCatalogAdvisoryLock(this.pool, `rag:source:${sourceId}`, handler);
}
async withSourceTryLock<T>(sourceId: string, handler: () => Promise<T>): Promise<T | undefined> {
return withCatalogTryAdvisoryLock(this.pool, `rag:source:${sourceId}`, handler);
}
async withVersionSharedLocks<T>(versionIds: string[], handler: () => Promise<T>): Promise<T> {
return withCatalogAdvisorySharedLocks(this.pool, versionIds.map((versionId) => `rag:version:${versionId}`), handler);
}
async withVersionExclusiveLock<T>(versionId: string, handler: () => Promise<T>): Promise<T> {
return withCatalogAdvisoryLock(this.pool, `rag:version:${versionId}`, handler);
}
async withVersionTryLock<T>(versionId: string, handler: () => Promise<T>): Promise<T | undefined> {
return withCatalogTryAdvisoryLock(this.pool, `rag:version:${versionId}`, handler);
}
async withGlobalTryLock<T>(lockName: string, handler: () => Promise<T>): Promise<T | undefined> {
return withCatalogTryAdvisoryLock(this.pool, lockName, handler);
}
async isMaintenanceEnabled(client?: PgPoolClient, lockRow = false): Promise<boolean> {
const executor = client ?? this.pool;
const result = await executor.query<{ maintenance: boolean }>(
`SELECT maintenance FROM rag_system_state WHERE singleton = true${lockRow ? " FOR UPDATE" : ""}`
);
return Boolean(result.rows[0]?.maintenance);
}
async beginAttempt(source: IngestSourceInput & { sourceId: string }): Promise<string> {
return withTransaction(this.pool, async (client) => {
if (await this.isMaintenanceEnabled(client, true)) {
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
}
const sourceResult = await client.query<{ source_type: SourceType; source_ref: string }>(
`INSERT INTO rag_sources(source_id, source_type, source_ref)
VALUES ($1, $2, $3)
ON CONFLICT (source_id) DO NOTHING
RETURNING source_type, source_ref`,
[source.sourceId, source.sourceType, source.sourceRef]
);
const existing = sourceResult.rowCount
? sourceResult.rows[0]
: (await client.query<{ source_type: SourceType; source_ref: string }>(
"SELECT source_type, source_ref FROM rag_sources WHERE source_id = $1",
[source.sourceId]
)).rows[0];
if (!existing || existing.source_type !== source.sourceType || existing.source_ref !== source.sourceRef) {
throw new CatalogError("sourceId already exists with a different sourceType or sourceRef", 409, "SOURCE_IDENTITY_CONFLICT");
}
const result = await client.query<{ attempt_id: string }>(
`INSERT INTO rag_ingestion_attempts(source_id, state, input_locator)
VALUES ($1, 'received', $2)
RETURNING attempt_id`,
[source.sourceId, source.readPath ?? source.sourceRef]
);
return result.rows[0].attempt_id;
});
}
async updateAttempt(attemptId: string, values: { state: "processing" | "completed" | "failed"; versionId?: string | null; inputHash?: string | null; errorCode?: string; errorDetail?: string }): Promise<void> {
await this.pool.query(
`UPDATE rag_ingestion_attempts
SET state = $2,
version_id = COALESCE($3, version_id),
input_hash = COALESCE($4, input_hash),
error_code = $5,
error_detail = $6,
completed_at = CASE WHEN $2 IN ('completed', 'failed') THEN now() ELSE completed_at END
WHERE attempt_id = $1`,
[attemptId, values.state, values.versionId ?? null, values.inputHash ?? null, values.errorCode ?? null, values.errorDetail ?? null]
);
}
async findReusableVersion(input: Pick<CreateVersionInput, "sourceId" | "sourceContentHash" | "processingFingerprint" | "metadataHash">): Promise<CatalogVersionRow | undefined> {
if (!input.sourceContentHash) {
return undefined;
}
const result = await this.pool.query<VersionDbRow>(
`SELECT * FROM rag_source_versions
WHERE source_id = $1
AND source_content_hash = $2
AND processing_fingerprint = $3
AND metadata_hash = $4
AND state NOT IN ('failed', 'rejected', 'purged')
ORDER BY version_number DESC
LIMIT 1`,
[input.sourceId, input.sourceContentHash, input.processingFingerprint, input.metadataHash]
);
return result.rows[0] ? toVersion(result.rows[0]) : undefined;
}
async createPendingVersion(input: CreateVersionInput): Promise<CatalogVersionRow> {
return withTransaction(this.pool, async (client) => {
if (await this.isMaintenanceEnabled(client, true)) {
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
}
await client.query(
`INSERT INTO rag_sources(source_id, source_type, source_ref)
VALUES ($1, $2, $3)
ON CONFLICT (source_id) DO NOTHING`,
[input.sourceId, input.sourceType, input.sourceRef]
);
const source = await client.query<{ active_version_id: string | null; source_type: SourceType; source_ref: string }>(
"SELECT active_version_id, source_type, source_ref FROM rag_sources WHERE source_id = $1 FOR UPDATE",
[input.sourceId]
);
if (!source.rowCount) {
throw new CatalogError("Source not found after creation", 500, "SOURCE_CREATE_FAILED");
}
if (source.rows[0].source_type !== input.sourceType || source.rows[0].source_ref !== input.sourceRef) {
throw new CatalogError("sourceId already exists with a different sourceType or sourceRef", 409, "SOURCE_IDENTITY_CONFLICT");
}
const baseActiveVersionId = source.rows[0].active_version_id;
const versionNumberResult = await client.query<{ next_version_number: string }>(
"SELECT COALESCE(MAX(version_number), 0) + 1 AS next_version_number FROM rag_source_versions WHERE source_id = $1",
[input.sourceId]
);
const versionNumber = Number(versionNumberResult.rows[0].next_version_number);
const versionResult = await client.query<VersionDbRow>(
`INSERT INTO rag_source_versions(
source_id, version_number, previous_version_id, state, original_manifest_hash,
source_content_hash, processing_fingerprint, metadata_hash, tags, activate_requested,
base_active_version_id, embedding_provider, embedding_model, embedding_dimensions,
qdrant_collection, expected_document_count, expected_point_count
) VALUES ($1, $2, $3, 'pending', $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16)
RETURNING *`,
[
input.sourceId,
versionNumber,
baseActiveVersionId,
input.originalManifestHash,
input.sourceContentHash,
input.processingFingerprint,
input.metadataHash,
input.tags,
input.activateRequested,
baseActiveVersionId,
input.embeddingProvider,
input.embeddingModel,
input.embeddingDimensions,
input.qdrantCollection,
input.expectedDocumentCount,
input.expectedPointCount
]
);
const version = toVersion(versionResult.rows[0]);
for (const document of input.documents) {
await client.query(
`INSERT INTO rag_version_documents(
version_id, document_id, document_key, original_hash, original_hash_kind,
content_hash, mime_type, title, index_state, chunk_count
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'pending', $9)`,
[
version.versionId,
document.documentId,
document.documentKey,
document.originalHash,
document.originalHashKind,
document.contentHash,
document.mimeType,
document.title,
document.chunkCount
]
);
}
return version;
});
}
async markIndexing(versionId: string): Promise<void> {
const result = await this.pool.query(
`UPDATE rag_source_versions SET state = 'indexing', indexing_started_at = now() WHERE version_id = $1 AND state = 'pending'`,
[versionId]
);
if (result.rowCount !== 1) {
throw new CatalogError("Version could not transition from pending to indexing", 409, "INVALID_VERSION_STATE");
}
await this.pool.query("UPDATE rag_version_documents SET index_state = 'indexing' WHERE version_id = $1", [versionId]);
}
async markReady(versionId: string, verifiedPointCount: number): Promise<CatalogVersionRow> {
return withTransaction(this.pool, async (client) => {
await client.query("UPDATE rag_version_documents SET index_state = 'ready' WHERE version_id = $1", [versionId]);
const result = await client.query<VersionDbRow>(
`UPDATE rag_source_versions
SET state = 'ready', verified_point_count = $2, ready_at = now()
WHERE version_id = $1 AND state = 'indexing' AND expected_point_count = $2
RETURNING *`,
[versionId, verifiedPointCount]
);
if (result.rowCount !== 1) {
throw new CatalogError("Version could not transition from indexing to ready", 409, "INVALID_VERSION_STATE");
}
return toVersion(result.rows[0]);
});
}
async markFailed(versionId: string | undefined, code: string, detail: string): Promise<void> {
if (!versionId) {
return;
}
const result = await this.pool.query(
`UPDATE rag_source_versions
SET state = 'failed', error_code = $2, error_detail = $3
WHERE version_id = $1 AND state IN ('pending', 'indexing')`,
[versionId, code, detail.slice(0, 2000)]
);
if (result.rowCount === 1) {
await this.pool.query("UPDATE rag_version_documents SET index_state = 'failed', error_detail = $2 WHERE version_id = $1", [versionId, detail.slice(0, 2000)]);
}
}
async activateVersion(sourceId: string, versionId: string, expectedActiveVersionId: string | null | undefined): Promise<CatalogVersionRow> {
const expected = normalizeExpectedActiveVersion(expectedActiveVersionId, true);
return withTransaction(this.pool, async (client) => {
if (await this.isMaintenanceEnabled(client, true)) {
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
}
const source = await client.query<{ active_version_id: string | null }>(
"SELECT active_version_id FROM rag_sources WHERE source_id = $1 FOR UPDATE",
[sourceId]
);
if (!source.rowCount) {
throw new CatalogError("Source not found", 404, "SOURCE_NOT_FOUND");
}
const currentActive = source.rows[0].active_version_id;
if (currentActive !== expected) {
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
}
const version = await client.query<VersionDbRow>(
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2 FOR UPDATE",
[sourceId, versionId]
);
if (!version.rowCount) {
throw new CatalogError("Version not found", 404, "VERSION_NOT_FOUND");
}
if (!["ready", "superseded", "active"].includes(version.rows[0].state)) {
throw new CatalogError("Only ready or superseded versions can be activated", 409, "VERSION_NOT_ACTIVATABLE");
}
if (version.rows[0].state === "active") {
if (currentActive !== versionId) {
throw new CatalogError("Active version invariant is inconsistent", 503, "ACTIVE_VERSION_INVARIANT_FAILED");
}
return toVersion(version.rows[0]);
}
if (currentActive) {
await client.query(
"UPDATE rag_source_versions SET state = 'superseded', superseded_at = now() WHERE source_id = $1 AND version_id = $2",
[sourceId, currentActive]
);
}
const updated = await client.query<VersionDbRow>(
`UPDATE rag_source_versions
SET state = 'active', activated_at = now(), error_code = NULL, error_detail = NULL
WHERE source_id = $1 AND version_id = $2
RETURNING *`,
[sourceId, versionId]
);
if (updated.rowCount !== 1) {
throw new CatalogError("Version activation failed", 500, "VERSION_ACTIVATION_FAILED");
}
await client.query("UPDATE rag_sources SET active_version_id = $2, needs_reingest = false WHERE source_id = $1", [sourceId, versionId]);
return toVersion(updated.rows[0]);
});
}
async assertActiveVersionPrecondition(sourceId: string, expectedActiveVersionId: string | null | undefined): Promise<void> {
const expected = normalizeExpectedActiveVersion(expectedActiveVersionId, true);
const result = await this.pool.query<{ active_version_id: string | null }>(
"SELECT active_version_id FROM rag_sources WHERE source_id = $1",
[sourceId]
);
if (!result.rowCount) {
throw new CatalogError("Source not found", 404, "SOURCE_NOT_FOUND");
}
if (result.rows[0].active_version_id !== expected) {
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
}
}
async listSources(): Promise<AvailableScope[]> {
const result = await this.pool.query<{
source_id: string;
source_ref: string;
tags: string[] | null;
active_version_id: string | null;
version_number: string | number | null;
state: SourceVersionState | null;
needs_reingest: boolean;
updated_at: Date;
}>(
`SELECT s.source_id, s.source_ref, v.tags, s.active_version_id, v.version_number, v.state, s.needs_reingest, s.updated_at
FROM rag_sources s
LEFT JOIN rag_source_versions v ON v.version_id = s.active_version_id
WHERE s.disabled_at IS NULL
ORDER BY s.source_ref ASC`
);
return result.rows.map((row) => ({
sourceId: row.source_id,
sourceRef: row.source_ref,
chunkModes: ["documental", "codigo"] as ChunkMode[],
tags: row.tags ?? [],
activeVersionId: row.active_version_id,
activeVersionNumber: row.version_number === null ? null : Number(row.version_number),
state: row.state,
needsReingest: row.needs_reingest,
updatedAt: row.updated_at.toISOString()
}));
}
async getSource(sourceId: string): Promise<AvailableScope | undefined> {
return (await this.listSources()).find((source) => source.sourceId === sourceId);
}
async listVersions(sourceId: string): Promise<CatalogVersionRow[]> {
const result = await this.pool.query<VersionDbRow>(
"SELECT * FROM rag_source_versions WHERE source_id = $1 ORDER BY version_number DESC",
[sourceId]
);
return result.rows.map(toVersion);
}
async getVersion(sourceId: string, versionId: string): Promise<CatalogVersionRow | undefined> {
const result = await this.pool.query<VersionDbRow>(
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2",
[sourceId, versionId]
);
return result.rows[0] ? toVersion(result.rows[0]) : undefined;
}
async resolveActiveVersions(scope?: RetrieveScope): Promise<ActiveVersionScope[]> {
const mustTags = scope?.tags ?? [];
const result = await this.pool.query<{
source_id: string;
version_id: string;
version_number: string | number;
expected_point_count: string | number;
embedding_dimensions: string | number;
qdrant_collection: string;
}>(
`SELECT s.source_id, v.version_id, v.version_number, v.expected_point_count, v.embedding_dimensions, v.qdrant_collection
FROM rag_sources s
JOIN rag_source_versions v ON v.version_id = s.active_version_id AND v.state = 'active'
WHERE s.disabled_at IS NULL
AND ($1::text IS NULL OR s.source_id = $1)
AND ($2::text IS NULL OR s.source_ref = $2)
AND ($3::text[] IS NULL OR v.tags @> $3::text[])`,
[scope?.sourceId ?? null, scope?.sourceRef ?? null, mustTags.length ? mustTags : null]
);
return result.rows.map((row) => ({
sourceId: row.source_id,
sourceVersionId: row.version_id,
sourceVersionNumber: Number(row.version_number),
expectedPointCount: Number(row.expected_point_count),
embeddingDimensions: Number(row.embedding_dimensions),
qdrantCollection: row.qdrant_collection
}));
}
async markNeedsReingest(sourceId: string): Promise<void> {
await this.pool.query("UPDATE rag_sources SET needs_reingest = true WHERE source_id = $1", [sourceId]);
}
async rollback(sourceId: string, targetVersionId: string, expectedActiveVersionId: string): Promise<CatalogVersionRow> {
return this.withSourceLock(sourceId, async () => this.withVersionExclusiveLock(targetVersionId, async () => {
const version = await this.activateVersion(sourceId, targetVersionId, expectedActiveVersionId);
const source = await this.getSource(sourceId);
if (source?.activeVersionId !== targetVersionId) {
throw new CatalogError("Rollback verification failed", 503, "ROLLBACK_VERIFICATION_FAILED");
}
return version;
}));
}
async markPurging(sourceId: string, versionId: string): Promise<CatalogVersionRow> {
return withTransaction(this.pool, async (client) => {
const version = await client.query<VersionDbRow>(
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2 FOR UPDATE",
[sourceId, versionId]
);
if (!version.rowCount) {
throw new CatalogError("Version not found", 404, "VERSION_NOT_FOUND");
}
if (version.rows[0].state === "active") {
throw new CatalogError("Active versions cannot be purged", 409, "ACTIVE_VERSION_PURGE_FORBIDDEN");
}
if (!["ready", "superseded", "failed", "rejected", "purging"].includes(version.rows[0].state)) {
throw new CatalogError("Version state cannot be purged", 409, "VERSION_NOT_PURGEABLE");
}
const updated = await client.query<VersionDbRow>(
"UPDATE rag_source_versions SET state = 'purging' WHERE source_id = $1 AND version_id = $2 RETURNING *",
[sourceId, versionId]
);
return toVersion(updated.rows[0]);
});
}
async markPurged(versionId: string): Promise<void> {
await withTransaction(this.pool, async (client) => {
await client.query("DELETE FROM rag_version_documents WHERE version_id = $1", [versionId]);
const result = await client.query(
`UPDATE rag_source_versions
SET state = 'purged', verified_point_count = 0, artifact_state = 'retention_deleted', purged_at = now()
WHERE version_id = $1 AND state = 'purging'`,
[versionId]
);
if (result.rowCount !== 1) {
throw new CatalogError("Version could not transition from purging to purged", 409, "INVALID_VERSION_STATE");
}
});
}
async validateActiveInvariant(): Promise<string[]> {
const result = await this.pool.query<{ source_id: string }>(
`SELECT s.source_id
FROM rag_sources s
LEFT JOIN rag_source_versions v
ON v.version_id = s.active_version_id
AND v.source_id = s.source_id
AND v.state = 'active'
WHERE (s.active_version_id IS NOT NULL AND v.version_id IS NULL)
UNION
SELECT v.source_id
FROM rag_source_versions v
JOIN rag_sources s ON s.source_id = v.source_id
WHERE v.state = 'active'
AND s.active_version_id IS DISTINCT FROM v.version_id`
);
return result.rows.map((row) => row.source_id);
}
async listOrphanedIndexingCandidates(maxAgeMs = 1800000): Promise<OrphanedVersionCandidate[]> {
const result = await this.pool.query<{
version_id: string;
source_id: string;
expected_document_count: string | number;
expected_point_count: string | number;
embedding_dimensions: string | number;
qdrant_collection: string;
}>(
`SELECT version_id, source_id, expected_document_count, expected_point_count, embedding_dimensions, qdrant_collection
FROM rag_source_versions
WHERE (
state = 'pending'
AND created_at < now() - ($1::text || ' milliseconds')::interval
) OR (
state = 'indexing'
AND indexing_started_at IS NOT NULL
AND indexing_started_at < now() - ($1::text || ' milliseconds')::interval
)`,
[String(maxAgeMs)]
);
return result.rows.map((row) => ({
sourceId: row.source_id,
versionId: row.version_id,
expectedDocumentCount: Number(row.expected_document_count),
expectedPointCount: Number(row.expected_point_count),
embeddingDimensions: Number(row.embedding_dimensions),
qdrantCollection: row.qdrant_collection
}));
}
async hasCompleteVersionDocuments(versionId: string, expectedDocumentCount: number, expectedPointCount: number): Promise<boolean> {
const result = await this.pool.query<{
document_count: string | number;
point_count: string | number;
missing_content_count: string | number;
}>(
`SELECT COUNT(*) AS document_count,
COALESCE(SUM(chunk_count), 0) AS point_count,
COUNT(*) FILTER (WHERE content_hash IS NULL) AS missing_content_count
FROM rag_version_documents
WHERE version_id = $1`,
[versionId]
);
const row = result.rows[0];
return Number(row?.document_count ?? 0) === expectedDocumentCount
&& Number(row?.point_count ?? 0) === expectedPointCount
&& Number(row?.missing_content_count ?? 0) === 0;
}
async recoverVersionReady(versionId: string, verifiedPointCount: number): Promise<boolean> {
return withTransaction(this.pool, async (client) => {
await client.query("UPDATE rag_version_documents SET index_state = 'ready' WHERE version_id = $1", [versionId]);
const result = await client.query(
`UPDATE rag_source_versions
SET state = 'ready', verified_point_count = $2, ready_at = now(), error_code = NULL, error_detail = NULL
WHERE version_id = $1 AND state IN ('pending', 'indexing') AND expected_point_count = $2`,
[versionId, verifiedPointCount]
);
return result.rowCount === 1;
});
}
async recoverOrphanedIndexingVersions(maxAgeMs = 1800000): Promise<Array<{ versionId: string; sourceId: string; action: "ready" | "failed" }>> {
const candidates = await this.listOrphanedIndexingCandidates(maxAgeMs);
const recovered: Array<{ versionId: string; sourceId: string; action: "ready" | "failed" }> = [];
for (const candidate of candidates) {
await this.markFailed(candidate.versionId, "ORPHANED_INDEXING", "Recovered by reconciler after stale pending/indexing state.");
recovered.push({ versionId: candidate.versionId, sourceId: candidate.sourceId, action: "failed" });
}
return recovered;
}
}
export { CatalogError };

338
src/modules/ingest/service.ts
View file

@ -1,25 +1,72 @@
import path from "node:path"; import path from "node:path";
import { readFile } from "node:fs/promises";
import { parseDocument, isSupportedDocument } from "../parsers/parser-registry.js"; import { parseDocument, isSupportedDocument } from "../parsers/parser-registry.js";
import { chunkDocument, codeChunkingPolicy, documentalChunkingPolicy } from "../process/chunking.js"; import { chunkDocument, codeChunkingPolicy, documentalChunkingPolicy } from "../process/chunking.js";
import type { EmbeddingProvider } from "../embeddings/provider.js"; import type { EmbeddingProvider } from "../embeddings/provider.js";
import type { VectorStoreClient } from "../vectorstore/client.js"; import type { VectorStoreClient } from "../vectorstore/client.js";
import type { IngestResult, IngestSourceInput, IngestedChunk, RetrieveScope } from "../../shared/types/rag.js"; import type { IngestResult, IngestSourceInput, IngestedChunk, RetrieveScope } from "../../shared/types/rag.js";
import { buildChunkId, buildDocumentId, buildQdrantPointId, buildSourceId, normalizeDocumentKey } from "../../shared/utils/ids.js"; import {
buildChunkId,
buildDocumentId,
buildMetadataHash,
buildProcessingFingerprint,
buildQdrantPointId,
buildSourceId,
buildVersionedQdrantPointId,
hashOrderedPairs,
normalizeContentForHash,
normalizeDocumentKey,
sha256Hex
} from "../../shared/utils/ids.js";
import { listFilesRecursively } from "../../shared/utils/files.js"; import { listFilesRecursively } from "../../shared/utils/files.js";
import { env } from "../../config/env.js"; import { env } from "../../config/env.js";
import { CatalogError, type CatalogDocumentInput, type CatalogRepository } from "../catalog/repository.js";
type PreparedDocument = CatalogDocumentInput & {
chunks: ReturnType<typeof chunkDocument>;
chunkMode: "documental" | "codigo";
content: string;
};
type OriginalDocument = {
filePath: string;
documentKey: string;
documentId: string;
bytes: Buffer;
originalHash: string;
};
export class IngestService { export class IngestService {
constructor( constructor(
private readonly embeddingProvider: EmbeddingProvider, private readonly embeddingProvider: EmbeddingProvider,
private readonly vectorStore: VectorStoreClient private readonly vectorStore: VectorStoreClient,
private readonly catalog?: CatalogRepository
) {} ) {}
async cleanup(scope: RetrieveScope): Promise<{ deleted: number }> { async cleanup(scope: RetrieveScope): Promise<{ deleted: number }> {
if (!env.ingestWritesEnabled) {
throw new CatalogError("Ingest writes are disabled during knowledge catalog maintenance", 503, "INGEST_WRITES_DISABLED");
}
const count = await this.vectorStore.deleteScope(scope); const count = await this.vectorStore.deleteScope(scope);
return { deleted: count }; return { deleted: count };
} }
async ingest(source: IngestSourceInput): Promise<IngestResult> { async ingest(source: IngestSourceInput): Promise<IngestResult> {
if (!env.ingestWritesEnabled) {
throw new CatalogError("Ingest writes are disabled during knowledge catalog maintenance", 503, "INGEST_WRITES_DISABLED");
}
if (env.knowledgeLifecycleEnforced) {
if (!this.catalog) {
throw new CatalogError("Knowledge lifecycle is enforced but PostgreSQL is not configured", 503, "CATALOG_UNAVAILABLE");
}
return this.ingestWithLifecycle(source, this.catalog);
}
return this.ingestLegacy(source);
}
private async ingestLegacy(source: IngestSourceInput): Promise<IngestResult> {
const sourceId = source.sourceId ?? buildSourceId(source.sourceType, source.sourceRef); const sourceId = source.sourceId ?? buildSourceId(source.sourceType, source.sourceRef);
const discoveredFiles = await this.resolveInputFiles(source); const discoveredFiles = await this.resolveInputFiles(source);
const supportedFiles = discoveredFiles.filter(isSupportedDocument); const supportedFiles = discoveredFiles.filter(isSupportedDocument);
@ -79,6 +126,7 @@ export class IngestService {
return { return {
accepted: true, accepted: true,
sourceId,
source, source,
filesDiscovered: supportedFiles.length, filesDiscovered: supportedFiles.length,
documentsProcessed, documentsProcessed,
@ -87,6 +135,292 @@ export class IngestService {
}; };
} }
private async ingestWithLifecycle(source: IngestSourceInput, catalog: CatalogRepository): Promise<IngestResult> {
const activate = source.activate ?? true;
if (activate && source.expectedActiveVersionId === undefined) {
throw new CatalogError("expectedActiveVersionId is required when activate is true", 400, "EXPECTED_ACTIVE_VERSION_REQUIRED");
}
const sourceId = source.sourceId ?? buildSourceId(source.sourceType, source.sourceRef);
const attemptId = await catalog.beginAttempt({ ...source, sourceId });
let versionId: string | undefined;
let canMarkVersionFailed = false;
try {
await catalog.updateAttempt(attemptId, { state: "processing" });
const discoveredFiles = await this.resolveInputFiles(source);
const supportedFiles = discoveredFiles.filter(isSupportedDocument);
const originalDocuments = await this.readOriginalDocuments(source, sourceId, supportedFiles);
if (originalDocuments.length === 0) {
throw new CatalogError("The source has no supported documents", 422, "EMPTY_SOURCE");
}
const originalManifestHash = hashOrderedPairs(originalDocuments.map((document) => [document.documentKey, document.originalHash]));
if (!originalManifestHash) {
throw new CatalogError("Original manifest hash could not be calculated", 422, "ORIGINAL_MANIFEST_HASH_UNAVAILABLE");
}
const preparedDocuments = await this.prepareDocuments(originalDocuments);
if (preparedDocuments.length === 0) {
throw new CatalogError("The source has no supported documents with useful content", 422, "EMPTY_SOURCE");
}
const tags = source.tags ?? [];
const sourceContentHash = hashOrderedPairs(preparedDocuments.map((document) => [document.documentKey, document.contentHash]));
const embeddingDimensions = this.embeddingProvider.dimensions ?? 4096;
const processingFingerprint = buildProcessingFingerprint({
parserVersion: "native-v1",
normalizationPolicy: "bom-crlf-trim-final-lf-v1",
chunking: {
code: codeChunkingPolicy,
documental: documentalChunkingPolicy
},
embeddingProvider: this.embeddingProvider.providerName,
embeddingModel: this.embeddingProvider.modelName,
embeddingDimensions
});
const metadataHash = buildMetadataHash(tags, { sourceType: source.sourceType, sourceRef: source.sourceRef });
return await catalog.withSourceLock(sourceId, async () => {
const reusable = await catalog.findReusableVersion({ sourceId, sourceContentHash, processingFingerprint, metadataHash });
if (reusable?.state === "active") {
if (activate) {
await catalog.assertActiveVersionPrecondition(sourceId, source.expectedActiveVersionId);
}
await catalog.updateAttempt(attemptId, { state: "completed", versionId: reusable.versionId, inputHash: originalManifestHash });
return {
accepted: true,
sourceId,
versionId: reusable.versionId,
versionNumber: reusable.versionNumber,
state: reusable.state,
previousVersionId: reusable.previousVersionId,
filesDiscovered: supportedFiles.length,
documentsProcessed: reusable.expectedDocumentCount,
chunksStored: 0,
activated: true,
noOp: true,
collectionName: env.qdrantCollection
};
}
if (reusable && ["pending", "indexing", "review_required"].includes(reusable.state)) {
throw new CatalogError("A reusable version is already being indexed", 409, "REUSABLE_VERSION_IN_PROGRESS");
}
if (reusable && ["ready", "superseded"].includes(reusable.state)) {
if (!activate) {
await catalog.updateAttempt(attemptId, { state: "completed", versionId: reusable.versionId, inputHash: originalManifestHash });
return {
accepted: true,
sourceId,
versionId: reusable.versionId,
versionNumber: reusable.versionNumber,
state: reusable.state,
previousVersionId: reusable.previousVersionId,
filesDiscovered: supportedFiles.length,
documentsProcessed: reusable.expectedDocumentCount,
chunksStored: 0,
activated: false,
noOp: false,
collectionName: env.qdrantCollection
};
}
const activated = await catalog.activateVersion(sourceId, reusable.versionId, source.expectedActiveVersionId);
await catalog.updateAttempt(attemptId, { state: "completed", versionId: activated.versionId, inputHash: originalManifestHash });
return {
accepted: true,
sourceId,
versionId: activated.versionId,
versionNumber: activated.versionNumber,
state: activated.state,
previousVersionId: activated.previousVersionId,
filesDiscovered: supportedFiles.length,
documentsProcessed: activated.expectedDocumentCount,
chunksStored: 0,
activated: true,
noOp: false,
collectionName: env.qdrantCollection
};
}
const expectedPointCount = preparedDocuments.reduce((total, document) => total + document.chunkCount, 0);
const created = await catalog.createPendingVersion({
sourceId,
sourceType: source.sourceType,
sourceRef: source.sourceRef,
originalManifestHash,
sourceContentHash,
processingFingerprint,
metadataHash,
tags,
activateRequested: activate,
embeddingProvider: this.embeddingProvider.providerName,
embeddingModel: this.embeddingProvider.modelName,
embeddingDimensions,
qdrantCollection: env.qdrantCollection,
expectedDocumentCount: preparedDocuments.length,
expectedPointCount,
documents: preparedDocuments
});
versionId = created.versionId;
canMarkVersionFailed = true;
await catalog.updateAttempt(attemptId, { state: "processing", versionId, inputHash: originalManifestHash });
await catalog.markIndexing(versionId);
const chunksStored = await this.writeVersionedChunks(source, sourceId, created.versionId, created.versionNumber, preparedDocuments, processingFingerprint, tags, embeddingDimensions);
const verifiedPointCount = await this.vectorStore.countVersionPoints(created.versionId);
if (verifiedPointCount !== expectedPointCount || chunksStored !== expectedPointCount) {
throw new CatalogError("Indexed point count does not match the expected catalog count", 503, "SOURCE_VERSION_INCONSISTENT");
}
const ready = await catalog.markReady(created.versionId, verifiedPointCount);
canMarkVersionFailed = false;
const finalVersion = activate
? await catalog.activateVersion(sourceId, ready.versionId, source.expectedActiveVersionId)
: ready;
await catalog.updateAttempt(attemptId, { state: "completed", versionId: finalVersion.versionId, inputHash: originalManifestHash });
return {
accepted: true,
sourceId,
versionId: finalVersion.versionId,
versionNumber: finalVersion.versionNumber,
state: finalVersion.state,
previousVersionId: finalVersion.previousVersionId,
filesDiscovered: supportedFiles.length,
documentsProcessed: preparedDocuments.length,
chunksStored,
activated: finalVersion.state === "active",
noOp: false,
collectionName: env.qdrantCollection
};
});
} catch (error) {
const detail = error instanceof Error ? error.message : "Unknown lifecycle ingest error";
if (canMarkVersionFailed) {
await catalog.markFailed(versionId, error instanceof CatalogError ? error.code : "INGEST_FAILED", detail).catch(() => undefined);
}
await catalog.updateAttempt(attemptId, {
state: "failed",
versionId,
errorCode: error instanceof CatalogError ? error.code : "INGEST_FAILED",
errorDetail: detail
}).catch(() => undefined);
throw error;
}
}
private async readOriginalDocuments(source: IngestSourceInput, sourceId: string, files: string[]): Promise<OriginalDocument[]> {
const originals: OriginalDocument[] = [];
for (const filePath of files) {
const bytes = await readFile(filePath);
const documentKey = this.buildDocumentKey(source, filePath);
originals.push({
filePath,
documentKey,
documentId: buildDocumentId(sourceId, documentKey),
bytes,
originalHash: sha256Hex(bytes)
});
}
return originals.sort((left, right) => Buffer.compare(Buffer.from(left.documentKey, "utf8"), Buffer.from(right.documentKey, "utf8")));
}
private async prepareDocuments(originals: OriginalDocument[]): Promise<PreparedDocument[]> {
const documents: PreparedDocument[] = [];
for (const original of originals) {
const parsed = await parseDocument(original.filePath);
const normalizedContent = normalizeContentForHash(parsed.content);
if (!normalizedContent.trim()) {
continue;
}
const chunkingPolicy = parsed.chunkMode === "codigo" ? codeChunkingPolicy : documentalChunkingPolicy;
const chunks = chunkDocument(parsed.title, normalizedContent, chunkingPolicy);
if (chunks.length === 0) {
continue;
}
documents.push({
documentId: original.documentId,
documentKey: original.documentKey,
originalHash: original.originalHash,
originalHashKind: "bytes",
contentHash: sha256Hex(normalizedContent),
mimeType: parsed.mimeType,
title: parsed.title,
chunkCount: chunks.length,
chunks,
chunkMode: chunkingPolicy.mode,
content: normalizedContent
});
}
return documents.sort((left, right) => Buffer.compare(Buffer.from(left.documentKey, "utf8"), Buffer.from(right.documentKey, "utf8")));
}
private buildDocumentKey(source: IngestSourceInput, filePath: string): string {
const logicalRef = source.sourceType === "folder"
? path.relative(path.resolve(source.readPath ?? source.sourceRef), path.resolve(filePath))
: source.sourceRef;
return normalizeDocumentKey(logicalRef || path.basename(filePath));
}
private async writeVersionedChunks(
source: IngestSourceInput,
sourceId: string,
versionId: string,
versionNumber: number,
documents: PreparedDocument[],
processingFingerprint: string,
tags: string[],
expectedEmbeddingDimensions: number
): Promise<number> {
let chunksStored = 0;
for (const document of documents) {
const embeddings = await this.embeddingProvider.embed(document.chunks.map((chunk) => chunk.content));
if (embeddings.length !== document.chunks.length || embeddings.some((embedding) => embedding.length !== expectedEmbeddingDimensions)) {
throw new CatalogError("Embedding provider returned invalid dimensions", 503, "EMBEDDING_DIMENSIONS_INVALID");
}
const qdrantChunks: IngestedChunk[] = document.chunks.map((chunk, index) => {
const chunkId = buildChunkId(document.documentId, document.chunkMode, chunk.index);
return {
id: buildVersionedQdrantPointId(versionId, chunkId),
vector: embeddings[index],
payload: {
chunk_id: chunkId,
source_id: sourceId,
source_version_id: versionId,
source_version_number: versionNumber,
source_type: source.sourceType,
source_ref: source.sourceRef,
document_id: document.documentId,
document_key: document.documentKey,
document_content_hash: document.contentHash,
title: document.title,
mime_type: document.mimeType,
section_title: chunk.sectionTitle,
chunk_mode: document.chunkMode,
chunk_index: chunk.index,
start_line: chunk.startLine,
end_line: chunk.endLine,
content: chunk.content,
tags,
embedding_provider: this.embeddingProvider.providerName,
embedding_model: this.embeddingProvider.modelName,
embedding_dimensions: embeddings[index]?.length ?? 0,
processing_fingerprint: processingFingerprint,
indexed_at: new Date().toISOString(),
write_state: "staged"
}
};
});
await this.vectorStore.upsert(qdrantChunks);
chunksStored += qdrantChunks.length;
}
return chunksStored;
}
private async resolveInputFiles(source: IngestSourceInput): Promise<string[]> { private async resolveInputFiles(source: IngestSourceInput): Promise<string[]> {
const targetPath = source.readPath ?? source.sourceRef; const targetPath = source.readPath ?? source.sourceRef;
if (source.sourceType === "file") { if (source.sourceType === "file") {

70
src/modules/retrieve/service.ts
View file

@ -1,6 +1,8 @@
import type { EmbeddingProvider } from "../embeddings/provider.js"; import type { EmbeddingProvider } from "../embeddings/provider.js";
import type { VectorStoreClient } from "../vectorstore/client.js"; import type { VectorStoreClient } from "../vectorstore/client.js";
import type { ChunkMode, RetrieveIntent, RetrieveResponse, RetrieveScope } from "../../shared/types/rag.js"; import type { ChunkMode, RetrieveIntent, RetrieveResponse, RetrieveScope } from "../../shared/types/rag.js";
import { env } from "../../config/env.js";
import { CatalogError, type ActiveVersionScope, type CatalogRepository } from "../catalog/repository.js";
function unique(values: string[]): string[] { function unique(values: string[]): string[] {
return [...new Set(values.filter(Boolean))]; return [...new Set(values.filter(Boolean))];
@ -176,13 +178,18 @@ async function embedQuery(provider: EmbeddingProvider, query: string): Promise<n
export class RetrieveService { export class RetrieveService {
constructor( constructor(
private readonly embeddingProvider: EmbeddingProvider, private readonly embeddingProvider: EmbeddingProvider,
private readonly vectorStore: VectorStoreClient private readonly vectorStore: VectorStoreClient,
private readonly catalog?: CatalogRepository
) {} ) {}
async retrieve(mode: ChunkMode, intent: RetrieveIntent, query: string, scope?: RetrieveScope): Promise<RetrieveResponse> { async retrieve(mode: ChunkMode, intent: RetrieveIntent, query: string, scope?: RetrieveScope): Promise<RetrieveResponse> {
const items = intent === "bootstrap" const activeVersions = await this.resolveActiveVersions(scope);
? await this.retrieveBootstrap(mode, query, scope) const items = await this.withActiveVersionReadLocks(activeVersions, async () => {
: await this.retrieveSpecific(mode, query, scope); await this.validateActiveVersions(activeVersions);
return intent === "bootstrap"
? this.retrieveBootstrap(mode, query, scope, activeVersions)
: this.retrieveSpecific(mode, query, scope, activeVersions);
});
const topics = unique(items.map((item) => item.title)).slice(0, 8); const topics = unique(items.map((item) => item.title)).slice(0, 8);
const criticalPoints = items const criticalPoints = items
@ -207,13 +214,58 @@ export class RetrieveService {
}; };
} }
private async retrieveSpecific(mode: ChunkMode, query: string, scope?: RetrieveScope) { private async withActiveVersionReadLocks<T>(activeVersions: ActiveVersionScope[] | undefined, handler: () => Promise<T>): Promise<T> {
if (!env.knowledgeLifecycleEnforced || !this.catalog || !activeVersions) {
return handler();
}
return this.catalog.withVersionSharedLocks(activeVersions.map((version) => version.sourceVersionId), handler);
}
private async resolveActiveVersions(scope?: RetrieveScope): Promise<ActiveVersionScope[] | undefined> {
if (!env.knowledgeLifecycleEnforced) {
return undefined;
}
if (!this.catalog) {
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
}
try {
return await this.catalog.resolveActiveVersions(scope);
} catch (error) {
if (error instanceof CatalogError) {
throw error;
}
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
}
}
private async validateActiveVersions(activeVersions: ActiveVersionScope[] | undefined): Promise<void> {
if (!env.knowledgeLifecycleEnforced || !this.catalog || !activeVersions) {
return;
}
for (const version of activeVersions) {
if (version.qdrantCollection !== env.qdrantCollection) {
await this.catalog.markNeedsReingest(version.sourceId);
throw new CatalogError("Active source version targets a different Qdrant collection", 503, "SOURCE_VERSION_INCONSISTENT");
}
await this.vectorStore.validateCollectionDimensions(version.embeddingDimensions);
const count = await this.vectorStore.countVersionPoints(version.sourceVersionId);
if (count !== version.expectedPointCount) {
await this.catalog.markNeedsReingest(version.sourceId);
throw new CatalogError("Active source version is inconsistent", 503, "SOURCE_VERSION_INCONSISTENT");
}
}
}
private async retrieveSpecific(mode: ChunkMode, query: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
const queries = buildSpecificQueries(query); const queries = buildSpecificQueries(query);
const merged = new Map<string, RetrieveResponse["items"][number]>(); const merged = new Map<string, RetrieveResponse["items"][number]>();
for (const subquery of queries) { for (const subquery of queries) {
const queryVector = await embedQuery(this.embeddingProvider, subquery); const queryVector = await embedQuery(this.embeddingProvider, subquery);
const results = await this.vectorStore.search(queryVector, 6, mode, scope); const results = await this.vectorStore.search(queryVector, 6, mode, scope, activeVersions);
for (const item of results) { for (const item of results) {
const existing = merged.get(item.chunkId); const existing = merged.get(item.chunkId);
@ -226,13 +278,13 @@ export class RetrieveService {
return rankItems(query, [...merged.values()]).slice(0, 8); return rankItems(query, [...merged.values()]).slice(0, 8);
} }
private async retrieveBootstrap(mode: ChunkMode, query: string, scope?: RetrieveScope) { private async retrieveBootstrap(mode: ChunkMode, query: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
const queries = buildBootstrapQueries(query); const queries = buildBootstrapQueries(query);
const merged = new Map<string, RetrieveResponse["items"][number]>(); const merged = new Map<string, RetrieveResponse["items"][number]>();
for (const subquery of queries) { for (const subquery of queries) {
const queryVector = await embedQuery(this.embeddingProvider, subquery); const queryVector = await embedQuery(this.embeddingProvider, subquery);
const results = await this.vectorStore.search(queryVector, 6, mode, scope); const results = await this.vectorStore.search(queryVector, 6, mode, scope, activeVersions);
for (const item of results) { for (const item of results) {
const existing = merged.get(item.chunkId); const existing = merged.get(item.chunkId);
@ -243,7 +295,7 @@ export class RetrieveService {
} }
if (merged.size === 0 && scope && (scope.sourceId || scope.sourceRef || (scope.tags && scope.tags.length > 0))) { if (merged.size === 0 && scope && (scope.sourceId || scope.sourceRef || (scope.tags && scope.tags.length > 0))) {
const explored = await this.vectorStore.browseScope(12, mode, scope); const explored = await this.vectorStore.browseScope(12, mode, scope, activeVersions);
for (const item of explored) { for (const item of explored) {
merged.set(item.chunkId, item); merged.set(item.chunkId, item);
} }

135
src/modules/vectorstore/client.ts
View file

@ -1,6 +1,7 @@
import { QdrantClient } from "@qdrant/js-client-rest"; import { QdrantClient } from "@qdrant/js-client-rest";
import { env } from "../../config/env.js"; import { env } from "../../config/env.js";
import type { AvailableScope, IngestedChunk, RetrieveScope, RetrievedItem } from "../../shared/types/rag.js"; import type { AvailableScope, IngestedChunk, RetrieveScope, RetrievedItem } from "../../shared/types/rag.js";
import type { ActiveVersionScope } from "../catalog/repository.js";
export function buildQdrantClient(): QdrantClient { export function buildQdrantClient(): QdrantClient {
const url = new URL(env.qdrantUrl); const url = new URL(env.qdrantUrl);
@ -17,14 +18,32 @@ export interface VectorStoreClient {
readonly kind: string; readonly kind: string;
healthcheck(): Promise<{ ok: boolean; kind: string }>; healthcheck(): Promise<{ ok: boolean; kind: string }>;
ensureCollection(vectorSize: number): Promise<void>; ensureCollection(vectorSize: number): Promise<void>;
ensurePayloadIndexes(): Promise<void>;
upsert(chunks: IngestedChunk[]): Promise<void>; upsert(chunks: IngestedChunk[]): Promise<void>;
search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]>; search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]>;
listScopes(): Promise<AvailableScope[]>; listScopes(): Promise<AvailableScope[]>;
browseScope(limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]>; browseScope(limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]>;
deleteScope(scope: RetrieveScope): Promise<number>; deleteScope(scope: RetrieveScope): Promise<number>;
validateCollectionDimensions(expectedDimensions: number): Promise<void>;
countVersionPoints(sourceVersionId: string): Promise<number>;
deleteVersionPoints(sourceVersionId: string): Promise<number>;
} }
function buildSearchFilter(mode?: string, scope?: RetrieveScope) { function extractVectorSize(collectionInfo: unknown): number | undefined {
const info = collectionInfo as { config?: { params?: { vectors?: unknown } } };
const vectors = info.config?.params?.vectors;
if (!vectors || typeof vectors !== "object") {
return undefined;
}
if ("size" in vectors && typeof vectors.size === "number") {
return vectors.size;
}
const namedVectors = Object.values(vectors as Record<string, unknown>);
const firstNamed = namedVectors[0] as { size?: unknown } | undefined;
return typeof firstNamed?.size === "number" ? firstNamed.size : undefined;
}
function buildSearchFilter(mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
const must: Array<Record<string, unknown>> = []; const must: Array<Record<string, unknown>> = [];
if (mode && mode !== "auto") { if (mode && mode !== "auto") {
@ -57,6 +76,13 @@ function buildSearchFilter(mode?: string, scope?: RetrieveScope) {
} }
} }
if (activeVersions) {
must.push({
key: "source_version_id",
match: { any: activeVersions.length > 0 ? activeVersions.map((version) => version.sourceVersionId) : ["__no_active_versions__"] }
});
}
return must.length > 0 ? { must } : undefined; return must.length > 0 ? { must } : undefined;
} }
@ -93,18 +119,47 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
distance: "Cosine" distance: "Cosine"
} }
}); });
} else {
await this.validateCollectionDimensions(vectorSize);
} }
await this.ensurePayloadIndexes();
this.collectionReady = true; this.collectionReady = true;
} }
async ensurePayloadIndexes(): Promise<void> {
const collections = await this.client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
if (!exists) {
return;
}
const collectionInfo = await this.client.getCollection(env.qdrantCollection) as { payload_schema?: Record<string, unknown> };
const payloadSchema = collectionInfo.payload_schema ?? {};
for (const fieldName of ["source_id", "source_version_id", "document_id", "chunk_mode", "tags"]) {
if (payloadSchema[fieldName]) {
continue;
}
await this.client.createPayloadIndex(env.qdrantCollection, {
field_name: fieldName,
field_schema: "keyword",
wait: true
});
}
}
async upsert(chunks: IngestedChunk[]): Promise<void> { async upsert(chunks: IngestedChunk[]): Promise<void> {
if (chunks.length === 0) { if (chunks.length === 0) {
return; return;
} }
await this.ensureCollection(chunks[0].vector.length); const vectorSize = chunks[0].vector.length;
await this.client.upsert(env.qdrantCollection, { if (vectorSize === 0 || chunks.some((chunk) => chunk.vector.length !== vectorSize)) {
throw new Error("Qdrant upsert rejected: vectors must be non-empty and have consistent dimensions");
}
await this.ensureCollection(vectorSize);
const result = await this.client.upsert(env.qdrantCollection, {
wait: true, wait: true,
points: chunks.map((chunk) => ({ points: chunks.map((chunk) => ({
id: chunk.id, id: chunk.id,
@ -112,21 +167,26 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
payload: chunk.payload payload: chunk.payload
})) }))
}); });
if (result.status !== "completed") {
throw new Error(`Qdrant upsert did not complete: ${result.status}`);
}
} }
async search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]> { async search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]> {
await this.ensureCollection(queryVector.length); await this.ensureCollection(queryVector.length);
const result = await this.client.search(env.qdrantCollection, { const result = await this.client.search(env.qdrantCollection, {
vector: queryVector, vector: queryVector,
limit, limit,
with_payload: true, with_payload: true,
filter: buildSearchFilter(mode, scope) filter: buildSearchFilter(mode, scope, activeVersions)
}); });
return result.map((point) => ({ return result.map((point) => ({
chunkId: String(point.payload?.chunk_id ?? point.id), chunkId: String(point.payload?.chunk_id ?? point.id),
documentId: String(point.payload?.document_id ?? ""), documentId: String(point.payload?.document_id ?? ""),
sourceId: String(point.payload?.source_id ?? ""), sourceId: String(point.payload?.source_id ?? ""),
sourceVersionId: point.payload?.source_version_id ? String(point.payload.source_version_id) : undefined,
sourceVersionNumber: point.payload?.source_version_number ? Number(point.payload.source_version_number) : undefined,
title: String(point.payload?.title ?? ""), title: String(point.payload?.title ?? ""),
sectionTitle: point.payload?.section_title ? String(point.payload.section_title) : undefined, sectionTitle: point.payload?.section_title ? String(point.payload.section_title) : undefined,
content: String(point.payload?.content ?? ""), content: String(point.payload?.content ?? ""),
@ -192,7 +252,7 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
return [...scopeMap.values()].sort((left, right) => left.sourceRef.localeCompare(right.sourceRef)); return [...scopeMap.values()].sort((left, right) => left.sourceRef.localeCompare(right.sourceRef));
} }
async browseScope(limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]> { async browseScope(limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]> {
const collections = await this.client.getCollections(); const collections = await this.client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection); const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
@ -203,13 +263,15 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
const response = await this.client.scroll(env.qdrantCollection, { const response = await this.client.scroll(env.qdrantCollection, {
limit, limit,
with_payload: true, with_payload: true,
filter: buildSearchFilter(mode, scope) filter: buildSearchFilter(mode, scope, activeVersions)
}); });
return response.points.map((point) => ({ return response.points.map((point) => ({
chunkId: String(point.payload?.chunk_id ?? point.id), chunkId: String(point.payload?.chunk_id ?? point.id),
documentId: String(point.payload?.document_id ?? ""), documentId: String(point.payload?.document_id ?? ""),
sourceId: String(point.payload?.source_id ?? ""), sourceId: String(point.payload?.source_id ?? ""),
sourceVersionId: point.payload?.source_version_id ? String(point.payload.source_version_id) : undefined,
sourceVersionNumber: point.payload?.source_version_number ? Number(point.payload.source_version_number) : undefined,
title: String(point.payload?.title ?? ""), title: String(point.payload?.title ?? ""),
sectionTitle: point.payload?.section_title ? String(point.payload.section_title) : undefined, sectionTitle: point.payload?.section_title ? String(point.payload.section_title) : undefined,
content: String(point.payload?.content ?? ""), content: String(point.payload?.content ?? ""),
@ -233,15 +295,64 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
return 0; return 0;
} }
// Contamos antes de borrar para devolver la cantidad
const countResult = await this.client.count(env.qdrantCollection, { const countResult = await this.client.count(env.qdrantCollection, {
filter filter
}); });
await this.client.delete(env.qdrantCollection, { const deleteResult = await this.client.delete(env.qdrantCollection, {
filter filter,
wait: true
}); });
if (deleteResult.status !== "completed") {
throw new Error(`Qdrant delete did not complete: ${deleteResult.status}`);
}
return countResult.count; return countResult.count;
} }
async validateCollectionDimensions(expectedDimensions: number): Promise<void> {
const collections = await this.client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
if (!exists) {
return;
}
const collectionInfo = await this.client.getCollection(env.qdrantCollection);
const actualDimensions = extractVectorSize(collectionInfo);
if (actualDimensions !== undefined && actualDimensions !== expectedDimensions) {
throw new Error(`Qdrant collection dimension mismatch: expected ${expectedDimensions}, got ${actualDimensions}`);
}
}
async countVersionPoints(sourceVersionId: string): Promise<number> {
const collections = await this.client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
if (!exists) {
return 0;
}
const result = await this.client.count(env.qdrantCollection, {
filter: {
must: [{ key: "source_version_id", match: { value: sourceVersionId } }]
},
exact: true
});
return result.count;
}
async deleteVersionPoints(sourceVersionId: string): Promise<number> {
const count = await this.countVersionPoints(sourceVersionId);
if (count === 0) {
return 0;
}
const deleteResult = await this.client.delete(env.qdrantCollection, {
filter: {
must: [{ key: "source_version_id", match: { value: sourceVersionId } }]
},
wait: true
});
if (deleteResult.status !== "completed") {
throw new Error(`Qdrant version delete did not complete: ${deleteResult.status}`);
}
return count;
}
} }

448
src/scripts/migrate-legacy-lifecycle.ts Normal file
View file

@ -0,0 +1,448 @@
import { buildCatalogPool, withCatalogAdvisoryLock, withTransaction, type PgPool } from "../modules/catalog/client.js";
import { pathToFileURL } from "node:url";
import { buildLegacyMigrationPlan, type LegacyPointSnapshot, type LegacySourcePlan } from "../modules/catalog/legacy-migration.js";
import { CatalogError } from "../modules/catalog/errors.js";
import { env } from "../config/env.js";
import { buildQdrantClient } from "../modules/vectorstore/client.js";
import { sha256Hex } from "../shared/utils/ids.js";
type QdrantClientInstance = ReturnType<typeof buildQdrantClient>;
function argValue(name: string): string | undefined {
const prefix = `${name}=`;
const entry = process.argv.find((arg) => arg.startsWith(prefix));
return entry?.slice(prefix.length);
}
function requireArg(name: string): string {
const value = argValue(name);
if (!value) {
throw new Error(`Missing required argument ${name}=...`);
}
return value;
}
async function readLegacyPoints(client: QdrantClientInstance, includeVersionIds = new Set<string>()): Promise<LegacyPointSnapshot[]> {
const collections = await client.getCollections();
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
if (!exists) {
return [];
}
const points: LegacyPointSnapshot[] = [];
let nextOffset = undefined;
do {
const response = await client.scroll(env.qdrantCollection, {
limit: 256,
offset: nextOffset,
with_payload: true,
with_vector: false
});
for (const point of response.points) {
const sourceVersionId = point.payload?.source_version_id ? String(point.payload.source_version_id) : "";
if (sourceVersionId && !includeVersionIds.has(sourceVersionId)) {
continue;
}
points.push({ id: point.id, payload: point.payload ?? {} });
}
nextOffset = response.next_page_offset;
} while (nextOffset);
return points;
}
async function assertMaintenance(pool: PgPool): Promise<void> {
if (env.ingestWritesEnabled) {
throw new CatalogError("INGEST_WRITES_ENABLED must be false during legacy migration", 503, "WRITES_NOT_DISABLED");
}
await withTransaction(pool, async (client) => {
const result = await client.query<{ maintenance: boolean }>("SELECT maintenance FROM rag_system_state WHERE singleton = true FOR UPDATE");
if (!result.rows[0]?.maintenance) {
throw new CatalogError("rag_system_state.maintenance must be true during legacy migration", 503, "MAINTENANCE_REQUIRED");
}
});
}
async function readBatchVersionIds(pool: PgPool, batchId: string): Promise<Set<string>> {
const result = await pool.query<{ version_id: string }>(
"SELECT version_id FROM rag_legacy_migration_items WHERE batch_id = $1 AND version_id IS NOT NULL",
[batchId]
);
return new Set(result.rows.map((row) => row.version_id));
}
async function createOrReuseBatch(pool: PgPool, snapshotRef: string, plan: { sources: LegacySourcePlan[] }, batchId?: string): Promise<string> {
if (batchId) {
const existing = await pool.query<{ state: string; snapshot_ref: string }>(
"SELECT state, snapshot_ref FROM rag_legacy_migration_batches WHERE batch_id = $1",
[batchId]
);
if (!existing.rowCount) {
throw new CatalogError(`Legacy migration batch not found: ${batchId}`, 404, "LEGACY_BATCH_NOT_FOUND");
}
if (existing.rows[0].snapshot_ref !== snapshotRef) {
throw new CatalogError("Legacy migration batch snapshot_ref does not match the requested snapshot", 409, "LEGACY_BATCH_SNAPSHOT_MISMATCH");
}
if (existing.rows[0].state === "rolled_back") {
throw new CatalogError("Rolled back legacy migration batches cannot be resumed", 409, "LEGACY_BATCH_ROLLED_BACK");
}
return batchId;
}
const batch = await pool.query<{ batch_id: string }>(
"INSERT INTO rag_legacy_migration_batches(state, snapshot_ref, report) VALUES ('planned', $1, $2) RETURNING batch_id",
[snapshotRef, { sourceCount: plan.sources.length, pointCount: plan.sources.reduce((total, source) => total + source.pointIds.length, 0) }]
);
return batch.rows[0].batch_id;
}
export async function createCatalogVersion(pool: PgPool, batchId: string, source: LegacySourcePlan, collection: string): Promise<{ versionId: string; versionNumber: number }> {
return withTransaction(pool, async (client) => {
const item = await client.query<{ version_id: string | null; expected_point_count: number }>(
"SELECT version_id, expected_point_count FROM rag_legacy_migration_items WHERE batch_id = $1 AND source_id = $2 FOR UPDATE",
[batchId, source.sourceId]
);
if (item.rows[0]?.version_id) {
const reusable = await client.query<{ version_id: string; version_number: string; expected_point_count: string; source_type: string; source_ref: string; qdrant_collection: string }>(
`SELECT v.version_id, v.version_number, v.expected_point_count, s.source_type, s.source_ref, v.qdrant_collection
FROM rag_source_versions v
JOIN rag_sources s ON s.source_id = v.source_id
WHERE v.version_id = $1 AND v.source_id = $2
FOR UPDATE OF v, s`,
[item.rows[0].version_id, source.sourceId]
);
if (!reusable.rowCount) {
throw new CatalogError(`Legacy migration item references a missing version for ${source.sourceId}`, 409, "LEGACY_VERSION_MISSING");
}
const row = reusable.rows[0];
if (row.source_type !== source.sourceType || row.source_ref !== source.sourceRef || row.qdrant_collection !== collection || Number(row.expected_point_count) !== source.pointIds.length) {
throw new CatalogError(`Legacy migration resume is not compatible for ${source.sourceId}`, 409, "LEGACY_RESUME_INCOMPATIBLE");
}
return { versionId: row.version_id, versionNumber: Number(row.version_number) };
}
const existing = await client.query("SELECT 1 FROM rag_sources WHERE source_id = $1", [source.sourceId]);
if (existing.rowCount) {
throw new CatalogError(`Source already exists in catalog: ${source.sourceId}`, 409, "LEGACY_SOURCE_ALREADY_CATALOGED");
}
await client.query(
"INSERT INTO rag_sources(source_id, source_type, source_ref) VALUES ($1, $2, $3)",
[source.sourceId, source.sourceType, source.sourceRef]
);
const version = await client.query<{ version_id: string; version_number: string }>(
`INSERT INTO rag_source_versions(
source_id, version_number, state, original_manifest_hash, source_content_hash,
processing_fingerprint, metadata_hash, tags, activate_requested, embedding_provider,
embedding_model, embedding_dimensions, qdrant_collection, expected_document_count,
expected_point_count, verified_point_count
) VALUES ($1, 1, 'ready', $2, $3, $4, $5, $6, true, $7, $8, $9, $10, $11, $12, 0)
RETURNING version_id, version_number`,
[
source.sourceId,
source.originalManifestHash,
source.sourceContentHash,
sha256Hex(`legacy:${batchId}`),
source.metadataHash,
source.tags,
source.embeddingProvider,
source.embeddingModel,
source.embeddingDimensions,
collection,
source.documents.length,
source.pointIds.length
]
);
const versionId = version.rows[0].version_id;
for (const document of source.documents) {
await client.query(
`INSERT INTO rag_version_documents(
version_id, document_id, document_key, original_hash, original_hash_kind,
content_hash, mime_type, title, index_state, chunk_count
) VALUES ($1, $2, $3, $4, 'legacy-derived', $5, $6, $7, 'ready', $8)`,
[versionId, document.documentId, document.documentKey, document.originalHash, document.contentHash, document.mimeType, document.title, document.chunkCount]
);
}
if (item.rowCount) {
await client.query(
`UPDATE rag_legacy_migration_items
SET version_id = $3, expected_point_count = $4, state = 'migrated', error_detail = NULL
WHERE batch_id = $1 AND source_id = $2`,
[batchId, source.sourceId, versionId, source.pointIds.length]
);
} else {
await client.query(
`INSERT INTO rag_legacy_migration_items(batch_id, source_id, version_id, expected_point_count, state)
VALUES ($1, $2, $3, $4, 'migrated')`,
[batchId, source.sourceId, versionId, source.pointIds.length]
);
}
return { versionId, versionNumber: Number(version.rows[0].version_number) };
});
}
async function setLifecyclePayload(client: QdrantClientInstance, source: LegacySourcePlan, versionId: string, versionNumber: number, batchId: string): Promise<void> {
const result = await client.setPayload(env.qdrantCollection, {
points: source.pointIds,
payload: {
source_version_id: versionId,
source_version_number: versionNumber,
processing_fingerprint: sha256Hex(`legacy:${batchId}`),
write_state: "legacy-migrated"
},
wait: true
});
if (result.status !== "completed") {
throw new Error(`Qdrant payload update did not complete for ${source.sourceId}: ${result.status}`);
}
}
async function countVersionPoints(client: QdrantClientInstance, versionId: string): Promise<number> {
const result = await client.count(env.qdrantCollection, {
filter: { must: [{ key: "source_version_id", match: { value: versionId } }] },
exact: true
});
return result.count;
}
async function countLegacySourcePoints(client: QdrantClientInstance, sourceId: string): Promise<number> {
let count = 0;
let nextOffset = undefined;
do {
const response = await client.scroll(env.qdrantCollection, {
limit: 256,
offset: nextOffset,
with_payload: ["source_id", "source_version_id"],
with_vector: false,
filter: { must: [{ key: "source_id", match: { value: sourceId } }] }
});
for (const point of response.points) {
if (!point.payload?.source_version_id) {
count += 1;
}
}
nextOffset = response.next_page_offset;
} while (nextOffset);
return count;
}
export async function activateMigratedVersion(pool: PgPool, sourceId: string, versionId: string, verifiedPointCount: number): Promise<void> {
await withTransaction(pool, async (client) => {
const existing = await client.query<{ state: string; active_version_id: string | null }>(
`SELECT v.state, s.active_version_id
FROM rag_source_versions v
JOIN rag_sources s ON s.source_id = v.source_id
WHERE v.source_id = $1 AND v.version_id = $2
FOR UPDATE OF v, s`,
[sourceId, versionId]
);
if (existing.rows[0]?.state === "active" && existing.rows[0].active_version_id === versionId) {
await client.query("UPDATE rag_source_versions SET verified_point_count = $2 WHERE source_id = $1 AND version_id = $3", [sourceId, verifiedPointCount, versionId]);
return;
}
const version = await client.query(
`UPDATE rag_source_versions
SET state = 'active', verified_point_count = $2, ready_at = now(), activated_at = now()
WHERE source_id = $1 AND version_id = $3 AND state = 'ready'
RETURNING version_id`,
[sourceId, verifiedPointCount, versionId]
);
if (version.rowCount !== 1) {
throw new CatalogError(`Could not activate migrated version for ${sourceId}`, 409, "LEGACY_ACTIVATION_FAILED");
}
await client.query("UPDATE rag_sources SET active_version_id = $2 WHERE source_id = $1", [sourceId, versionId]);
});
}
export async function rollbackMigratedBatch(
pool: PgPool,
qdrant: QdrantClientInstance,
batchId: string,
snapshotRef: string,
snapshotChecksum?: string
): Promise<{ ok: true; mode: "rollback"; batchId: string; sources: string[] }> {
const itemsWithVersions = await withTransaction(pool, async (client) => {
const batch = await client.query<{ state: string; snapshot_ref: string }>(
"SELECT state, snapshot_ref FROM rag_legacy_migration_batches WHERE batch_id = $1 FOR UPDATE",
[batchId]
);
if (!batch.rowCount) {
throw new CatalogError(`Legacy migration batch not found: ${batchId}`, 404, "LEGACY_BATCH_NOT_FOUND");
}
if (batch.rows[0].snapshot_ref !== snapshotRef) {
throw new CatalogError("Legacy migration batch snapshot_ref does not match the requested snapshot", 409, "LEGACY_BATCH_SNAPSHOT_MISMATCH");
}
const items = await client.query<{ source_id: string; version_id: string | null; expected_point_count: string | number }>(
"SELECT source_id, version_id, expected_point_count FROM rag_legacy_migration_items WHERE batch_id = $1 FOR UPDATE",
[batchId]
);
const rollbackItems = items.rows.filter((item): item is { source_id: string; version_id: string; expected_point_count: string | number } => Boolean(item.version_id));
if (rollbackItems.length === 0 && batch.rows[0].state === "rolled_back") {
return [];
}
await client.query(
`SELECT s.source_id
FROM rag_sources s
JOIN rag_legacy_migration_items i ON i.source_id = s.source_id
WHERE i.batch_id = $1 AND i.version_id IS NOT NULL
FOR UPDATE OF s`,
[batchId]
);
await client.query(
`SELECT v.version_id
FROM rag_source_versions v
JOIN rag_legacy_migration_items i ON i.version_id = v.version_id
WHERE i.batch_id = $1 AND i.version_id IS NOT NULL
FOR UPDATE OF v`,
[batchId]
);
await client.query(
`UPDATE rag_sources s
SET active_version_id = NULL
FROM rag_legacy_migration_items i
WHERE i.batch_id = $1
AND i.source_id = s.source_id
AND i.version_id IS NOT NULL
AND s.active_version_id = i.version_id`,
[batchId]
);
await client.query(
`UPDATE rag_source_versions v
SET state = 'purging'
FROM rag_legacy_migration_items i
WHERE i.batch_id = $1
AND i.version_id = v.version_id`,
[batchId]
);
const activePointers = await client.query<{ count: string }>(
`SELECT COUNT(*) AS count
FROM rag_sources s
JOIN rag_legacy_migration_items i ON i.source_id = s.source_id
WHERE i.batch_id = $1
AND i.version_id IS NOT NULL
AND s.active_version_id = i.version_id`,
[batchId]
);
if (Number(activePointers.rows[0]?.count ?? 0) !== 0) {
throw new CatalogError("Legacy rollback preparation left active source pointers", 503, "LEGACY_ROLLBACK_ACTIVE_POINTERS_REMAIN");
}
return rollbackItems.map((item) => ({
sourceId: item.source_id,
versionId: item.version_id,
expectedPointCount: Number(item.expected_point_count)
}));
});
if (itemsWithVersions.length === 0) {
return { ok: true, mode: "rollback", batchId, sources: [] };
}
const recovered = await qdrant.recoverSnapshot(env.qdrantCollection, {
location: snapshotRef,
priority: "snapshot",
checksum: snapshotChecksum
});
if (!recovered) {
throw new Error("Qdrant snapshot recovery returned false");
}
for (const item of itemsWithVersions) {
const legacyCount = await countLegacySourcePoints(qdrant, item.sourceId);
if (legacyCount !== item.expectedPointCount) {
throw new CatalogError(`Restored legacy point count mismatch for ${item.sourceId}`, 503, "LEGACY_RESTORE_POINT_COUNT_MISMATCH");
}
}
await withTransaction(pool, async (client) => {
for (const item of itemsWithVersions) {
await client.query("DELETE FROM rag_version_documents WHERE version_id = $1", [item.versionId]);
await client.query("DELETE FROM rag_source_versions WHERE version_id = $1", [item.versionId]);
await client.query("DELETE FROM rag_sources WHERE source_id = $1", [item.sourceId]);
await client.query("UPDATE rag_legacy_migration_items SET version_id = NULL, state = 'rolled_back' WHERE batch_id = $1 AND source_id = $2", [batchId, item.sourceId]);
}
await client.query("UPDATE rag_legacy_migration_batches SET state = 'rolled_back', completed_at = now() WHERE batch_id = $1", [batchId]);
});
return { ok: true, mode: "rollback", batchId, sources: itemsWithVersions.map((item) => item.sourceId) };
}
async function dryRun() {
const qdrant = buildQdrantClient();
const points = await readLegacyPoints(qdrant);
const plan = buildLegacyMigrationPlan(points);
return { mode: "dry-run", collection: env.qdrantCollection, legacyPoints: points.length, ...plan };
}
async function applyMigration() {
const snapshotRef = requireArg("--snapshot-ref");
const requestedBatchId = argValue("--batch-id");
const pool = buildCatalogPool();
const qdrant = buildQdrantClient();
let batchId = requestedBatchId;
try {
return await withCatalogAdvisoryLock(pool, "rag:knowledge-lifecycle:legacy-migration", async () => {
await assertMaintenance(pool);
const includeVersionIds = requestedBatchId ? await readBatchVersionIds(pool, requestedBatchId) : new Set<string>();
const plan = buildLegacyMigrationPlan(await readLegacyPoints(qdrant, includeVersionIds));
if (!plan.ok) {
return { ok: false, mode: "apply", batchId: requestedBatchId, blockedReasons: plan.blockedReasons };
}
batchId = await createOrReuseBatch(pool, snapshotRef, plan, requestedBatchId);
const migrated: Array<{ sourceId: string; versionId: string; pointCount: number }> = [];
for (const source of plan.sources) {
const { versionId, versionNumber } = await createCatalogVersion(pool, batchId, source, env.qdrantCollection);
await setLifecyclePayload(qdrant, source, versionId, versionNumber, batchId);
const verifiedPointCount = await countVersionPoints(qdrant, versionId);
if (verifiedPointCount !== source.pointIds.length) {
throw new CatalogError(`Point count mismatch for ${source.sourceId}`, 503, "LEGACY_POINT_COUNT_MISMATCH");
}
await activateMigratedVersion(pool, source.sourceId, versionId, verifiedPointCount);
await pool.query("UPDATE rag_legacy_migration_items SET state = 'verified', error_detail = NULL WHERE batch_id = $1 AND source_id = $2", [batchId, source.sourceId]);
migrated.push({ sourceId: source.sourceId, versionId, pointCount: verifiedPointCount });
}
await pool.query("UPDATE rag_legacy_migration_batches SET state = 'verified', completed_at = now(), report = report || $2::jsonb WHERE batch_id = $1", [batchId, JSON.stringify({ migrated })]);
return { ok: true, mode: "apply", batchId, migrated };
});
} catch (error) {
if (batchId) {
await pool.query("UPDATE rag_legacy_migration_batches SET state = 'failed', completed_at = now(), report = report || $2::jsonb WHERE batch_id = $1 AND state <> 'verified'", [batchId, JSON.stringify({ error: error instanceof Error ? error.message : "Unknown legacy migration error" })]).catch(() => undefined);
}
throw error;
} finally {
await pool.end();
}
}
async function rollbackMigration() {
const batchId = requireArg("--batch-id");
const snapshotRef = requireArg("--snapshot-ref");
const snapshotChecksum = argValue("--snapshot-checksum");
const pool = buildCatalogPool();
const qdrant = buildQdrantClient();
try {
return await withCatalogAdvisoryLock(pool, "rag:knowledge-lifecycle:legacy-migration", async () => {
await assertMaintenance(pool);
return rollbackMigratedBatch(pool, qdrant, batchId, snapshotRef, snapshotChecksum);
});
} finally {
await pool.end();
}
}
async function main() {
const report = process.argv.includes("--apply")
? await applyMigration()
: process.argv.includes("--rollback")
? await rollbackMigration()
: await dryRun();
console.log(JSON.stringify(report, null, 2));
if (!report.ok) process.exitCode = 1;
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
main().catch((error) => {
console.error(error instanceof Error ? error.message : error);
process.exit(1);
});
}

32
src/shared/types/rag.ts
View file

@ -6,6 +6,18 @@ export type IngestMode = "mechanical" | "interactive";
export type SourceType = "file" | "folder"; export type SourceType = "file" | "folder";
export type SourceVersionState =
| "pending"
| "indexing"
| "review_required"
| "ready"
| "active"
| "superseded"
| "failed"
| "rejected"
| "purging"
| "purged";
export interface IngestSourceInput { export interface IngestSourceInput {
sourceId?: string; sourceId?: string;
sourceType: SourceType; sourceType: SourceType;
@ -13,6 +25,8 @@ export interface IngestSourceInput {
readPath?: string; readPath?: string;
mode?: IngestMode; mode?: IngestMode;
tags?: string[]; tags?: string[];
activate?: boolean;
expectedActiveVersionId?: string | null;
} }
export interface IngestedChunk { export interface IngestedChunk {
@ -23,10 +37,17 @@ export interface IngestedChunk {
export interface IngestResult { export interface IngestResult {
accepted: true; accepted: true;
source: IngestSourceInput; source?: IngestSourceInput;
sourceId: string;
versionId?: string;
versionNumber?: number;
state?: SourceVersionState;
previousVersionId?: string | null;
filesDiscovered: number; filesDiscovered: number;
documentsProcessed: number; documentsProcessed: number;
chunksStored: number; chunksStored: number;
activated?: boolean;
noOp?: boolean;
collectionName: string; collectionName: string;
} }
@ -34,6 +55,8 @@ export interface RetrievedItem {
chunkId: string; chunkId: string;
documentId: string; documentId: string;
sourceId: string; sourceId: string;
sourceVersionId?: string;
sourceVersionNumber?: number;
title: string; title: string;
sectionTitle?: string; sectionTitle?: string;
content: string; content: string;
@ -54,6 +77,11 @@ export interface AvailableScope {
sourceRef: string; sourceRef: string;
chunkModes: ChunkMode[]; chunkModes: ChunkMode[];
tags: string[]; tags: string[];
activeVersionId?: string | null;
activeVersionNumber?: number | null;
state?: SourceVersionState | null;
needsReingest?: boolean;
updatedAt?: string;
} }
export interface RetrieveResponse { export interface RetrieveResponse {
@ -80,6 +108,8 @@ export interface AnswerResponse {
citations: Array<{ citations: Array<{
chunkId: string; chunkId: string;
documentId: string; documentId: string;
sourceVersionId?: string;
sourceVersionNumber?: number;
title: string; title: string;
sectionTitle?: string; sectionTitle?: string;
startLine?: number; startLine?: number;

72
src/shared/utils/ids.ts
View file

@ -1,6 +1,17 @@
import path from "node:path"; import path from "node:path";
import { createHash } from "node:crypto"; import { createHash } from "node:crypto";
const UUID_NAMESPACE_URL = "6ba7b811-9dad-11d1-80b4-00c04fd430c8";
function uuidToBytes(uuid: string): Buffer {
return Buffer.from(uuid.replace(/-/g, ""), "hex");
}
function bytesToUuid(bytes: Buffer): string {
const hex = bytes.toString("hex");
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
}
function asciiNormalize(value: string): string { function asciiNormalize(value: string): string {
return value return value
.normalize("NFD") .normalize("NFD")
@ -17,6 +28,58 @@ export function normalizeDocumentKey(value: string): string {
.toLowerCase(); .toLowerCase();
} }
export function sha256Hex(value: string | Buffer): string {
return createHash("sha256").update(value).digest("hex");
}
export function normalizeContentForHash(content: string): string {
const normalized = content
.replace(/^\uFEFF/, "")
.replace(/\r\n/g, "\n")
.replace(/\r/g, "\n")
.split("\n")
.map((line) => line.replace(/[ \t]+$/g, ""))
.join("\n")
.replace(/^\n+|\n+$/g, "");
return `${normalized}\n`;
}
export function canonicalJson(value: unknown): string {
if (Array.isArray(value)) {
return `[${value.map(canonicalJson).join(",")}]`;
}
if (value && typeof value === "object") {
const entries = Object.entries(value as Record<string, unknown>)
.filter(([, entryValue]) => entryValue !== undefined)
.sort(([left], [right]) => Buffer.compare(Buffer.from(left, "utf8"), Buffer.from(right, "utf8")));
return `{${entries.map(([key, entryValue]) => `${JSON.stringify(key)}:${canonicalJson(entryValue)}`).join(",")}}`;
}
return JSON.stringify(value);
}
export function hashOrderedPairs(pairs: Array<[string, string | null]>): string | null {
if (pairs.some(([, value]) => value === null)) {
return null;
}
const serialized = [...pairs]
.sort(([left], [right]) => Buffer.compare(Buffer.from(left, "utf8"), Buffer.from(right, "utf8")))
.map(([key, value]) => `${key}\0${value ?? ""}`)
.join("\n");
return sha256Hex(serialized);
}
export function buildProcessingFingerprint(input: Record<string, unknown>): string {
return sha256Hex(canonicalJson(input));
}
export function buildMetadataHash(tags: string[], metadata: Record<string, unknown> = {}): string {
return sha256Hex(canonicalJson({ metadata, tags: [...tags].sort() }));
}
export function buildSourceId(sourceType: string, sourceRef: string): string { export function buildSourceId(sourceType: string, sourceRef: string): string {
const readable = normalizeDocumentKey(path.basename(sourceRef) || sourceRef).replace(/\.[^.]+$/, "") || "source"; const readable = normalizeDocumentKey(path.basename(sourceRef) || sourceRef).replace(/\.[^.]+$/, "") || "source";
const suffix = createHash("sha1").update(sourceRef).digest("hex").slice(0, 8); const suffix = createHash("sha1").update(sourceRef).digest("hex").slice(0, 8);
@ -36,3 +99,12 @@ export function buildQdrantPointId(value: string): string {
const hex = createHash("sha1").update(value).digest("hex").slice(0, 32); const hex = createHash("sha1").update(value).digest("hex").slice(0, 32);
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`; return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
} }
export function buildVersionedQdrantPointId(sourceVersionId: string, chunkId: string): string {
const namespace = uuidToBytes(UUID_NAMESPACE_URL);
const name = Buffer.from(`rag-qdrant-point:${sourceVersionId}:${chunkId}`, "utf8");
const hash = createHash("sha1").update(Buffer.concat([namespace, name])).digest().subarray(0, 16);
hash[6] = (hash[6] & 0x0f) | 0x50;
hash[8] = (hash[8] & 0x3f) | 0x80;
return bytesToUuid(hash);
}

514
tests/lifecycle-services.test.ts Normal file
View file

@ -0,0 +1,514 @@
import test from "node:test";
import assert from "node:assert/strict";
import { mkdtemp, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { createApp } from "../src/app.js";
import { openApiDocument } from "../src/api/openapi.js";
import { env } from "../src/config/env.js";
import { CatalogError } from "../src/modules/catalog/errors.js";
import { buildLegacyMigrationPlan } from "../src/modules/catalog/legacy-migration.js";
import { KnowledgeLifecycleReconciler } from "../src/modules/catalog/reconciler.js";
import { CatalogRepository } from "../src/modules/catalog/repository.js";
import { activateMigratedVersion, createCatalogVersion, rollbackMigratedBatch } from "../src/scripts/migrate-legacy-lifecycle.js";
import { IngestService } from "../src/modules/ingest/service.js";
import { RetrieveService } from "../src/modules/retrieve/service.js";
import type { EmbeddingProvider } from "../src/modules/embeddings/provider.js";
import type { VectorStoreClient } from "../src/modules/vectorstore/client.js";
function setEnvFlag(name: keyof typeof env, value: unknown) {
(env as unknown as Record<string, unknown>)[name] = value;
}
async function withEnvFlags(updates: Partial<Record<keyof typeof env, unknown>>, handler: () => Promise<void>) {
const previous = new Map<keyof typeof env, unknown>();
for (const [name, value] of Object.entries(updates) as Array<[keyof typeof env, unknown]>) {
previous.set(name, env[name]);
setEnvFlag(name, value);
}
try {
await handler();
} finally {
for (const [name, value] of previous) {
setEnvFlag(name, value);
}
}
}
async function withTempMarkdown(content: string, handler: (filePath: string) => Promise<void>) {
const dir = await mkdtemp(path.join(os.tmpdir(), "rag-lifecycle-test-"));
try {
const filePath = path.join(dir, "source.md");
await writeFile(filePath, content, "utf8");
await handler(filePath);
} finally {
await rm(dir, { recursive: true, force: true });
}
}
function fakeEmbeddingProvider(): EmbeddingProvider {
return {
providerName: "test-provider",
modelName: "test-model",
dimensions: 3,
async embed(input: string[]) {
return input.map(() => [0.1, 0.2, 0.3]);
}
};
}
function fakeVectorStore() {
const chunks: unknown[] = [];
const vectorStore: Partial<VectorStoreClient> = {
kind: "fake",
async upsert(input) {
chunks.push(...input);
},
async countVersionPoints() {
return chunks.length;
},
async validateCollectionDimensions() {
return undefined;
}
};
return { vectorStore: vectorStore as VectorStoreClient, chunks };
}
function fakeTransactionPool(handler: (sql: string, params?: unknown[]) => Promise<{ rowCount: number; rows: unknown[] }>) {
const client = {
async query(sql: string, params?: unknown[]) {
if (/^(BEGIN|COMMIT|ROLLBACK|SET CONSTRAINTS)/.test(sql.trim())) {
return { rowCount: 0, rows: [] };
}
return handler(sql, params);
},
release() {
return undefined;
}
};
return {
async connect() {
return client;
},
async query(sql: string, params?: unknown[]) {
return handler(sql, params);
}
};
}
test("activation failure leaves a newly indexed version ready instead of failed", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
const { vectorStore } = fakeVectorStore();
const calls: string[] = [];
const catalog = {
async beginAttempt() { return "attempt-1"; },
async updateAttempt() { return undefined; },
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
async findReusableVersion() { return undefined; },
async createPendingVersion() { return { versionId: "version-1", versionNumber: 1, state: "pending", previousVersionId: null }; },
async markIndexing() { calls.push("indexing"); },
async markReady() { calls.push("ready"); return { versionId: "version-1", versionNumber: 1, state: "ready", previousVersionId: null }; },
async activateVersion() { throw new CatalogError("precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED"); },
async markFailed() { calls.push("failed"); }
};
const service = new IngestService(fakeEmbeddingProvider(), vectorStore, catalog as never);
await assert.rejects(
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: "other" }),
/precondition failed/
);
assert.deepEqual(calls, ["indexing", "ready"]);
}));
});
test("active no-op validates the expected active version precondition", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
const { vectorStore, chunks } = fakeVectorStore();
let preconditionChecked = false;
const catalog = {
async beginAttempt() { return "attempt-1"; },
async updateAttempt() { return undefined; },
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
async findReusableVersion() { return { versionId: "active-1", versionNumber: 1, state: "active", previousVersionId: null, expectedDocumentCount: 1 }; },
async assertActiveVersionPrecondition() {
preconditionChecked = true;
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
},
async markFailed() { throw new Error("markFailed must not run for no-op precondition errors"); }
};
const service = new IngestService(fakeEmbeddingProvider(), vectorStore, catalog as never);
await assert.rejects(
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: "wrong" }),
/precondition failed/i
);
assert.equal(preconditionChecked, true);
assert.equal(chunks.length, 0);
}));
});
test("/sources fails closed and /cleanup refuses deletion when lifecycle enforcement is enabled", async () => {
setEnvFlag("knowledgeLifecycleEnforced", true);
setEnvFlag("lifecycleAdminToken", "test-token");
const server = createApp().listen(0);
try {
const address = server.address();
assert.ok(address && typeof address === "object");
const baseUrl = `http://127.0.0.1:${address.port}`;
const sources = await fetch(`${baseUrl}/sources`);
assert.equal(sources.status, 503);
const cleanupWithoutAdmin = await fetch(`${baseUrl}/cleanup`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ scope: { sourceId: "src:default:folder:docs" } })
});
assert.equal(cleanupWithoutAdmin.status, 401);
const cleanup = await fetch(`${baseUrl}/cleanup`, {
method: "POST",
headers: { "Content-Type": "application/json", Authorization: "Bearer test-token" },
body: JSON.stringify({ scope: { sourceId: "src:default:folder:docs" } })
});
assert.equal(cleanup.status, 410);
} finally {
server.close();
setEnvFlag("knowledgeLifecycleEnforced", false);
setEnvFlag("lifecycleAdminToken", "");
}
});
test("retrieval fails closed when lifecycle is enforced without a catalog", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
const service = new RetrieveService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore);
await assert.rejects(
service.retrieve("documental", "specific", "hello"),
(error) => error instanceof CatalogError && error.statusCode === 503 && error.code === "CATALOG_UNAVAILABLE"
);
});
});
test("repository state guards do not mark non-pending/indexing versions as failed", async () => {
const sqlCalls: string[] = [];
const pool = {
async query(sql: string) {
sqlCalls.push(sql);
return sql.includes("UPDATE rag_source_versions") ? { rowCount: 0, rows: [] } : { rowCount: 1, rows: [] };
}
};
const repository = new CatalogRepository(pool as never);
await repository.markFailed("ready-version", "TEST", "must not change ready");
assert.equal(sqlCalls.some((sql) => sql.includes("UPDATE rag_version_documents")), false);
assert.match(sqlCalls[0], /state IN \('pending', 'indexing'\)/);
});
test("reconciler reports invariant violations and recovers orphaned versions", async () => {
const calls: string[] = [];
const catalog = {
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
async resolveActiveVersions() { return [{ sourceId: "src:one", sourceVersionId: "v1", expectedPointCount: 2, sourceVersionNumber: 1, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }]; },
async validateActiveInvariant() { return ["src:bad"]; },
async listOrphanedIndexingCandidates() { return []; },
async markNeedsReingest(sourceId: string) { calls.push(sourceId); }
};
const vectorStore = { async validateCollectionDimensions() { return undefined; }, async countVersionPoints() { return 1; } };
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
const status = await reconciler.runOnce();
assert.equal(status.ok, false);
assert.deepEqual(status.inconsistentSources, ["src:one"]);
assert.deepEqual(status.invariantViolations, ["src:bad"]);
assert.equal(status.orphanedVersionsRecovered, 0);
assert.deepEqual(calls, ["src:one"]);
});
test("reconciler recovers stale complete indexing versions to ready", async () => {
const calls: string[] = [];
const catalog = {
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
async resolveActiveVersions() { return []; },
async validateActiveInvariant() { return []; },
async listOrphanedIndexingCandidates() {
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
},
async withSourceTryLock(_sourceId: string, handler: () => Promise<unknown>) { calls.push("source-lock"); return handler(); },
async withVersionTryLock(_versionId: string, handler: () => Promise<unknown>) { calls.push("version-lock"); return handler(); },
async hasCompleteVersionDocuments() { return true; },
async recoverVersionReady() { calls.push("ready"); return true; },
async markFailed() { calls.push("failed"); }
};
const vectorStore = { async validateCollectionDimensions() { calls.push("dimensions"); }, async countVersionPoints() { calls.push("count"); return 2; } };
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
const status = await reconciler.runOnce();
assert.equal(status.ok, true);
assert.equal(status.orphanedVersionsRecovered, 1);
assert.equal(status.orphanedVersionsFailed, 0);
assert.deepEqual(calls, ["source-lock", "version-lock", "dimensions", "count", "ready"]);
});
test("reconciler marks stale partial indexing versions as failed", async () => {
const calls: string[] = [];
const catalog = {
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
async resolveActiveVersions() { return []; },
async validateActiveInvariant() { return []; },
async listOrphanedIndexingCandidates() {
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
},
async withSourceTryLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
async withVersionTryLock(_versionId: string, handler: () => Promise<unknown>) { return handler(); },
async hasCompleteVersionDocuments() { return true; },
async recoverVersionReady() { calls.push("ready"); return true; },
async markFailed() { calls.push("failed"); }
};
const vectorStore = { async validateCollectionDimensions() { return undefined; }, async countVersionPoints() { return 1; } };
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
const status = await reconciler.runOnce();
assert.equal(status.orphanedVersionsRecovered, 0);
assert.equal(status.orphanedVersionsFailed, 1);
assert.deepEqual(calls, ["failed"]);
});
test("reconciler does not touch stale versions while a source lock is held", async () => {
const calls: string[] = [];
const catalog = {
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
async resolveActiveVersions() { return []; },
async validateActiveInvariant() { return []; },
async listOrphanedIndexingCandidates() {
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
},
async withSourceTryLock() { return undefined; },
async markFailed() { calls.push("failed"); }
};
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, fakeVectorStore().vectorStore);
const status = await reconciler.runOnce();
assert.equal(status.orphanedVersionsRecovered, 0);
assert.equal(status.orphanedVersionsFailed, 0);
assert.deepEqual(calls, []);
});
test("reconciler skips when another reconciler holds the advisory lock", async () => {
const catalog = {
async withGlobalTryLock() { return undefined; },
async resolveActiveVersions() { throw new Error("must not run without the lock"); }
};
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, fakeVectorStore().vectorStore);
const status = await reconciler.runOnce();
assert.equal(status.ok, true);
assert.equal(status.inconsistentSources.length, 0);
});
test("retrieval rejects active versions from a different Qdrant collection", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
const catalog = {
async resolveActiveVersions() {
return [{ sourceId: "src:one", sourceVersionId: "v1", sourceVersionNumber: 1, expectedPointCount: 1, embeddingDimensions: 3, qdrantCollection: "other_collection" }];
},
async markNeedsReingest() { return undefined; },
async withVersionSharedLocks(_versionIds: string[], handler: () => Promise<unknown>) { return handler(); }
};
const service = new RetrieveService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore, catalog as never);
await assert.rejects(
service.retrieve("documental", "specific", "hello"),
(error) => error instanceof CatalogError && error.code === "SOURCE_VERSION_INCONSISTENT"
);
});
});
test("retrieval holds shared locks while validating counts and searching Qdrant", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
const calls: string[] = [];
const catalog = {
async resolveActiveVersions() {
calls.push("resolve");
return [{ sourceId: "src:one", sourceVersionId: "v1", sourceVersionNumber: 1, expectedPointCount: 1, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
},
async withVersionSharedLocks(_versionIds: string[], handler: () => Promise<unknown>) {
calls.push("lock:start");
const result = await handler();
calls.push("lock:end");
return result;
},
async markNeedsReingest() { calls.push("needs-reingest"); }
};
const vectorStore = {
kind: "fake",
async validateCollectionDimensions() { calls.push("dimensions"); },
async countVersionPoints() { calls.push("count"); return 1; },
async search() { calls.push("search"); return []; },
async browseScope() { return []; }
};
const service = new RetrieveService(fakeEmbeddingProvider(), vectorStore as never, catalog as never);
await service.retrieve("documental", "specific", "hello");
assert.deepEqual(calls, ["resolve", "lock:start", "dimensions", "count", "search", "lock:end"]);
});
});
test("lifecycle ingest rejects vectors that do not match configured embedding dimensions before upsert", async () => {
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
const calls: string[] = [];
const provider: EmbeddingProvider = {
providerName: "test-provider",
modelName: "test-model",
dimensions: 3,
async embed(input: string[]) {
return input.map(() => [0.1, 0.2]);
}
};
const vectorStore = {
kind: "fake",
async upsert() { calls.push("upsert"); },
async countVersionPoints() { return 0; }
};
const catalog = {
async beginAttempt() { return "attempt-1"; },
async updateAttempt() { return undefined; },
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
async findReusableVersion() { return undefined; },
async createPendingVersion() { return { versionId: "version-1", versionNumber: 1, state: "pending", previousVersionId: null }; },
async markIndexing() { calls.push("indexing"); },
async markReady() { calls.push("ready"); return { versionId: "version-1", versionNumber: 1, state: "ready", previousVersionId: null }; },
async markFailed() { calls.push("failed"); }
};
const service = new IngestService(provider, vectorStore as never, catalog as never);
await assert.rejects(
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: null }),
(error) => error instanceof CatalogError && error.code === "EMBEDDING_DIMENSIONS_INVALID"
);
assert.deepEqual(calls, ["indexing", "failed"]);
}));
});
test("cleanup respects disabled ingest writes even on the legacy path", async () => {
await withEnvFlags({ ingestWritesEnabled: false, knowledgeLifecycleEnforced: false }, async () => {
const service = new IngestService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore);
await assert.rejects(
service.cleanup({ sourceId: "src:default:folder:docs" }),
(error) => error instanceof CatalogError && error.code === "INGEST_WRITES_DISABLED"
);
});
});
test("markPurged fails when the version is no longer in purging state", async () => {
const pool = fakeTransactionPool(async (sql) => {
if (sql.includes("UPDATE rag_source_versions")) {
return { rowCount: 0, rows: [] };
}
return { rowCount: 1, rows: [] };
});
const repository = new CatalogRepository(pool as never);
await assert.rejects(
repository.markPurged("version-1"),
(error) => error instanceof CatalogError && error.code === "INVALID_VERSION_STATE"
);
});
test("legacy migration apply reuses a catalog version created before payload update", async () => {
const source = buildLegacyMigrationPlan([{ id: "p1", payload: {
source_id: "src:default:file:source.md",
source_type: "file",
source_ref: "source.md",
document_id: "doc:src:default:file:source.md:source.md",
document_key: "source.md",
chunk_id: "chk:doc:src:default:file:source.md:source.md:documental:0001",
content: "content",
embedding_provider: "openrouter",
embedding_model: "qwen",
embedding_dimensions: 3
} }]).sources[0];
const sqlCalls: string[] = [];
const pool = fakeTransactionPool(async (sql) => {
sqlCalls.push(sql);
if (sql.includes("FROM rag_legacy_migration_items")) {
return { rowCount: 1, rows: [{ version_id: "version-1", expected_point_count: 1 }] };
}
if (sql.includes("FROM rag_source_versions v")) {
return { rowCount: 1, rows: [{ version_id: "version-1", version_number: "1", expected_point_count: "1", source_type: "file", source_ref: "source.md", qdrant_collection: env.qdrantCollection }] };
}
return { rowCount: 1, rows: [] };
});
const result = await createCatalogVersion(pool as never, "batch-1", source, env.qdrantCollection);
assert.deepEqual(result, { versionId: "version-1", versionNumber: 1 });
assert.equal(sqlCalls.some((sql) => sql.includes("INSERT INTO rag_sources")), false);
});
test("legacy migration activation is idempotent after payload update and prior activation", async () => {
const sqlCalls: string[] = [];
const pool = fakeTransactionPool(async (sql) => {
sqlCalls.push(sql);
if (sql.includes("SELECT v.state")) {
return { rowCount: 1, rows: [{ state: "active", active_version_id: "version-1" }] };
}
return { rowCount: 1, rows: [] };
});
await activateMigratedVersion(pool as never, "src:default:file:source.md", "version-1", 1);
assert.equal(sqlCalls.some((sql) => sql.includes("SET state = 'active'")), false);
});
test("legacy rollback prepares the whole batch before restoring and validates restored legacy counts before catalog cleanup", async () => {
const calls: string[] = [];
const pool = fakeTransactionPool(async (sql) => {
if (sql.includes("SELECT state, snapshot_ref")) {
calls.push("batch-lock");
return { rowCount: 1, rows: [{ state: "verified", snapshot_ref: "snapshot-url" }] };
}
if (sql.includes("SELECT source_id, version_id, expected_point_count")) {
calls.push("items-lock");
return { rowCount: 1, rows: [{ source_id: "src:default:file:source.md", version_id: "version-1", expected_point_count: 1 }] };
}
if (sql.includes("SET active_version_id = NULL")) calls.push("active-null");
if (sql.includes("SET state = 'purging'")) calls.push("purging");
if (sql.includes("COUNT(*) AS count")) {
calls.push("zero-active-check");
return { rowCount: 1, rows: [{ count: "0" }] };
}
if (sql.includes("DELETE FROM rag_version_documents")) calls.push("delete-docs");
if (sql.includes("UPDATE rag_legacy_migration_items SET version_id = NULL")) calls.push("items-rolled-back");
return { rowCount: 1, rows: [] };
});
const qdrant = {
async recoverSnapshot() { calls.push("restore-snapshot"); return true; },
async scroll() {
calls.push("verify-legacy-count");
return { points: [{ id: "p1", payload: { source_id: "src:default:file:source.md" } }], next_page_offset: undefined };
}
};
const result = await rollbackMigratedBatch(pool as never, qdrant as never, "batch-1", "snapshot-url");
assert.equal(result.ok, true);
assert.ok(calls.indexOf("zero-active-check") < calls.indexOf("restore-snapshot"));
assert.ok(calls.indexOf("verify-legacy-count") < calls.indexOf("delete-docs"));
assert.ok(calls.includes("items-rolled-back"));
});
test("OpenAPI documents /sources 503 responses", () => {
const sources = openApiDocument.paths["/sources"].get.responses;
assert.ok("503" in sources);
});
test("legacy migration dry-run blocks weak document identity", () => {
const plan = buildLegacyMigrationPlan([
{
id: "p1",
payload: {
source_id: "src:default:folder:docs",
source_ref: "docs",
source_type: "folder",
document_id: "doc:src:default:folder:docs:readme.md",
chunk_id: "chk:doc:src:default:folder:docs:readme.md:documental:0001",
content: "content",
embedding_provider: "openrouter",
embedding_model: "qwen",
embedding_dimensions: 3
}
}
]);
assert.equal(plan.ok, false);
assert.match(plan.blockedReasons.join("\n"), /weak document identity/);
});

61
tests/lifecycle-utils.test.ts Normal file
View file

@ -0,0 +1,61 @@
import test from "node:test";
import assert from "node:assert/strict";
import {
buildMetadataHash,
buildProcessingFingerprint,
buildVersionedQdrantPointId,
hashOrderedPairs,
normalizeContentForHash,
sha256Hex
} from "../src/shared/utils/ids.js";
import { CatalogError } from "../src/modules/catalog/errors.js";
import { normalizeExpectedActiveVersion } from "../src/modules/catalog/lifecycle.js";
test("versioned Qdrant point IDs are deterministic UUIDv5 values", () => {
const versionId = "123e4567-e89b-12d3-a456-426614174000";
const chunkId = "chk:doc:src:default:folder:docs:readme.md:documental:0001";
const first = buildVersionedQdrantPointId(versionId, chunkId);
const second = buildVersionedQdrantPointId(versionId, chunkId);
const differentVersion = buildVersionedQdrantPointId("123e4567-e89b-12d3-a456-426614174001", chunkId);
assert.equal(first, second);
assert.notEqual(first, differentVersion);
assert.match(first, /^[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/);
});
test("content hashing normalization is canonical", () => {
const left = normalizeContentForHash("\uFEFFTitle\r\nBody \r\n\r\n");
const right = normalizeContentForHash("Title\nBody\n");
assert.equal(left, "Title\nBody\n");
assert.equal(sha256Hex(left), sha256Hex(right));
});
test("ordered pair hashes are independent from input order", () => {
const first = hashOrderedPairs([["b.md", "hash-b"], ["a.md", "hash-a"]]);
const second = hashOrderedPairs([["a.md", "hash-a"], ["b.md", "hash-b"]]);
assert.equal(first, second);
assert.equal(hashOrderedPairs([["a.md", null]]), null);
});
test("processing and metadata fingerprints are stable with sorted keys", () => {
assert.equal(
buildProcessingFingerprint({ model: "qwen", provider: "openrouter" }),
buildProcessingFingerprint({ provider: "openrouter", model: "qwen" })
);
assert.equal(
buildMetadataHash(["b", "a"], { sourceRef: "docs" }),
buildMetadataHash(["a", "b"], { sourceRef: "docs" })
);
});
test("activation precondition is mandatory when activation is requested", () => {
assert.throws(
() => normalizeExpectedActiveVersion(undefined, true),
(error) => error instanceof CatalogError && error.statusCode === 400
);
assert.equal(normalizeExpectedActiveVersion(null, true), null);
assert.equal(normalizeExpectedActiveVersion(undefined, false), undefined);
});