feat(rag): add knowledge lifecycle management
This commit is contained in:
parent
e5f739eb6c
commit
551cfe8816
33 changed files with 5308 additions and 261 deletions
|
|
@ -11,3 +11,10 @@ ANSWER_PROVIDER=openrouter
|
|||
ANSWER_MODEL=openai/gpt-4.1-mini
|
||||
ANSWER_BASE_URL=https://openrouter.ai/api/v1
|
||||
ANSWER_API_KEY=
|
||||
POSTGRES_URL=postgresql://rag:rag@postgres:5432/rag
|
||||
POSTGRES_SSL=false
|
||||
LIFECYCLE_ADMIN_TOKEN=
|
||||
KNOWLEDGE_LIFECYCLE_ENFORCED=false
|
||||
INGEST_WRITES_ENABLED=true
|
||||
LIFECYCLE_RECONCILE_INTERVAL_MS=300000
|
||||
LIFECYCLE_INDEXING_STALE_TIMEOUT_MS=1800000
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ WORKDIR /app
|
|||
COPY package.json package-lock.json tsconfig.json ./
|
||||
RUN npm ci
|
||||
COPY src ./src
|
||||
COPY migrations ./migrations
|
||||
COPY public ./public
|
||||
RUN npm run build
|
||||
|
||||
|
|
@ -12,6 +13,7 @@ ENV NODE_ENV=production
|
|||
COPY package.json package-lock.json ./
|
||||
RUN npm ci --omit=dev
|
||||
COPY --from=build /app/dist ./dist
|
||||
COPY --from=build /app/migrations ./migrations
|
||||
COPY public ./public
|
||||
EXPOSE 3000
|
||||
CMD ["node", "dist/server.js"]
|
||||
CMD ["sh", "-c", "node dist/modules/catalog/migrations.js && node dist/server.js"]
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@
|
|||
|
||||
**Proyecto:** Workspace de tools IA para empresas
|
||||
**Modulo:** RAG
|
||||
**Ultima actualizacion:** 2026-09-08
|
||||
**Ultima modificacion por:** Agente RAG 2
|
||||
**Estado:** Operativa y validada en produccion
|
||||
**Ultima actualizacion:** 2026-09-11
|
||||
**Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
|
||||
**Estado:** Punto 2 implementado localmente; pendiente de migracion y validacion en produccion
|
||||
|
||||
Esta guia explica como descubrir y consumir la API. El contrato tecnico completo y canonico se publica en OpenAPI 3.1.1; este documento prioriza el camino rapido, las decisiones de uso y los ejemplos habituales.
|
||||
|
||||
|
|
@ -24,7 +24,7 @@ https://rag.por-correo.com
|
|||
|
||||
`GET /help` indica expresamente que debe consultarse `/openapi.json` para obtener los parametros, cuerpos, respuestas, errores y ejemplos completos de cada endpoint.
|
||||
|
||||
La API no requiere autenticacion actualmente. No debe asumirse que esta situacion es definitiva: la autenticacion y la proteccion de operaciones sensibles forman parte del pendiente 5.
|
||||
La API general sigue sin autenticacion completa. Las operaciones administrativas del ciclo de vida usan una proteccion acotada con `Authorization: Bearer <LIFECYCLE_ADMIN_TOKEN>`; esto no sustituye la autenticacion general pendiente.
|
||||
|
||||
## Elegir la operacion correcta
|
||||
|
||||
|
|
@ -32,18 +32,20 @@ La API no requiere autenticacion actualmente. No debe asumirse que esta situacio
|
|||
|---|---|
|
||||
| Comprobar servicio y capacidades | `GET /health` |
|
||||
| Ver scopes ingeridos | `GET /sources` |
|
||||
| Ver versiones de una fuente | `GET /sources/{sourceId}/versions` |
|
||||
| Activar, rollback o purgar versiones | `/sources/*` con token administrativo |
|
||||
| Recuperar contexto para otro agente | `POST /retrieve` |
|
||||
| Obtener una respuesta final con citas | `POST /answer` |
|
||||
| Comparar una respuesta sin RAG | `POST /answer/direct` |
|
||||
| Conversar con bootstrap e historial | `POST /chat` |
|
||||
| Ingerir una ruta accesible por el servidor | `POST /ingest` |
|
||||
| Subir un archivo o carpeta ZIP | `POST /ingest/upload` |
|
||||
| Eliminar una fuente antes de reingestar | `POST /cleanup` |
|
||||
| Eliminar una fuente antes de reingestar | `POST /cleanup` legacy, o purga de version no activa |
|
||||
| Consultar o actualizar evaluaciones | `/logs/*` |
|
||||
|
||||
## Catalogo completo
|
||||
|
||||
La API queda formada por 16 operaciones tras incorporar el descubrimiento:
|
||||
La API queda formada por las operaciones de descubrimiento, ingesta, retrieval, evaluacion y ciclo de vida:
|
||||
|
||||
| Metodo | Ruta | Descripcion |
|
||||
|---|---|---|
|
||||
|
|
@ -52,6 +54,13 @@ La API queda formada por 16 operaciones tras incorporar el descubrimiento:
|
|||
| `GET` | `/playground` | Interfaz web de prueba |
|
||||
| `GET` | `/health` | Estado, proveedores, parsers y chunking |
|
||||
| `GET` | `/sources` | Scopes disponibles en Qdrant |
|
||||
| `GET` | `/sources/{sourceId}` | Fuente registrada en el catalogo |
|
||||
| `GET` | `/sources/{sourceId}/versions` | Versiones de una fuente |
|
||||
| `GET` | `/sources/{sourceId}/versions/{versionId}` | Detalle de una version |
|
||||
| `POST` | `/sources/{sourceId}/versions/{versionId}/activate` | Activacion administrativa con precondicion |
|
||||
| `POST` | `/sources/{sourceId}/rollback` | Rollback administrativo sin recalcular embeddings |
|
||||
| `POST` | `/sources/{sourceId}/versions/{versionId}/retry` | Reintento explicito de version fallida o huerfana con fuente durable |
|
||||
| `DELETE` | `/sources/{sourceId}/versions/{versionId}` | Purga administrativa de una version no activa |
|
||||
| `GET` | `/models/answer` | Modelo por defecto y modelos seleccionables |
|
||||
| `POST` | `/ingest` | Ingesta desde una ruta del servidor |
|
||||
| `POST` | `/ingest/upload` | Upload `multipart/form-data` |
|
||||
|
|
@ -102,6 +111,15 @@ Los errores JSON siguen esta forma:
|
|||
|
||||
Actualmente la mayoria de fallos de validacion o de proveedores se devuelven como `500`. `POST /cleanup` y `POST /ingest/upload` tambien pueden devolver `400`, y `PATCH /logs/{id}` devuelve `404` si no encuentra el log. Mejorar la validacion queda fuera de este bloque para no cambiar silenciosamente el comportamiento existente.
|
||||
|
||||
Con `KNOWLEDGE_LIFECYCLE_ENFORCED=true`, la API devuelve errores mas especificos para el catalogo:
|
||||
|
||||
| Codigo HTTP | Caso |
|
||||
|---|---|
|
||||
| `400` | Falta `expectedActiveVersionId` cuando `activate=true` |
|
||||
| `409` | Cambio concurrente de version activa, version en curso o intento de purgar una activa |
|
||||
| `422` | Fuente vacia, sin documentos soportados o sin contenido util |
|
||||
| `503` | PostgreSQL, Qdrant, embeddings o version activa inconsistente |
|
||||
|
||||
## Recuperar contexto
|
||||
|
||||
`POST /retrieve` es la operacion recomendada cuando otro agente o flujo generara la respuesta final.
|
||||
|
|
@ -248,12 +266,22 @@ Respuesta:
|
|||
"sourceType": "folder",
|
||||
"sourceRef": "/data/project-docs",
|
||||
"mode": "mechanical",
|
||||
"tags": ["project", "documentation"]
|
||||
"tags": ["project", "documentation"],
|
||||
"activate": true,
|
||||
"expectedActiveVersionId": null
|
||||
}
|
||||
```
|
||||
|
||||
`readPath` es un campo interno para separar la ruta fisica de lectura de la referencia logica. Los consumidores normales no deben enviarlo.
|
||||
|
||||
Con ciclo de vida activo:
|
||||
|
||||
- `activate` por defecto es `true`.
|
||||
- si `activate=true`, `expectedActiveVersionId` es obligatorio: `null` para primera activacion y UUID para reemplazar una version existente.
|
||||
- una ingesta identica a la version activa responde `noOp: true` y no escribe nuevos puntos.
|
||||
- `activate=false` deja la version en `ready` para activarla despues con una operacion administrativa.
|
||||
- la respuesta exitosa cambia a `201 Created` e incluye `sourceId`, `versionId`, `versionNumber`, `state`, `previousVersionId`, `activated` y `noOp`.
|
||||
|
||||
### Upload
|
||||
|
||||
`POST /ingest/upload` usa `multipart/form-data`:
|
||||
|
|
@ -265,6 +293,8 @@ Respuesta:
|
|||
| `mode` | No | `mechanical` o `interactive` |
|
||||
| `tags` | No | Lista separada por comas |
|
||||
| `isZipFolder` | No | Texto `true` o `false` |
|
||||
| `activate` | No | Texto `true` o `false` |
|
||||
| `expectedActiveVersionId` | Si activa | UUID o texto `null` |
|
||||
|
||||
```bash
|
||||
curl -sS -X POST "https://rag.por-correo.com/ingest/upload" \
|
||||
|
|
@ -274,11 +304,52 @@ curl -sS -X POST "https://rag.por-correo.com/ingest/upload" \
|
|||
-F "tags=customer-a,support"
|
||||
```
|
||||
|
||||
Ambas variantes responden `202` con el origen aceptado, archivos descubiertos, documentos procesados, chunks almacenados y coleccion. El upload añade `uploadedResource`.
|
||||
Con enforcement desactivado, ambas variantes conservan `202`. Con enforcement activado, responden `201` y usan el catalogo PostgreSQL como autoridad de version activa. El upload añade `uploadedResource`.
|
||||
|
||||
## Ciclo de vida de fuentes
|
||||
|
||||
El catalogo PostgreSQL es la autoridad para saber que fuentes existen, que versiones tienen y cual es la version activa. Qdrant conserva los puntos fisicos versionados, pero retrieval solo consulta versiones activas resueltas desde PostgreSQL.
|
||||
|
||||
### Listado y detalle
|
||||
|
||||
```text
|
||||
GET /sources
|
||||
GET /sources/{sourceId}
|
||||
GET /sources/{sourceId}/versions
|
||||
GET /sources/{sourceId}/versions/{versionId}
|
||||
```
|
||||
|
||||
`GET /sources` mantiene los campos usados por el playground (`sourceId`, `sourceRef`, `chunkModes`, `tags`) y añade `activeVersionId`, `activeVersionNumber`, `state`, `needsReingest` y `updatedAt`.
|
||||
|
||||
### Activacion y rollback
|
||||
|
||||
```bash
|
||||
curl -sS -X POST "https://rag.por-correo.com/sources/src:default:folder:docs/versions/<version-id>/activate" \
|
||||
-H "Authorization: Bearer $LIFECYCLE_ADMIN_TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"expectedActiveVersionId":"<active-version-id>"}'
|
||||
```
|
||||
|
||||
Rollback usa la misma activacion atomica del puntero PostgreSQL:
|
||||
|
||||
```json
|
||||
{
|
||||
"targetVersionId": "uuid-version-anterior",
|
||||
"expectedActiveVersionId": "uuid-version-activa-actual"
|
||||
}
|
||||
```
|
||||
|
||||
### Purga
|
||||
|
||||
`DELETE /sources/{sourceId}/versions/{versionId}` exige token administrativo y solo acepta versiones no activas. La purga marca la version como `purging`, borra puntos Qdrant con `wait=true`, verifica que no quedan puntos y finalmente deja la version como `purged`.
|
||||
|
||||
### Reintento explicito
|
||||
|
||||
`POST /sources/{sourceId}/versions/{versionId}/retry` acepta versiones `failed`, `pending` o `indexing`. Si recibe una fuente durable (`sourceType`, `sourceRef` y opcionalmente `readPath`) cierra versiones huerfanas como fallidas y crea una version nueva mediante `/ingest`. Si no recibe fuente durable, devuelve `202` con instrucciones: los uploads temporales no se reintentan en sitio porque el original ya no esta garantizado.
|
||||
|
||||
## Limpieza controlada
|
||||
|
||||
`POST /cleanup` elimina los chunks que coinciden con un scope:
|
||||
`POST /cleanup` elimina los chunks que coinciden con un scope solo en modo legacy y con `INGEST_WRITES_ENABLED=true`:
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
|
|||
1169
docs/CONTRATO_CICLO_VIDA_Y_OCR.md
Normal file
1169
docs/CONTRATO_CICLO_VIDA_Y_OCR.md
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -2,9 +2,9 @@
|
|||
|
||||
**Proyecto:** Workspace de tools IA para empresas
|
||||
**Modulo:** RAG
|
||||
**Ultima actualizacion:** 2026-04-05
|
||||
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales
|
||||
**Estado:** V1 desplegada en VPS2
|
||||
**Ultima actualizacion:** 2026-09-11
|
||||
**Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
|
||||
**Estado:** V1 desplegada en VPS2; ciclo de vida implementado localmente pendiente de migracion productiva
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -200,6 +200,12 @@ Dato importante para `RAG`:
|
|||
- `ANSWER_MODEL=openai/gpt-4.1-mini`
|
||||
- `ANSWER_BASE_URL=https://openrouter.ai/api/v1`
|
||||
- `ANSWER_API_KEY=<valor real>`
|
||||
- `POSTGRES_URL=<conexion interna a PostgreSQL>`
|
||||
- `POSTGRES_SSL=false` salvo que la conexion lo exija
|
||||
- `LIFECYCLE_ADMIN_TOKEN=<secreto administrativo>`
|
||||
- `KNOWLEDGE_LIFECYCLE_ENFORCED=false` durante despliegue inicial y migracion
|
||||
- `INGEST_WRITES_ENABLED=false` durante la ventana de migracion legacy
|
||||
- `LIFECYCLE_RECONCILE_INTERVAL_MS=300000`
|
||||
|
||||
### Variables de entorno con las que quedo operativo en VPS2
|
||||
|
||||
|
|
@ -219,6 +225,15 @@ Valores funcionales confirmados en EasyPanel:
|
|||
- `ANSWER_BASE_URL=https://openrouter.ai/api/v1`
|
||||
- `ANSWER_API_KEY=<secreto>`
|
||||
|
||||
Variables pendientes de completar para activar el punto 2 en produccion:
|
||||
|
||||
- `POSTGRES_URL=<secreto>`
|
||||
- `POSTGRES_SSL=false`
|
||||
- `LIFECYCLE_ADMIN_TOKEN=<secreto>`
|
||||
- `KNOWLEDGE_LIFECYCLE_ENFORCED=false` hasta completar la migracion legacy
|
||||
- `INGEST_WRITES_ENABLED=true` en operacion normal; `false` durante migracion
|
||||
- `LIFECYCLE_RECONCILE_INTERVAL_MS=300000`
|
||||
|
||||
Para respaldo operativo local fuera de EasyPanel se deja un archivo ignorado por Git:
|
||||
|
||||
- `RAG/.env.easypanel.local`
|
||||
|
|
@ -228,7 +243,27 @@ Ese archivo no debe versionarse porque contiene secretos reales.
|
|||
### Volumen persistente
|
||||
|
||||
- no es obligatorio para la app RAG en esta fase
|
||||
- el estado persistente principal vive en `qdrant`
|
||||
- el estado persistente vectorial vive en `qdrant`
|
||||
- desde el punto 2, el catalogo de fuentes y versiones vive en PostgreSQL
|
||||
|
||||
### Arranque con migraciones
|
||||
|
||||
La imagen arranca ejecutando primero:
|
||||
|
||||
```bash
|
||||
node dist/modules/catalog/migrations.js && node dist/server.js
|
||||
```
|
||||
|
||||
Si `POSTGRES_URL` no esta configurada, el migrador informa que omite migraciones. Si PostgreSQL esta configurado y una migracion falla o cambia su checksum, el servicio no debe completar el arranque.
|
||||
|
||||
### Secuencia operativa del punto 2
|
||||
|
||||
1. Desplegar el codigo con `KNOWLEDGE_LIFECYCLE_ENFORCED=false`.
|
||||
2. Bloquear escrituras con `INGEST_WRITES_ENABLED=false` durante la migracion.
|
||||
3. Ejecutar `npm run migrate:legacy:lifecycle` en modo dry-run y revisar colisiones.
|
||||
4. Tomar snapshot Qdrant antes de cualquier migracion destructiva.
|
||||
5. Catalogar el corpus legacy y verificar recuentos PostgreSQL/Qdrant. Si el proceso cae, reanudar con `npm run migrate:legacy:lifecycle -- --apply --snapshot-ref=<snapshot> --batch-id=<batch-id>`.
|
||||
6. Activar `KNOWLEDGE_LIFECYCLE_ENFORCED=true` solo despues de validar retrieval y playground.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -93,3 +93,123 @@ Dar continuidad al RAG en `RAG/` a partir del estado actual documentado.
|
|||
- Pendiente 1 completado y validado definitivamente en `https://rag.por-correo.com`.
|
||||
- La API dispone de descubrimiento rapido y contrato OpenAPI para sus 16 operaciones.
|
||||
- Siguiente prioridad: ciclo de vida del conocimiento.
|
||||
|
||||
---
|
||||
|
||||
### 2026-09-08 - Agente RAG 2
|
||||
|
||||
**Modelo:** gpt-5.6-sol
|
||||
**Session ID OpenCode:** `ses_29bdbd003ffeLrLjUlFgnp08Y7`
|
||||
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
|
||||
|
||||
**Subagentes de analisis:**
|
||||
- `Subagente Extraccion de versiones PaddleOCR`: contraste de versiones PyPI compatibles para el benchmark CPU.
|
||||
- `Subagente Validacion del ciclo de vida`: revision adversarial del catalogo PostgreSQL y versionado Qdrant.
|
||||
- `Subagente Validacion OCR`: revision adversarial de deteccion, servicio privado, quality gates y revision humana.
|
||||
- `Subagente Auditoria de claridad del contrato`: detecto contradicciones de hashes, idempotencia, migracion y revision.
|
||||
- `Subagente Revalidacion de ejecutabilidad`: detecto bloqueos restantes en metadata, concurrencia y artefactos.
|
||||
- `Subagente Revalidacion de resiliencia`: reviso carreras, barreras de migracion y recuperacion de jobs.
|
||||
- `Subagente Revision final del contrato`: verifico y cerro la secuencia segura de rollback legacy.
|
||||
|
||||
**Trabajo realizado:**
|
||||
- Auditoria del pipeline actual de ingesta, IDs, Qdrant, parser PDF, API e infraestructura EasyPanel.
|
||||
- Confirmacion de PostgreSQL 17 como catalogo transaccional y Qdrant como almacenamiento vectorial versionado.
|
||||
- Benchmark aislado en VPS2 con el PDF mixto real de FacturaTech.
|
||||
- Descarte de Tesseract como motor unico por omitir identificadores criticos.
|
||||
- Seleccion de PaddleOCR como motor principal y deteccion de errores alfanumericos de alta confianza que obligan a revision humana.
|
||||
- Creacion de `docs/CONTRATO_CICLO_VIDA_Y_OCR.md` con esquema, estados, APIs, migracion, OCR, tareas, pruebas, despliegue y rollback.
|
||||
- Enlace del contrato desde el backlog canonico.
|
||||
- Correccion del contrato mediante revisiones adversariales hasta eliminar sus bloqueos criticos conocidos.
|
||||
- Limpieza de los artefactos e imagenes temporales usados en el benchmark local y de VPS2.
|
||||
|
||||
**Estado final:**
|
||||
- Diseño de los pendientes 2 y 3 cerrado y listo para implementacion secuencial.
|
||||
- El punto 2 debe completarse y validarse en produccion antes de iniciar el punto 3.
|
||||
- No se modifico codigo ni produccion durante esta fase de diseño.
|
||||
|
||||
**Archivos modificados:**
|
||||
- `RAG/docs/CONTRATO_CICLO_VIDA_Y_OCR.md`
|
||||
- `RAG/docs/PENDIENTES_RAG.md`
|
||||
- `RAG/docs/HISTORIAL_SESIONES.md`
|
||||
- `docs/INDICE_DOCUMENTACION.md`
|
||||
- `docs/HISTORIAL_SESIONES.md`
|
||||
|
||||
---
|
||||
|
||||
### 2026-09-11 - Subagente Implementacion ciclo de vida del conocimiento
|
||||
|
||||
**Modelo:** openai/gpt-5.5
|
||||
**Session ID OpenCode:** `ses_f6fe0ad30ffeiottYsATsRemCo`
|
||||
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
|
||||
|
||||
**Rol asumido:**
|
||||
Ejecutar la fase apply del punto 2 del RAG y corregir los bloqueadores finales detectados antes de commit, push o despliegue.
|
||||
|
||||
**Trabajo realizado:**
|
||||
- Ajuste del migrador legacy para permitir reanudacion por `--batch-id`, reutilizando versiones ya creadas y completando activacion tras actualizaciones de payload ya aplicadas.
|
||||
- Endurecimiento de rollback legacy con lock global de migracion, comprobacion de snapshot y operaciones serializadas sobre fuentes/versiones.
|
||||
- Reconciliador protegido con advisory lock no bloqueante, timeout configurable basado en `indexing_started_at` y validacion de coleccion/dimensiones.
|
||||
- Protecciones adicionales: `POST /cleanup` respeta `INGEST_WRITES_ENABLED`, `markPurged` valida `rowCount`, rollback usa locks y OpenAPI refleja `503` en health con `ok` booleano.
|
||||
- Script `migrate:legacy:lifecycle` ajustado para ejecutarse desde `dist/` con Node tras build.
|
||||
- Añadidos tests unitarios para reanudacion legacy, activacion idempotente, cleanup con escrituras deshabilitadas, reconciliador, mismatch de coleccion y `markPurged`.
|
||||
|
||||
**Validacion local:**
|
||||
- `npm run check` correcto.
|
||||
- `npm test` correcto: 18 tests pasan.
|
||||
- `npm run build` correcto.
|
||||
- `git diff --check` correcto.
|
||||
|
||||
**Estado final:**
|
||||
- Punto 2 implementado y validado localmente.
|
||||
- No se hizo commit, push, deploy ni lectura de `.env`.
|
||||
- Sigue pendiente la verificacion independiente y la validacion/migracion en produccion antes de marcar el punto 2 como cerrado.
|
||||
|
||||
**Archivos modificados en esta correccion:**
|
||||
- `.env.example`
|
||||
- `package.json`
|
||||
- `package-lock.json`
|
||||
- `src/config/env.ts`
|
||||
- `src/modules/catalog/client.ts`
|
||||
- `src/modules/catalog/repository.ts`
|
||||
- `src/modules/catalog/reconciler.ts`
|
||||
- `src/modules/vectorstore/client.ts`
|
||||
- `src/modules/retrieve/service.ts`
|
||||
- `src/modules/ingest/service.ts`
|
||||
- `src/scripts/migrate-legacy-lifecycle.ts`
|
||||
- `src/api/openapi.ts`
|
||||
- `tests/lifecycle-services.test.ts`
|
||||
- `docs/API_RAG.md`
|
||||
- `docs/DESPLIEGUE_EASYPANEL.md`
|
||||
- `docs/SISTEMA_RAG_BASE.md`
|
||||
- `RAG/docs/HISTORIAL_SESIONES.md`
|
||||
|
||||
---
|
||||
|
||||
### 2026-09-11 (reanudacion final) - Subagente Implementacion ciclo de vida del conocimiento
|
||||
|
||||
**Modelo:** openai/gpt-5.5
|
||||
**Session ID OpenCode:** `ses_f6fe0ad30ffeiottYsATsRemCo`
|
||||
**Directorio:** `/home/pancho/Documentos/Empresa/Desarrollo/IA/RAG`
|
||||
|
||||
**Rol asumido:**
|
||||
Completar los 4 bloqueos del sdd-verify y cerrar la validacion local del punto 2.
|
||||
|
||||
**Trabajo realizado:**
|
||||
- Reconciliador: recupera versiones huerfanas `pending/indexing` a `ready` contando puntos Qdrant y validando documentos, coleccion y dimensiones; marca `failed` solo si quedan incompletas; try-locks evitan matar ingestas vivas.
|
||||
- Ingesta: rechaza embeddings con dimensiones distintas de las esperadas antes del upsert y antes de alcanzar `ready`.
|
||||
- Rollback legacy: transaccion diferida del lote, confirmacion de cero punteros activos, restauracion de snapshot con verificacion de recuentos y limpieza posterior del catalogo.
|
||||
- OpenAPI: `/sources` documenta `503`.
|
||||
- Retrieval: locks compartidos adquiridos antes de validar recuento/dimensiones y mantenidos durante toda la consulta Qdrant.
|
||||
- Interrupcion por limite de uso a mitad de la validacion; reanudada tras cambio de cuenta y completada hasta el final.
|
||||
|
||||
**Validacion local:**
|
||||
- `npm run check` correcto.
|
||||
- `npm test` correcto: 25/25 tests pasan.
|
||||
- `npm run build` correcto.
|
||||
- `git diff --check` correcto.
|
||||
- Verificacion independiente (sdd-verify): PASS local.
|
||||
|
||||
**Estado final:**
|
||||
- Implementacion local del punto 2 verificada de forma independiente: PASS local.
|
||||
- No se hizo commit, push, deploy ni lectura de `.env`.
|
||||
- Pendiente: commit/push, deploy con enforcement desactivado, migracion legacy y validacion en produccion antes de cerrar el punto 2.
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@
|
|||
|
||||
**Proyecto:** Workspace de tools IA para empresas
|
||||
**Modulo:** RAG
|
||||
**Ultima actualizacion:** 2026-04-02
|
||||
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales
|
||||
**Estado:** En definicion
|
||||
**Ultima actualizacion:** 2026-09-11
|
||||
**Ultima modificacion por:** Subagente Implementacion ciclo de vida del conocimiento
|
||||
**Estado:** Punto 2 implementado localmente; pendiente de migracion y validacion en produccion
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -531,3 +531,48 @@ El enriquecimiento ligero debe ser:
|
|||
2. Como construiremos exactamente `source_id`, `document_id` y `chunk_id`.
|
||||
3. Que nombre tendran las fuentes logicas por defecto para documentos sueltos.
|
||||
4. Como distinguiremos entre fuente nueva, fuente actualizada y fuente obsoleta.
|
||||
|
||||
---
|
||||
|
||||
## Ciclo de vida del conocimiento implementado localmente
|
||||
|
||||
El punto 2 introduce un catalogo PostgreSQL como autoridad de fuentes y versiones. Qdrant deja de decidir que conocimiento esta activo: cada punto fisico queda asociado a `source_version_id`, y retrieval filtra por las versiones activas resueltas desde PostgreSQL cuando `KNOWLEDGE_LIFECYCLE_ENFORCED=true`.
|
||||
|
||||
### Flujo de ingesta versionada
|
||||
|
||||
1. Resolver `source_id` y registrar un intento en `rag_ingestion_attempts`.
|
||||
2. Descubrir documentos soportados y calcular hashes de originales.
|
||||
3. Normalizar contenido y calcular hashes de contenido, fingerprint de procesamiento y hash de metadatos.
|
||||
4. Detectar si ya existe una version reutilizable.
|
||||
5. Crear una version `pending`, pasarla a `indexing`, escribir puntos Qdrant versionados y verificar recuentos.
|
||||
6. Pasar la version a `ready`.
|
||||
7. Si `activate=true`, activar con precondicion `expectedActiveVersionId`.
|
||||
|
||||
Una escritura parcial en Qdrant no queda visible para retrieval hasta que PostgreSQL mueve atomica y correctamente el puntero `active_version_id`.
|
||||
|
||||
### Nuevos campos de ingesta
|
||||
|
||||
```json
|
||||
{
|
||||
"activate": true,
|
||||
"expectedActiveVersionId": null
|
||||
}
|
||||
```
|
||||
|
||||
- `activate=true` exige `expectedActiveVersionId`.
|
||||
- Para primera activacion se envia `null`.
|
||||
- Para reemplazo se envia el UUID de la version activa que el cliente espera reemplazar.
|
||||
- `activate=false` permite dejar una version preparada en estado `ready`.
|
||||
|
||||
### Identidad y hashes
|
||||
|
||||
- `document_key` de uploads se deriva de `source_ref`, no de rutas temporales.
|
||||
- `original_hash` usa SHA-256 de bytes originales.
|
||||
- `content_hash` usa SHA-256 del contenido normalizado.
|
||||
- `source_content_hash` y `original_manifest_hash` usan pares ordenados `document_key + NUL + hash`.
|
||||
- `processing_fingerprint` cambia cuando cambian parser, normalizacion, chunking, proveedor, modelo o dimensiones de embeddings.
|
||||
- El ID fisico de Qdrant usa UUIDv5 con `source_version_id + chunk_id`.
|
||||
|
||||
### Compatibilidad temporal
|
||||
|
||||
Con `KNOWLEDGE_LIFECYCLE_ENFORCED=false`, el camino legacy se conserva para permitir despliegue, dry-run y migracion controlada. La validacion final del punto 2 exige activar enforcement despues de catalogar el corpus legacy.
|
||||
|
|
|
|||
|
|
@ -17,6 +17,8 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
|
|||
|
||||
## 2. Ciclo de vida del conocimiento
|
||||
|
||||
**Diseño:** Contrato de implementacion cerrado en [`CONTRATO_CICLO_VIDA_Y_OCR.md`](./CONTRATO_CICLO_VIDA_Y_OCR.md). Pendiente de implementar y validar en produccion antes de iniciar el punto 3.
|
||||
|
||||
- Crear un catalogo de fuentes y versiones ingeridas.
|
||||
- Saber que documento esta vigente, obsoleto, reemplazado o pendiente de reingesta.
|
||||
- Evitar duplicados y permitir actualizaciones incrementales controladas.
|
||||
|
|
@ -25,6 +27,8 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
|
|||
|
||||
## 3. OCR integrado en la ingesta
|
||||
|
||||
**Diseño:** Contrato de implementacion cerrado en [`CONTRATO_CICLO_VIDA_Y_OCR.md`](./CONTRATO_CICLO_VIDA_Y_OCR.md). Su ejecucion queda bloqueada hasta completar el punto 2.
|
||||
|
||||
- Detectar PDFs con capturas, imagenes o una capa textual insuficiente.
|
||||
- Ejecutar OCR automaticamente o bloquear la ingesta para revision cuando no pueda garantizarse la cobertura.
|
||||
- Generar un documento intermedio auditable cuando sea necesario.
|
||||
|
|
@ -46,6 +50,7 @@ Este documento es la fuente canonica del orden de trabajo pendiente del modulo R
|
|||
- Aplicar rate limiting y limites de tamano o tipo de archivo.
|
||||
- Revisar CORS, exposicion publica, registros sensibles y permisos por operacion.
|
||||
- Sustituir la configuracion sin credenciales de n8n cuando exista el mecanismo de autenticacion definitivo.
|
||||
- Actualizar Express/`qs` para cerrar los avisos moderados pendientes de `npm audit`.
|
||||
|
||||
## 6. Pruebas automatizadas
|
||||
|
||||
|
|
|
|||
|
|
@ -2,9 +2,9 @@
|
|||
|
||||
**Proyecto:** Workspace de tools IA para empresas
|
||||
**Modulo:** RAG
|
||||
**Ultima actualizacion:** 2026-04-05
|
||||
**Ultima modificacion por:** Agente tools IA para potenciar servicios empresariales
|
||||
**Estado:** V1 operativa desplegada
|
||||
**Ultima actualizacion:** 2026-09-11
|
||||
**Ultima modificacion por:** Agente RAG 2
|
||||
**Estado:** V1 operativa; ciclo de vida implementado localmente y pendiente de despliegue
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -105,6 +105,14 @@ Pruebas funcionales ya superadas:
|
|||
- consulta conceptual sobre caracteristicas del RAG
|
||||
- consulta tecnica en modo codigo sobre la construccion de `source_id`
|
||||
|
||||
### Ciclo de vida del conocimiento
|
||||
|
||||
Cada fuente dispone de versiones controladas en PostgreSQL, mientras Qdrant conserva sus chunks y embeddings. Solo la version activa participa en las consultas.
|
||||
|
||||
Esto permite actualizar una fuente sin mezclar contenido antiguo y nuevo, detectar ingestas identicas, conservar versiones anteriores y hacer rollback sin recalcular embeddings. Los parametros de consulta actuales no cambian.
|
||||
|
||||
Esta mejora esta implementada y validada localmente. Falta desplegarla y registrar el corpus existente como version inicial antes de activarla en produccion.
|
||||
|
||||
Los pendientes vigentes, su prioridad y su estado se mantienen en [`PENDIENTES_RAG.md`](./PENDIENTES_RAG.md).
|
||||
|
||||
---
|
||||
|
|
|
|||
206
migrations/001_knowledge_lifecycle.sql
Normal file
206
migrations/001_knowledge_lifecycle.sql
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
CREATE EXTENSION IF NOT EXISTS pgcrypto;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rag_sources (
|
||||
source_id text PRIMARY KEY,
|
||||
tenant text NOT NULL DEFAULT 'default',
|
||||
source_type text NOT NULL CHECK (source_type IN ('file', 'folder')),
|
||||
source_ref text NOT NULL,
|
||||
active_version_id uuid NULL,
|
||||
needs_reingest boolean NOT NULL DEFAULT false,
|
||||
disabled_at timestamptz NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rag_source_versions (
|
||||
version_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
source_id text NOT NULL REFERENCES rag_sources(source_id) ON DELETE RESTRICT,
|
||||
version_number bigint NOT NULL CHECK (version_number > 0),
|
||||
previous_version_id uuid NULL,
|
||||
state text NOT NULL CHECK (state IN (
|
||||
'pending', 'indexing', 'review_required', 'ready', 'active',
|
||||
'superseded', 'failed', 'rejected', 'purging', 'purged'
|
||||
)),
|
||||
original_manifest_hash char(64) NOT NULL,
|
||||
source_content_hash char(64) NULL,
|
||||
processing_fingerprint char(64) NOT NULL,
|
||||
metadata_hash char(64) NOT NULL,
|
||||
tags text[] NOT NULL DEFAULT '{}',
|
||||
activate_requested boolean NOT NULL DEFAULT true,
|
||||
base_active_version_id uuid NULL,
|
||||
embedding_provider text NOT NULL,
|
||||
embedding_model text NOT NULL,
|
||||
embedding_dimensions integer NOT NULL CHECK (embedding_dimensions > 0),
|
||||
qdrant_collection text NOT NULL,
|
||||
expected_document_count integer NOT NULL DEFAULT 0 CHECK (expected_document_count >= 0),
|
||||
expected_point_count integer NOT NULL DEFAULT 0 CHECK (expected_point_count >= 0),
|
||||
verified_point_count integer NOT NULL DEFAULT 0 CHECK (verified_point_count >= 0),
|
||||
error_code text NULL,
|
||||
error_detail text NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
indexing_started_at timestamptz NULL,
|
||||
ready_at timestamptz NULL,
|
||||
activated_at timestamptz NULL,
|
||||
superseded_at timestamptz NULL,
|
||||
reviewed_at timestamptz NULL,
|
||||
reviewed_by text NULL,
|
||||
artifact_state text NOT NULL DEFAULT 'none'
|
||||
CHECK (artifact_state IN ('none', 'present', 'retention_deleting', 'retention_deleted')),
|
||||
retention_due_at timestamptz NULL,
|
||||
purged_at timestamptz NULL,
|
||||
UNIQUE (source_id, version_number),
|
||||
UNIQUE (version_id, source_id),
|
||||
FOREIGN KEY (previous_version_id, source_id)
|
||||
REFERENCES rag_source_versions(version_id, source_id)
|
||||
DEFERRABLE INITIALLY DEFERRED,
|
||||
FOREIGN KEY (base_active_version_id, source_id)
|
||||
REFERENCES rag_source_versions(version_id, source_id)
|
||||
DEFERRABLE INITIALLY DEFERRED
|
||||
);
|
||||
|
||||
DO $$
|
||||
BEGIN
|
||||
IF NOT EXISTS (
|
||||
SELECT 1 FROM pg_constraint WHERE conname = 'rag_sources_active_version_fk'
|
||||
) THEN
|
||||
ALTER TABLE rag_sources
|
||||
ADD CONSTRAINT rag_sources_active_version_fk
|
||||
FOREIGN KEY (active_version_id, source_id)
|
||||
REFERENCES rag_source_versions(version_id, source_id)
|
||||
DEFERRABLE INITIALLY DEFERRED;
|
||||
END IF;
|
||||
END $$;
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS rag_one_active_version_per_source
|
||||
ON rag_source_versions(source_id)
|
||||
WHERE state = 'active';
|
||||
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS rag_one_reusable_content_version
|
||||
ON rag_source_versions(source_id, source_content_hash, processing_fingerprint, metadata_hash)
|
||||
WHERE source_content_hash IS NOT NULL
|
||||
AND state NOT IN ('failed', 'rejected', 'purged');
|
||||
|
||||
CREATE INDEX IF NOT EXISTS rag_source_versions_source_state_idx
|
||||
ON rag_source_versions(source_id, state);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rag_version_documents (
|
||||
version_id uuid NOT NULL REFERENCES rag_source_versions(version_id) ON DELETE RESTRICT,
|
||||
document_id text NOT NULL,
|
||||
document_key text NOT NULL,
|
||||
original_hash char(64) NOT NULL,
|
||||
original_hash_kind text NOT NULL CHECK (original_hash_kind IN ('bytes', 'legacy-derived')),
|
||||
content_hash char(64) NULL,
|
||||
mime_type text NOT NULL,
|
||||
title text NOT NULL,
|
||||
extraction_method text NOT NULL DEFAULT 'native',
|
||||
index_state text NOT NULL CHECK (index_state IN ('pending', 'indexing', 'ready', 'failed')),
|
||||
chunk_count integer NOT NULL DEFAULT 0 CHECK (chunk_count >= 0),
|
||||
artifact_manifest_path text NULL,
|
||||
error_detail text NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (version_id, document_id),
|
||||
UNIQUE (version_id, document_key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS rag_version_documents_document_idx
|
||||
ON rag_version_documents(document_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rag_ingestion_attempts (
|
||||
attempt_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
source_id text NOT NULL REFERENCES rag_sources(source_id) ON DELETE RESTRICT,
|
||||
version_id uuid NULL REFERENCES rag_source_versions(version_id) ON DELETE SET NULL,
|
||||
state text NOT NULL CHECK (state IN ('received', 'processing', 'completed', 'failed')),
|
||||
input_locator text NOT NULL,
|
||||
input_hash char(64) NULL,
|
||||
error_code text NULL,
|
||||
error_detail text NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
completed_at timestamptz NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rag_system_state (
|
||||
singleton boolean PRIMARY KEY DEFAULT true CHECK (singleton),
|
||||
maintenance boolean NOT NULL DEFAULT false,
|
||||
maintenance_reason text NULL,
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
INSERT INTO rag_system_state(singleton) VALUES (true)
|
||||
ON CONFLICT (singleton) DO NOTHING;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rag_legacy_migration_batches (
|
||||
batch_id uuid PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
state text NOT NULL CHECK (state IN ('planned', 'running', 'verified', 'rolled_back', 'failed')),
|
||||
snapshot_ref text NOT NULL,
|
||||
report jsonb NOT NULL DEFAULT '{}',
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
completed_at timestamptz NULL
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS rag_legacy_migration_items (
|
||||
batch_id uuid NOT NULL REFERENCES rag_legacy_migration_batches(batch_id) ON DELETE RESTRICT,
|
||||
source_id text NOT NULL,
|
||||
version_id uuid NULL REFERENCES rag_source_versions(version_id) ON DELETE SET NULL,
|
||||
expected_point_count integer NOT NULL CHECK (expected_point_count >= 0),
|
||||
state text NOT NULL CHECK (state IN ('planned', 'migrated', 'verified', 'rolled_back', 'failed')),
|
||||
error_detail text NULL,
|
||||
PRIMARY KEY (batch_id, source_id)
|
||||
);
|
||||
|
||||
CREATE OR REPLACE FUNCTION rag_touch_updated_at()
|
||||
RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN
|
||||
NEW.updated_at = now();
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$;
|
||||
|
||||
DROP TRIGGER IF EXISTS rag_sources_touch_updated_at ON rag_sources;
|
||||
CREATE TRIGGER rag_sources_touch_updated_at
|
||||
BEFORE UPDATE ON rag_sources
|
||||
FOR EACH ROW EXECUTE FUNCTION rag_touch_updated_at();
|
||||
|
||||
CREATE OR REPLACE FUNCTION rag_validate_active_source_pointer()
|
||||
RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
DECLARE
|
||||
invalid_pointer_count integer;
|
||||
orphan_active_count integer;
|
||||
BEGIN
|
||||
SELECT count(*) INTO invalid_pointer_count
|
||||
FROM rag_sources s
|
||||
LEFT JOIN rag_source_versions v
|
||||
ON v.version_id = s.active_version_id
|
||||
AND v.source_id = s.source_id
|
||||
AND v.state = 'active'
|
||||
WHERE s.active_version_id IS NOT NULL
|
||||
AND v.version_id IS NULL;
|
||||
|
||||
IF invalid_pointer_count > 0 THEN
|
||||
RAISE EXCEPTION 'rag_sources.active_version_id must point to an active version in the same source';
|
||||
END IF;
|
||||
|
||||
SELECT count(*) INTO orphan_active_count
|
||||
FROM rag_source_versions v
|
||||
JOIN rag_sources s ON s.source_id = v.source_id
|
||||
WHERE v.state = 'active'
|
||||
AND s.active_version_id IS DISTINCT FROM v.version_id;
|
||||
|
||||
IF orphan_active_count > 0 THEN
|
||||
RAISE EXCEPTION 'every active rag_source_versions row must be the source active_version_id';
|
||||
END IF;
|
||||
|
||||
RETURN NULL;
|
||||
END;
|
||||
$$;
|
||||
|
||||
DROP TRIGGER IF EXISTS rag_sources_validate_active_pointer ON rag_sources;
|
||||
CREATE CONSTRAINT TRIGGER rag_sources_validate_active_pointer
|
||||
AFTER INSERT OR UPDATE ON rag_sources
|
||||
DEFERRABLE INITIALLY DEFERRED
|
||||
FOR EACH ROW EXECUTE FUNCTION rag_validate_active_source_pointer();
|
||||
|
||||
DROP TRIGGER IF EXISTS rag_versions_validate_active_pointer ON rag_source_versions;
|
||||
CREATE CONSTRAINT TRIGGER rag_versions_validate_active_pointer
|
||||
AFTER INSERT OR UPDATE ON rag_source_versions
|
||||
DEFERRABLE INITIALLY DEFERRED
|
||||
FOR EACH ROW EXECUTE FUNCTION rag_validate_active_source_pointer();
|
||||
509
package-lock.json
generated
509
package-lock.json
generated
|
|
@ -9,12 +9,13 @@
|
|||
"version": "0.1.0",
|
||||
"dependencies": {
|
||||
"@qdrant/js-client-rest": "^1.15.0",
|
||||
"adm-zip": "^0.5.17",
|
||||
"adm-zip": "^0.6.1",
|
||||
"dotenv": "^16.4.5",
|
||||
"express": "^4.21.2",
|
||||
"multer": "^2.0.0",
|
||||
"multer": "^2.3.0",
|
||||
"openai": "^4.104.0",
|
||||
"pdf-parse": "^1.1.1"
|
||||
"pdf-parse": "^1.1.1",
|
||||
"pg": "^8.23.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/adm-zip": "^0.5.8",
|
||||
|
|
@ -22,14 +23,15 @@
|
|||
"@types/multer": "^1.4.12",
|
||||
"@types/node": "^22.15.3",
|
||||
"@types/pdf-parse": "^1.1.4",
|
||||
"@types/pg": "^8.23.1",
|
||||
"tsx": "^4.19.3",
|
||||
"typescript": "^5.8.3"
|
||||
}
|
||||
},
|
||||
"node_modules/@esbuild/aix-ppc64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.27.7.tgz",
|
||||
"integrity": "sha512-EKX3Qwmhz1eMdEJokhALr0YiD0lhQNwDqkPYyPhiSwKrh7/4KRjQc04sZ8db+5DVVnZ1LmbNDI1uAMPEUBnQPg==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/aix-ppc64/-/aix-ppc64-0.28.2.tgz",
|
||||
"integrity": "sha512-XExcO+dvLKvVtNTibSTBej1NCAbaGhWn9Ww1ZPx80qsahhPFe/8jgWP0IchNe0F3HwkU7n8ejhH8bjonqht8mQ==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
|
|
@ -44,9 +46,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/android-arm": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.27.7.tgz",
|
||||
"integrity": "sha512-jbPXvB4Yj2yBV7HUfE2KHe4GJX51QplCN1pGbYjvsyCZbQmies29EoJbkEc+vYuU5o45AfQn37vZlyXy4YJ8RQ==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-arm/-/android-arm-0.28.2.tgz",
|
||||
"integrity": "sha512-kXXoiPVVGQcnIYGOeaovwOURpniDBpSq4A03qkQ+BMQqtGG6HYap3xne9C1O1yo4TR3qxlCX5IqqmX6fFo2Lqg==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
|
|
@ -61,9 +63,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/android-arm64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.27.7.tgz",
|
||||
"integrity": "sha512-62dPZHpIXzvChfvfLJow3q5dDtiNMkwiRzPylSCfriLvZeq0a1bWChrGx/BbUbPwOrsWKMn8idSllklzBy+dgQ==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-arm64/-/android-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-5YfKeeI8qWfBZIX+u2xZC3Zlb3Os/gLS2sbEKM+I4ZOcsWmHS2WLysCcQZDAFRslDUU5Oiq44gf6PYN1vGwG5A==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
|
|
@ -78,9 +80,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/android-x64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.27.7.tgz",
|
||||
"integrity": "sha512-x5VpMODneVDb70PYV2VQOmIUUiBtY3D3mPBG8NxVk5CogneYhkR7MmM3yR/uMdITLrC1ml/NV1rj4bMJuy9MCg==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/android-x64/-/android-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-O387ite7SzUyCcy3JQX4P4bLtEA7bLLkx+esve5JHnyYfNTxcVpXZo9jhdB0lTKN44gztELTdU7nS8Nr16Fs1Q==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
|
|
@ -95,9 +97,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/darwin-arm64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.27.7.tgz",
|
||||
"integrity": "sha512-5lckdqeuBPlKUwvoCXIgI2D9/ABmPq3Rdp7IfL70393YgaASt7tbju3Ac+ePVi3KDH6N2RqePfHnXkaDtY9fkw==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/darwin-arm64/-/darwin-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-n4KqkOQrraxHJcgjM1RvwbigfQKIKJVpM7xp+KsxiyUSrRdIXnt73VhrPAx0fV44hgfmIVKjxMN9J1t5jySVkw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
|
|
@ -112,9 +114,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/darwin-x64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.27.7.tgz",
|
||||
"integrity": "sha512-rYnXrKcXuT7Z+WL5K980jVFdvVKhCHhUwid+dDYQpH+qu+TefcomiMAJpIiC2EM3Rjtq0sO3StMV/+3w3MyyqQ==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/darwin-x64/-/darwin-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-uq6suIWYP37qzGddBKPw5QEQPi6HiLGsO7UmkpfyaYNQ3D+rN6w6WfwH+nuqcGXWvawGwxOEroO4YGnFh95azw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
|
|
@ -129,9 +131,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/freebsd-arm64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.27.7.tgz",
|
||||
"integrity": "sha512-B48PqeCsEgOtzME2GbNM2roU29AMTuOIN91dsMO30t+Ydis3z/3Ngoj5hhnsOSSwNzS+6JppqWsuhTp6E82l2w==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-arm64/-/freebsd-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-n+I0BTSRIoy+d6RPKnEVwql5UwBJolytvY4mAOIEJorKlqgPII8ix6slVVrfZ5Tnj7glIZvloylbB/EJPMWEXw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
|
|
@ -146,9 +148,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/freebsd-x64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.27.7.tgz",
|
||||
"integrity": "sha512-jOBDK5XEjA4m5IJK3bpAQF9/Lelu/Z9ZcdhTRLf4cajlB+8VEhFFRjWgfy3M1O4rO2GQ/b2dLwCUGpiF/eATNQ==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/freebsd-x64/-/freebsd-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-78XJTJkvPs0kz2w61301PJjXl4g7q3JqiYMZ/M/yVI73EHBrCRTgkhu9oqG7vPqq+a/yadEW8aD+agKlk5xrmg==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
|
|
@ -163,9 +165,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-arm": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.27.7.tgz",
|
||||
"integrity": "sha512-RkT/YXYBTSULo3+af8Ib0ykH8u2MBh57o7q/DAs3lTJlyVQkgQvlrPTnjIzzRPQyavxtPtfg0EopvDyIt0j1rA==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm/-/linux-arm-0.28.2.tgz",
|
||||
"integrity": "sha512-XlDnu2q5yoqems+xay6wSAcg9DDD7K9RLKZEBOMZm3ckNpJBvOX20tSfby8KfrrhINDyv9V2YVZKY/SpoGJI8w==",
|
||||
"cpu": [
|
||||
"arm"
|
||||
],
|
||||
|
|
@ -180,9 +182,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-arm64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.27.7.tgz",
|
||||
"integrity": "sha512-RZPHBoxXuNnPQO9rvjh5jdkRmVizktkT7TCDkDmQ0W2SwHInKCAV95GRuvdSvA7w4VMwfCjUiPwDi0ZO6Nfe9A==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-arm64/-/linux-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-pW4AC0P3it8c7do9MVM4p51FzHzdM/TZrerurgRcHJ2WTa1VQ1CIq18xncfpBJw4ojkiZZrKW2yIBWBP92j6Ug==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
|
|
@ -197,9 +199,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-ia32": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.27.7.tgz",
|
||||
"integrity": "sha512-GA48aKNkyQDbd3KtkplYWT102C5sn/EZTY4XROkxONgruHPU72l+gW+FfF8tf2cFjeHaRbWpOYa/uRBz/Xq1Pg==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-ia32/-/linux-ia32-0.28.2.tgz",
|
||||
"integrity": "sha512-CYbnj78HsIeA+DhgUKgFCfvNsTHFhMMrinUrMZpDXJXKN8T3XViTZ/+wtHeVxEWY8ewSzTFN+nRmSwO2tZaLUQ==",
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
|
|
@ -214,9 +216,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-loong64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.27.7.tgz",
|
||||
"integrity": "sha512-a4POruNM2oWsD4WKvBSEKGIiWQF8fZOAsycHOt6JBpZ+JN2n2JH9WAv56SOyu9X5IqAjqSIPTaJkqN8F7XOQ5Q==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-loong64/-/linux-loong64-0.28.2.tgz",
|
||||
"integrity": "sha512-buwkd8nsph4R+ajRvw0qM5Hja/TXQow3ptzWO2EbG/cqcIkHloRrdlBtQlshyYGTNFvfkfJ5tpPLVkY4DtsPfQ==",
|
||||
"cpu": [
|
||||
"loong64"
|
||||
],
|
||||
|
|
@ -231,9 +233,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-mips64el": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.27.7.tgz",
|
||||
"integrity": "sha512-KabT5I6StirGfIz0FMgl1I+R1H73Gp0ofL9A3nG3i/cYFJzKHhouBV5VWK1CSgKvVaG4q1RNpCTR2LuTVB3fIw==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-mips64el/-/linux-mips64el-0.28.2.tgz",
|
||||
"integrity": "sha512-ZVykbDyk7519VwiNb9Lcj9m8XM6v5V9uKPvrEMkkEedVewf+0itkhahp4HDpgERXhwLRpWFypsGbG/J8s0QjJA==",
|
||||
"cpu": [
|
||||
"mips64el"
|
||||
],
|
||||
|
|
@ -248,9 +250,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-ppc64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.27.7.tgz",
|
||||
"integrity": "sha512-gRsL4x6wsGHGRqhtI+ifpN/vpOFTQtnbsupUF5R5YTAg+y/lKelYR1hXbnBdzDjGbMYjVJLJTd2OFmMewAgwlQ==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-ppc64/-/linux-ppc64-0.28.2.tgz",
|
||||
"integrity": "sha512-CAXl+Dtd9UUuJd8pKKdwh6MLm3MUMiqMPmhZ3tTSXPqfyQ3vDl6R5hZdZ/kYojK4ofXtdfSv1tFq8XzWx3heNQ==",
|
||||
"cpu": [
|
||||
"ppc64"
|
||||
],
|
||||
|
|
@ -265,9 +267,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-riscv64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.27.7.tgz",
|
||||
"integrity": "sha512-hL25LbxO1QOngGzu2U5xeXtxXcW+/GvMN3ejANqXkxZ/opySAZMrc+9LY/WyjAan41unrR3YrmtTsUpwT66InQ==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-riscv64/-/linux-riscv64-0.28.2.tgz",
|
||||
"integrity": "sha512-GeXCej4IQtU1B+QlDV8W/RRvbzI3O/Stss+/bCXv4lZls5WGRtu2a+3JkA3i4qIUlMXpcHebWpF8AkJhATowuA==",
|
||||
"cpu": [
|
||||
"riscv64"
|
||||
],
|
||||
|
|
@ -282,9 +284,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-s390x": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.27.7.tgz",
|
||||
"integrity": "sha512-2k8go8Ycu1Kb46vEelhu1vqEP+UeRVj2zY1pSuPdgvbd5ykAw82Lrro28vXUrRmzEsUV0NzCf54yARIK8r0fdw==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-s390x/-/linux-s390x-0.28.2.tgz",
|
||||
"integrity": "sha512-3H1weTYZPxt/WOhByszQZybS9w5lKzUn1FDMsgEChbHWQwHYQQRfBxgCcZvPhjHfKyJjIievvMmEUawJrdY9Dg==",
|
||||
"cpu": [
|
||||
"s390x"
|
||||
],
|
||||
|
|
@ -299,9 +301,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/linux-x64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.27.7.tgz",
|
||||
"integrity": "sha512-hzznmADPt+OmsYzw1EE33ccA+HPdIqiCRq7cQeL1Jlq2gb1+OyWBkMCrYGBJ+sxVzve2ZJEVeePbLM2iEIZSxA==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/linux-x64/-/linux-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-4xTZr1FUmSoQW4XIWmit3tzQrUTZM+N3P0XV8xROKYF50XfI7xeO90+1bZvNwxIufQ9hDQVRJH5YhgPVF8A/HQ==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
|
|
@ -316,9 +318,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/netbsd-arm64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.27.7.tgz",
|
||||
"integrity": "sha512-b6pqtrQdigZBwZxAn1UpazEisvwaIDvdbMbmrly7cDTMFnw/+3lVxxCTGOrkPVnsYIosJJXAsILG9XcQS+Yu6w==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-arm64/-/netbsd-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-sSATRjPeDBg3pdgHoQfoYBob11Kk1FGa9lui5RIHZCoCkJa9QKlvl3/vKz2usCmYYjs7ymJR/2Nnsqe+Hjt5nw==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
|
|
@ -333,9 +335,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/netbsd-x64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.27.7.tgz",
|
||||
"integrity": "sha512-OfatkLojr6U+WN5EDYuoQhtM+1xco+/6FSzJJnuWiUw5eVcicbyK3dq5EeV/QHT1uy6GoDhGbFpprUiHUYggrw==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/netbsd-x64/-/netbsd-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-lqnzCV+mM0gIADaKihiCg6ifgfU2L3h5E33rNQBN1Y4MaVGnzryzmvvf7UHxprpQdE8hpqLolJ9Rl+SkIRDpyw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
|
|
@ -350,9 +352,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/openbsd-arm64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.27.7.tgz",
|
||||
"integrity": "sha512-AFuojMQTxAz75Fo8idVcqoQWEHIXFRbOc1TrVcFSgCZtQfSdc1RXgB3tjOn/krRHENUB4j00bfGjyl2mJrU37A==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-arm64/-/openbsd-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-AL2qJILH7lNjrDmCQDvdxMfAUIv8KMNZOvrwAQ8i8//ntL9FflhOyMJ8OZSMBb8/AWXe3/5v5S20y3zCoZWKoQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
|
|
@ -367,9 +369,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/openbsd-x64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.27.7.tgz",
|
||||
"integrity": "sha512-+A1NJmfM8WNDv5CLVQYJ5PshuRm/4cI6WMZRg1by1GwPIQPCTs1GLEUHwiiQGT5zDdyLiRM/l1G0Pv54gvtKIg==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openbsd-x64/-/openbsd-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-QtiuPytchRyC4rwUKhexJdQKvDuZ6hWloi3igqPQNUJCS1/v9EiO3UTOXR6A3FoMo4fnAKbWJdqaIwhOzh8qEw==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
|
|
@ -384,9 +386,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/openharmony-arm64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.27.7.tgz",
|
||||
"integrity": "sha512-+KrvYb/C8zA9CU/g0sR6w2RBw7IGc5J2BPnc3dYc5VJxHCSF1yNMxTV5LQ7GuKteQXZtspjFbiuW5/dOj7H4Yw==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/openharmony-arm64/-/openharmony-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-WkhYDmpTjLvGlScA1rwjRUmhl4k8oXR3cIbtqWmELgU/dFeHHlEllxDvdWcNJV9rbzCexB5vz8gtNewWLgCT7Q==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
|
|
@ -401,9 +403,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/sunos-x64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.27.7.tgz",
|
||||
"integrity": "sha512-ikktIhFBzQNt/QDyOL580ti9+5mL/YZeUPKU2ivGtGjdTYoqz6jObj6nOMfhASpS4GU4Q/Clh1QtxWAvcYKamA==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/sunos-x64/-/sunos-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-GPMSkTOtMnv2U2F8gxe4Io6qmVs+YKyp832Etqqxr0hFngmXQ3rzwytelm3GIn7T4VviRUlf3sOgBOiTdvaf7g==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
|
|
@ -418,9 +420,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/win32-arm64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.27.7.tgz",
|
||||
"integrity": "sha512-7yRhbHvPqSpRUV7Q20VuDwbjW5kIMwTHpptuUzV+AA46kiPze5Z7qgt6CLCK3pWFrHeNfDd1VKgyP4O+ng17CA==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-arm64/-/win32-arm64-0.28.2.tgz",
|
||||
"integrity": "sha512-PIhhEkE9uPBleRBrQEJpUn7MBnibZzbGzYWPmY3x+YoVg/95zbjB4CxPPOQ8l5tYYM4mMaCthF8/1DIfBQQyWQ==",
|
||||
"cpu": [
|
||||
"arm64"
|
||||
],
|
||||
|
|
@ -435,9 +437,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/win32-ia32": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.27.7.tgz",
|
||||
"integrity": "sha512-SmwKXe6VHIyZYbBLJrhOoCJRB/Z1tckzmgTLfFYOfpMAx63BJEaL9ExI8x7v0oAO3Zh6D/Oi1gVxEYr5oUCFhw==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-ia32/-/win32-ia32-0.28.2.tgz",
|
||||
"integrity": "sha512-YmJbfTlvU7Sdn9BB+4PRES4oB6pxgS37MAONj+hBr/cpXS1aBPKXxNnDbu+QCWPj0o9dgyxeq79g6c5P8KeuYA==",
|
||||
"cpu": [
|
||||
"ia32"
|
||||
],
|
||||
|
|
@ -452,9 +454,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/@esbuild/win32-x64": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.27.7.tgz",
|
||||
"integrity": "sha512-56hiAJPhwQ1R4i+21FVF7V8kSD5zZTdHcVuRFMW0hn753vVfQN8xlx4uOPT4xoGH0Z/oVATuR82AiqSTDIpaHg==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/@esbuild/win32-x64/-/win32-x64-0.28.2.tgz",
|
||||
"integrity": "sha512-5ebpxr3nWMzrL/rnUI755Jkuee0bHL/Gq0WTF9lvcpv73wAp5eu8MfBUgWK9bhWvZjj7yX8etf/8tI8Ney695g==",
|
||||
"cpu": [
|
||||
"x64"
|
||||
],
|
||||
|
|
@ -597,6 +599,18 @@
|
|||
"@types/node": "*"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/pg": {
|
||||
"version": "8.23.1",
|
||||
"resolved": "https://registry.npmjs.org/@types/pg/-/pg-8.23.1.tgz",
|
||||
"integrity": "sha512-fKVHpikPdg4GKks3JuLEhvwSyvwzF23hnabPy6DD8ljVbC7+6J5dQzdv4arV6jqq57djnMgs1HKBxX4P8aBI3A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@types/node": "*",
|
||||
"pg-protocol": "*",
|
||||
"pg-types": "^2.2.0"
|
||||
}
|
||||
},
|
||||
"node_modules/@types/qs": {
|
||||
"version": "6.15.0",
|
||||
"resolved": "https://registry.npmjs.org/@types/qs/-/qs-6.15.0.tgz",
|
||||
|
|
@ -658,12 +672,12 @@
|
|||
}
|
||||
},
|
||||
"node_modules/adm-zip": {
|
||||
"version": "0.5.17",
|
||||
"resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.5.17.tgz",
|
||||
"integrity": "sha512-+Ut8d9LLqwEvHHJl1+PIHqoyDxFgVN847JTVM3Izi3xHDWPE4UtzzXysMZQs64DMcrJfBeS/uoEP4AD3HQHnQQ==",
|
||||
"version": "0.6.1",
|
||||
"resolved": "https://registry.npmjs.org/adm-zip/-/adm-zip-0.6.1.tgz",
|
||||
"integrity": "sha512-Xwrja8nx9e5o2N1my4DsKCeKpdrnACyr1wtbPxBDgGzKzKyE9kRtBFA8mWldI+RVlD7CBZNWY/wQ2+ydwOR6kQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=12.0"
|
||||
"node": ">=14.0"
|
||||
}
|
||||
},
|
||||
"node_modules/agentkeepalive": {
|
||||
|
|
@ -697,9 +711,9 @@
|
|||
"license": "MIT"
|
||||
},
|
||||
"node_modules/body-parser": {
|
||||
"version": "1.20.4",
|
||||
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.4.tgz",
|
||||
"integrity": "sha512-ZTgYYLMOXY9qKU/57FAo8F+HA2dGX7bqGc71txDRC1rS4frdFI5R7NhluHxH6M0YItAP0sHB4uqAOcYKxO6uGA==",
|
||||
"version": "1.20.8",
|
||||
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-1.20.8.tgz",
|
||||
"integrity": "sha512-JNcyFQ64OiijEkPzUBTCe+hyPXUD/3LEldGQ6iF5LR1w00mx9o7xtDWHXBY2iItjdCFGoilOLNQbH943ut7pHA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"bytes": "~3.1.2",
|
||||
|
|
@ -710,7 +724,7 @@
|
|||
"http-errors": "~2.0.1",
|
||||
"iconv-lite": "~0.4.24",
|
||||
"on-finished": "~2.4.1",
|
||||
"qs": "~6.14.0",
|
||||
"qs": "~6.16.0",
|
||||
"raw-body": "~2.5.3",
|
||||
"type-is": "~1.6.18",
|
||||
"unpipe": "~1.0.0"
|
||||
|
|
@ -720,6 +734,22 @@
|
|||
"npm": "1.2.8000 || >= 1.4.16"
|
||||
}
|
||||
},
|
||||
"node_modules/body-parser/node_modules/qs": {
|
||||
"version": "6.16.0",
|
||||
"resolved": "https://registry.npmjs.org/qs/-/qs-6.16.0.tgz",
|
||||
"integrity": "sha512-h6fhOIaRrID2CbEY2fqs+7t+UXZo+MLAnU5gRIq85uFtdiUPCdsApMlHhXogKVM4HM2DVbIjGNTTYH2OcmP1vA==",
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"es-define-property": "^1.0.1",
|
||||
"side-channel": "^1.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.6"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/buffer-from": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
|
||||
|
|
@ -962,9 +992,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/esbuild": {
|
||||
"version": "0.27.7",
|
||||
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.27.7.tgz",
|
||||
"integrity": "sha512-IxpibTjyVnmrIQo5aqNpCgoACA/dTKLTlhMHihVHhdkxKyPO1uBBthumT0rdHmcsk9uMonIWS0m4FljWzILh3w==",
|
||||
"version": "0.28.2",
|
||||
"resolved": "https://registry.npmjs.org/esbuild/-/esbuild-0.28.2.tgz",
|
||||
"integrity": "sha512-HKVLS8dvII+xoKW9kmqxbRKrnWEXfJJr/FZhhJmiqIB0e053QNYFqOBouTMO/k5sID4MvCiUCvv8b9M4h32wIA==",
|
||||
"dev": true,
|
||||
"hasInstallScript": true,
|
||||
"license": "MIT",
|
||||
|
|
@ -975,32 +1005,32 @@
|
|||
"node": ">=18"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"@esbuild/aix-ppc64": "0.27.7",
|
||||
"@esbuild/android-arm": "0.27.7",
|
||||
"@esbuild/android-arm64": "0.27.7",
|
||||
"@esbuild/android-x64": "0.27.7",
|
||||
"@esbuild/darwin-arm64": "0.27.7",
|
||||
"@esbuild/darwin-x64": "0.27.7",
|
||||
"@esbuild/freebsd-arm64": "0.27.7",
|
||||
"@esbuild/freebsd-x64": "0.27.7",
|
||||
"@esbuild/linux-arm": "0.27.7",
|
||||
"@esbuild/linux-arm64": "0.27.7",
|
||||
"@esbuild/linux-ia32": "0.27.7",
|
||||
"@esbuild/linux-loong64": "0.27.7",
|
||||
"@esbuild/linux-mips64el": "0.27.7",
|
||||
"@esbuild/linux-ppc64": "0.27.7",
|
||||
"@esbuild/linux-riscv64": "0.27.7",
|
||||
"@esbuild/linux-s390x": "0.27.7",
|
||||
"@esbuild/linux-x64": "0.27.7",
|
||||
"@esbuild/netbsd-arm64": "0.27.7",
|
||||
"@esbuild/netbsd-x64": "0.27.7",
|
||||
"@esbuild/openbsd-arm64": "0.27.7",
|
||||
"@esbuild/openbsd-x64": "0.27.7",
|
||||
"@esbuild/openharmony-arm64": "0.27.7",
|
||||
"@esbuild/sunos-x64": "0.27.7",
|
||||
"@esbuild/win32-arm64": "0.27.7",
|
||||
"@esbuild/win32-ia32": "0.27.7",
|
||||
"@esbuild/win32-x64": "0.27.7"
|
||||
"@esbuild/aix-ppc64": "0.28.2",
|
||||
"@esbuild/android-arm": "0.28.2",
|
||||
"@esbuild/android-arm64": "0.28.2",
|
||||
"@esbuild/android-x64": "0.28.2",
|
||||
"@esbuild/darwin-arm64": "0.28.2",
|
||||
"@esbuild/darwin-x64": "0.28.2",
|
||||
"@esbuild/freebsd-arm64": "0.28.2",
|
||||
"@esbuild/freebsd-x64": "0.28.2",
|
||||
"@esbuild/linux-arm": "0.28.2",
|
||||
"@esbuild/linux-arm64": "0.28.2",
|
||||
"@esbuild/linux-ia32": "0.28.2",
|
||||
"@esbuild/linux-loong64": "0.28.2",
|
||||
"@esbuild/linux-mips64el": "0.28.2",
|
||||
"@esbuild/linux-ppc64": "0.28.2",
|
||||
"@esbuild/linux-riscv64": "0.28.2",
|
||||
"@esbuild/linux-s390x": "0.28.2",
|
||||
"@esbuild/linux-x64": "0.28.2",
|
||||
"@esbuild/netbsd-arm64": "0.28.2",
|
||||
"@esbuild/netbsd-x64": "0.28.2",
|
||||
"@esbuild/openbsd-arm64": "0.28.2",
|
||||
"@esbuild/openbsd-x64": "0.28.2",
|
||||
"@esbuild/openharmony-arm64": "0.28.2",
|
||||
"@esbuild/sunos-x64": "0.28.2",
|
||||
"@esbuild/win32-arm64": "0.28.2",
|
||||
"@esbuild/win32-ia32": "0.28.2",
|
||||
"@esbuild/win32-x64": "0.28.2"
|
||||
}
|
||||
},
|
||||
"node_modules/escape-html": {
|
||||
|
|
@ -1028,14 +1058,14 @@
|
|||
}
|
||||
},
|
||||
"node_modules/express": {
|
||||
"version": "4.22.1",
|
||||
"resolved": "https://registry.npmjs.org/express/-/express-4.22.1.tgz",
|
||||
"integrity": "sha512-F2X8g9P1X7uCPZMA3MVf9wcTqlyNp7IhH5qPCI0izhaOIYXaW9L535tGA3qmjRzpH+bZczqq7hVKxTR4NWnu+g==",
|
||||
"version": "4.22.2",
|
||||
"resolved": "https://registry.npmjs.org/express/-/express-4.22.2.tgz",
|
||||
"integrity": "sha512-IuL+Elrou2ZvCFHs18/CIzy2Nzvo25nZ1/D2eIZlz7c+QUayAcYoiM2BthCjs+EBHVpjYjcuLDAiCWgeIX3X1Q==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"accepts": "~1.3.8",
|
||||
"array-flatten": "1.1.1",
|
||||
"body-parser": "~1.20.3",
|
||||
"body-parser": "~1.20.5",
|
||||
"content-disposition": "~0.5.4",
|
||||
"content-type": "~1.0.4",
|
||||
"cookie": "~0.7.1",
|
||||
|
|
@ -1054,7 +1084,7 @@
|
|||
"parseurl": "~1.3.3",
|
||||
"path-to-regexp": "~0.1.12",
|
||||
"proxy-addr": "~2.0.7",
|
||||
"qs": "~6.14.0",
|
||||
"qs": "~6.15.1",
|
||||
"range-parser": "~1.2.1",
|
||||
"safe-buffer": "5.2.1",
|
||||
"send": "~0.19.0",
|
||||
|
|
@ -1092,16 +1122,16 @@
|
|||
}
|
||||
},
|
||||
"node_modules/form-data": {
|
||||
"version": "4.0.5",
|
||||
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.5.tgz",
|
||||
"integrity": "sha512-8RipRLol37bNs2bhoV67fiTEvdTrbMUYcFTiy3+wuuOnUog2QBHCZWXDRijWQfAkhBj2Uf5UnVaiWwA5vdd82w==",
|
||||
"version": "4.0.6",
|
||||
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.6.tgz",
|
||||
"integrity": "sha512-vKatAh4SlVfgbv+YtmhiRjhEMJsYpsG1Y2rMQtR+SVSbytsSD1YGzDIcrAJmdFec88u/+VoGmxnl+80gL1tRCQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"asynckit": "^0.4.0",
|
||||
"combined-stream": "^1.0.8",
|
||||
"es-set-tostringtag": "^2.1.0",
|
||||
"hasown": "^2.0.2",
|
||||
"mime-types": "^2.1.12"
|
||||
"hasown": "^2.0.4",
|
||||
"mime-types": "^2.1.35"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 6"
|
||||
|
|
@ -1205,19 +1235,6 @@
|
|||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/get-tsconfig": {
|
||||
"version": "4.13.7",
|
||||
"resolved": "https://registry.npmjs.org/get-tsconfig/-/get-tsconfig-4.13.7.tgz",
|
||||
"integrity": "sha512-7tN6rFgBlMgpBML5j8typ92BKFi2sFQvIdpAqLA2beia5avZDrMs0FLZiM5etShWq5irVyGcGMEA1jcDaK7A/Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"resolve-pkg-maps": "^1.0.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/privatenumber/get-tsconfig?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/gopd": {
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
|
||||
|
|
@ -1258,9 +1275,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/hasown": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz",
|
||||
"integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==",
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz",
|
||||
"integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"function-bind": "^1.1.2"
|
||||
|
|
@ -1401,9 +1418,9 @@
|
|||
"license": "MIT"
|
||||
},
|
||||
"node_modules/multer": {
|
||||
"version": "2.1.1",
|
||||
"resolved": "https://registry.npmjs.org/multer/-/multer-2.1.1.tgz",
|
||||
"integrity": "sha512-mo+QTzKlx8R7E5ylSXxWzGoXoZbOsRMpyitcht8By2KHvMbf3tjwosZ/Mu/XYU6UuJ3VZnODIrak5ZrPiPyB6A==",
|
||||
"version": "2.3.0",
|
||||
"resolved": "https://registry.npmjs.org/multer/-/multer-2.3.0.tgz",
|
||||
"integrity": "sha512-cjNbm3sttszgZeGfJR124D+jFEfkXCVAsoPBmFn9X7UxmDSFHWqE2CoEj0vrmSpuAFnqWR1Szcm9QTsiHr60Xw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"append-field": "^1.0.0",
|
||||
|
|
@ -1574,6 +1591,134 @@
|
|||
"url": "https://github.com/sponsors/mehmet-kozan"
|
||||
}
|
||||
},
|
||||
"node_modules/pg": {
|
||||
"version": "8.23.0",
|
||||
"resolved": "https://registry.npmjs.org/pg/-/pg-8.23.0.tgz",
|
||||
"integrity": "sha512-Ip2EQCngowJLGOfCwkFhPXU7/ljlhn6Rxlmy4XYfL2Y+vyRM59+8uR2xqRWKdYmbXmxCFOAmKxBuSUCdF34qLg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"pg-connection-string": "^2.14.0",
|
||||
"pg-pool": "^3.14.0",
|
||||
"pg-protocol": "^1.16.0",
|
||||
"pg-types": "2.2.0",
|
||||
"pgpass": "1.0.5"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 16.0.0"
|
||||
},
|
||||
"optionalDependencies": {
|
||||
"pg-cloudflare": "^1.4.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"pg-native": ">=3.0.1"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"pg-native": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/pg-cloudflare": {
|
||||
"version": "1.4.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-cloudflare/-/pg-cloudflare-1.4.0.tgz",
|
||||
"integrity": "sha512-Vo7z/6rrQYxpNRylp4Tlob2elzbh+N/MOQbxFVWCxS7oEx6jF53GTJFxK2WWpKuBRkmiin4Mt+xofFDjx09R0A==",
|
||||
"license": "MIT",
|
||||
"optional": true
|
||||
},
|
||||
"node_modules/pg-connection-string": {
|
||||
"version": "2.14.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-connection-string/-/pg-connection-string-2.14.0.tgz",
|
||||
"integrity": "sha512-XwWDGcLRGCXAR8F/AM5bG7Q+A3Wm2s6QeEjlOKZLlH3UYcguiqCWKyWXVag5TLTIjR7oOJUY8kcADaZgWPyLeg==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/pg-int8": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/pg-int8/-/pg-int8-1.0.1.tgz",
|
||||
"integrity": "sha512-WCtabS6t3c8SkpDBUlb1kjOs7l66xsGdKpIPZsg4wR+B3+u9UAum2odSsF9tnvxg80h4ZxLWMy4pRjOsFIqQpw==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">=4.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/pg-pool": {
|
||||
"version": "3.14.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-pool/-/pg-pool-3.14.0.tgz",
|
||||
"integrity": "sha512-gKtPkFdQPU3DksooVLi9LsjZxrsBUZIpa+7aVx+LV5pNh0KzP4Zleud2po+ConrxbuXGBJ6Hfer6hdgpIBpBaw==",
|
||||
"license": "MIT",
|
||||
"peerDependencies": {
|
||||
"pg": ">=8.0"
|
||||
}
|
||||
},
|
||||
"node_modules/pg-protocol": {
|
||||
"version": "1.16.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-protocol/-/pg-protocol-1.16.0.tgz",
|
||||
"integrity": "sha512-sILXutLVjCLjcDuOmvhX5e2Z4cS5qG/6Bu3VkpFwdf/633ElGLpEh9bgmuI5I4sqKqkifQiGyiCcx1HdtrK7tg==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/pg-types": {
|
||||
"version": "2.2.0",
|
||||
"resolved": "https://registry.npmjs.org/pg-types/-/pg-types-2.2.0.tgz",
|
||||
"integrity": "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"pg-int8": "1.0.1",
|
||||
"postgres-array": "~2.0.0",
|
||||
"postgres-bytea": "~1.0.0",
|
||||
"postgres-date": "~1.0.4",
|
||||
"postgres-interval": "^1.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=4"
|
||||
}
|
||||
},
|
||||
"node_modules/pgpass": {
|
||||
"version": "1.0.5",
|
||||
"resolved": "https://registry.npmjs.org/pgpass/-/pgpass-1.0.5.tgz",
|
||||
"integrity": "sha512-FdW9r/jQZhSeohs1Z3sI1yxFQNFvMcnmfuj4WBMUTxOrAyLMaTcE1aAMBiTlbMNaXvBCQuVi0R7hd8udDSP7ug==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"split2": "^4.1.0"
|
||||
}
|
||||
},
|
||||
"node_modules/postgres-array": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/postgres-array/-/postgres-array-2.0.0.tgz",
|
||||
"integrity": "sha512-VpZrUqU5A69eQyW2c5CA1jtLecCsN2U/bD6VilrFDWq5+5UIEVO7nazS3TEcHf1zuPYO/sqGvUvW62g86RXZuA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=4"
|
||||
}
|
||||
},
|
||||
"node_modules/postgres-bytea": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/postgres-bytea/-/postgres-bytea-1.0.1.tgz",
|
||||
"integrity": "sha512-5+5HqXnsZPE65IJZSMkZtURARZelel2oXUEO8rH83VS/hxH5vv1uHquPg5wZs8yMAfdv971IU+kcPUczi7NVBQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/postgres-date": {
|
||||
"version": "1.0.7",
|
||||
"resolved": "https://registry.npmjs.org/postgres-date/-/postgres-date-1.0.7.tgz",
|
||||
"integrity": "sha512-suDmjLVQg78nMK2UZ454hAG+OAW+HQPZ6n++TNDUX+L0+uUlLywnoxJKDou51Zm+zTCjrCl0Nq6J9C5hP9vK/Q==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/postgres-interval": {
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/postgres-interval/-/postgres-interval-1.2.0.tgz",
|
||||
"integrity": "sha512-9ZhXKM/rw350N1ovuWHbGxnGh/SNJ4cnxHiM0rxE4VN41wsg8P8zWn9hv/buK00RP4WvlOyr/RBDiptyxVbkZQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"xtend": "^4.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
},
|
||||
"node_modules/proxy-addr": {
|
||||
"version": "2.0.7",
|
||||
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
|
||||
|
|
@ -1588,12 +1733,13 @@
|
|||
}
|
||||
},
|
||||
"node_modules/qs": {
|
||||
"version": "6.14.2",
|
||||
"resolved": "https://registry.npmjs.org/qs/-/qs-6.14.2.tgz",
|
||||
"integrity": "sha512-V/yCWTTF7VJ9hIh18Ugr2zhJMP01MY7c5kh4J870L7imm6/DIzBsNLTXzMwUA3yZ5b/KBqLx8Kp3uRvd7xSe3Q==",
|
||||
"version": "6.15.3",
|
||||
"resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz",
|
||||
"integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==",
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"side-channel": "^1.1.0"
|
||||
"es-define-property": "^1.0.1",
|
||||
"side-channel": "^1.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.6"
|
||||
|
|
@ -1640,16 +1786,6 @@
|
|||
"node": ">= 6"
|
||||
}
|
||||
},
|
||||
"node_modules/resolve-pkg-maps": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/resolve-pkg-maps/-/resolve-pkg-maps-1.0.0.tgz",
|
||||
"integrity": "sha512-seS2Tj26TBVOC2NIc2rOe2y2ZO7efxITtLZcGSOnHHNOQ7CkiUBfw0Iw2ck6xkIhPwLhKNLS8BO+hEpngQlqzw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"url": "https://github.com/privatenumber/resolve-pkg-maps?sponsor=1"
|
||||
}
|
||||
},
|
||||
"node_modules/safe-buffer": {
|
||||
"version": "5.2.1",
|
||||
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
|
||||
|
|
@ -1728,14 +1864,14 @@
|
|||
"license": "ISC"
|
||||
},
|
||||
"node_modules/side-channel": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.0.tgz",
|
||||
"integrity": "sha512-ZX99e6tRweoUXqR+VBrslhda51Nh5MTQwou5tnUDgbtyM0dBgmhEDtWGP/xbKn6hqfPRHujUNwz5fy/wbbhnpw==",
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz",
|
||||
"integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"es-errors": "^1.3.0",
|
||||
"object-inspect": "^1.13.3",
|
||||
"side-channel-list": "^1.0.0",
|
||||
"object-inspect": "^1.13.4",
|
||||
"side-channel-list": "^1.0.1",
|
||||
"side-channel-map": "^1.0.1",
|
||||
"side-channel-weakmap": "^1.0.2"
|
||||
},
|
||||
|
|
@ -1747,13 +1883,13 @@
|
|||
}
|
||||
},
|
||||
"node_modules/side-channel-list": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.0.tgz",
|
||||
"integrity": "sha512-FCLHtRD/gnpCiCHEiJLOwdmFP+wzCmDEkc9y7NsYxeF4u7Btsn1ZuwgwJGxImImHicJArLP4R0yX4c2KCrMrTA==",
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",
|
||||
"integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"es-errors": "^1.3.0",
|
||||
"object-inspect": "^1.13.3"
|
||||
"object-inspect": "^1.13.4"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
|
|
@ -1799,6 +1935,15 @@
|
|||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/split2": {
|
||||
"version": "4.2.0",
|
||||
"resolved": "https://registry.npmjs.org/split2/-/split2-4.2.0.tgz",
|
||||
"integrity": "sha512-UcjcJOWknrNkF6PLX83qcHM6KHgVKNkV62Y8a5uYDVv9ydGQVwAHMKqHdJje1VTWpljG0WYpCDhrCdAOYH4TWg==",
|
||||
"license": "ISC",
|
||||
"engines": {
|
||||
"node": ">= 10.x"
|
||||
}
|
||||
},
|
||||
"node_modules/statuses": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
|
||||
|
|
@ -1841,14 +1986,13 @@
|
|||
"license": "MIT"
|
||||
},
|
||||
"node_modules/tsx": {
|
||||
"version": "4.21.0",
|
||||
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.21.0.tgz",
|
||||
"integrity": "sha512-5C1sg4USs1lfG0GFb2RLXsdpXqBSEhAaA/0kPL01wxzpMqLILNxIxIOKiILz+cdg/pLnOUxFYOR5yhHU666wbw==",
|
||||
"version": "4.23.13",
|
||||
"resolved": "https://registry.npmjs.org/tsx/-/tsx-4.23.13.tgz",
|
||||
"integrity": "sha512-BL5MGkRln6aDYhb0xbQlEAGw743BaZYWdbWtdJOBriYJboKgUUYCadFp2/FpBBZquBC/ezNBn7wMMPx7FDZUDw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"esbuild": "~0.27.0",
|
||||
"get-tsconfig": "^4.7.5"
|
||||
"esbuild": "~0.28.0"
|
||||
},
|
||||
"bin": {
|
||||
"tsx": "dist/cli.mjs"
|
||||
|
|
@ -1893,9 +2037,9 @@
|
|||
}
|
||||
},
|
||||
"node_modules/undici": {
|
||||
"version": "6.24.1",
|
||||
"resolved": "https://registry.npmjs.org/undici/-/undici-6.24.1.tgz",
|
||||
"integrity": "sha512-sC+b0tB1whOCzbtlx20fx3WgCXwkW627p4EA9uM+/tNNPkSS+eSEld6pAs9nDv7WbY1UUljBMYPtu9BCOrCWKA==",
|
||||
"version": "6.28.1",
|
||||
"resolved": "https://registry.npmjs.org/undici/-/undici-6.28.1.tgz",
|
||||
"integrity": "sha512-zWpdTVD54H48CIybL0rWQ3ukpb9d23wM7eH5RtfdmeP70cWHNjtfo7P4vZX+5CoDcO53J4Pu5uXp7lNfjc6DRA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18.17"
|
||||
|
|
@ -1964,6 +2108,15 @@
|
|||
"tr46": "~0.0.3",
|
||||
"webidl-conversions": "^3.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/xtend": {
|
||||
"version": "4.0.2",
|
||||
"resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz",
|
||||
"integrity": "sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.4"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
13
package.json
13
package.json
|
|
@ -6,17 +6,21 @@
|
|||
"scripts": {
|
||||
"dev": "tsx watch src/server.ts",
|
||||
"build": "tsc -p tsconfig.json",
|
||||
"start": "node dist/server.js",
|
||||
"start": "node dist/modules/catalog/migrations.js && node dist/server.js",
|
||||
"migrate:lifecycle": "node dist/modules/catalog/migrations.js",
|
||||
"migrate:legacy:lifecycle": "node dist/scripts/migrate-legacy-lifecycle.js",
|
||||
"test": "NODE_ENV=test tsx --test tests/**/*.test.ts",
|
||||
"check": "tsc --noEmit -p tsconfig.json"
|
||||
},
|
||||
"dependencies": {
|
||||
"@qdrant/js-client-rest": "^1.15.0",
|
||||
"adm-zip": "^0.5.17",
|
||||
"adm-zip": "^0.6.1",
|
||||
"dotenv": "^16.4.5",
|
||||
"express": "^4.21.2",
|
||||
"multer": "^2.0.0",
|
||||
"multer": "^2.3.0",
|
||||
"openai": "^4.104.0",
|
||||
"pdf-parse": "^1.1.1"
|
||||
"pdf-parse": "^1.1.1",
|
||||
"pg": "^8.23.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/adm-zip": "^0.5.8",
|
||||
|
|
@ -24,6 +28,7 @@
|
|||
"@types/multer": "^1.4.12",
|
||||
"@types/node": "^22.15.3",
|
||||
"@types/pdf-parse": "^1.1.4",
|
||||
"@types/pg": "^8.23.1",
|
||||
"tsx": "^4.19.3",
|
||||
"typescript": "^5.8.3"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -38,6 +38,8 @@ const btnClearUpload = document.getElementById("btnClearUpload");
|
|||
const uploadStatusText = document.getElementById("uploadStatusText");
|
||||
const ingestMode = document.getElementById("ingestMode");
|
||||
const ingestTags = document.getElementById("ingestTags");
|
||||
const ingestActivate = document.getElementById("ingestActivate");
|
||||
const ingestExpectedActiveVersionId = document.getElementById("ingestExpectedActiveVersionId");
|
||||
const ingestModeHint = document.getElementById("ingestModeHint");
|
||||
|
||||
const cleanupScopeSelect = document.getElementById("cleanupScopeSelect");
|
||||
|
|
@ -80,7 +82,17 @@ function splitTags(value) {
|
|||
|
||||
function buildScopeLabel(scope) {
|
||||
const modes = scope.chunkModes.join(", ") || "sin modo";
|
||||
return `${scope.sourceRef} [${modes}]`;
|
||||
const version = scope.activeVersionNumber ? ` v${scope.activeVersionNumber}` : "";
|
||||
const state = scope.state ? ` ${scope.state}` : "";
|
||||
return `${scope.sourceRef}${version}${state} [${modes}]`;
|
||||
}
|
||||
|
||||
function normalizeExpectedActiveVersionInput() {
|
||||
const value = ingestExpectedActiveVersionId.value.trim();
|
||||
if (!value) {
|
||||
return undefined;
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
let previousSourceRef = ""; // Para guardar el valor original de SourceRef
|
||||
|
|
@ -363,6 +375,11 @@ ingestButton.addEventListener("click", async () => {
|
|||
formData.append("file", ingestUploadFile.files[0]);
|
||||
formData.append("mode", ingestMode.value);
|
||||
formData.append("tags", splitTags(ingestTags.value).join(","));
|
||||
formData.append("activate", ingestActivate.value);
|
||||
const expectedActiveVersionId = normalizeExpectedActiveVersionInput();
|
||||
if (expectedActiveVersionId !== undefined) {
|
||||
formData.append("expectedActiveVersionId", expectedActiveVersionId);
|
||||
}
|
||||
if (ingestScopeMode.value === "custom" && ingestSourceId.value.trim()) {
|
||||
formData.append("sourceId", ingestSourceId.value.trim());
|
||||
}
|
||||
|
|
@ -414,6 +431,11 @@ ingestButton.addEventListener("click", async () => {
|
|||
formData.append("isZipFolder", "true");
|
||||
formData.append("mode", ingestMode.value);
|
||||
formData.append("tags", splitTags(ingestTags.value).join(","));
|
||||
formData.append("activate", ingestActivate.value);
|
||||
const expectedActiveVersionId = normalizeExpectedActiveVersionInput();
|
||||
if (expectedActiveVersionId !== undefined) {
|
||||
formData.append("expectedActiveVersionId", expectedActiveVersionId);
|
||||
}
|
||||
if (ingestScopeMode.value === "custom" && ingestSourceId.value.trim()) {
|
||||
formData.append("sourceId", ingestSourceId.value.trim());
|
||||
}
|
||||
|
|
@ -431,7 +453,9 @@ ingestButton.addEventListener("click", async () => {
|
|||
sourceType: ingestSourceType.value,
|
||||
sourceRef: ingestSourceRef.value,
|
||||
mode: ingestMode.value,
|
||||
tags: splitTags(ingestTags.value)
|
||||
tags: splitTags(ingestTags.value),
|
||||
activate: ingestActivate.value === "true",
|
||||
expectedActiveVersionId: normalizeExpectedActiveVersionInput() === "null" ? null : normalizeExpectedActiveVersionInput()
|
||||
});
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -75,8 +75,17 @@
|
|||
<label>Tags (coma separada)
|
||||
<input id="ingestTags" value="workspace,global-docs" />
|
||||
</label>
|
||||
<label>Activar al terminar
|
||||
<select id="ingestActivate">
|
||||
<option value="true">si, activar version</option>
|
||||
<option value="false">no, dejar en ready</option>
|
||||
</select>
|
||||
</label>
|
||||
<label>Version activa esperada
|
||||
<input id="ingestExpectedActiveVersionId" placeholder="uuid actual o null para primera activacion" />
|
||||
</label>
|
||||
</div>
|
||||
<p class="helper" id="ingestModeHint">Si seleccionas un archivo local, el playground usara upload directo y podras aislarlo con un `sourceId` propio para no mezclarlo con otros scopes.</p>
|
||||
<p class="helper" id="ingestModeHint">Si el ciclo de vida esta activo, una ingesta con activacion exige indicar la version activa esperada; usa <code>null</code> para la primera activacion de una fuente.</p>
|
||||
<div class="actions">
|
||||
<button id="ingestButton">Lanzar ingesta</button>
|
||||
</div>
|
||||
|
|
@ -87,7 +96,7 @@
|
|||
<section id="tab-cleanup" class="tab-panel">
|
||||
<article class="panel">
|
||||
<h2>Limpieza controlada</h2>
|
||||
<p class="helper">Permite eliminar contexto ya ingerido del RAG antes de reingestar una fuente, para evitar duplicados y versiones mezcladas.</p>
|
||||
<p class="helper">Esta operacion queda como compatibilidad legacy. Con ciclo de vida, usa la purga administrativa de versiones no activas.</p>
|
||||
<div class="grid single-grid">
|
||||
<label>Scope disponible a limpiar
|
||||
<select id="cleanupScopeSelect">
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ export const openApiDocument = {
|
|||
{ name: "Discovery", description: "API contract and browser playground." },
|
||||
{ name: "Status", description: "Service capabilities and available resources." },
|
||||
{ name: "Ingestion", description: "Knowledge ingestion and cleanup." },
|
||||
{ name: "Lifecycle", description: "Knowledge source versions, activation, rollback, and purge." },
|
||||
{ name: "Retrieval", description: "Context retrieval and model-backed answers." },
|
||||
{ name: "Evaluation", description: "Evaluation log capture and review." }
|
||||
],
|
||||
|
|
@ -70,7 +71,8 @@ export const openApiDocument = {
|
|||
tags: ["Status"],
|
||||
summary: "Check service health and capabilities",
|
||||
responses: {
|
||||
"200": jsonResponse("Current service status.", ref("HealthResponse"))
|
||||
"200": jsonResponse("Current service status.", ref("HealthResponse")),
|
||||
"503": jsonResponse("One or more required services are unavailable.", ref("HealthResponse"))
|
||||
}
|
||||
}
|
||||
},
|
||||
|
|
@ -83,10 +85,117 @@ export const openApiDocument = {
|
|||
type: "array",
|
||||
items: ref("AvailableScope")
|
||||
}),
|
||||
"503": serverError,
|
||||
"500": serverError
|
||||
}
|
||||
}
|
||||
},
|
||||
"/sources/{sourceId}": {
|
||||
get: {
|
||||
tags: ["Lifecycle"],
|
||||
summary: "Get a catalog source",
|
||||
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
|
||||
responses: {
|
||||
"200": jsonResponse("Catalog source.", ref("AvailableScope")),
|
||||
"404": jsonResponse("Source not found.", ref("Error")),
|
||||
"503": serverError
|
||||
}
|
||||
}
|
||||
},
|
||||
"/sources/{sourceId}/versions": {
|
||||
get: {
|
||||
tags: ["Lifecycle"],
|
||||
summary: "List source versions",
|
||||
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
|
||||
responses: {
|
||||
"200": jsonResponse("Source versions.", { type: "array", items: ref("SourceVersion") }),
|
||||
"503": serverError
|
||||
}
|
||||
}
|
||||
},
|
||||
"/sources/{sourceId}/versions/{versionId}": {
|
||||
get: {
|
||||
tags: ["Lifecycle"],
|
||||
summary: "Get one source version",
|
||||
parameters: [
|
||||
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
|
||||
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
|
||||
],
|
||||
responses: {
|
||||
"200": jsonResponse("Source version.", ref("SourceVersion")),
|
||||
"404": jsonResponse("Version not found.", ref("Error")),
|
||||
"503": serverError
|
||||
}
|
||||
},
|
||||
delete: {
|
||||
tags: ["Lifecycle"],
|
||||
summary: "Purge a non-active source version",
|
||||
security: [{ bearerAuth: [] }],
|
||||
parameters: [
|
||||
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
|
||||
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
|
||||
],
|
||||
responses: {
|
||||
"200": jsonResponse("Version purged.", ref("PurgeResponse")),
|
||||
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
|
||||
"409": jsonResponse("Active versions cannot be purged.", ref("Error")),
|
||||
"503": serverError
|
||||
}
|
||||
}
|
||||
},
|
||||
"/sources/{sourceId}/versions/{versionId}/activate": {
|
||||
post: {
|
||||
tags: ["Lifecycle"],
|
||||
summary: "Activate a ready or superseded version",
|
||||
security: [{ bearerAuth: [] }],
|
||||
parameters: [
|
||||
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
|
||||
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
|
||||
],
|
||||
requestBody: { required: true, content: jsonContent(ref("ActivateVersionRequest")) },
|
||||
responses: {
|
||||
"200": jsonResponse("Activated source version.", ref("SourceVersion")),
|
||||
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
|
||||
"409": jsonResponse("Active version precondition failed.", ref("Error")),
|
||||
"503": serverError
|
||||
}
|
||||
}
|
||||
},
|
||||
"/sources/{sourceId}/rollback": {
|
||||
post: {
|
||||
tags: ["Lifecycle"],
|
||||
summary: "Rollback a source to a previous version",
|
||||
security: [{ bearerAuth: [] }],
|
||||
parameters: [{ name: "sourceId", in: "path", required: true, schema: { type: "string" } }],
|
||||
requestBody: { required: true, content: jsonContent(ref("RollbackRequest")) },
|
||||
responses: {
|
||||
"200": jsonResponse("Rollback activated source version.", ref("SourceVersion")),
|
||||
"400": jsonResponse("Required rollback fields are missing.", ref("Error")),
|
||||
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
|
||||
"409": jsonResponse("Active version precondition failed.", ref("Error")),
|
||||
"503": serverError
|
||||
}
|
||||
}
|
||||
},
|
||||
"/sources/{sourceId}/versions/{versionId}/retry": {
|
||||
post: {
|
||||
tags: ["Lifecycle"],
|
||||
summary: "Explicitly retry a failed or orphaned version by re-submitting a durable source",
|
||||
security: [{ bearerAuth: [] }],
|
||||
parameters: [
|
||||
{ name: "sourceId", in: "path", required: true, schema: { type: "string" } },
|
||||
{ name: "versionId", in: "path", required: true, schema: { type: "string", format: "uuid" } }
|
||||
],
|
||||
requestBody: { required: false, content: jsonContent(ref("RetryVersionRequest")) },
|
||||
responses: {
|
||||
"201": jsonResponse("Replacement version created by explicit retry.", ref("IngestResponse")),
|
||||
"202": jsonResponse("Retry instructions returned because no durable source payload was supplied.", ref("RetryVersionResponse")),
|
||||
"401": jsonResponse("Missing or invalid lifecycle admin token.", ref("Error")),
|
||||
"409": jsonResponse("Version state is not retryable.", ref("Error")),
|
||||
"503": serverError
|
||||
}
|
||||
}
|
||||
},
|
||||
"/models/answer": {
|
||||
get: {
|
||||
tags: ["Status"],
|
||||
|
|
@ -112,7 +221,12 @@ export const openApiDocument = {
|
|||
})
|
||||
},
|
||||
responses: {
|
||||
"202": jsonResponse("Ingestion completed and accepted.", ref("IngestResponse")),
|
||||
"201": jsonResponse("Lifecycle ingestion completed and committed.", ref("IngestResponse")),
|
||||
"202": jsonResponse("Legacy ingestion completed and accepted while lifecycle enforcement is disabled.", ref("IngestResponse")),
|
||||
"400": jsonResponse("Invalid lifecycle activation request.", ref("Error")),
|
||||
"409": jsonResponse("Concurrent active version change or reusable version in progress.", ref("Error")),
|
||||
"422": jsonResponse("Empty or unsupported source.", ref("Error")),
|
||||
"503": serverError,
|
||||
"500": serverError
|
||||
}
|
||||
}
|
||||
|
|
@ -130,8 +244,12 @@ export const openApiDocument = {
|
|||
}
|
||||
},
|
||||
responses: {
|
||||
"202": jsonResponse("Upload ingested successfully.", ref("UploadIngestResponse")),
|
||||
"201": jsonResponse("Lifecycle upload ingested successfully.", ref("UploadIngestResponse")),
|
||||
"202": jsonResponse("Legacy upload ingested successfully while lifecycle enforcement is disabled.", ref("UploadIngestResponse")),
|
||||
"400": jsonResponse("The file field is missing.", ref("Error"), { ok: false, error: "Missing file upload" }),
|
||||
"409": jsonResponse("Concurrent active version change or reusable version in progress.", ref("Error")),
|
||||
"422": jsonResponse("Empty or unsupported source.", ref("Error")),
|
||||
"503": serverError,
|
||||
"500": serverError
|
||||
}
|
||||
}
|
||||
|
|
@ -150,6 +268,9 @@ export const openApiDocument = {
|
|||
responses: {
|
||||
"200": jsonResponse("Matching chunks deleted.", ref("CleanupResponse")),
|
||||
"400": jsonResponse("Scope is missing or unsafe for cleanup.", ref("Error")),
|
||||
"401": jsonResponse("Missing or invalid lifecycle admin token when lifecycle enforcement is active.", ref("Error")),
|
||||
"410": jsonResponse("Legacy cleanup is disabled while lifecycle enforcement is active.", ref("Error")),
|
||||
"503": serverError,
|
||||
"500": serverError
|
||||
}
|
||||
}
|
||||
|
|
@ -272,6 +393,9 @@ export const openApiDocument = {
|
|||
}
|
||||
},
|
||||
components: {
|
||||
securitySchemes: {
|
||||
bearerAuth: { type: "http", scheme: "bearer" }
|
||||
},
|
||||
schemas: {
|
||||
Error: {
|
||||
type: "object",
|
||||
|
|
@ -289,6 +413,10 @@ export const openApiDocument = {
|
|||
tags: { type: "array", items: { type: "string" } }
|
||||
}
|
||||
},
|
||||
SourceVersionState: {
|
||||
type: "string",
|
||||
enum: ["pending", "indexing", "review_required", "ready", "active", "superseded", "failed", "rejected", "purging", "purged"]
|
||||
},
|
||||
AvailableScope: {
|
||||
type: "object",
|
||||
required: ["sourceId", "sourceRef", "chunkModes", "tags"],
|
||||
|
|
@ -296,7 +424,12 @@ export const openApiDocument = {
|
|||
sourceId: { type: "string" },
|
||||
sourceRef: { type: "string" },
|
||||
chunkModes: { type: "array", items: { $ref: "#/components/schemas/ChunkMode" } },
|
||||
tags: { type: "array", items: { type: "string" } }
|
||||
tags: { type: "array", items: { type: "string" } },
|
||||
activeVersionId: { type: ["string", "null"], format: "uuid" },
|
||||
activeVersionNumber: { type: ["integer", "null"] },
|
||||
state: { ...ref("SourceVersionState"), nullable: true },
|
||||
needsReingest: { type: "boolean" },
|
||||
updatedAt: { type: "string", format: "date-time" }
|
||||
}
|
||||
},
|
||||
ChunkMode: { type: "string", enum: ["documental", "codigo", "auto"] },
|
||||
|
|
@ -310,18 +443,27 @@ export const openApiDocument = {
|
|||
sourceRef: { type: "string" },
|
||||
readPath: { type: "string", description: "Internal filesystem path used when the logical sourceRef differs from the read location." },
|
||||
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
|
||||
tags: { type: "array", items: { type: "string" } }
|
||||
tags: { type: "array", items: { type: "string" } },
|
||||
activate: { type: "boolean", default: true },
|
||||
expectedActiveVersionId: { type: ["string", "null"], format: "uuid", description: "Required when activate is true. Use null for first activation." }
|
||||
}
|
||||
},
|
||||
IngestResponse: {
|
||||
type: "object",
|
||||
required: ["accepted", "source", "filesDiscovered", "documentsProcessed", "chunksStored", "collectionName"],
|
||||
required: ["accepted", "sourceId", "filesDiscovered", "documentsProcessed", "chunksStored", "collectionName"],
|
||||
properties: {
|
||||
accepted: { type: "boolean", const: true },
|
||||
source: ref("IngestRequest"),
|
||||
sourceId: { type: "string" },
|
||||
versionId: { type: "string", format: "uuid" },
|
||||
versionNumber: { type: "integer" },
|
||||
state: ref("SourceVersionState"),
|
||||
previousVersionId: { type: ["string", "null"], format: "uuid" },
|
||||
filesDiscovered: { type: "integer" },
|
||||
documentsProcessed: { type: "integer" },
|
||||
chunksStored: { type: "integer" },
|
||||
activated: { type: "boolean" },
|
||||
noOp: { type: "boolean" },
|
||||
collectionName: { type: "string" }
|
||||
}
|
||||
},
|
||||
|
|
@ -333,7 +475,9 @@ export const openApiDocument = {
|
|||
sourceId: { type: "string" },
|
||||
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
|
||||
tags: { type: "string", description: "Comma-separated tags." },
|
||||
isZipFolder: { type: "string", enum: ["true", "false"], default: "false" }
|
||||
isZipFolder: { type: "string", enum: ["true", "false"], default: "false" },
|
||||
activate: { type: "string", enum: ["true", "false"], default: "true" },
|
||||
expectedActiveVersionId: { type: "string", description: "UUID or literal null when activate is true." }
|
||||
}
|
||||
},
|
||||
UploadIngestResponse: {
|
||||
|
|
@ -355,7 +499,7 @@ export const openApiDocument = {
|
|||
type: "object",
|
||||
required: ["ok", "deleted"],
|
||||
properties: {
|
||||
ok: { type: "boolean", const: true },
|
||||
ok: { type: "boolean" },
|
||||
deleted: { type: "integer" }
|
||||
}
|
||||
},
|
||||
|
|
@ -389,6 +533,8 @@ export const openApiDocument = {
|
|||
chunkId: { type: "string" },
|
||||
documentId: { type: "string" },
|
||||
sourceId: { type: "string" },
|
||||
sourceVersionId: { type: "string", format: "uuid" },
|
||||
sourceVersionNumber: { type: "integer" },
|
||||
title: { type: "string" },
|
||||
sectionTitle: { type: "string" },
|
||||
content: { type: "string" },
|
||||
|
|
@ -433,6 +579,8 @@ export const openApiDocument = {
|
|||
properties: {
|
||||
chunkId: { type: "string" },
|
||||
documentId: { type: "string" },
|
||||
sourceVersionId: { type: "string", format: "uuid" },
|
||||
sourceVersionNumber: { type: "integer" },
|
||||
title: { type: "string" },
|
||||
sectionTitle: { type: "string" },
|
||||
startLine: { type: "integer" },
|
||||
|
|
@ -571,18 +719,87 @@ export const openApiDocument = {
|
|||
},
|
||||
HealthResponse: {
|
||||
type: "object",
|
||||
required: ["ok", "service", "environment", "embeddings", "answer", "vectorStore", "parsers", "chunking"],
|
||||
required: ["ok", "service", "environment", "embeddings", "answer", "vectorStore", "postgres", "knowledgeLifecycle", "parsers", "chunking"],
|
||||
properties: {
|
||||
ok: { type: "boolean", const: true },
|
||||
ok: { type: "boolean" },
|
||||
service: { type: "string", const: "rag" },
|
||||
environment: { type: "string" },
|
||||
embeddings: ref("ProviderModel"),
|
||||
answer: ref("ProviderModel"),
|
||||
vectorStore: { type: "object", additionalProperties: true },
|
||||
postgres: { type: "object", additionalProperties: true },
|
||||
knowledgeLifecycle: { type: "object", additionalProperties: true },
|
||||
parsers: { type: "array", items: { type: "string" } },
|
||||
chunking: { type: "object", additionalProperties: true }
|
||||
}
|
||||
},
|
||||
SourceVersion: {
|
||||
type: "object",
|
||||
required: ["versionId", "sourceId", "versionNumber", "state", "tags", "embeddingProvider", "embeddingModel", "embeddingDimensions", "expectedDocumentCount", "expectedPointCount", "verifiedPointCount", "qdrantCollection"],
|
||||
properties: {
|
||||
versionId: { type: "string", format: "uuid" },
|
||||
sourceId: { type: "string" },
|
||||
versionNumber: { type: "integer" },
|
||||
previousVersionId: { type: ["string", "null"], format: "uuid" },
|
||||
state: ref("SourceVersionState"),
|
||||
tags: { type: "array", items: { type: "string" } },
|
||||
sourceContentHash: { type: ["string", "null"] },
|
||||
processingFingerprint: { type: "string" },
|
||||
metadataHash: { type: "string" },
|
||||
embeddingProvider: { type: "string" },
|
||||
embeddingModel: { type: "string" },
|
||||
embeddingDimensions: { type: "integer" },
|
||||
expectedDocumentCount: { type: "integer" },
|
||||
expectedPointCount: { type: "integer" },
|
||||
verifiedPointCount: { type: "integer" },
|
||||
qdrantCollection: { type: "string" }
|
||||
}
|
||||
},
|
||||
ActivateVersionRequest: {
|
||||
type: "object",
|
||||
required: ["expectedActiveVersionId"],
|
||||
properties: { expectedActiveVersionId: { type: ["string", "null"], format: "uuid" } }
|
||||
},
|
||||
RollbackRequest: {
|
||||
type: "object",
|
||||
required: ["targetVersionId", "expectedActiveVersionId"],
|
||||
properties: {
|
||||
targetVersionId: { type: "string", format: "uuid" },
|
||||
expectedActiveVersionId: { type: "string", format: "uuid" }
|
||||
}
|
||||
},
|
||||
RetryVersionRequest: {
|
||||
type: "object",
|
||||
properties: {
|
||||
sourceType: { type: "string", enum: ["file", "folder"] },
|
||||
sourceRef: { type: "string" },
|
||||
readPath: { type: "string" },
|
||||
mode: { type: "string", enum: ["mechanical", "interactive"], default: "mechanical" },
|
||||
tags: { type: "array", items: { type: "string" } },
|
||||
activate: { type: "boolean", default: true },
|
||||
expectedActiveVersionId: { type: ["string", "null"], format: "uuid" }
|
||||
}
|
||||
},
|
||||
RetryVersionResponse: {
|
||||
type: "object",
|
||||
required: ["ok", "retryRequired", "sourceId", "versionId", "message"],
|
||||
properties: {
|
||||
ok: { type: "boolean", const: true },
|
||||
retryRequired: { type: "boolean", const: true },
|
||||
sourceId: { type: "string" },
|
||||
versionId: { type: "string", format: "uuid" },
|
||||
message: { type: "string" }
|
||||
}
|
||||
},
|
||||
PurgeResponse: {
|
||||
type: "object",
|
||||
required: ["ok", "purged", "versionId"],
|
||||
properties: {
|
||||
ok: { type: "boolean", const: true },
|
||||
purged: { type: "boolean", const: true },
|
||||
versionId: { type: "string", format: "uuid" }
|
||||
}
|
||||
},
|
||||
ProviderModel: {
|
||||
type: "object",
|
||||
required: ["provider", "model"],
|
||||
|
|
|
|||
238
src/app.ts
238
src/app.ts
|
|
@ -4,6 +4,7 @@ import type { Request } from "express";
|
|||
import { writeFile, unlink, mkdtemp, rm } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { timingSafeEqual } from "node:crypto";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import AdmZip from "adm-zip";
|
||||
import { buildApiHelp, openApiDocument } from "./api/openapi.js";
|
||||
|
|
@ -16,6 +17,9 @@ import { documentalChunkingPolicy } from "./modules/process/chunking.js";
|
|||
import { RetrieveService } from "./modules/retrieve/service.js";
|
||||
import { supportedParserExtensions } from "./modules/parsers/parser-registry.js";
|
||||
import { QdrantVectorStoreClient } from "./modules/vectorstore/client.js";
|
||||
import { getCatalogPool } from "./modules/catalog/client.js";
|
||||
import { KnowledgeLifecycleReconciler } from "./modules/catalog/reconciler.js";
|
||||
import { CatalogError, CatalogRepository } from "./modules/catalog/repository.js";
|
||||
import type { ChatMessage, ChunkMode, RetrieveIntent, RetrieveScope } from "./shared/types/rag.js";
|
||||
|
||||
type UploadRequest = Request & {
|
||||
|
|
@ -30,10 +34,50 @@ export function createApp() {
|
|||
const upload = multer({ storage: multer.memoryStorage() });
|
||||
const embeddingProvider = new OpenRouterEmbeddingProvider();
|
||||
const vectorStore = new QdrantVectorStoreClient();
|
||||
const catalogPool = getCatalogPool();
|
||||
const catalog = catalogPool ? new CatalogRepository(catalogPool) : undefined;
|
||||
const reconciler = new KnowledgeLifecycleReconciler(catalog, vectorStore);
|
||||
const evaluationLogs = new EvaluationLogService(embeddingProvider);
|
||||
const ingestService = new IngestService(embeddingProvider, vectorStore);
|
||||
const retrieveService = new RetrieveService(embeddingProvider, vectorStore);
|
||||
const ingestService = new IngestService(embeddingProvider, vectorStore, catalog);
|
||||
const retrieveService = new RetrieveService(embeddingProvider, vectorStore, catalog);
|
||||
const answerService = new AnswerService(retrieveService);
|
||||
reconciler.start();
|
||||
|
||||
function sendError(res: express.Response, error: unknown, fallback: string) {
|
||||
const upstreamUnavailable = error instanceof Error && /(ECONNREFUSED|ETIMEDOUT|timeout|connection|database|postgres|qdrant)/i.test(error.message);
|
||||
const statusCode = error instanceof CatalogError ? error.statusCode : upstreamUnavailable ? 503 : 500;
|
||||
const body = {
|
||||
ok: false,
|
||||
error: error instanceof Error ? error.message : fallback,
|
||||
code: error instanceof CatalogError ? error.code : undefined
|
||||
};
|
||||
res.status(statusCode).json(body);
|
||||
}
|
||||
|
||||
function requireLifecycleAdmin(req: express.Request, res: express.Response): boolean {
|
||||
const header = req.header("authorization") ?? "";
|
||||
const token = header.startsWith("Bearer ") ? header.slice("Bearer ".length) : "";
|
||||
if (!env.lifecycleAdminToken || !token) {
|
||||
res.status(401).json({ ok: false, error: "Lifecycle admin token is required" });
|
||||
return false;
|
||||
}
|
||||
const expected = Buffer.from(env.lifecycleAdminToken);
|
||||
const actual = Buffer.from(token);
|
||||
const valid = expected.length === actual.length && timingSafeEqual(expected, actual);
|
||||
if (!valid) {
|
||||
res.status(401).json({ ok: false, error: "Lifecycle admin token is invalid" });
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
function requireCatalog(res: express.Response): CatalogRepository | undefined {
|
||||
if (!catalog) {
|
||||
res.status(503).json({ ok: false, error: "Knowledge catalog is not configured", code: "CATALOG_UNAVAILABLE" });
|
||||
return undefined;
|
||||
}
|
||||
return catalog;
|
||||
}
|
||||
|
||||
function needsContextLog(summary?: string, itemsCount = 0, answer?: string) {
|
||||
if (itemsCount === 0) {
|
||||
|
|
@ -76,8 +120,11 @@ export function createApp() {
|
|||
|
||||
app.get("/health", async (_req, res) => {
|
||||
const vectorHealth = await vectorStore.healthcheck();
|
||||
res.json({
|
||||
ok: true,
|
||||
const postgresHealth = catalog ? await catalog.healthcheck() : { ok: false, kind: "postgres" as const };
|
||||
const reconcilerStatus = reconciler.getStatus();
|
||||
const ok = vectorHealth.ok && (!env.knowledgeLifecycleEnforced || (postgresHealth.ok && reconcilerStatus.ok));
|
||||
res.status(ok ? 200 : 503).json({
|
||||
ok,
|
||||
service: "rag",
|
||||
environment: env.nodeEnv,
|
||||
embeddings: {
|
||||
|
|
@ -89,6 +136,12 @@ export function createApp() {
|
|||
model: env.answerModel
|
||||
},
|
||||
vectorStore: vectorHealth,
|
||||
postgres: postgresHealth,
|
||||
knowledgeLifecycle: {
|
||||
enforced: env.knowledgeLifecycleEnforced,
|
||||
ingestWritesEnabled: env.ingestWritesEnabled,
|
||||
reconciler: reconcilerStatus
|
||||
},
|
||||
parsers: supportedParserExtensions(),
|
||||
chunking: documentalChunkingPolicy
|
||||
});
|
||||
|
|
@ -96,10 +149,55 @@ export function createApp() {
|
|||
|
||||
app.get("/sources", async (_req, res) => {
|
||||
try {
|
||||
const scopes = await vectorStore.listScopes();
|
||||
if (env.knowledgeLifecycleEnforced && !catalog) {
|
||||
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
|
||||
}
|
||||
const scopes = env.knowledgeLifecycleEnforced && catalog
|
||||
? await catalog.listSources()
|
||||
: await vectorStore.listScopes();
|
||||
res.json(scopes);
|
||||
} catch (error) {
|
||||
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown sources error" });
|
||||
sendError(res, error, "Unknown sources error");
|
||||
}
|
||||
});
|
||||
|
||||
app.get("/sources/:sourceId", async (req, res) => {
|
||||
const repository = requireCatalog(res);
|
||||
if (!repository) return;
|
||||
try {
|
||||
const source = await repository.getSource(String(req.params.sourceId));
|
||||
if (!source) {
|
||||
res.status(404).json({ ok: false, error: "Source not found" });
|
||||
return;
|
||||
}
|
||||
res.json(source);
|
||||
} catch (error) {
|
||||
sendError(res, error, "Unknown source error");
|
||||
}
|
||||
});
|
||||
|
||||
app.get("/sources/:sourceId/versions", async (req, res) => {
|
||||
const repository = requireCatalog(res);
|
||||
if (!repository) return;
|
||||
try {
|
||||
res.json(await repository.listVersions(String(req.params.sourceId)));
|
||||
} catch (error) {
|
||||
sendError(res, error, "Unknown versions error");
|
||||
}
|
||||
});
|
||||
|
||||
app.get("/sources/:sourceId/versions/:versionId", async (req, res) => {
|
||||
const repository = requireCatalog(res);
|
||||
if (!repository) return;
|
||||
try {
|
||||
const version = await repository.getVersion(String(req.params.sourceId), String(req.params.versionId));
|
||||
if (!version) {
|
||||
res.status(404).json({ ok: false, error: "Version not found" });
|
||||
return;
|
||||
}
|
||||
res.json(version);
|
||||
} catch (error) {
|
||||
sendError(res, error, "Unknown version error");
|
||||
}
|
||||
});
|
||||
|
||||
|
|
@ -128,14 +226,23 @@ export function createApp() {
|
|||
app.post("/ingest", async (req, res) => {
|
||||
try {
|
||||
const result = await ingestService.ingest(req.body);
|
||||
res.status(202).json(result);
|
||||
res.status(env.knowledgeLifecycleEnforced ? 201 : 202).json(result);
|
||||
} catch (error) {
|
||||
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown ingest error" });
|
||||
sendError(res, error, "Unknown ingest error");
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/cleanup", async (req, res) => {
|
||||
try {
|
||||
if (env.knowledgeLifecycleEnforced) {
|
||||
if (!requireLifecycleAdmin(req, res)) return;
|
||||
res.status(410).json({
|
||||
ok: false,
|
||||
error: "POST /cleanup is deprecated while knowledge lifecycle is enforced; purge non-active versions with DELETE /sources/:sourceId/versions/:versionId.",
|
||||
code: "CLEANUP_DEPRECATED_UNDER_LIFECYCLE"
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (!req.body.scope) {
|
||||
res.status(400).json({ ok: false, error: "Missing scope for cleanup" });
|
||||
return;
|
||||
|
|
@ -155,7 +262,100 @@ export function createApp() {
|
|||
const result = await ingestService.cleanup(scope);
|
||||
res.status(200).json({ ok: true, ...result });
|
||||
} catch (error) {
|
||||
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown cleanup error" });
|
||||
sendError(res, error, "Unknown cleanup error");
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/sources/:sourceId/versions/:versionId/activate", async (req, res) => {
|
||||
if (!requireLifecycleAdmin(req, res)) return;
|
||||
const repository = requireCatalog(res);
|
||||
if (!repository) return;
|
||||
try {
|
||||
const version = await repository.activateVersion(String(req.params.sourceId), String(req.params.versionId), req.body.expectedActiveVersionId);
|
||||
res.json(version);
|
||||
} catch (error) {
|
||||
sendError(res, error, "Unknown activation error");
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/sources/:sourceId/rollback", async (req, res) => {
|
||||
if (!requireLifecycleAdmin(req, res)) return;
|
||||
const repository = requireCatalog(res);
|
||||
if (!repository) return;
|
||||
try {
|
||||
if (!req.body.targetVersionId || !req.body.expectedActiveVersionId) {
|
||||
res.status(400).json({ ok: false, error: "targetVersionId and expectedActiveVersionId are required" });
|
||||
return;
|
||||
}
|
||||
const version = await repository.rollback(String(req.params.sourceId), String(req.body.targetVersionId), String(req.body.expectedActiveVersionId));
|
||||
res.json(version);
|
||||
} catch (error) {
|
||||
sendError(res, error, "Unknown rollback error");
|
||||
}
|
||||
});
|
||||
|
||||
app.post("/sources/:sourceId/versions/:versionId/retry", async (req, res) => {
|
||||
if (!requireLifecycleAdmin(req, res)) return;
|
||||
const repository = requireCatalog(res);
|
||||
if (!repository) return;
|
||||
try {
|
||||
const version = await repository.getVersion(String(req.params.sourceId), String(req.params.versionId));
|
||||
if (!version) {
|
||||
res.status(404).json({ ok: false, error: "Version not found" });
|
||||
return;
|
||||
}
|
||||
if (!["failed", "pending", "indexing"].includes(version.state)) {
|
||||
res.status(409).json({ ok: false, error: "Only failed or orphaned pending/indexing versions can be retried" });
|
||||
return;
|
||||
}
|
||||
if (req.body.sourceType && req.body.sourceRef) {
|
||||
if (["pending", "indexing"].includes(version.state)) {
|
||||
await repository.markFailed(version.versionId, "EXPLICIT_RETRY_REPLACED", "Explicit retry requested; the orphaned version was closed before creating a replacement.");
|
||||
}
|
||||
const result = await ingestService.ingest({
|
||||
sourceId: version.sourceId,
|
||||
sourceType: req.body.sourceType,
|
||||
sourceRef: String(req.body.sourceRef),
|
||||
readPath: req.body.readPath ? String(req.body.readPath) : undefined,
|
||||
mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
|
||||
tags: Array.isArray(req.body.tags) ? req.body.tags.map(String) : version.tags,
|
||||
activate: req.body.activate === undefined ? true : Boolean(req.body.activate),
|
||||
expectedActiveVersionId: req.body.expectedActiveVersionId === undefined ? null : req.body.expectedActiveVersionId
|
||||
});
|
||||
res.status(201).json(result);
|
||||
return;
|
||||
}
|
||||
res.status(202).json({
|
||||
ok: true,
|
||||
retryRequired: true,
|
||||
sourceId: version.sourceId,
|
||||
versionId: version.versionId,
|
||||
message: "Explicit retry is safe only by re-submitting the original source to /ingest with the same sourceId and expectedActiveVersionId. Temporary uploads without durable originals cannot be retried in place."
|
||||
});
|
||||
} catch (error) {
|
||||
sendError(res, error, "Unknown retry error");
|
||||
}
|
||||
});
|
||||
|
||||
app.delete("/sources/:sourceId/versions/:versionId", async (req, res) => {
|
||||
if (!requireLifecycleAdmin(req, res)) return;
|
||||
const repository = requireCatalog(res);
|
||||
if (!repository) return;
|
||||
try {
|
||||
const sourceId = String(req.params.sourceId);
|
||||
const versionId = String(req.params.versionId);
|
||||
await repository.withSourceLock(sourceId, async () => repository.withVersionExclusiveLock(versionId, async () => {
|
||||
await repository.markPurging(sourceId, versionId);
|
||||
await vectorStore.deleteVersionPoints(versionId);
|
||||
const remaining = await vectorStore.countVersionPoints(versionId);
|
||||
if (remaining !== 0) {
|
||||
throw new CatalogError("Version purge did not delete all points", 503, "VERSION_PURGE_INCOMPLETE");
|
||||
}
|
||||
await repository.markPurged(versionId);
|
||||
}));
|
||||
res.json({ ok: true, purged: true, versionId });
|
||||
} catch (error) {
|
||||
sendError(res, error, "Unknown purge error");
|
||||
}
|
||||
});
|
||||
|
||||
|
|
@ -181,7 +381,6 @@ export function createApp() {
|
|||
let result;
|
||||
|
||||
if (isZipFolder) {
|
||||
// Logica para carpetas subidas como ZIP
|
||||
extractDirPath = await mkdtemp(path.join(tempDirBase, "rag-upload-"));
|
||||
const zip = new AdmZip(tempFilePath);
|
||||
zip.extractAllTo(extractDirPath, true);
|
||||
|
|
@ -189,29 +388,32 @@ export function createApp() {
|
|||
result = await ingestService.ingest({
|
||||
sourceId: req.body.sourceId ? String(req.body.sourceId) : undefined,
|
||||
sourceType: "folder",
|
||||
sourceRef: req.file.originalname.replace(/\.zip$/i, ""), // Usamos el nombre original sin .zip como ref
|
||||
sourceRef: req.file.originalname.replace(/\.zip$/i, ""),
|
||||
readPath: extractDirPath,
|
||||
mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
|
||||
tags
|
||||
tags,
|
||||
activate: req.body.activate === undefined ? undefined : req.body.activate === "true",
|
||||
expectedActiveVersionId: req.body.expectedActiveVersionId === "null" ? null : req.body.expectedActiveVersionId ? String(req.body.expectedActiveVersionId) : undefined
|
||||
});
|
||||
} else {
|
||||
// Logica para archivos sueltos
|
||||
result = await ingestService.ingest({
|
||||
sourceId: req.body.sourceId ? String(req.body.sourceId) : undefined,
|
||||
sourceType: "file",
|
||||
sourceRef: req.file.originalname,
|
||||
readPath: tempFilePath,
|
||||
mode: req.body.mode === "interactive" ? "interactive" : "mechanical",
|
||||
tags
|
||||
tags,
|
||||
activate: req.body.activate === undefined ? undefined : req.body.activate === "true",
|
||||
expectedActiveVersionId: req.body.expectedActiveVersionId === "null" ? null : req.body.expectedActiveVersionId ? String(req.body.expectedActiveVersionId) : undefined
|
||||
});
|
||||
}
|
||||
|
||||
res.status(202).json({
|
||||
res.status(env.knowledgeLifecycleEnforced ? 201 : 202).json({
|
||||
...result,
|
||||
uploadedResource: req.file.originalname
|
||||
});
|
||||
} catch (error) {
|
||||
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown upload ingest error" });
|
||||
sendError(res, error, "Unknown upload ingest error");
|
||||
} finally {
|
||||
if (tempFilePath) {
|
||||
await unlink(tempFilePath).catch(() => undefined);
|
||||
|
|
@ -284,7 +486,7 @@ export function createApp() {
|
|||
|
||||
res.json(result);
|
||||
} catch (error) {
|
||||
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown retrieve error" });
|
||||
sendError(res, error, "Unknown retrieve error");
|
||||
}
|
||||
});
|
||||
|
||||
|
|
@ -335,7 +537,7 @@ export function createApp() {
|
|||
|
||||
res.json(result);
|
||||
} catch (error) {
|
||||
res.status(500).json({ ok: false, error: error instanceof Error ? error.message : "Unknown answer error" });
|
||||
sendError(res, error, "Unknown answer error");
|
||||
}
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,9 @@
|
|||
import { config as loadEnv } from "dotenv";
|
||||
|
||||
loadEnv();
|
||||
loadEnv({ path: ".env.local", override: true });
|
||||
if (process.env.NODE_ENV !== "test") {
|
||||
loadEnv();
|
||||
loadEnv({ path: ".env.local", override: true });
|
||||
}
|
||||
|
||||
function requireEnv(name: string, fallback?: string): string {
|
||||
const value = process.env[name] ?? fallback;
|
||||
|
|
@ -11,6 +13,24 @@ function requireEnv(name: string, fallback?: string): string {
|
|||
return value;
|
||||
}
|
||||
|
||||
function optionalEnv(...names: string[]): string {
|
||||
for (const name of names) {
|
||||
const value = process.env[name];
|
||||
if (value) {
|
||||
return value;
|
||||
}
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
function booleanEnv(name: string, fallback: boolean): boolean {
|
||||
const value = process.env[name];
|
||||
if (value === undefined) {
|
||||
return fallback;
|
||||
}
|
||||
return ["1", "true", "yes", "on"].includes(value.toLowerCase());
|
||||
}
|
||||
|
||||
export const env = {
|
||||
nodeEnv: process.env.NODE_ENV ?? "development",
|
||||
port: Number(process.env.PORT ?? 3000),
|
||||
|
|
@ -25,5 +45,12 @@ export const env = {
|
|||
answerProvider: requireEnv("ANSWER_PROVIDER", "openrouter"),
|
||||
answerModel: requireEnv("ANSWER_MODEL", "openai/gpt-4.1-mini"),
|
||||
answerBaseUrl: requireEnv("ANSWER_BASE_URL", "https://openrouter.ai/api/v1"),
|
||||
answerApiKey: process.env.ANSWER_API_KEY ?? process.env.EMBEDDING_API_KEY ?? ""
|
||||
answerApiKey: process.env.ANSWER_API_KEY ?? process.env.EMBEDDING_API_KEY ?? "",
|
||||
postgresUrl: optionalEnv("POSTGRES_URL", "DATABASE_URL"),
|
||||
postgresSsl: booleanEnv("POSTGRES_SSL", false),
|
||||
lifecycleAdminToken: process.env.LIFECYCLE_ADMIN_TOKEN ?? "",
|
||||
knowledgeLifecycleEnforced: booleanEnv("KNOWLEDGE_LIFECYCLE_ENFORCED", false),
|
||||
ingestWritesEnabled: booleanEnv("INGEST_WRITES_ENABLED", true),
|
||||
lifecycleReconcileIntervalMs: Number(process.env.LIFECYCLE_RECONCILE_INTERVAL_MS ?? 300000),
|
||||
lifecycleIndexingStaleTimeoutMs: Number(process.env.LIFECYCLE_INDEXING_STALE_TIMEOUT_MS ?? 1800000)
|
||||
} as const;
|
||||
|
|
|
|||
|
|
@ -116,6 +116,8 @@ export class AnswerService {
|
|||
citations: retrieved.items.map((item) => ({
|
||||
chunkId: item.chunkId,
|
||||
documentId: item.documentId,
|
||||
sourceVersionId: item.sourceVersionId,
|
||||
sourceVersionNumber: item.sourceVersionNumber,
|
||||
title: item.title,
|
||||
sectionTitle: item.sectionTitle,
|
||||
startLine: item.startLine,
|
||||
|
|
|
|||
94
src/modules/catalog/client.ts
Normal file
94
src/modules/catalog/client.ts
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
import pg from "pg";
|
||||
import { env } from "../../config/env.js";
|
||||
|
||||
const { Pool } = pg;
|
||||
|
||||
export type PgPool = pg.Pool;
|
||||
export type PgPoolClient = pg.PoolClient;
|
||||
|
||||
let sharedPool: PgPool | undefined;
|
||||
|
||||
export function hasCatalogConnection(): boolean {
|
||||
return Boolean(env.postgresUrl);
|
||||
}
|
||||
|
||||
export function buildCatalogPool(): PgPool {
|
||||
if (!env.postgresUrl) {
|
||||
throw new Error("Missing POSTGRES_URL or DATABASE_URL for knowledge catalog");
|
||||
}
|
||||
|
||||
return new Pool({
|
||||
connectionString: env.postgresUrl,
|
||||
ssl: env.postgresSsl ? { rejectUnauthorized: false } : undefined
|
||||
});
|
||||
}
|
||||
|
||||
export function getCatalogPool(): PgPool | undefined {
|
||||
if (!hasCatalogConnection()) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
sharedPool ??= buildCatalogPool();
|
||||
return sharedPool;
|
||||
}
|
||||
|
||||
export async function withTransaction<T>(pool: PgPool, handler: (client: PgPoolClient) => Promise<T>): Promise<T> {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query("BEGIN");
|
||||
await client.query("SET CONSTRAINTS ALL DEFERRED");
|
||||
const result = await handler(client);
|
||||
await client.query("COMMIT");
|
||||
return result;
|
||||
} catch (error) {
|
||||
await client.query("ROLLBACK").catch(() => undefined);
|
||||
throw error;
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
export async function withCatalogAdvisoryLock<T>(pool: PgPool, lockName: string, handler: () => Promise<T>): Promise<T> {
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query("SELECT pg_advisory_lock(hashtext($1))", [lockName]);
|
||||
return await handler();
|
||||
} finally {
|
||||
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [lockName]).catch(() => undefined);
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
export async function withCatalogTryAdvisoryLock<T>(pool: PgPool, lockName: string, handler: () => Promise<T>): Promise<T | undefined> {
|
||||
const client = await pool.connect();
|
||||
let locked = false;
|
||||
try {
|
||||
const result = await client.query<{ locked: boolean }>("SELECT pg_try_advisory_lock(hashtext($1)) AS locked", [lockName]);
|
||||
locked = Boolean(result.rows[0]?.locked);
|
||||
if (!locked) {
|
||||
return undefined;
|
||||
}
|
||||
return await handler();
|
||||
} finally {
|
||||
if (locked) {
|
||||
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [lockName]).catch(() => undefined);
|
||||
}
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
export async function withCatalogAdvisorySharedLocks<T>(pool: PgPool, lockNames: string[], handler: () => Promise<T>): Promise<T> {
|
||||
const names = [...new Set(lockNames)].sort((left, right) => Buffer.compare(Buffer.from(left), Buffer.from(right)));
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
for (const name of names) {
|
||||
await client.query("SELECT pg_advisory_lock_shared(hashtext($1))", [name]);
|
||||
}
|
||||
return await handler();
|
||||
} finally {
|
||||
for (const name of names.reverse()) {
|
||||
await client.query("SELECT pg_advisory_unlock_shared(hashtext($1))", [name]).catch(() => undefined);
|
||||
}
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
9
src/modules/catalog/errors.ts
Normal file
9
src/modules/catalog/errors.ts
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
export class CatalogError extends Error {
|
||||
constructor(
|
||||
message: string,
|
||||
readonly statusCode = 500,
|
||||
readonly code = "CATALOG_ERROR"
|
||||
) {
|
||||
super(message);
|
||||
}
|
||||
}
|
||||
150
src/modules/catalog/legacy-migration.ts
Normal file
150
src/modules/catalog/legacy-migration.ts
Normal file
|
|
@ -0,0 +1,150 @@
|
|||
import { buildDocumentId, buildMetadataHash, hashOrderedPairs, sha256Hex } from "../../shared/utils/ids.js";
|
||||
import type { SourceType } from "../../shared/types/rag.js";
|
||||
|
||||
export type LegacyPointId = string | number;
|
||||
|
||||
export interface LegacyPointSnapshot {
|
||||
id: LegacyPointId;
|
||||
payload: Record<string, unknown>;
|
||||
}
|
||||
|
||||
export interface LegacyDocumentPlan {
|
||||
documentId: string;
|
||||
documentKey: string;
|
||||
originalHash: string;
|
||||
contentHash: string;
|
||||
mimeType: string;
|
||||
title: string;
|
||||
chunkCount: number;
|
||||
}
|
||||
|
||||
export interface LegacySourcePlan {
|
||||
sourceId: string;
|
||||
sourceType: SourceType;
|
||||
sourceRef: string;
|
||||
pointIds: LegacyPointId[];
|
||||
tags: string[];
|
||||
documents: LegacyDocumentPlan[];
|
||||
originalManifestHash: string;
|
||||
sourceContentHash: string;
|
||||
metadataHash: string;
|
||||
embeddingProvider: string;
|
||||
embeddingModel: string;
|
||||
embeddingDimensions: number;
|
||||
blockedReasons: string[];
|
||||
}
|
||||
|
||||
export interface LegacyMigrationPlan {
|
||||
ok: boolean;
|
||||
sources: LegacySourcePlan[];
|
||||
blockedReasons: string[];
|
||||
}
|
||||
|
||||
function stringValue(value: unknown): string {
|
||||
return typeof value === "string" ? value.trim() : "";
|
||||
}
|
||||
|
||||
function stringArray(value: unknown): string[] {
|
||||
return Array.isArray(value) ? value.map(String).filter(Boolean).sort() : [];
|
||||
}
|
||||
|
||||
function unique(values: string[]): string[] {
|
||||
return [...new Set(values.filter(Boolean))];
|
||||
}
|
||||
|
||||
function sortedByUtf8<T>(values: T[], getKey: (value: T) => string): T[] {
|
||||
return [...values].sort((left, right) => Buffer.compare(Buffer.from(getKey(left), "utf8"), Buffer.from(getKey(right), "utf8")));
|
||||
}
|
||||
|
||||
export function buildLegacyMigrationPlan(points: LegacyPointSnapshot[]): LegacyMigrationPlan {
|
||||
const groups = new Map<string, LegacyPointSnapshot[]>();
|
||||
const globalBlocked: string[] = [];
|
||||
|
||||
for (const point of points) {
|
||||
const sourceId = stringValue(point.payload.source_id);
|
||||
if (!sourceId) {
|
||||
globalBlocked.push(`Point ${String(point.id)} is missing source_id.`);
|
||||
continue;
|
||||
}
|
||||
const entries = groups.get(sourceId) ?? [];
|
||||
entries.push(point);
|
||||
groups.set(sourceId, entries);
|
||||
}
|
||||
|
||||
const sources = sortedByUtf8([...groups.entries()], ([sourceId]) => sourceId).map(([sourceId, sourcePoints]) => {
|
||||
const blockedReasons: string[] = [];
|
||||
const sourceRefs = unique(sourcePoints.map((point) => stringValue(point.payload.source_ref)));
|
||||
const sourceTypes = unique(sourcePoints.map((point) => stringValue(point.payload.source_type)));
|
||||
const providers = unique(sourcePoints.map((point) => stringValue(point.payload.embedding_provider)));
|
||||
const models = unique(sourcePoints.map((point) => stringValue(point.payload.embedding_model)));
|
||||
const dimensions = unique(sourcePoints.map((point) => String(Number(point.payload.embedding_dimensions ?? 0))).filter((value) => value !== "0"));
|
||||
const tagSnapshot = unique(sourcePoints.flatMap((point) => stringArray(point.payload.tags)));
|
||||
|
||||
if (sourceRefs.length !== 1) blockedReasons.push("Source has zero or multiple source_ref values.");
|
||||
if (sourceTypes.length !== 1 || !["file", "folder"].includes(sourceTypes[0])) blockedReasons.push("Source has no unique file/folder source_type.");
|
||||
if (providers.length !== 1) blockedReasons.push("Source has no unique embedding provider.");
|
||||
if (models.length !== 1) blockedReasons.push("Source has no unique embedding model.");
|
||||
if (dimensions.length !== 1) blockedReasons.push("Source has no unique non-zero embedding dimensions.");
|
||||
|
||||
const documentGroups = new Map<string, LegacyPointSnapshot[]>();
|
||||
for (const point of sourcePoints) {
|
||||
const documentId = stringValue(point.payload.document_id);
|
||||
const documentKey = stringValue(point.payload.document_key);
|
||||
const chunkId = stringValue(point.payload.chunk_id);
|
||||
const content = stringValue(point.payload.content);
|
||||
if (!documentId || !documentKey || !chunkId || !content) {
|
||||
blockedReasons.push(`Point ${String(point.id)} has weak document identity or empty content.`);
|
||||
continue;
|
||||
}
|
||||
if (documentId !== buildDocumentId(sourceId, documentKey)) {
|
||||
blockedReasons.push(`Point ${String(point.id)} document_id does not match source_id and document_key.`);
|
||||
}
|
||||
const existing = documentGroups.get(documentId) ?? [];
|
||||
existing.push(point);
|
||||
documentGroups.set(documentId, existing);
|
||||
}
|
||||
|
||||
const documents: LegacyDocumentPlan[] = sortedByUtf8([...documentGroups.entries()], ([documentId]) => documentId).map(([documentId, documentPoints]) => {
|
||||
const documentKeys = unique(documentPoints.map((point) => stringValue(point.payload.document_key)));
|
||||
const mimeTypes = unique(documentPoints.map((point) => stringValue(point.payload.mime_type) || "text/plain"));
|
||||
const titles = unique(documentPoints.map((point) => stringValue(point.payload.title) || documentKeys[0]));
|
||||
if (documentKeys.length !== 1) blockedReasons.push(`Document ${documentId} has no unique document_key.`);
|
||||
if (mimeTypes.length !== 1) blockedReasons.push(`Document ${documentId} has no unique mime_type.`);
|
||||
const orderedContents = sortedByUtf8(documentPoints, (point) => stringValue(point.payload.chunk_id))
|
||||
.map((point) => stringValue(point.payload.content))
|
||||
.join("\n");
|
||||
const contentHash = sha256Hex(orderedContents);
|
||||
return {
|
||||
documentId,
|
||||
documentKey: documentKeys[0] ?? "",
|
||||
originalHash: contentHash,
|
||||
contentHash,
|
||||
mimeType: mimeTypes[0] ?? "text/plain",
|
||||
title: titles[0] ?? documentKeys[0] ?? documentId,
|
||||
chunkCount: documentPoints.length
|
||||
};
|
||||
});
|
||||
|
||||
const originalManifestHash = hashOrderedPairs(documents.map((document) => [document.documentKey, document.originalHash])) ?? "";
|
||||
const sourceContentHash = hashOrderedPairs(documents.map((document) => [document.documentKey, document.contentHash])) ?? "";
|
||||
|
||||
return {
|
||||
sourceId,
|
||||
sourceType: sourceTypes[0] as SourceType,
|
||||
sourceRef: sourceRefs[0] ?? "",
|
||||
pointIds: sourcePoints.map((point) => point.id),
|
||||
tags: tagSnapshot,
|
||||
documents,
|
||||
originalManifestHash,
|
||||
sourceContentHash,
|
||||
metadataHash: buildMetadataHash(tagSnapshot, { sourceType: sourceTypes[0], sourceRef: sourceRefs[0] }),
|
||||
embeddingProvider: providers[0] ?? "",
|
||||
embeddingModel: models[0] ?? "",
|
||||
embeddingDimensions: Number(dimensions[0] ?? 0),
|
||||
blockedReasons: unique(blockedReasons)
|
||||
};
|
||||
});
|
||||
|
||||
const blockedReasons = [...globalBlocked, ...sources.flatMap((source) => source.blockedReasons.map((reason) => `${source.sourceId}: ${reason}`))];
|
||||
return { ok: blockedReasons.length === 0, sources, blockedReasons };
|
||||
}
|
||||
11
src/modules/catalog/lifecycle.ts
Normal file
11
src/modules/catalog/lifecycle.ts
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
import { CatalogError } from "./errors.js";
|
||||
|
||||
export function normalizeExpectedActiveVersion(value: string | null | undefined, activate: boolean): string | null | undefined {
|
||||
if (!activate) {
|
||||
return undefined;
|
||||
}
|
||||
if (value === undefined) {
|
||||
throw new CatalogError("expectedActiveVersionId is required when activate is true", 400, "EXPECTED_ACTIVE_VERSION_REQUIRED");
|
||||
}
|
||||
return value;
|
||||
}
|
||||
90
src/modules/catalog/migrations.ts
Normal file
90
src/modules/catalog/migrations.ts
Normal file
|
|
@ -0,0 +1,90 @@
|
|||
import { createHash } from "node:crypto";
|
||||
import { readdir, readFile } from "node:fs/promises";
|
||||
import path from "node:path";
|
||||
import { fileURLToPath, pathToFileURL } from "node:url";
|
||||
import { buildCatalogPool, hasCatalogConnection, type PgPool } from "./client.js";
|
||||
|
||||
const GLOBAL_MIGRATION_LOCK = "rag:knowledge-lifecycle:migrations";
|
||||
|
||||
function sha256(value: string): string {
|
||||
return createHash("sha256").update(value).digest("hex");
|
||||
}
|
||||
|
||||
function resolveMigrationsDir(): string {
|
||||
const currentFile = fileURLToPath(import.meta.url);
|
||||
return path.resolve(path.dirname(currentFile), "../../../migrations");
|
||||
}
|
||||
|
||||
export async function runKnowledgeLifecycleMigrations(pool: PgPool = buildCatalogPool()): Promise<void> {
|
||||
const migrationsDir = resolveMigrationsDir();
|
||||
const files = (await readdir(migrationsDir))
|
||||
.filter((file) => /^\d+_.*\.sql$/.test(file))
|
||||
.sort();
|
||||
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query("SELECT pg_advisory_lock(hashtext($1))", [GLOBAL_MIGRATION_LOCK]);
|
||||
await client.query(`
|
||||
CREATE TABLE IF NOT EXISTS rag_schema_migrations (
|
||||
name text PRIMARY KEY,
|
||||
checksum char(64) NOT NULL,
|
||||
applied_at timestamptz NOT NULL DEFAULT now()
|
||||
)
|
||||
`);
|
||||
|
||||
for (const file of files) {
|
||||
const sql = await readFile(path.join(migrationsDir, file), "utf8");
|
||||
const checksum = sha256(sql);
|
||||
const existing = await client.query<{ checksum: string }>(
|
||||
"SELECT checksum FROM rag_schema_migrations WHERE name = $1",
|
||||
[file]
|
||||
);
|
||||
|
||||
if (existing.rowCount && existing.rows[0].checksum !== checksum) {
|
||||
throw new Error(`Migration checksum mismatch: ${file}`);
|
||||
}
|
||||
|
||||
if (existing.rowCount) {
|
||||
continue;
|
||||
}
|
||||
|
||||
await client.query("BEGIN");
|
||||
try {
|
||||
await client.query(sql);
|
||||
await client.query(
|
||||
"INSERT INTO rag_schema_migrations(name, checksum) VALUES ($1, $2)",
|
||||
[file, checksum]
|
||||
);
|
||||
await client.query("COMMIT");
|
||||
} catch (error) {
|
||||
await client.query("ROLLBACK").catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
await client.query("SELECT pg_advisory_unlock(hashtext($1))", [GLOBAL_MIGRATION_LOCK]).catch(() => undefined);
|
||||
client.release();
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
if (!hasCatalogConnection()) {
|
||||
console.warn("Knowledge lifecycle migrations skipped: POSTGRES_URL or DATABASE_URL is not configured.");
|
||||
return;
|
||||
}
|
||||
|
||||
const pool = buildCatalogPool();
|
||||
try {
|
||||
await runKnowledgeLifecycleMigrations(pool);
|
||||
console.log("Knowledge lifecycle migrations completed.");
|
||||
} finally {
|
||||
await pool.end();
|
||||
}
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.message : error);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
123
src/modules/catalog/reconciler.ts
Normal file
123
src/modules/catalog/reconciler.ts
Normal file
|
|
@ -0,0 +1,123 @@
|
|||
import { env } from "../../config/env.js";
|
||||
import type { VectorStoreClient } from "../vectorstore/client.js";
|
||||
import type { CatalogRepository } from "./repository.js";
|
||||
|
||||
export interface ReconcilerStatus {
|
||||
ok: boolean;
|
||||
lastRunAt?: string;
|
||||
lastError?: string;
|
||||
inconsistentSources: string[];
|
||||
orphanedVersionsRecovered?: number;
|
||||
orphanedVersionsFailed?: number;
|
||||
invariantViolations?: string[];
|
||||
}
|
||||
|
||||
export class KnowledgeLifecycleReconciler {
|
||||
private status: ReconcilerStatus = { ok: true, inconsistentSources: [] };
|
||||
private timer: NodeJS.Timeout | undefined;
|
||||
|
||||
constructor(
|
||||
private readonly catalog: CatalogRepository | undefined,
|
||||
private readonly vectorStore: VectorStoreClient
|
||||
) {}
|
||||
|
||||
getStatus(): ReconcilerStatus {
|
||||
return this.status;
|
||||
}
|
||||
|
||||
start(): void {
|
||||
if (!env.knowledgeLifecycleEnforced || !this.catalog || this.timer) {
|
||||
return;
|
||||
}
|
||||
|
||||
void this.runOnce();
|
||||
this.timer = setInterval(() => {
|
||||
void this.runOnce();
|
||||
}, env.lifecycleReconcileIntervalMs);
|
||||
this.timer.unref();
|
||||
}
|
||||
|
||||
async runOnce(): Promise<ReconcilerStatus> {
|
||||
if (!this.catalog) {
|
||||
this.status = { ok: false, lastRunAt: new Date().toISOString(), lastError: "Knowledge catalog is not configured", inconsistentSources: [] };
|
||||
return this.status;
|
||||
}
|
||||
|
||||
const catalog = this.catalog;
|
||||
try {
|
||||
const lockedStatus = await catalog.withGlobalTryLock("rag:knowledge-lifecycle:reconciler", async () => {
|
||||
const activeVersions = await catalog.resolveActiveVersions();
|
||||
const inconsistentSources: string[] = [];
|
||||
const invariantViolations = await catalog.validateActiveInvariant();
|
||||
const orphanRecovery = await this.recoverOrphanedVersions(catalog);
|
||||
|
||||
for (const version of activeVersions) {
|
||||
if (version.qdrantCollection !== env.qdrantCollection) {
|
||||
inconsistentSources.push(version.sourceId);
|
||||
await catalog.markNeedsReingest(version.sourceId);
|
||||
continue;
|
||||
}
|
||||
await this.vectorStore.validateCollectionDimensions(version.embeddingDimensions);
|
||||
const count = await this.vectorStore.countVersionPoints(version.sourceVersionId);
|
||||
if (count !== version.expectedPointCount) {
|
||||
inconsistentSources.push(version.sourceId);
|
||||
await catalog.markNeedsReingest(version.sourceId);
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
ok: inconsistentSources.length === 0 && invariantViolations.length === 0,
|
||||
lastRunAt: new Date().toISOString(),
|
||||
inconsistentSources,
|
||||
orphanedVersionsRecovered: orphanRecovery.ready,
|
||||
orphanedVersionsFailed: orphanRecovery.failed,
|
||||
invariantViolations
|
||||
};
|
||||
});
|
||||
|
||||
this.status = lockedStatus ?? { ...this.status, lastRunAt: new Date().toISOString() };
|
||||
return this.status;
|
||||
} catch (error) {
|
||||
this.status = {
|
||||
ok: false,
|
||||
lastRunAt: new Date().toISOString(),
|
||||
lastError: error instanceof Error ? error.message : "Unknown reconciler error",
|
||||
inconsistentSources: []
|
||||
};
|
||||
return this.status;
|
||||
}
|
||||
}
|
||||
|
||||
private async recoverOrphanedVersions(catalog: CatalogRepository): Promise<{ ready: number; failed: number }> {
|
||||
const candidates = await catalog.listOrphanedIndexingCandidates(env.lifecycleIndexingStaleTimeoutMs);
|
||||
let ready = 0;
|
||||
let failed = 0;
|
||||
|
||||
for (const candidate of candidates) {
|
||||
const recovered = await catalog.withSourceTryLock(candidate.sourceId, async () => catalog.withVersionTryLock(candidate.versionId, async () => {
|
||||
const documentsComplete = await catalog.hasCompleteVersionDocuments(candidate.versionId, candidate.expectedDocumentCount, candidate.expectedPointCount);
|
||||
const collectionMatches = candidate.qdrantCollection === env.qdrantCollection;
|
||||
if (collectionMatches) {
|
||||
await this.vectorStore.validateCollectionDimensions(candidate.embeddingDimensions);
|
||||
}
|
||||
const pointCount = collectionMatches ? await this.vectorStore.countVersionPoints(candidate.versionId) : 0;
|
||||
if (collectionMatches && documentsComplete && pointCount === candidate.expectedPointCount) {
|
||||
const changed = await catalog.recoverVersionReady(candidate.versionId, pointCount);
|
||||
if (changed) {
|
||||
return "ready" as const;
|
||||
}
|
||||
}
|
||||
await catalog.markFailed(candidate.versionId, "ORPHANED_INDEXING_INCOMPLETE", "Stale pending/indexing version has incomplete documents, dimensions, collection, or point count.");
|
||||
return "failed" as const;
|
||||
}));
|
||||
|
||||
if (recovered === "ready") {
|
||||
ready += 1;
|
||||
} else if (recovered === "failed") {
|
||||
failed += 1;
|
||||
}
|
||||
}
|
||||
|
||||
return { ready, failed };
|
||||
}
|
||||
}
|
||||
641
src/modules/catalog/repository.ts
Normal file
641
src/modules/catalog/repository.ts
Normal file
|
|
@ -0,0 +1,641 @@
|
|||
import type { AvailableScope, ChunkMode, IngestSourceInput, SourceType, SourceVersionState, RetrieveScope } from "../../shared/types/rag.js";
|
||||
import { withCatalogAdvisoryLock, withCatalogAdvisorySharedLocks, withCatalogTryAdvisoryLock, withTransaction, type PgPool, type PgPoolClient } from "./client.js";
|
||||
import { CatalogError } from "./errors.js";
|
||||
import { normalizeExpectedActiveVersion } from "./lifecycle.js";
|
||||
|
||||
export interface CatalogDocumentInput {
|
||||
documentId: string;
|
||||
documentKey: string;
|
||||
originalHash: string;
|
||||
originalHashKind: "bytes" | "legacy-derived";
|
||||
contentHash: string | null;
|
||||
mimeType: string;
|
||||
title: string;
|
||||
chunkCount: number;
|
||||
}
|
||||
|
||||
export interface CreateVersionInput {
|
||||
sourceId: string;
|
||||
sourceType: SourceType;
|
||||
sourceRef: string;
|
||||
originalManifestHash: string;
|
||||
sourceContentHash: string | null;
|
||||
processingFingerprint: string;
|
||||
metadataHash: string;
|
||||
tags: string[];
|
||||
activateRequested: boolean;
|
||||
embeddingProvider: string;
|
||||
embeddingModel: string;
|
||||
embeddingDimensions: number;
|
||||
qdrantCollection: string;
|
||||
expectedDocumentCount: number;
|
||||
expectedPointCount: number;
|
||||
documents: CatalogDocumentInput[];
|
||||
}
|
||||
|
||||
export interface CatalogVersionRow {
|
||||
versionId: string;
|
||||
sourceId: string;
|
||||
versionNumber: number;
|
||||
previousVersionId: string | null;
|
||||
state: SourceVersionState;
|
||||
tags: string[];
|
||||
sourceContentHash: string | null;
|
||||
processingFingerprint: string;
|
||||
metadataHash: string;
|
||||
embeddingProvider: string;
|
||||
embeddingModel: string;
|
||||
embeddingDimensions: number;
|
||||
expectedDocumentCount: number;
|
||||
expectedPointCount: number;
|
||||
verifiedPointCount: number;
|
||||
qdrantCollection: string;
|
||||
}
|
||||
|
||||
export interface ActiveVersionScope {
|
||||
sourceId: string;
|
||||
sourceVersionId: string;
|
||||
sourceVersionNumber: number;
|
||||
expectedPointCount: number;
|
||||
embeddingDimensions: number;
|
||||
qdrantCollection: string;
|
||||
}
|
||||
|
||||
export interface OrphanedVersionCandidate {
|
||||
sourceId: string;
|
||||
versionId: string;
|
||||
expectedDocumentCount: number;
|
||||
expectedPointCount: number;
|
||||
embeddingDimensions: number;
|
||||
qdrantCollection: string;
|
||||
}
|
||||
|
||||
type VersionDbRow = {
|
||||
version_id: string;
|
||||
source_id: string;
|
||||
version_number: string | number;
|
||||
previous_version_id: string | null;
|
||||
state: SourceVersionState;
|
||||
tags: string[];
|
||||
source_content_hash: string | null;
|
||||
processing_fingerprint: string;
|
||||
metadata_hash: string;
|
||||
embedding_provider: string;
|
||||
embedding_model: string;
|
||||
embedding_dimensions: string | number;
|
||||
expected_document_count: string | number;
|
||||
expected_point_count: string | number;
|
||||
verified_point_count: string | number;
|
||||
qdrant_collection: string;
|
||||
};
|
||||
|
||||
function toVersion(row: VersionDbRow): CatalogVersionRow {
|
||||
return {
|
||||
versionId: row.version_id,
|
||||
sourceId: row.source_id,
|
||||
versionNumber: Number(row.version_number),
|
||||
previousVersionId: row.previous_version_id,
|
||||
state: row.state,
|
||||
tags: row.tags ?? [],
|
||||
sourceContentHash: row.source_content_hash,
|
||||
processingFingerprint: row.processing_fingerprint,
|
||||
metadataHash: row.metadata_hash,
|
||||
embeddingProvider: row.embedding_provider,
|
||||
embeddingModel: row.embedding_model,
|
||||
embeddingDimensions: Number(row.embedding_dimensions),
|
||||
expectedDocumentCount: Number(row.expected_document_count),
|
||||
expectedPointCount: Number(row.expected_point_count),
|
||||
verifiedPointCount: Number(row.verified_point_count),
|
||||
qdrantCollection: row.qdrant_collection
|
||||
};
|
||||
}
|
||||
|
||||
export class CatalogRepository {
|
||||
constructor(private readonly pool: PgPool) {}
|
||||
|
||||
async healthcheck(): Promise<{ ok: boolean; kind: "postgres" }> {
|
||||
try {
|
||||
await this.pool.query("SELECT 1");
|
||||
return { ok: true, kind: "postgres" };
|
||||
} catch {
|
||||
return { ok: false, kind: "postgres" };
|
||||
}
|
||||
}
|
||||
|
||||
async withSourceLock<T>(sourceId: string, handler: () => Promise<T>): Promise<T> {
|
||||
return withCatalogAdvisoryLock(this.pool, `rag:source:${sourceId}`, handler);
|
||||
}
|
||||
|
||||
async withSourceTryLock<T>(sourceId: string, handler: () => Promise<T>): Promise<T | undefined> {
|
||||
return withCatalogTryAdvisoryLock(this.pool, `rag:source:${sourceId}`, handler);
|
||||
}
|
||||
|
||||
async withVersionSharedLocks<T>(versionIds: string[], handler: () => Promise<T>): Promise<T> {
|
||||
return withCatalogAdvisorySharedLocks(this.pool, versionIds.map((versionId) => `rag:version:${versionId}`), handler);
|
||||
}
|
||||
|
||||
async withVersionExclusiveLock<T>(versionId: string, handler: () => Promise<T>): Promise<T> {
|
||||
return withCatalogAdvisoryLock(this.pool, `rag:version:${versionId}`, handler);
|
||||
}
|
||||
|
||||
async withVersionTryLock<T>(versionId: string, handler: () => Promise<T>): Promise<T | undefined> {
|
||||
return withCatalogTryAdvisoryLock(this.pool, `rag:version:${versionId}`, handler);
|
||||
}
|
||||
|
||||
async withGlobalTryLock<T>(lockName: string, handler: () => Promise<T>): Promise<T | undefined> {
|
||||
return withCatalogTryAdvisoryLock(this.pool, lockName, handler);
|
||||
}
|
||||
|
||||
async isMaintenanceEnabled(client?: PgPoolClient, lockRow = false): Promise<boolean> {
|
||||
const executor = client ?? this.pool;
|
||||
const result = await executor.query<{ maintenance: boolean }>(
|
||||
`SELECT maintenance FROM rag_system_state WHERE singleton = true${lockRow ? " FOR UPDATE" : ""}`
|
||||
);
|
||||
return Boolean(result.rows[0]?.maintenance);
|
||||
}
|
||||
|
||||
async beginAttempt(source: IngestSourceInput & { sourceId: string }): Promise<string> {
|
||||
return withTransaction(this.pool, async (client) => {
|
||||
if (await this.isMaintenanceEnabled(client, true)) {
|
||||
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
|
||||
}
|
||||
|
||||
const sourceResult = await client.query<{ source_type: SourceType; source_ref: string }>(
|
||||
`INSERT INTO rag_sources(source_id, source_type, source_ref)
|
||||
VALUES ($1, $2, $3)
|
||||
ON CONFLICT (source_id) DO NOTHING
|
||||
RETURNING source_type, source_ref`,
|
||||
[source.sourceId, source.sourceType, source.sourceRef]
|
||||
);
|
||||
const existing = sourceResult.rowCount
|
||||
? sourceResult.rows[0]
|
||||
: (await client.query<{ source_type: SourceType; source_ref: string }>(
|
||||
"SELECT source_type, source_ref FROM rag_sources WHERE source_id = $1",
|
||||
[source.sourceId]
|
||||
)).rows[0];
|
||||
|
||||
if (!existing || existing.source_type !== source.sourceType || existing.source_ref !== source.sourceRef) {
|
||||
throw new CatalogError("sourceId already exists with a different sourceType or sourceRef", 409, "SOURCE_IDENTITY_CONFLICT");
|
||||
}
|
||||
|
||||
const result = await client.query<{ attempt_id: string }>(
|
||||
`INSERT INTO rag_ingestion_attempts(source_id, state, input_locator)
|
||||
VALUES ($1, 'received', $2)
|
||||
RETURNING attempt_id`,
|
||||
[source.sourceId, source.readPath ?? source.sourceRef]
|
||||
);
|
||||
return result.rows[0].attempt_id;
|
||||
});
|
||||
}
|
||||
|
||||
async updateAttempt(attemptId: string, values: { state: "processing" | "completed" | "failed"; versionId?: string | null; inputHash?: string | null; errorCode?: string; errorDetail?: string }): Promise<void> {
|
||||
await this.pool.query(
|
||||
`UPDATE rag_ingestion_attempts
|
||||
SET state = $2,
|
||||
version_id = COALESCE($3, version_id),
|
||||
input_hash = COALESCE($4, input_hash),
|
||||
error_code = $5,
|
||||
error_detail = $6,
|
||||
completed_at = CASE WHEN $2 IN ('completed', 'failed') THEN now() ELSE completed_at END
|
||||
WHERE attempt_id = $1`,
|
||||
[attemptId, values.state, values.versionId ?? null, values.inputHash ?? null, values.errorCode ?? null, values.errorDetail ?? null]
|
||||
);
|
||||
}
|
||||
|
||||
async findReusableVersion(input: Pick<CreateVersionInput, "sourceId" | "sourceContentHash" | "processingFingerprint" | "metadataHash">): Promise<CatalogVersionRow | undefined> {
|
||||
if (!input.sourceContentHash) {
|
||||
return undefined;
|
||||
}
|
||||
const result = await this.pool.query<VersionDbRow>(
|
||||
`SELECT * FROM rag_source_versions
|
||||
WHERE source_id = $1
|
||||
AND source_content_hash = $2
|
||||
AND processing_fingerprint = $3
|
||||
AND metadata_hash = $4
|
||||
AND state NOT IN ('failed', 'rejected', 'purged')
|
||||
ORDER BY version_number DESC
|
||||
LIMIT 1`,
|
||||
[input.sourceId, input.sourceContentHash, input.processingFingerprint, input.metadataHash]
|
||||
);
|
||||
return result.rows[0] ? toVersion(result.rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async createPendingVersion(input: CreateVersionInput): Promise<CatalogVersionRow> {
|
||||
return withTransaction(this.pool, async (client) => {
|
||||
if (await this.isMaintenanceEnabled(client, true)) {
|
||||
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
|
||||
}
|
||||
|
||||
await client.query(
|
||||
`INSERT INTO rag_sources(source_id, source_type, source_ref)
|
||||
VALUES ($1, $2, $3)
|
||||
ON CONFLICT (source_id) DO NOTHING`,
|
||||
[input.sourceId, input.sourceType, input.sourceRef]
|
||||
);
|
||||
const source = await client.query<{ active_version_id: string | null; source_type: SourceType; source_ref: string }>(
|
||||
"SELECT active_version_id, source_type, source_ref FROM rag_sources WHERE source_id = $1 FOR UPDATE",
|
||||
[input.sourceId]
|
||||
);
|
||||
if (!source.rowCount) {
|
||||
throw new CatalogError("Source not found after creation", 500, "SOURCE_CREATE_FAILED");
|
||||
}
|
||||
if (source.rows[0].source_type !== input.sourceType || source.rows[0].source_ref !== input.sourceRef) {
|
||||
throw new CatalogError("sourceId already exists with a different sourceType or sourceRef", 409, "SOURCE_IDENTITY_CONFLICT");
|
||||
}
|
||||
const baseActiveVersionId = source.rows[0].active_version_id;
|
||||
const versionNumberResult = await client.query<{ next_version_number: string }>(
|
||||
"SELECT COALESCE(MAX(version_number), 0) + 1 AS next_version_number FROM rag_source_versions WHERE source_id = $1",
|
||||
[input.sourceId]
|
||||
);
|
||||
const versionNumber = Number(versionNumberResult.rows[0].next_version_number);
|
||||
const versionResult = await client.query<VersionDbRow>(
|
||||
`INSERT INTO rag_source_versions(
|
||||
source_id, version_number, previous_version_id, state, original_manifest_hash,
|
||||
source_content_hash, processing_fingerprint, metadata_hash, tags, activate_requested,
|
||||
base_active_version_id, embedding_provider, embedding_model, embedding_dimensions,
|
||||
qdrant_collection, expected_document_count, expected_point_count
|
||||
) VALUES ($1, $2, $3, 'pending', $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16)
|
||||
RETURNING *`,
|
||||
[
|
||||
input.sourceId,
|
||||
versionNumber,
|
||||
baseActiveVersionId,
|
||||
input.originalManifestHash,
|
||||
input.sourceContentHash,
|
||||
input.processingFingerprint,
|
||||
input.metadataHash,
|
||||
input.tags,
|
||||
input.activateRequested,
|
||||
baseActiveVersionId,
|
||||
input.embeddingProvider,
|
||||
input.embeddingModel,
|
||||
input.embeddingDimensions,
|
||||
input.qdrantCollection,
|
||||
input.expectedDocumentCount,
|
||||
input.expectedPointCount
|
||||
]
|
||||
);
|
||||
|
||||
const version = toVersion(versionResult.rows[0]);
|
||||
for (const document of input.documents) {
|
||||
await client.query(
|
||||
`INSERT INTO rag_version_documents(
|
||||
version_id, document_id, document_key, original_hash, original_hash_kind,
|
||||
content_hash, mime_type, title, index_state, chunk_count
|
||||
) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, 'pending', $9)`,
|
||||
[
|
||||
version.versionId,
|
||||
document.documentId,
|
||||
document.documentKey,
|
||||
document.originalHash,
|
||||
document.originalHashKind,
|
||||
document.contentHash,
|
||||
document.mimeType,
|
||||
document.title,
|
||||
document.chunkCount
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
return version;
|
||||
});
|
||||
}
|
||||
|
||||
async markIndexing(versionId: string): Promise<void> {
|
||||
const result = await this.pool.query(
|
||||
`UPDATE rag_source_versions SET state = 'indexing', indexing_started_at = now() WHERE version_id = $1 AND state = 'pending'`,
|
||||
[versionId]
|
||||
);
|
||||
if (result.rowCount !== 1) {
|
||||
throw new CatalogError("Version could not transition from pending to indexing", 409, "INVALID_VERSION_STATE");
|
||||
}
|
||||
await this.pool.query("UPDATE rag_version_documents SET index_state = 'indexing' WHERE version_id = $1", [versionId]);
|
||||
}
|
||||
|
||||
async markReady(versionId: string, verifiedPointCount: number): Promise<CatalogVersionRow> {
|
||||
return withTransaction(this.pool, async (client) => {
|
||||
await client.query("UPDATE rag_version_documents SET index_state = 'ready' WHERE version_id = $1", [versionId]);
|
||||
const result = await client.query<VersionDbRow>(
|
||||
`UPDATE rag_source_versions
|
||||
SET state = 'ready', verified_point_count = $2, ready_at = now()
|
||||
WHERE version_id = $1 AND state = 'indexing' AND expected_point_count = $2
|
||||
RETURNING *`,
|
||||
[versionId, verifiedPointCount]
|
||||
);
|
||||
if (result.rowCount !== 1) {
|
||||
throw new CatalogError("Version could not transition from indexing to ready", 409, "INVALID_VERSION_STATE");
|
||||
}
|
||||
return toVersion(result.rows[0]);
|
||||
});
|
||||
}
|
||||
|
||||
async markFailed(versionId: string | undefined, code: string, detail: string): Promise<void> {
|
||||
if (!versionId) {
|
||||
return;
|
||||
}
|
||||
const result = await this.pool.query(
|
||||
`UPDATE rag_source_versions
|
||||
SET state = 'failed', error_code = $2, error_detail = $3
|
||||
WHERE version_id = $1 AND state IN ('pending', 'indexing')`,
|
||||
[versionId, code, detail.slice(0, 2000)]
|
||||
);
|
||||
if (result.rowCount === 1) {
|
||||
await this.pool.query("UPDATE rag_version_documents SET index_state = 'failed', error_detail = $2 WHERE version_id = $1", [versionId, detail.slice(0, 2000)]);
|
||||
}
|
||||
}
|
||||
|
||||
async activateVersion(sourceId: string, versionId: string, expectedActiveVersionId: string | null | undefined): Promise<CatalogVersionRow> {
|
||||
const expected = normalizeExpectedActiveVersion(expectedActiveVersionId, true);
|
||||
return withTransaction(this.pool, async (client) => {
|
||||
if (await this.isMaintenanceEnabled(client, true)) {
|
||||
throw new CatalogError("Knowledge catalog is in maintenance mode", 503, "CATALOG_MAINTENANCE");
|
||||
}
|
||||
const source = await client.query<{ active_version_id: string | null }>(
|
||||
"SELECT active_version_id FROM rag_sources WHERE source_id = $1 FOR UPDATE",
|
||||
[sourceId]
|
||||
);
|
||||
if (!source.rowCount) {
|
||||
throw new CatalogError("Source not found", 404, "SOURCE_NOT_FOUND");
|
||||
}
|
||||
const currentActive = source.rows[0].active_version_id;
|
||||
if (currentActive !== expected) {
|
||||
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
|
||||
}
|
||||
|
||||
const version = await client.query<VersionDbRow>(
|
||||
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2 FOR UPDATE",
|
||||
[sourceId, versionId]
|
||||
);
|
||||
if (!version.rowCount) {
|
||||
throw new CatalogError("Version not found", 404, "VERSION_NOT_FOUND");
|
||||
}
|
||||
if (!["ready", "superseded", "active"].includes(version.rows[0].state)) {
|
||||
throw new CatalogError("Only ready or superseded versions can be activated", 409, "VERSION_NOT_ACTIVATABLE");
|
||||
}
|
||||
if (version.rows[0].state === "active") {
|
||||
if (currentActive !== versionId) {
|
||||
throw new CatalogError("Active version invariant is inconsistent", 503, "ACTIVE_VERSION_INVARIANT_FAILED");
|
||||
}
|
||||
return toVersion(version.rows[0]);
|
||||
}
|
||||
|
||||
if (currentActive) {
|
||||
await client.query(
|
||||
"UPDATE rag_source_versions SET state = 'superseded', superseded_at = now() WHERE source_id = $1 AND version_id = $2",
|
||||
[sourceId, currentActive]
|
||||
);
|
||||
}
|
||||
const updated = await client.query<VersionDbRow>(
|
||||
`UPDATE rag_source_versions
|
||||
SET state = 'active', activated_at = now(), error_code = NULL, error_detail = NULL
|
||||
WHERE source_id = $1 AND version_id = $2
|
||||
RETURNING *`,
|
||||
[sourceId, versionId]
|
||||
);
|
||||
if (updated.rowCount !== 1) {
|
||||
throw new CatalogError("Version activation failed", 500, "VERSION_ACTIVATION_FAILED");
|
||||
}
|
||||
await client.query("UPDATE rag_sources SET active_version_id = $2, needs_reingest = false WHERE source_id = $1", [sourceId, versionId]);
|
||||
return toVersion(updated.rows[0]);
|
||||
});
|
||||
}
|
||||
|
||||
async assertActiveVersionPrecondition(sourceId: string, expectedActiveVersionId: string | null | undefined): Promise<void> {
|
||||
const expected = normalizeExpectedActiveVersion(expectedActiveVersionId, true);
|
||||
const result = await this.pool.query<{ active_version_id: string | null }>(
|
||||
"SELECT active_version_id FROM rag_sources WHERE source_id = $1",
|
||||
[sourceId]
|
||||
);
|
||||
if (!result.rowCount) {
|
||||
throw new CatalogError("Source not found", 404, "SOURCE_NOT_FOUND");
|
||||
}
|
||||
if (result.rows[0].active_version_id !== expected) {
|
||||
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
|
||||
}
|
||||
}
|
||||
|
||||
async listSources(): Promise<AvailableScope[]> {
|
||||
const result = await this.pool.query<{
|
||||
source_id: string;
|
||||
source_ref: string;
|
||||
tags: string[] | null;
|
||||
active_version_id: string | null;
|
||||
version_number: string | number | null;
|
||||
state: SourceVersionState | null;
|
||||
needs_reingest: boolean;
|
||||
updated_at: Date;
|
||||
}>(
|
||||
`SELECT s.source_id, s.source_ref, v.tags, s.active_version_id, v.version_number, v.state, s.needs_reingest, s.updated_at
|
||||
FROM rag_sources s
|
||||
LEFT JOIN rag_source_versions v ON v.version_id = s.active_version_id
|
||||
WHERE s.disabled_at IS NULL
|
||||
ORDER BY s.source_ref ASC`
|
||||
);
|
||||
return result.rows.map((row) => ({
|
||||
sourceId: row.source_id,
|
||||
sourceRef: row.source_ref,
|
||||
chunkModes: ["documental", "codigo"] as ChunkMode[],
|
||||
tags: row.tags ?? [],
|
||||
activeVersionId: row.active_version_id,
|
||||
activeVersionNumber: row.version_number === null ? null : Number(row.version_number),
|
||||
state: row.state,
|
||||
needsReingest: row.needs_reingest,
|
||||
updatedAt: row.updated_at.toISOString()
|
||||
}));
|
||||
}
|
||||
|
||||
async getSource(sourceId: string): Promise<AvailableScope | undefined> {
|
||||
return (await this.listSources()).find((source) => source.sourceId === sourceId);
|
||||
}
|
||||
|
||||
async listVersions(sourceId: string): Promise<CatalogVersionRow[]> {
|
||||
const result = await this.pool.query<VersionDbRow>(
|
||||
"SELECT * FROM rag_source_versions WHERE source_id = $1 ORDER BY version_number DESC",
|
||||
[sourceId]
|
||||
);
|
||||
return result.rows.map(toVersion);
|
||||
}
|
||||
|
||||
async getVersion(sourceId: string, versionId: string): Promise<CatalogVersionRow | undefined> {
|
||||
const result = await this.pool.query<VersionDbRow>(
|
||||
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2",
|
||||
[sourceId, versionId]
|
||||
);
|
||||
return result.rows[0] ? toVersion(result.rows[0]) : undefined;
|
||||
}
|
||||
|
||||
async resolveActiveVersions(scope?: RetrieveScope): Promise<ActiveVersionScope[]> {
|
||||
const mustTags = scope?.tags ?? [];
|
||||
const result = await this.pool.query<{
|
||||
source_id: string;
|
||||
version_id: string;
|
||||
version_number: string | number;
|
||||
expected_point_count: string | number;
|
||||
embedding_dimensions: string | number;
|
||||
qdrant_collection: string;
|
||||
}>(
|
||||
`SELECT s.source_id, v.version_id, v.version_number, v.expected_point_count, v.embedding_dimensions, v.qdrant_collection
|
||||
FROM rag_sources s
|
||||
JOIN rag_source_versions v ON v.version_id = s.active_version_id AND v.state = 'active'
|
||||
WHERE s.disabled_at IS NULL
|
||||
AND ($1::text IS NULL OR s.source_id = $1)
|
||||
AND ($2::text IS NULL OR s.source_ref = $2)
|
||||
AND ($3::text[] IS NULL OR v.tags @> $3::text[])`,
|
||||
[scope?.sourceId ?? null, scope?.sourceRef ?? null, mustTags.length ? mustTags : null]
|
||||
);
|
||||
return result.rows.map((row) => ({
|
||||
sourceId: row.source_id,
|
||||
sourceVersionId: row.version_id,
|
||||
sourceVersionNumber: Number(row.version_number),
|
||||
expectedPointCount: Number(row.expected_point_count),
|
||||
embeddingDimensions: Number(row.embedding_dimensions),
|
||||
qdrantCollection: row.qdrant_collection
|
||||
}));
|
||||
}
|
||||
|
||||
async markNeedsReingest(sourceId: string): Promise<void> {
|
||||
await this.pool.query("UPDATE rag_sources SET needs_reingest = true WHERE source_id = $1", [sourceId]);
|
||||
}
|
||||
|
||||
async rollback(sourceId: string, targetVersionId: string, expectedActiveVersionId: string): Promise<CatalogVersionRow> {
|
||||
return this.withSourceLock(sourceId, async () => this.withVersionExclusiveLock(targetVersionId, async () => {
|
||||
const version = await this.activateVersion(sourceId, targetVersionId, expectedActiveVersionId);
|
||||
const source = await this.getSource(sourceId);
|
||||
if (source?.activeVersionId !== targetVersionId) {
|
||||
throw new CatalogError("Rollback verification failed", 503, "ROLLBACK_VERIFICATION_FAILED");
|
||||
}
|
||||
return version;
|
||||
}));
|
||||
}
|
||||
|
||||
async markPurging(sourceId: string, versionId: string): Promise<CatalogVersionRow> {
|
||||
return withTransaction(this.pool, async (client) => {
|
||||
const version = await client.query<VersionDbRow>(
|
||||
"SELECT * FROM rag_source_versions WHERE source_id = $1 AND version_id = $2 FOR UPDATE",
|
||||
[sourceId, versionId]
|
||||
);
|
||||
if (!version.rowCount) {
|
||||
throw new CatalogError("Version not found", 404, "VERSION_NOT_FOUND");
|
||||
}
|
||||
if (version.rows[0].state === "active") {
|
||||
throw new CatalogError("Active versions cannot be purged", 409, "ACTIVE_VERSION_PURGE_FORBIDDEN");
|
||||
}
|
||||
if (!["ready", "superseded", "failed", "rejected", "purging"].includes(version.rows[0].state)) {
|
||||
throw new CatalogError("Version state cannot be purged", 409, "VERSION_NOT_PURGEABLE");
|
||||
}
|
||||
const updated = await client.query<VersionDbRow>(
|
||||
"UPDATE rag_source_versions SET state = 'purging' WHERE source_id = $1 AND version_id = $2 RETURNING *",
|
||||
[sourceId, versionId]
|
||||
);
|
||||
return toVersion(updated.rows[0]);
|
||||
});
|
||||
}
|
||||
|
||||
async markPurged(versionId: string): Promise<void> {
|
||||
await withTransaction(this.pool, async (client) => {
|
||||
await client.query("DELETE FROM rag_version_documents WHERE version_id = $1", [versionId]);
|
||||
const result = await client.query(
|
||||
`UPDATE rag_source_versions
|
||||
SET state = 'purged', verified_point_count = 0, artifact_state = 'retention_deleted', purged_at = now()
|
||||
WHERE version_id = $1 AND state = 'purging'`,
|
||||
[versionId]
|
||||
);
|
||||
if (result.rowCount !== 1) {
|
||||
throw new CatalogError("Version could not transition from purging to purged", 409, "INVALID_VERSION_STATE");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async validateActiveInvariant(): Promise<string[]> {
|
||||
const result = await this.pool.query<{ source_id: string }>(
|
||||
`SELECT s.source_id
|
||||
FROM rag_sources s
|
||||
LEFT JOIN rag_source_versions v
|
||||
ON v.version_id = s.active_version_id
|
||||
AND v.source_id = s.source_id
|
||||
AND v.state = 'active'
|
||||
WHERE (s.active_version_id IS NOT NULL AND v.version_id IS NULL)
|
||||
UNION
|
||||
SELECT v.source_id
|
||||
FROM rag_source_versions v
|
||||
JOIN rag_sources s ON s.source_id = v.source_id
|
||||
WHERE v.state = 'active'
|
||||
AND s.active_version_id IS DISTINCT FROM v.version_id`
|
||||
);
|
||||
return result.rows.map((row) => row.source_id);
|
||||
}
|
||||
|
||||
async listOrphanedIndexingCandidates(maxAgeMs = 1800000): Promise<OrphanedVersionCandidate[]> {
|
||||
const result = await this.pool.query<{
|
||||
version_id: string;
|
||||
source_id: string;
|
||||
expected_document_count: string | number;
|
||||
expected_point_count: string | number;
|
||||
embedding_dimensions: string | number;
|
||||
qdrant_collection: string;
|
||||
}>(
|
||||
`SELECT version_id, source_id, expected_document_count, expected_point_count, embedding_dimensions, qdrant_collection
|
||||
FROM rag_source_versions
|
||||
WHERE (
|
||||
state = 'pending'
|
||||
AND created_at < now() - ($1::text || ' milliseconds')::interval
|
||||
) OR (
|
||||
state = 'indexing'
|
||||
AND indexing_started_at IS NOT NULL
|
||||
AND indexing_started_at < now() - ($1::text || ' milliseconds')::interval
|
||||
)`,
|
||||
[String(maxAgeMs)]
|
||||
);
|
||||
return result.rows.map((row) => ({
|
||||
sourceId: row.source_id,
|
||||
versionId: row.version_id,
|
||||
expectedDocumentCount: Number(row.expected_document_count),
|
||||
expectedPointCount: Number(row.expected_point_count),
|
||||
embeddingDimensions: Number(row.embedding_dimensions),
|
||||
qdrantCollection: row.qdrant_collection
|
||||
}));
|
||||
}
|
||||
|
||||
async hasCompleteVersionDocuments(versionId: string, expectedDocumentCount: number, expectedPointCount: number): Promise<boolean> {
|
||||
const result = await this.pool.query<{
|
||||
document_count: string | number;
|
||||
point_count: string | number;
|
||||
missing_content_count: string | number;
|
||||
}>(
|
||||
`SELECT COUNT(*) AS document_count,
|
||||
COALESCE(SUM(chunk_count), 0) AS point_count,
|
||||
COUNT(*) FILTER (WHERE content_hash IS NULL) AS missing_content_count
|
||||
FROM rag_version_documents
|
||||
WHERE version_id = $1`,
|
||||
[versionId]
|
||||
);
|
||||
const row = result.rows[0];
|
||||
return Number(row?.document_count ?? 0) === expectedDocumentCount
|
||||
&& Number(row?.point_count ?? 0) === expectedPointCount
|
||||
&& Number(row?.missing_content_count ?? 0) === 0;
|
||||
}
|
||||
|
||||
async recoverVersionReady(versionId: string, verifiedPointCount: number): Promise<boolean> {
|
||||
return withTransaction(this.pool, async (client) => {
|
||||
await client.query("UPDATE rag_version_documents SET index_state = 'ready' WHERE version_id = $1", [versionId]);
|
||||
const result = await client.query(
|
||||
`UPDATE rag_source_versions
|
||||
SET state = 'ready', verified_point_count = $2, ready_at = now(), error_code = NULL, error_detail = NULL
|
||||
WHERE version_id = $1 AND state IN ('pending', 'indexing') AND expected_point_count = $2`,
|
||||
[versionId, verifiedPointCount]
|
||||
);
|
||||
return result.rowCount === 1;
|
||||
});
|
||||
}
|
||||
|
||||
async recoverOrphanedIndexingVersions(maxAgeMs = 1800000): Promise<Array<{ versionId: string; sourceId: string; action: "ready" | "failed" }>> {
|
||||
const candidates = await this.listOrphanedIndexingCandidates(maxAgeMs);
|
||||
const recovered: Array<{ versionId: string; sourceId: string; action: "ready" | "failed" }> = [];
|
||||
for (const candidate of candidates) {
|
||||
await this.markFailed(candidate.versionId, "ORPHANED_INDEXING", "Recovered by reconciler after stale pending/indexing state.");
|
||||
recovered.push({ versionId: candidate.versionId, sourceId: candidate.sourceId, action: "failed" });
|
||||
}
|
||||
return recovered;
|
||||
}
|
||||
}
|
||||
export { CatalogError };
|
||||
|
|
@ -1,25 +1,72 @@
|
|||
import path from "node:path";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { parseDocument, isSupportedDocument } from "../parsers/parser-registry.js";
|
||||
import { chunkDocument, codeChunkingPolicy, documentalChunkingPolicy } from "../process/chunking.js";
|
||||
import type { EmbeddingProvider } from "../embeddings/provider.js";
|
||||
import type { VectorStoreClient } from "../vectorstore/client.js";
|
||||
import type { IngestResult, IngestSourceInput, IngestedChunk, RetrieveScope } from "../../shared/types/rag.js";
|
||||
import { buildChunkId, buildDocumentId, buildQdrantPointId, buildSourceId, normalizeDocumentKey } from "../../shared/utils/ids.js";
|
||||
import {
|
||||
buildChunkId,
|
||||
buildDocumentId,
|
||||
buildMetadataHash,
|
||||
buildProcessingFingerprint,
|
||||
buildQdrantPointId,
|
||||
buildSourceId,
|
||||
buildVersionedQdrantPointId,
|
||||
hashOrderedPairs,
|
||||
normalizeContentForHash,
|
||||
normalizeDocumentKey,
|
||||
sha256Hex
|
||||
} from "../../shared/utils/ids.js";
|
||||
import { listFilesRecursively } from "../../shared/utils/files.js";
|
||||
import { env } from "../../config/env.js";
|
||||
import { CatalogError, type CatalogDocumentInput, type CatalogRepository } from "../catalog/repository.js";
|
||||
|
||||
type PreparedDocument = CatalogDocumentInput & {
|
||||
chunks: ReturnType<typeof chunkDocument>;
|
||||
chunkMode: "documental" | "codigo";
|
||||
content: string;
|
||||
};
|
||||
|
||||
type OriginalDocument = {
|
||||
filePath: string;
|
||||
documentKey: string;
|
||||
documentId: string;
|
||||
bytes: Buffer;
|
||||
originalHash: string;
|
||||
};
|
||||
|
||||
export class IngestService {
|
||||
constructor(
|
||||
private readonly embeddingProvider: EmbeddingProvider,
|
||||
private readonly vectorStore: VectorStoreClient
|
||||
private readonly vectorStore: VectorStoreClient,
|
||||
private readonly catalog?: CatalogRepository
|
||||
) {}
|
||||
|
||||
async cleanup(scope: RetrieveScope): Promise<{ deleted: number }> {
|
||||
if (!env.ingestWritesEnabled) {
|
||||
throw new CatalogError("Ingest writes are disabled during knowledge catalog maintenance", 503, "INGEST_WRITES_DISABLED");
|
||||
}
|
||||
const count = await this.vectorStore.deleteScope(scope);
|
||||
return { deleted: count };
|
||||
}
|
||||
|
||||
async ingest(source: IngestSourceInput): Promise<IngestResult> {
|
||||
if (!env.ingestWritesEnabled) {
|
||||
throw new CatalogError("Ingest writes are disabled during knowledge catalog maintenance", 503, "INGEST_WRITES_DISABLED");
|
||||
}
|
||||
|
||||
if (env.knowledgeLifecycleEnforced) {
|
||||
if (!this.catalog) {
|
||||
throw new CatalogError("Knowledge lifecycle is enforced but PostgreSQL is not configured", 503, "CATALOG_UNAVAILABLE");
|
||||
}
|
||||
return this.ingestWithLifecycle(source, this.catalog);
|
||||
}
|
||||
|
||||
return this.ingestLegacy(source);
|
||||
}
|
||||
|
||||
private async ingestLegacy(source: IngestSourceInput): Promise<IngestResult> {
|
||||
const sourceId = source.sourceId ?? buildSourceId(source.sourceType, source.sourceRef);
|
||||
const discoveredFiles = await this.resolveInputFiles(source);
|
||||
const supportedFiles = discoveredFiles.filter(isSupportedDocument);
|
||||
|
|
@ -79,6 +126,7 @@ export class IngestService {
|
|||
|
||||
return {
|
||||
accepted: true,
|
||||
sourceId,
|
||||
source,
|
||||
filesDiscovered: supportedFiles.length,
|
||||
documentsProcessed,
|
||||
|
|
@ -87,6 +135,292 @@ export class IngestService {
|
|||
};
|
||||
}
|
||||
|
||||
private async ingestWithLifecycle(source: IngestSourceInput, catalog: CatalogRepository): Promise<IngestResult> {
|
||||
const activate = source.activate ?? true;
|
||||
if (activate && source.expectedActiveVersionId === undefined) {
|
||||
throw new CatalogError("expectedActiveVersionId is required when activate is true", 400, "EXPECTED_ACTIVE_VERSION_REQUIRED");
|
||||
}
|
||||
|
||||
const sourceId = source.sourceId ?? buildSourceId(source.sourceType, source.sourceRef);
|
||||
const attemptId = await catalog.beginAttempt({ ...source, sourceId });
|
||||
let versionId: string | undefined;
|
||||
let canMarkVersionFailed = false;
|
||||
|
||||
try {
|
||||
await catalog.updateAttempt(attemptId, { state: "processing" });
|
||||
const discoveredFiles = await this.resolveInputFiles(source);
|
||||
const supportedFiles = discoveredFiles.filter(isSupportedDocument);
|
||||
const originalDocuments = await this.readOriginalDocuments(source, sourceId, supportedFiles);
|
||||
if (originalDocuments.length === 0) {
|
||||
throw new CatalogError("The source has no supported documents", 422, "EMPTY_SOURCE");
|
||||
}
|
||||
const originalManifestHash = hashOrderedPairs(originalDocuments.map((document) => [document.documentKey, document.originalHash]));
|
||||
if (!originalManifestHash) {
|
||||
throw new CatalogError("Original manifest hash could not be calculated", 422, "ORIGINAL_MANIFEST_HASH_UNAVAILABLE");
|
||||
}
|
||||
const preparedDocuments = await this.prepareDocuments(originalDocuments);
|
||||
|
||||
if (preparedDocuments.length === 0) {
|
||||
throw new CatalogError("The source has no supported documents with useful content", 422, "EMPTY_SOURCE");
|
||||
}
|
||||
|
||||
const tags = source.tags ?? [];
|
||||
const sourceContentHash = hashOrderedPairs(preparedDocuments.map((document) => [document.documentKey, document.contentHash]));
|
||||
|
||||
const embeddingDimensions = this.embeddingProvider.dimensions ?? 4096;
|
||||
const processingFingerprint = buildProcessingFingerprint({
|
||||
parserVersion: "native-v1",
|
||||
normalizationPolicy: "bom-crlf-trim-final-lf-v1",
|
||||
chunking: {
|
||||
code: codeChunkingPolicy,
|
||||
documental: documentalChunkingPolicy
|
||||
},
|
||||
embeddingProvider: this.embeddingProvider.providerName,
|
||||
embeddingModel: this.embeddingProvider.modelName,
|
||||
embeddingDimensions
|
||||
});
|
||||
const metadataHash = buildMetadataHash(tags, { sourceType: source.sourceType, sourceRef: source.sourceRef });
|
||||
|
||||
return await catalog.withSourceLock(sourceId, async () => {
|
||||
const reusable = await catalog.findReusableVersion({ sourceId, sourceContentHash, processingFingerprint, metadataHash });
|
||||
if (reusable?.state === "active") {
|
||||
if (activate) {
|
||||
await catalog.assertActiveVersionPrecondition(sourceId, source.expectedActiveVersionId);
|
||||
}
|
||||
await catalog.updateAttempt(attemptId, { state: "completed", versionId: reusable.versionId, inputHash: originalManifestHash });
|
||||
return {
|
||||
accepted: true,
|
||||
sourceId,
|
||||
versionId: reusable.versionId,
|
||||
versionNumber: reusable.versionNumber,
|
||||
state: reusable.state,
|
||||
previousVersionId: reusable.previousVersionId,
|
||||
filesDiscovered: supportedFiles.length,
|
||||
documentsProcessed: reusable.expectedDocumentCount,
|
||||
chunksStored: 0,
|
||||
activated: true,
|
||||
noOp: true,
|
||||
collectionName: env.qdrantCollection
|
||||
};
|
||||
}
|
||||
|
||||
if (reusable && ["pending", "indexing", "review_required"].includes(reusable.state)) {
|
||||
throw new CatalogError("A reusable version is already being indexed", 409, "REUSABLE_VERSION_IN_PROGRESS");
|
||||
}
|
||||
|
||||
if (reusable && ["ready", "superseded"].includes(reusable.state)) {
|
||||
if (!activate) {
|
||||
await catalog.updateAttempt(attemptId, { state: "completed", versionId: reusable.versionId, inputHash: originalManifestHash });
|
||||
return {
|
||||
accepted: true,
|
||||
sourceId,
|
||||
versionId: reusable.versionId,
|
||||
versionNumber: reusable.versionNumber,
|
||||
state: reusable.state,
|
||||
previousVersionId: reusable.previousVersionId,
|
||||
filesDiscovered: supportedFiles.length,
|
||||
documentsProcessed: reusable.expectedDocumentCount,
|
||||
chunksStored: 0,
|
||||
activated: false,
|
||||
noOp: false,
|
||||
collectionName: env.qdrantCollection
|
||||
};
|
||||
}
|
||||
const activated = await catalog.activateVersion(sourceId, reusable.versionId, source.expectedActiveVersionId);
|
||||
await catalog.updateAttempt(attemptId, { state: "completed", versionId: activated.versionId, inputHash: originalManifestHash });
|
||||
return {
|
||||
accepted: true,
|
||||
sourceId,
|
||||
versionId: activated.versionId,
|
||||
versionNumber: activated.versionNumber,
|
||||
state: activated.state,
|
||||
previousVersionId: activated.previousVersionId,
|
||||
filesDiscovered: supportedFiles.length,
|
||||
documentsProcessed: activated.expectedDocumentCount,
|
||||
chunksStored: 0,
|
||||
activated: true,
|
||||
noOp: false,
|
||||
collectionName: env.qdrantCollection
|
||||
};
|
||||
}
|
||||
|
||||
const expectedPointCount = preparedDocuments.reduce((total, document) => total + document.chunkCount, 0);
|
||||
const created = await catalog.createPendingVersion({
|
||||
sourceId,
|
||||
sourceType: source.sourceType,
|
||||
sourceRef: source.sourceRef,
|
||||
originalManifestHash,
|
||||
sourceContentHash,
|
||||
processingFingerprint,
|
||||
metadataHash,
|
||||
tags,
|
||||
activateRequested: activate,
|
||||
embeddingProvider: this.embeddingProvider.providerName,
|
||||
embeddingModel: this.embeddingProvider.modelName,
|
||||
embeddingDimensions,
|
||||
qdrantCollection: env.qdrantCollection,
|
||||
expectedDocumentCount: preparedDocuments.length,
|
||||
expectedPointCount,
|
||||
documents: preparedDocuments
|
||||
});
|
||||
versionId = created.versionId;
|
||||
canMarkVersionFailed = true;
|
||||
await catalog.updateAttempt(attemptId, { state: "processing", versionId, inputHash: originalManifestHash });
|
||||
await catalog.markIndexing(versionId);
|
||||
|
||||
const chunksStored = await this.writeVersionedChunks(source, sourceId, created.versionId, created.versionNumber, preparedDocuments, processingFingerprint, tags, embeddingDimensions);
|
||||
const verifiedPointCount = await this.vectorStore.countVersionPoints(created.versionId);
|
||||
if (verifiedPointCount !== expectedPointCount || chunksStored !== expectedPointCount) {
|
||||
throw new CatalogError("Indexed point count does not match the expected catalog count", 503, "SOURCE_VERSION_INCONSISTENT");
|
||||
}
|
||||
|
||||
const ready = await catalog.markReady(created.versionId, verifiedPointCount);
|
||||
canMarkVersionFailed = false;
|
||||
const finalVersion = activate
|
||||
? await catalog.activateVersion(sourceId, ready.versionId, source.expectedActiveVersionId)
|
||||
: ready;
|
||||
await catalog.updateAttempt(attemptId, { state: "completed", versionId: finalVersion.versionId, inputHash: originalManifestHash });
|
||||
|
||||
return {
|
||||
accepted: true,
|
||||
sourceId,
|
||||
versionId: finalVersion.versionId,
|
||||
versionNumber: finalVersion.versionNumber,
|
||||
state: finalVersion.state,
|
||||
previousVersionId: finalVersion.previousVersionId,
|
||||
filesDiscovered: supportedFiles.length,
|
||||
documentsProcessed: preparedDocuments.length,
|
||||
chunksStored,
|
||||
activated: finalVersion.state === "active",
|
||||
noOp: false,
|
||||
collectionName: env.qdrantCollection
|
||||
};
|
||||
});
|
||||
} catch (error) {
|
||||
const detail = error instanceof Error ? error.message : "Unknown lifecycle ingest error";
|
||||
if (canMarkVersionFailed) {
|
||||
await catalog.markFailed(versionId, error instanceof CatalogError ? error.code : "INGEST_FAILED", detail).catch(() => undefined);
|
||||
}
|
||||
await catalog.updateAttempt(attemptId, {
|
||||
state: "failed",
|
||||
versionId,
|
||||
errorCode: error instanceof CatalogError ? error.code : "INGEST_FAILED",
|
||||
errorDetail: detail
|
||||
}).catch(() => undefined);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
private async readOriginalDocuments(source: IngestSourceInput, sourceId: string, files: string[]): Promise<OriginalDocument[]> {
|
||||
const originals: OriginalDocument[] = [];
|
||||
for (const filePath of files) {
|
||||
const bytes = await readFile(filePath);
|
||||
const documentKey = this.buildDocumentKey(source, filePath);
|
||||
originals.push({
|
||||
filePath,
|
||||
documentKey,
|
||||
documentId: buildDocumentId(sourceId, documentKey),
|
||||
bytes,
|
||||
originalHash: sha256Hex(bytes)
|
||||
});
|
||||
}
|
||||
return originals.sort((left, right) => Buffer.compare(Buffer.from(left.documentKey, "utf8"), Buffer.from(right.documentKey, "utf8")));
|
||||
}
|
||||
|
||||
private async prepareDocuments(originals: OriginalDocument[]): Promise<PreparedDocument[]> {
|
||||
const documents: PreparedDocument[] = [];
|
||||
for (const original of originals) {
|
||||
const parsed = await parseDocument(original.filePath);
|
||||
const normalizedContent = normalizeContentForHash(parsed.content);
|
||||
if (!normalizedContent.trim()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const chunkingPolicy = parsed.chunkMode === "codigo" ? codeChunkingPolicy : documentalChunkingPolicy;
|
||||
const chunks = chunkDocument(parsed.title, normalizedContent, chunkingPolicy);
|
||||
if (chunks.length === 0) {
|
||||
continue;
|
||||
}
|
||||
|
||||
documents.push({
|
||||
documentId: original.documentId,
|
||||
documentKey: original.documentKey,
|
||||
originalHash: original.originalHash,
|
||||
originalHashKind: "bytes",
|
||||
contentHash: sha256Hex(normalizedContent),
|
||||
mimeType: parsed.mimeType,
|
||||
title: parsed.title,
|
||||
chunkCount: chunks.length,
|
||||
chunks,
|
||||
chunkMode: chunkingPolicy.mode,
|
||||
content: normalizedContent
|
||||
});
|
||||
}
|
||||
return documents.sort((left, right) => Buffer.compare(Buffer.from(left.documentKey, "utf8"), Buffer.from(right.documentKey, "utf8")));
|
||||
}
|
||||
|
||||
private buildDocumentKey(source: IngestSourceInput, filePath: string): string {
|
||||
const logicalRef = source.sourceType === "folder"
|
||||
? path.relative(path.resolve(source.readPath ?? source.sourceRef), path.resolve(filePath))
|
||||
: source.sourceRef;
|
||||
return normalizeDocumentKey(logicalRef || path.basename(filePath));
|
||||
}
|
||||
|
||||
private async writeVersionedChunks(
|
||||
source: IngestSourceInput,
|
||||
sourceId: string,
|
||||
versionId: string,
|
||||
versionNumber: number,
|
||||
documents: PreparedDocument[],
|
||||
processingFingerprint: string,
|
||||
tags: string[],
|
||||
expectedEmbeddingDimensions: number
|
||||
): Promise<number> {
|
||||
let chunksStored = 0;
|
||||
for (const document of documents) {
|
||||
const embeddings = await this.embeddingProvider.embed(document.chunks.map((chunk) => chunk.content));
|
||||
if (embeddings.length !== document.chunks.length || embeddings.some((embedding) => embedding.length !== expectedEmbeddingDimensions)) {
|
||||
throw new CatalogError("Embedding provider returned invalid dimensions", 503, "EMBEDDING_DIMENSIONS_INVALID");
|
||||
}
|
||||
const qdrantChunks: IngestedChunk[] = document.chunks.map((chunk, index) => {
|
||||
const chunkId = buildChunkId(document.documentId, document.chunkMode, chunk.index);
|
||||
return {
|
||||
id: buildVersionedQdrantPointId(versionId, chunkId),
|
||||
vector: embeddings[index],
|
||||
payload: {
|
||||
chunk_id: chunkId,
|
||||
source_id: sourceId,
|
||||
source_version_id: versionId,
|
||||
source_version_number: versionNumber,
|
||||
source_type: source.sourceType,
|
||||
source_ref: source.sourceRef,
|
||||
document_id: document.documentId,
|
||||
document_key: document.documentKey,
|
||||
document_content_hash: document.contentHash,
|
||||
title: document.title,
|
||||
mime_type: document.mimeType,
|
||||
section_title: chunk.sectionTitle,
|
||||
chunk_mode: document.chunkMode,
|
||||
chunk_index: chunk.index,
|
||||
start_line: chunk.startLine,
|
||||
end_line: chunk.endLine,
|
||||
content: chunk.content,
|
||||
tags,
|
||||
embedding_provider: this.embeddingProvider.providerName,
|
||||
embedding_model: this.embeddingProvider.modelName,
|
||||
embedding_dimensions: embeddings[index]?.length ?? 0,
|
||||
processing_fingerprint: processingFingerprint,
|
||||
indexed_at: new Date().toISOString(),
|
||||
write_state: "staged"
|
||||
}
|
||||
};
|
||||
});
|
||||
await this.vectorStore.upsert(qdrantChunks);
|
||||
chunksStored += qdrantChunks.length;
|
||||
}
|
||||
return chunksStored;
|
||||
}
|
||||
|
||||
private async resolveInputFiles(source: IngestSourceInput): Promise<string[]> {
|
||||
const targetPath = source.readPath ?? source.sourceRef;
|
||||
if (source.sourceType === "file") {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
import type { EmbeddingProvider } from "../embeddings/provider.js";
|
||||
import type { VectorStoreClient } from "../vectorstore/client.js";
|
||||
import type { ChunkMode, RetrieveIntent, RetrieveResponse, RetrieveScope } from "../../shared/types/rag.js";
|
||||
import { env } from "../../config/env.js";
|
||||
import { CatalogError, type ActiveVersionScope, type CatalogRepository } from "../catalog/repository.js";
|
||||
|
||||
function unique(values: string[]): string[] {
|
||||
return [...new Set(values.filter(Boolean))];
|
||||
|
|
@ -176,13 +178,18 @@ async function embedQuery(provider: EmbeddingProvider, query: string): Promise<n
|
|||
export class RetrieveService {
|
||||
constructor(
|
||||
private readonly embeddingProvider: EmbeddingProvider,
|
||||
private readonly vectorStore: VectorStoreClient
|
||||
private readonly vectorStore: VectorStoreClient,
|
||||
private readonly catalog?: CatalogRepository
|
||||
) {}
|
||||
|
||||
async retrieve(mode: ChunkMode, intent: RetrieveIntent, query: string, scope?: RetrieveScope): Promise<RetrieveResponse> {
|
||||
const items = intent === "bootstrap"
|
||||
? await this.retrieveBootstrap(mode, query, scope)
|
||||
: await this.retrieveSpecific(mode, query, scope);
|
||||
const activeVersions = await this.resolveActiveVersions(scope);
|
||||
const items = await this.withActiveVersionReadLocks(activeVersions, async () => {
|
||||
await this.validateActiveVersions(activeVersions);
|
||||
return intent === "bootstrap"
|
||||
? this.retrieveBootstrap(mode, query, scope, activeVersions)
|
||||
: this.retrieveSpecific(mode, query, scope, activeVersions);
|
||||
});
|
||||
|
||||
const topics = unique(items.map((item) => item.title)).slice(0, 8);
|
||||
const criticalPoints = items
|
||||
|
|
@ -207,13 +214,58 @@ export class RetrieveService {
|
|||
};
|
||||
}
|
||||
|
||||
private async retrieveSpecific(mode: ChunkMode, query: string, scope?: RetrieveScope) {
|
||||
private async withActiveVersionReadLocks<T>(activeVersions: ActiveVersionScope[] | undefined, handler: () => Promise<T>): Promise<T> {
|
||||
if (!env.knowledgeLifecycleEnforced || !this.catalog || !activeVersions) {
|
||||
return handler();
|
||||
}
|
||||
return this.catalog.withVersionSharedLocks(activeVersions.map((version) => version.sourceVersionId), handler);
|
||||
}
|
||||
|
||||
private async resolveActiveVersions(scope?: RetrieveScope): Promise<ActiveVersionScope[] | undefined> {
|
||||
if (!env.knowledgeLifecycleEnforced) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
if (!this.catalog) {
|
||||
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
|
||||
}
|
||||
|
||||
try {
|
||||
return await this.catalog.resolveActiveVersions(scope);
|
||||
} catch (error) {
|
||||
if (error instanceof CatalogError) {
|
||||
throw error;
|
||||
}
|
||||
throw new CatalogError("Knowledge catalog is unavailable", 503, "CATALOG_UNAVAILABLE");
|
||||
}
|
||||
}
|
||||
|
||||
private async validateActiveVersions(activeVersions: ActiveVersionScope[] | undefined): Promise<void> {
|
||||
if (!env.knowledgeLifecycleEnforced || !this.catalog || !activeVersions) {
|
||||
return;
|
||||
}
|
||||
|
||||
for (const version of activeVersions) {
|
||||
if (version.qdrantCollection !== env.qdrantCollection) {
|
||||
await this.catalog.markNeedsReingest(version.sourceId);
|
||||
throw new CatalogError("Active source version targets a different Qdrant collection", 503, "SOURCE_VERSION_INCONSISTENT");
|
||||
}
|
||||
await this.vectorStore.validateCollectionDimensions(version.embeddingDimensions);
|
||||
const count = await this.vectorStore.countVersionPoints(version.sourceVersionId);
|
||||
if (count !== version.expectedPointCount) {
|
||||
await this.catalog.markNeedsReingest(version.sourceId);
|
||||
throw new CatalogError("Active source version is inconsistent", 503, "SOURCE_VERSION_INCONSISTENT");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async retrieveSpecific(mode: ChunkMode, query: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
|
||||
const queries = buildSpecificQueries(query);
|
||||
const merged = new Map<string, RetrieveResponse["items"][number]>();
|
||||
|
||||
for (const subquery of queries) {
|
||||
const queryVector = await embedQuery(this.embeddingProvider, subquery);
|
||||
const results = await this.vectorStore.search(queryVector, 6, mode, scope);
|
||||
const results = await this.vectorStore.search(queryVector, 6, mode, scope, activeVersions);
|
||||
|
||||
for (const item of results) {
|
||||
const existing = merged.get(item.chunkId);
|
||||
|
|
@ -226,13 +278,13 @@ export class RetrieveService {
|
|||
return rankItems(query, [...merged.values()]).slice(0, 8);
|
||||
}
|
||||
|
||||
private async retrieveBootstrap(mode: ChunkMode, query: string, scope?: RetrieveScope) {
|
||||
private async retrieveBootstrap(mode: ChunkMode, query: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
|
||||
const queries = buildBootstrapQueries(query);
|
||||
const merged = new Map<string, RetrieveResponse["items"][number]>();
|
||||
|
||||
for (const subquery of queries) {
|
||||
const queryVector = await embedQuery(this.embeddingProvider, subquery);
|
||||
const results = await this.vectorStore.search(queryVector, 6, mode, scope);
|
||||
const results = await this.vectorStore.search(queryVector, 6, mode, scope, activeVersions);
|
||||
|
||||
for (const item of results) {
|
||||
const existing = merged.get(item.chunkId);
|
||||
|
|
@ -243,7 +295,7 @@ export class RetrieveService {
|
|||
}
|
||||
|
||||
if (merged.size === 0 && scope && (scope.sourceId || scope.sourceRef || (scope.tags && scope.tags.length > 0))) {
|
||||
const explored = await this.vectorStore.browseScope(12, mode, scope);
|
||||
const explored = await this.vectorStore.browseScope(12, mode, scope, activeVersions);
|
||||
for (const item of explored) {
|
||||
merged.set(item.chunkId, item);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
import { QdrantClient } from "@qdrant/js-client-rest";
|
||||
import { env } from "../../config/env.js";
|
||||
import type { AvailableScope, IngestedChunk, RetrieveScope, RetrievedItem } from "../../shared/types/rag.js";
|
||||
import type { ActiveVersionScope } from "../catalog/repository.js";
|
||||
|
||||
export function buildQdrantClient(): QdrantClient {
|
||||
const url = new URL(env.qdrantUrl);
|
||||
|
|
@ -17,14 +18,32 @@ export interface VectorStoreClient {
|
|||
readonly kind: string;
|
||||
healthcheck(): Promise<{ ok: boolean; kind: string }>;
|
||||
ensureCollection(vectorSize: number): Promise<void>;
|
||||
ensurePayloadIndexes(): Promise<void>;
|
||||
upsert(chunks: IngestedChunk[]): Promise<void>;
|
||||
search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]>;
|
||||
search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]>;
|
||||
listScopes(): Promise<AvailableScope[]>;
|
||||
browseScope(limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]>;
|
||||
browseScope(limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]>;
|
||||
deleteScope(scope: RetrieveScope): Promise<number>;
|
||||
validateCollectionDimensions(expectedDimensions: number): Promise<void>;
|
||||
countVersionPoints(sourceVersionId: string): Promise<number>;
|
||||
deleteVersionPoints(sourceVersionId: string): Promise<number>;
|
||||
}
|
||||
|
||||
function buildSearchFilter(mode?: string, scope?: RetrieveScope) {
|
||||
function extractVectorSize(collectionInfo: unknown): number | undefined {
|
||||
const info = collectionInfo as { config?: { params?: { vectors?: unknown } } };
|
||||
const vectors = info.config?.params?.vectors;
|
||||
if (!vectors || typeof vectors !== "object") {
|
||||
return undefined;
|
||||
}
|
||||
if ("size" in vectors && typeof vectors.size === "number") {
|
||||
return vectors.size;
|
||||
}
|
||||
const namedVectors = Object.values(vectors as Record<string, unknown>);
|
||||
const firstNamed = namedVectors[0] as { size?: unknown } | undefined;
|
||||
return typeof firstNamed?.size === "number" ? firstNamed.size : undefined;
|
||||
}
|
||||
|
||||
function buildSearchFilter(mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]) {
|
||||
const must: Array<Record<string, unknown>> = [];
|
||||
|
||||
if (mode && mode !== "auto") {
|
||||
|
|
@ -57,6 +76,13 @@ function buildSearchFilter(mode?: string, scope?: RetrieveScope) {
|
|||
}
|
||||
}
|
||||
|
||||
if (activeVersions) {
|
||||
must.push({
|
||||
key: "source_version_id",
|
||||
match: { any: activeVersions.length > 0 ? activeVersions.map((version) => version.sourceVersionId) : ["__no_active_versions__"] }
|
||||
});
|
||||
}
|
||||
|
||||
return must.length > 0 ? { must } : undefined;
|
||||
}
|
||||
|
||||
|
|
@ -93,18 +119,47 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
|
|||
distance: "Cosine"
|
||||
}
|
||||
});
|
||||
} else {
|
||||
await this.validateCollectionDimensions(vectorSize);
|
||||
}
|
||||
|
||||
await this.ensurePayloadIndexes();
|
||||
this.collectionReady = true;
|
||||
}
|
||||
|
||||
async ensurePayloadIndexes(): Promise<void> {
|
||||
const collections = await this.client.getCollections();
|
||||
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
|
||||
if (!exists) {
|
||||
return;
|
||||
}
|
||||
|
||||
const collectionInfo = await this.client.getCollection(env.qdrantCollection) as { payload_schema?: Record<string, unknown> };
|
||||
const payloadSchema = collectionInfo.payload_schema ?? {};
|
||||
for (const fieldName of ["source_id", "source_version_id", "document_id", "chunk_mode", "tags"]) {
|
||||
if (payloadSchema[fieldName]) {
|
||||
continue;
|
||||
}
|
||||
await this.client.createPayloadIndex(env.qdrantCollection, {
|
||||
field_name: fieldName,
|
||||
field_schema: "keyword",
|
||||
wait: true
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
async upsert(chunks: IngestedChunk[]): Promise<void> {
|
||||
if (chunks.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
await this.ensureCollection(chunks[0].vector.length);
|
||||
await this.client.upsert(env.qdrantCollection, {
|
||||
const vectorSize = chunks[0].vector.length;
|
||||
if (vectorSize === 0 || chunks.some((chunk) => chunk.vector.length !== vectorSize)) {
|
||||
throw new Error("Qdrant upsert rejected: vectors must be non-empty and have consistent dimensions");
|
||||
}
|
||||
|
||||
await this.ensureCollection(vectorSize);
|
||||
const result = await this.client.upsert(env.qdrantCollection, {
|
||||
wait: true,
|
||||
points: chunks.map((chunk) => ({
|
||||
id: chunk.id,
|
||||
|
|
@ -112,21 +167,26 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
|
|||
payload: chunk.payload
|
||||
}))
|
||||
});
|
||||
if (result.status !== "completed") {
|
||||
throw new Error(`Qdrant upsert did not complete: ${result.status}`);
|
||||
}
|
||||
}
|
||||
|
||||
async search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]> {
|
||||
async search(queryVector: number[], limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]> {
|
||||
await this.ensureCollection(queryVector.length);
|
||||
const result = await this.client.search(env.qdrantCollection, {
|
||||
vector: queryVector,
|
||||
limit,
|
||||
with_payload: true,
|
||||
filter: buildSearchFilter(mode, scope)
|
||||
filter: buildSearchFilter(mode, scope, activeVersions)
|
||||
});
|
||||
|
||||
return result.map((point) => ({
|
||||
chunkId: String(point.payload?.chunk_id ?? point.id),
|
||||
documentId: String(point.payload?.document_id ?? ""),
|
||||
sourceId: String(point.payload?.source_id ?? ""),
|
||||
sourceVersionId: point.payload?.source_version_id ? String(point.payload.source_version_id) : undefined,
|
||||
sourceVersionNumber: point.payload?.source_version_number ? Number(point.payload.source_version_number) : undefined,
|
||||
title: String(point.payload?.title ?? ""),
|
||||
sectionTitle: point.payload?.section_title ? String(point.payload.section_title) : undefined,
|
||||
content: String(point.payload?.content ?? ""),
|
||||
|
|
@ -192,7 +252,7 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
|
|||
return [...scopeMap.values()].sort((left, right) => left.sourceRef.localeCompare(right.sourceRef));
|
||||
}
|
||||
|
||||
async browseScope(limit: number, mode?: string, scope?: RetrieveScope): Promise<RetrievedItem[]> {
|
||||
async browseScope(limit: number, mode?: string, scope?: RetrieveScope, activeVersions?: ActiveVersionScope[]): Promise<RetrievedItem[]> {
|
||||
const collections = await this.client.getCollections();
|
||||
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
|
||||
|
||||
|
|
@ -203,13 +263,15 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
|
|||
const response = await this.client.scroll(env.qdrantCollection, {
|
||||
limit,
|
||||
with_payload: true,
|
||||
filter: buildSearchFilter(mode, scope)
|
||||
filter: buildSearchFilter(mode, scope, activeVersions)
|
||||
});
|
||||
|
||||
return response.points.map((point) => ({
|
||||
chunkId: String(point.payload?.chunk_id ?? point.id),
|
||||
documentId: String(point.payload?.document_id ?? ""),
|
||||
sourceId: String(point.payload?.source_id ?? ""),
|
||||
sourceVersionId: point.payload?.source_version_id ? String(point.payload.source_version_id) : undefined,
|
||||
sourceVersionNumber: point.payload?.source_version_number ? Number(point.payload.source_version_number) : undefined,
|
||||
title: String(point.payload?.title ?? ""),
|
||||
sectionTitle: point.payload?.section_title ? String(point.payload.section_title) : undefined,
|
||||
content: String(point.payload?.content ?? ""),
|
||||
|
|
@ -233,15 +295,64 @@ export class QdrantVectorStoreClient implements VectorStoreClient {
|
|||
return 0;
|
||||
}
|
||||
|
||||
// Contamos antes de borrar para devolver la cantidad
|
||||
const countResult = await this.client.count(env.qdrantCollection, {
|
||||
filter
|
||||
});
|
||||
|
||||
await this.client.delete(env.qdrantCollection, {
|
||||
filter
|
||||
const deleteResult = await this.client.delete(env.qdrantCollection, {
|
||||
filter,
|
||||
wait: true
|
||||
});
|
||||
if (deleteResult.status !== "completed") {
|
||||
throw new Error(`Qdrant delete did not complete: ${deleteResult.status}`);
|
||||
}
|
||||
|
||||
return countResult.count;
|
||||
}
|
||||
|
||||
async validateCollectionDimensions(expectedDimensions: number): Promise<void> {
|
||||
const collections = await this.client.getCollections();
|
||||
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
|
||||
if (!exists) {
|
||||
return;
|
||||
}
|
||||
const collectionInfo = await this.client.getCollection(env.qdrantCollection);
|
||||
const actualDimensions = extractVectorSize(collectionInfo);
|
||||
if (actualDimensions !== undefined && actualDimensions !== expectedDimensions) {
|
||||
throw new Error(`Qdrant collection dimension mismatch: expected ${expectedDimensions}, got ${actualDimensions}`);
|
||||
}
|
||||
}
|
||||
|
||||
async countVersionPoints(sourceVersionId: string): Promise<number> {
|
||||
const collections = await this.client.getCollections();
|
||||
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
|
||||
if (!exists) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
const result = await this.client.count(env.qdrantCollection, {
|
||||
filter: {
|
||||
must: [{ key: "source_version_id", match: { value: sourceVersionId } }]
|
||||
},
|
||||
exact: true
|
||||
});
|
||||
return result.count;
|
||||
}
|
||||
|
||||
async deleteVersionPoints(sourceVersionId: string): Promise<number> {
|
||||
const count = await this.countVersionPoints(sourceVersionId);
|
||||
if (count === 0) {
|
||||
return 0;
|
||||
}
|
||||
const deleteResult = await this.client.delete(env.qdrantCollection, {
|
||||
filter: {
|
||||
must: [{ key: "source_version_id", match: { value: sourceVersionId } }]
|
||||
},
|
||||
wait: true
|
||||
});
|
||||
if (deleteResult.status !== "completed") {
|
||||
throw new Error(`Qdrant version delete did not complete: ${deleteResult.status}`);
|
||||
}
|
||||
return count;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
448
src/scripts/migrate-legacy-lifecycle.ts
Normal file
448
src/scripts/migrate-legacy-lifecycle.ts
Normal file
|
|
@ -0,0 +1,448 @@
|
|||
import { buildCatalogPool, withCatalogAdvisoryLock, withTransaction, type PgPool } from "../modules/catalog/client.js";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { buildLegacyMigrationPlan, type LegacyPointSnapshot, type LegacySourcePlan } from "../modules/catalog/legacy-migration.js";
|
||||
import { CatalogError } from "../modules/catalog/errors.js";
|
||||
import { env } from "../config/env.js";
|
||||
import { buildQdrantClient } from "../modules/vectorstore/client.js";
|
||||
import { sha256Hex } from "../shared/utils/ids.js";
|
||||
|
||||
type QdrantClientInstance = ReturnType<typeof buildQdrantClient>;
|
||||
|
||||
function argValue(name: string): string | undefined {
|
||||
const prefix = `${name}=`;
|
||||
const entry = process.argv.find((arg) => arg.startsWith(prefix));
|
||||
return entry?.slice(prefix.length);
|
||||
}
|
||||
|
||||
function requireArg(name: string): string {
|
||||
const value = argValue(name);
|
||||
if (!value) {
|
||||
throw new Error(`Missing required argument ${name}=...`);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
|
||||
async function readLegacyPoints(client: QdrantClientInstance, includeVersionIds = new Set<string>()): Promise<LegacyPointSnapshot[]> {
|
||||
const collections = await client.getCollections();
|
||||
const exists = collections.collections.some((collection) => collection.name === env.qdrantCollection);
|
||||
if (!exists) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const points: LegacyPointSnapshot[] = [];
|
||||
let nextOffset = undefined;
|
||||
do {
|
||||
const response = await client.scroll(env.qdrantCollection, {
|
||||
limit: 256,
|
||||
offset: nextOffset,
|
||||
with_payload: true,
|
||||
with_vector: false
|
||||
});
|
||||
for (const point of response.points) {
|
||||
const sourceVersionId = point.payload?.source_version_id ? String(point.payload.source_version_id) : "";
|
||||
if (sourceVersionId && !includeVersionIds.has(sourceVersionId)) {
|
||||
continue;
|
||||
}
|
||||
points.push({ id: point.id, payload: point.payload ?? {} });
|
||||
}
|
||||
nextOffset = response.next_page_offset;
|
||||
} while (nextOffset);
|
||||
return points;
|
||||
}
|
||||
|
||||
async function assertMaintenance(pool: PgPool): Promise<void> {
|
||||
if (env.ingestWritesEnabled) {
|
||||
throw new CatalogError("INGEST_WRITES_ENABLED must be false during legacy migration", 503, "WRITES_NOT_DISABLED");
|
||||
}
|
||||
await withTransaction(pool, async (client) => {
|
||||
const result = await client.query<{ maintenance: boolean }>("SELECT maintenance FROM rag_system_state WHERE singleton = true FOR UPDATE");
|
||||
if (!result.rows[0]?.maintenance) {
|
||||
throw new CatalogError("rag_system_state.maintenance must be true during legacy migration", 503, "MAINTENANCE_REQUIRED");
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async function readBatchVersionIds(pool: PgPool, batchId: string): Promise<Set<string>> {
|
||||
const result = await pool.query<{ version_id: string }>(
|
||||
"SELECT version_id FROM rag_legacy_migration_items WHERE batch_id = $1 AND version_id IS NOT NULL",
|
||||
[batchId]
|
||||
);
|
||||
return new Set(result.rows.map((row) => row.version_id));
|
||||
}
|
||||
|
||||
async function createOrReuseBatch(pool: PgPool, snapshotRef: string, plan: { sources: LegacySourcePlan[] }, batchId?: string): Promise<string> {
|
||||
if (batchId) {
|
||||
const existing = await pool.query<{ state: string; snapshot_ref: string }>(
|
||||
"SELECT state, snapshot_ref FROM rag_legacy_migration_batches WHERE batch_id = $1",
|
||||
[batchId]
|
||||
);
|
||||
if (!existing.rowCount) {
|
||||
throw new CatalogError(`Legacy migration batch not found: ${batchId}`, 404, "LEGACY_BATCH_NOT_FOUND");
|
||||
}
|
||||
if (existing.rows[0].snapshot_ref !== snapshotRef) {
|
||||
throw new CatalogError("Legacy migration batch snapshot_ref does not match the requested snapshot", 409, "LEGACY_BATCH_SNAPSHOT_MISMATCH");
|
||||
}
|
||||
if (existing.rows[0].state === "rolled_back") {
|
||||
throw new CatalogError("Rolled back legacy migration batches cannot be resumed", 409, "LEGACY_BATCH_ROLLED_BACK");
|
||||
}
|
||||
return batchId;
|
||||
}
|
||||
|
||||
const batch = await pool.query<{ batch_id: string }>(
|
||||
"INSERT INTO rag_legacy_migration_batches(state, snapshot_ref, report) VALUES ('planned', $1, $2) RETURNING batch_id",
|
||||
[snapshotRef, { sourceCount: plan.sources.length, pointCount: plan.sources.reduce((total, source) => total + source.pointIds.length, 0) }]
|
||||
);
|
||||
return batch.rows[0].batch_id;
|
||||
}
|
||||
|
||||
export async function createCatalogVersion(pool: PgPool, batchId: string, source: LegacySourcePlan, collection: string): Promise<{ versionId: string; versionNumber: number }> {
|
||||
return withTransaction(pool, async (client) => {
|
||||
const item = await client.query<{ version_id: string | null; expected_point_count: number }>(
|
||||
"SELECT version_id, expected_point_count FROM rag_legacy_migration_items WHERE batch_id = $1 AND source_id = $2 FOR UPDATE",
|
||||
[batchId, source.sourceId]
|
||||
);
|
||||
if (item.rows[0]?.version_id) {
|
||||
const reusable = await client.query<{ version_id: string; version_number: string; expected_point_count: string; source_type: string; source_ref: string; qdrant_collection: string }>(
|
||||
`SELECT v.version_id, v.version_number, v.expected_point_count, s.source_type, s.source_ref, v.qdrant_collection
|
||||
FROM rag_source_versions v
|
||||
JOIN rag_sources s ON s.source_id = v.source_id
|
||||
WHERE v.version_id = $1 AND v.source_id = $2
|
||||
FOR UPDATE OF v, s`,
|
||||
[item.rows[0].version_id, source.sourceId]
|
||||
);
|
||||
if (!reusable.rowCount) {
|
||||
throw new CatalogError(`Legacy migration item references a missing version for ${source.sourceId}`, 409, "LEGACY_VERSION_MISSING");
|
||||
}
|
||||
const row = reusable.rows[0];
|
||||
if (row.source_type !== source.sourceType || row.source_ref !== source.sourceRef || row.qdrant_collection !== collection || Number(row.expected_point_count) !== source.pointIds.length) {
|
||||
throw new CatalogError(`Legacy migration resume is not compatible for ${source.sourceId}`, 409, "LEGACY_RESUME_INCOMPATIBLE");
|
||||
}
|
||||
return { versionId: row.version_id, versionNumber: Number(row.version_number) };
|
||||
}
|
||||
|
||||
const existing = await client.query("SELECT 1 FROM rag_sources WHERE source_id = $1", [source.sourceId]);
|
||||
if (existing.rowCount) {
|
||||
throw new CatalogError(`Source already exists in catalog: ${source.sourceId}`, 409, "LEGACY_SOURCE_ALREADY_CATALOGED");
|
||||
}
|
||||
await client.query(
|
||||
"INSERT INTO rag_sources(source_id, source_type, source_ref) VALUES ($1, $2, $3)",
|
||||
[source.sourceId, source.sourceType, source.sourceRef]
|
||||
);
|
||||
const version = await client.query<{ version_id: string; version_number: string }>(
|
||||
`INSERT INTO rag_source_versions(
|
||||
source_id, version_number, state, original_manifest_hash, source_content_hash,
|
||||
processing_fingerprint, metadata_hash, tags, activate_requested, embedding_provider,
|
||||
embedding_model, embedding_dimensions, qdrant_collection, expected_document_count,
|
||||
expected_point_count, verified_point_count
|
||||
) VALUES ($1, 1, 'ready', $2, $3, $4, $5, $6, true, $7, $8, $9, $10, $11, $12, 0)
|
||||
RETURNING version_id, version_number`,
|
||||
[
|
||||
source.sourceId,
|
||||
source.originalManifestHash,
|
||||
source.sourceContentHash,
|
||||
sha256Hex(`legacy:${batchId}`),
|
||||
source.metadataHash,
|
||||
source.tags,
|
||||
source.embeddingProvider,
|
||||
source.embeddingModel,
|
||||
source.embeddingDimensions,
|
||||
collection,
|
||||
source.documents.length,
|
||||
source.pointIds.length
|
||||
]
|
||||
);
|
||||
const versionId = version.rows[0].version_id;
|
||||
for (const document of source.documents) {
|
||||
await client.query(
|
||||
`INSERT INTO rag_version_documents(
|
||||
version_id, document_id, document_key, original_hash, original_hash_kind,
|
||||
content_hash, mime_type, title, index_state, chunk_count
|
||||
) VALUES ($1, $2, $3, $4, 'legacy-derived', $5, $6, $7, 'ready', $8)`,
|
||||
[versionId, document.documentId, document.documentKey, document.originalHash, document.contentHash, document.mimeType, document.title, document.chunkCount]
|
||||
);
|
||||
}
|
||||
if (item.rowCount) {
|
||||
await client.query(
|
||||
`UPDATE rag_legacy_migration_items
|
||||
SET version_id = $3, expected_point_count = $4, state = 'migrated', error_detail = NULL
|
||||
WHERE batch_id = $1 AND source_id = $2`,
|
||||
[batchId, source.sourceId, versionId, source.pointIds.length]
|
||||
);
|
||||
} else {
|
||||
await client.query(
|
||||
`INSERT INTO rag_legacy_migration_items(batch_id, source_id, version_id, expected_point_count, state)
|
||||
VALUES ($1, $2, $3, $4, 'migrated')`,
|
||||
[batchId, source.sourceId, versionId, source.pointIds.length]
|
||||
);
|
||||
}
|
||||
return { versionId, versionNumber: Number(version.rows[0].version_number) };
|
||||
});
|
||||
}
|
||||
|
||||
async function setLifecyclePayload(client: QdrantClientInstance, source: LegacySourcePlan, versionId: string, versionNumber: number, batchId: string): Promise<void> {
|
||||
const result = await client.setPayload(env.qdrantCollection, {
|
||||
points: source.pointIds,
|
||||
payload: {
|
||||
source_version_id: versionId,
|
||||
source_version_number: versionNumber,
|
||||
processing_fingerprint: sha256Hex(`legacy:${batchId}`),
|
||||
write_state: "legacy-migrated"
|
||||
},
|
||||
wait: true
|
||||
});
|
||||
if (result.status !== "completed") {
|
||||
throw new Error(`Qdrant payload update did not complete for ${source.sourceId}: ${result.status}`);
|
||||
}
|
||||
}
|
||||
|
||||
async function countVersionPoints(client: QdrantClientInstance, versionId: string): Promise<number> {
|
||||
const result = await client.count(env.qdrantCollection, {
|
||||
filter: { must: [{ key: "source_version_id", match: { value: versionId } }] },
|
||||
exact: true
|
||||
});
|
||||
return result.count;
|
||||
}
|
||||
|
||||
async function countLegacySourcePoints(client: QdrantClientInstance, sourceId: string): Promise<number> {
|
||||
let count = 0;
|
||||
let nextOffset = undefined;
|
||||
do {
|
||||
const response = await client.scroll(env.qdrantCollection, {
|
||||
limit: 256,
|
||||
offset: nextOffset,
|
||||
with_payload: ["source_id", "source_version_id"],
|
||||
with_vector: false,
|
||||
filter: { must: [{ key: "source_id", match: { value: sourceId } }] }
|
||||
});
|
||||
for (const point of response.points) {
|
||||
if (!point.payload?.source_version_id) {
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
nextOffset = response.next_page_offset;
|
||||
} while (nextOffset);
|
||||
return count;
|
||||
}
|
||||
|
||||
export async function activateMigratedVersion(pool: PgPool, sourceId: string, versionId: string, verifiedPointCount: number): Promise<void> {
|
||||
await withTransaction(pool, async (client) => {
|
||||
const existing = await client.query<{ state: string; active_version_id: string | null }>(
|
||||
`SELECT v.state, s.active_version_id
|
||||
FROM rag_source_versions v
|
||||
JOIN rag_sources s ON s.source_id = v.source_id
|
||||
WHERE v.source_id = $1 AND v.version_id = $2
|
||||
FOR UPDATE OF v, s`,
|
||||
[sourceId, versionId]
|
||||
);
|
||||
if (existing.rows[0]?.state === "active" && existing.rows[0].active_version_id === versionId) {
|
||||
await client.query("UPDATE rag_source_versions SET verified_point_count = $2 WHERE source_id = $1 AND version_id = $3", [sourceId, verifiedPointCount, versionId]);
|
||||
return;
|
||||
}
|
||||
const version = await client.query(
|
||||
`UPDATE rag_source_versions
|
||||
SET state = 'active', verified_point_count = $2, ready_at = now(), activated_at = now()
|
||||
WHERE source_id = $1 AND version_id = $3 AND state = 'ready'
|
||||
RETURNING version_id`,
|
||||
[sourceId, verifiedPointCount, versionId]
|
||||
);
|
||||
if (version.rowCount !== 1) {
|
||||
throw new CatalogError(`Could not activate migrated version for ${sourceId}`, 409, "LEGACY_ACTIVATION_FAILED");
|
||||
}
|
||||
await client.query("UPDATE rag_sources SET active_version_id = $2 WHERE source_id = $1", [sourceId, versionId]);
|
||||
});
|
||||
}
|
||||
|
||||
export async function rollbackMigratedBatch(
|
||||
pool: PgPool,
|
||||
qdrant: QdrantClientInstance,
|
||||
batchId: string,
|
||||
snapshotRef: string,
|
||||
snapshotChecksum?: string
|
||||
): Promise<{ ok: true; mode: "rollback"; batchId: string; sources: string[] }> {
|
||||
const itemsWithVersions = await withTransaction(pool, async (client) => {
|
||||
const batch = await client.query<{ state: string; snapshot_ref: string }>(
|
||||
"SELECT state, snapshot_ref FROM rag_legacy_migration_batches WHERE batch_id = $1 FOR UPDATE",
|
||||
[batchId]
|
||||
);
|
||||
if (!batch.rowCount) {
|
||||
throw new CatalogError(`Legacy migration batch not found: ${batchId}`, 404, "LEGACY_BATCH_NOT_FOUND");
|
||||
}
|
||||
if (batch.rows[0].snapshot_ref !== snapshotRef) {
|
||||
throw new CatalogError("Legacy migration batch snapshot_ref does not match the requested snapshot", 409, "LEGACY_BATCH_SNAPSHOT_MISMATCH");
|
||||
}
|
||||
|
||||
const items = await client.query<{ source_id: string; version_id: string | null; expected_point_count: string | number }>(
|
||||
"SELECT source_id, version_id, expected_point_count FROM rag_legacy_migration_items WHERE batch_id = $1 FOR UPDATE",
|
||||
[batchId]
|
||||
);
|
||||
const rollbackItems = items.rows.filter((item): item is { source_id: string; version_id: string; expected_point_count: string | number } => Boolean(item.version_id));
|
||||
if (rollbackItems.length === 0 && batch.rows[0].state === "rolled_back") {
|
||||
return [];
|
||||
}
|
||||
|
||||
await client.query(
|
||||
`SELECT s.source_id
|
||||
FROM rag_sources s
|
||||
JOIN rag_legacy_migration_items i ON i.source_id = s.source_id
|
||||
WHERE i.batch_id = $1 AND i.version_id IS NOT NULL
|
||||
FOR UPDATE OF s`,
|
||||
[batchId]
|
||||
);
|
||||
await client.query(
|
||||
`SELECT v.version_id
|
||||
FROM rag_source_versions v
|
||||
JOIN rag_legacy_migration_items i ON i.version_id = v.version_id
|
||||
WHERE i.batch_id = $1 AND i.version_id IS NOT NULL
|
||||
FOR UPDATE OF v`,
|
||||
[batchId]
|
||||
);
|
||||
await client.query(
|
||||
`UPDATE rag_sources s
|
||||
SET active_version_id = NULL
|
||||
FROM rag_legacy_migration_items i
|
||||
WHERE i.batch_id = $1
|
||||
AND i.source_id = s.source_id
|
||||
AND i.version_id IS NOT NULL
|
||||
AND s.active_version_id = i.version_id`,
|
||||
[batchId]
|
||||
);
|
||||
await client.query(
|
||||
`UPDATE rag_source_versions v
|
||||
SET state = 'purging'
|
||||
FROM rag_legacy_migration_items i
|
||||
WHERE i.batch_id = $1
|
||||
AND i.version_id = v.version_id`,
|
||||
[batchId]
|
||||
);
|
||||
const activePointers = await client.query<{ count: string }>(
|
||||
`SELECT COUNT(*) AS count
|
||||
FROM rag_sources s
|
||||
JOIN rag_legacy_migration_items i ON i.source_id = s.source_id
|
||||
WHERE i.batch_id = $1
|
||||
AND i.version_id IS NOT NULL
|
||||
AND s.active_version_id = i.version_id`,
|
||||
[batchId]
|
||||
);
|
||||
if (Number(activePointers.rows[0]?.count ?? 0) !== 0) {
|
||||
throw new CatalogError("Legacy rollback preparation left active source pointers", 503, "LEGACY_ROLLBACK_ACTIVE_POINTERS_REMAIN");
|
||||
}
|
||||
|
||||
return rollbackItems.map((item) => ({
|
||||
sourceId: item.source_id,
|
||||
versionId: item.version_id,
|
||||
expectedPointCount: Number(item.expected_point_count)
|
||||
}));
|
||||
});
|
||||
|
||||
if (itemsWithVersions.length === 0) {
|
||||
return { ok: true, mode: "rollback", batchId, sources: [] };
|
||||
}
|
||||
|
||||
const recovered = await qdrant.recoverSnapshot(env.qdrantCollection, {
|
||||
location: snapshotRef,
|
||||
priority: "snapshot",
|
||||
checksum: snapshotChecksum
|
||||
});
|
||||
if (!recovered) {
|
||||
throw new Error("Qdrant snapshot recovery returned false");
|
||||
}
|
||||
|
||||
for (const item of itemsWithVersions) {
|
||||
const legacyCount = await countLegacySourcePoints(qdrant, item.sourceId);
|
||||
if (legacyCount !== item.expectedPointCount) {
|
||||
throw new CatalogError(`Restored legacy point count mismatch for ${item.sourceId}`, 503, "LEGACY_RESTORE_POINT_COUNT_MISMATCH");
|
||||
}
|
||||
}
|
||||
|
||||
await withTransaction(pool, async (client) => {
|
||||
for (const item of itemsWithVersions) {
|
||||
await client.query("DELETE FROM rag_version_documents WHERE version_id = $1", [item.versionId]);
|
||||
await client.query("DELETE FROM rag_source_versions WHERE version_id = $1", [item.versionId]);
|
||||
await client.query("DELETE FROM rag_sources WHERE source_id = $1", [item.sourceId]);
|
||||
await client.query("UPDATE rag_legacy_migration_items SET version_id = NULL, state = 'rolled_back' WHERE batch_id = $1 AND source_id = $2", [batchId, item.sourceId]);
|
||||
}
|
||||
await client.query("UPDATE rag_legacy_migration_batches SET state = 'rolled_back', completed_at = now() WHERE batch_id = $1", [batchId]);
|
||||
});
|
||||
|
||||
return { ok: true, mode: "rollback", batchId, sources: itemsWithVersions.map((item) => item.sourceId) };
|
||||
}
|
||||
|
||||
async function dryRun() {
|
||||
const qdrant = buildQdrantClient();
|
||||
const points = await readLegacyPoints(qdrant);
|
||||
const plan = buildLegacyMigrationPlan(points);
|
||||
return { mode: "dry-run", collection: env.qdrantCollection, legacyPoints: points.length, ...plan };
|
||||
}
|
||||
|
||||
async function applyMigration() {
|
||||
const snapshotRef = requireArg("--snapshot-ref");
|
||||
const requestedBatchId = argValue("--batch-id");
|
||||
const pool = buildCatalogPool();
|
||||
const qdrant = buildQdrantClient();
|
||||
let batchId = requestedBatchId;
|
||||
try {
|
||||
return await withCatalogAdvisoryLock(pool, "rag:knowledge-lifecycle:legacy-migration", async () => {
|
||||
await assertMaintenance(pool);
|
||||
const includeVersionIds = requestedBatchId ? await readBatchVersionIds(pool, requestedBatchId) : new Set<string>();
|
||||
const plan = buildLegacyMigrationPlan(await readLegacyPoints(qdrant, includeVersionIds));
|
||||
if (!plan.ok) {
|
||||
return { ok: false, mode: "apply", batchId: requestedBatchId, blockedReasons: plan.blockedReasons };
|
||||
}
|
||||
batchId = await createOrReuseBatch(pool, snapshotRef, plan, requestedBatchId);
|
||||
const migrated: Array<{ sourceId: string; versionId: string; pointCount: number }> = [];
|
||||
for (const source of plan.sources) {
|
||||
const { versionId, versionNumber } = await createCatalogVersion(pool, batchId, source, env.qdrantCollection);
|
||||
await setLifecyclePayload(qdrant, source, versionId, versionNumber, batchId);
|
||||
const verifiedPointCount = await countVersionPoints(qdrant, versionId);
|
||||
if (verifiedPointCount !== source.pointIds.length) {
|
||||
throw new CatalogError(`Point count mismatch for ${source.sourceId}`, 503, "LEGACY_POINT_COUNT_MISMATCH");
|
||||
}
|
||||
await activateMigratedVersion(pool, source.sourceId, versionId, verifiedPointCount);
|
||||
await pool.query("UPDATE rag_legacy_migration_items SET state = 'verified', error_detail = NULL WHERE batch_id = $1 AND source_id = $2", [batchId, source.sourceId]);
|
||||
migrated.push({ sourceId: source.sourceId, versionId, pointCount: verifiedPointCount });
|
||||
}
|
||||
await pool.query("UPDATE rag_legacy_migration_batches SET state = 'verified', completed_at = now(), report = report || $2::jsonb WHERE batch_id = $1", [batchId, JSON.stringify({ migrated })]);
|
||||
return { ok: true, mode: "apply", batchId, migrated };
|
||||
});
|
||||
} catch (error) {
|
||||
if (batchId) {
|
||||
await pool.query("UPDATE rag_legacy_migration_batches SET state = 'failed', completed_at = now(), report = report || $2::jsonb WHERE batch_id = $1 AND state <> 'verified'", [batchId, JSON.stringify({ error: error instanceof Error ? error.message : "Unknown legacy migration error" })]).catch(() => undefined);
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
await pool.end();
|
||||
}
|
||||
}
|
||||
|
||||
async function rollbackMigration() {
|
||||
const batchId = requireArg("--batch-id");
|
||||
const snapshotRef = requireArg("--snapshot-ref");
|
||||
const snapshotChecksum = argValue("--snapshot-checksum");
|
||||
const pool = buildCatalogPool();
|
||||
const qdrant = buildQdrantClient();
|
||||
try {
|
||||
return await withCatalogAdvisoryLock(pool, "rag:knowledge-lifecycle:legacy-migration", async () => {
|
||||
await assertMaintenance(pool);
|
||||
return rollbackMigratedBatch(pool, qdrant, batchId, snapshotRef, snapshotChecksum);
|
||||
});
|
||||
} finally {
|
||||
await pool.end();
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const report = process.argv.includes("--apply")
|
||||
? await applyMigration()
|
||||
: process.argv.includes("--rollback")
|
||||
? await rollbackMigration()
|
||||
: await dryRun();
|
||||
console.log(JSON.stringify(report, null, 2));
|
||||
if (!report.ok) process.exitCode = 1;
|
||||
}
|
||||
|
||||
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
|
||||
main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.message : error);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
|
@ -6,6 +6,18 @@ export type IngestMode = "mechanical" | "interactive";
|
|||
|
||||
export type SourceType = "file" | "folder";
|
||||
|
||||
export type SourceVersionState =
|
||||
| "pending"
|
||||
| "indexing"
|
||||
| "review_required"
|
||||
| "ready"
|
||||
| "active"
|
||||
| "superseded"
|
||||
| "failed"
|
||||
| "rejected"
|
||||
| "purging"
|
||||
| "purged";
|
||||
|
||||
export interface IngestSourceInput {
|
||||
sourceId?: string;
|
||||
sourceType: SourceType;
|
||||
|
|
@ -13,6 +25,8 @@ export interface IngestSourceInput {
|
|||
readPath?: string;
|
||||
mode?: IngestMode;
|
||||
tags?: string[];
|
||||
activate?: boolean;
|
||||
expectedActiveVersionId?: string | null;
|
||||
}
|
||||
|
||||
export interface IngestedChunk {
|
||||
|
|
@ -23,10 +37,17 @@ export interface IngestedChunk {
|
|||
|
||||
export interface IngestResult {
|
||||
accepted: true;
|
||||
source: IngestSourceInput;
|
||||
source?: IngestSourceInput;
|
||||
sourceId: string;
|
||||
versionId?: string;
|
||||
versionNumber?: number;
|
||||
state?: SourceVersionState;
|
||||
previousVersionId?: string | null;
|
||||
filesDiscovered: number;
|
||||
documentsProcessed: number;
|
||||
chunksStored: number;
|
||||
activated?: boolean;
|
||||
noOp?: boolean;
|
||||
collectionName: string;
|
||||
}
|
||||
|
||||
|
|
@ -34,6 +55,8 @@ export interface RetrievedItem {
|
|||
chunkId: string;
|
||||
documentId: string;
|
||||
sourceId: string;
|
||||
sourceVersionId?: string;
|
||||
sourceVersionNumber?: number;
|
||||
title: string;
|
||||
sectionTitle?: string;
|
||||
content: string;
|
||||
|
|
@ -54,6 +77,11 @@ export interface AvailableScope {
|
|||
sourceRef: string;
|
||||
chunkModes: ChunkMode[];
|
||||
tags: string[];
|
||||
activeVersionId?: string | null;
|
||||
activeVersionNumber?: number | null;
|
||||
state?: SourceVersionState | null;
|
||||
needsReingest?: boolean;
|
||||
updatedAt?: string;
|
||||
}
|
||||
|
||||
export interface RetrieveResponse {
|
||||
|
|
@ -80,6 +108,8 @@ export interface AnswerResponse {
|
|||
citations: Array<{
|
||||
chunkId: string;
|
||||
documentId: string;
|
||||
sourceVersionId?: string;
|
||||
sourceVersionNumber?: number;
|
||||
title: string;
|
||||
sectionTitle?: string;
|
||||
startLine?: number;
|
||||
|
|
|
|||
|
|
@ -1,6 +1,17 @@
|
|||
import path from "node:path";
|
||||
import { createHash } from "node:crypto";
|
||||
|
||||
const UUID_NAMESPACE_URL = "6ba7b811-9dad-11d1-80b4-00c04fd430c8";
|
||||
|
||||
function uuidToBytes(uuid: string): Buffer {
|
||||
return Buffer.from(uuid.replace(/-/g, ""), "hex");
|
||||
}
|
||||
|
||||
function bytesToUuid(bytes: Buffer): string {
|
||||
const hex = bytes.toString("hex");
|
||||
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
|
||||
}
|
||||
|
||||
function asciiNormalize(value: string): string {
|
||||
return value
|
||||
.normalize("NFD")
|
||||
|
|
@ -17,6 +28,58 @@ export function normalizeDocumentKey(value: string): string {
|
|||
.toLowerCase();
|
||||
}
|
||||
|
||||
export function sha256Hex(value: string | Buffer): string {
|
||||
return createHash("sha256").update(value).digest("hex");
|
||||
}
|
||||
|
||||
export function normalizeContentForHash(content: string): string {
|
||||
const normalized = content
|
||||
.replace(/^\uFEFF/, "")
|
||||
.replace(/\r\n/g, "\n")
|
||||
.replace(/\r/g, "\n")
|
||||
.split("\n")
|
||||
.map((line) => line.replace(/[ \t]+$/g, ""))
|
||||
.join("\n")
|
||||
.replace(/^\n+|\n+$/g, "");
|
||||
|
||||
return `${normalized}\n`;
|
||||
}
|
||||
|
||||
export function canonicalJson(value: unknown): string {
|
||||
if (Array.isArray(value)) {
|
||||
return `[${value.map(canonicalJson).join(",")}]`;
|
||||
}
|
||||
|
||||
if (value && typeof value === "object") {
|
||||
const entries = Object.entries(value as Record<string, unknown>)
|
||||
.filter(([, entryValue]) => entryValue !== undefined)
|
||||
.sort(([left], [right]) => Buffer.compare(Buffer.from(left, "utf8"), Buffer.from(right, "utf8")));
|
||||
return `{${entries.map(([key, entryValue]) => `${JSON.stringify(key)}:${canonicalJson(entryValue)}`).join(",")}}`;
|
||||
}
|
||||
|
||||
return JSON.stringify(value);
|
||||
}
|
||||
|
||||
export function hashOrderedPairs(pairs: Array<[string, string | null]>): string | null {
|
||||
if (pairs.some(([, value]) => value === null)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const serialized = [...pairs]
|
||||
.sort(([left], [right]) => Buffer.compare(Buffer.from(left, "utf8"), Buffer.from(right, "utf8")))
|
||||
.map(([key, value]) => `${key}\0${value ?? ""}`)
|
||||
.join("\n");
|
||||
return sha256Hex(serialized);
|
||||
}
|
||||
|
||||
export function buildProcessingFingerprint(input: Record<string, unknown>): string {
|
||||
return sha256Hex(canonicalJson(input));
|
||||
}
|
||||
|
||||
export function buildMetadataHash(tags: string[], metadata: Record<string, unknown> = {}): string {
|
||||
return sha256Hex(canonicalJson({ metadata, tags: [...tags].sort() }));
|
||||
}
|
||||
|
||||
export function buildSourceId(sourceType: string, sourceRef: string): string {
|
||||
const readable = normalizeDocumentKey(path.basename(sourceRef) || sourceRef).replace(/\.[^.]+$/, "") || "source";
|
||||
const suffix = createHash("sha1").update(sourceRef).digest("hex").slice(0, 8);
|
||||
|
|
@ -36,3 +99,12 @@ export function buildQdrantPointId(value: string): string {
|
|||
const hex = createHash("sha1").update(value).digest("hex").slice(0, 32);
|
||||
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20, 32)}`;
|
||||
}
|
||||
|
||||
export function buildVersionedQdrantPointId(sourceVersionId: string, chunkId: string): string {
|
||||
const namespace = uuidToBytes(UUID_NAMESPACE_URL);
|
||||
const name = Buffer.from(`rag-qdrant-point:${sourceVersionId}:${chunkId}`, "utf8");
|
||||
const hash = createHash("sha1").update(Buffer.concat([namespace, name])).digest().subarray(0, 16);
|
||||
hash[6] = (hash[6] & 0x0f) | 0x50;
|
||||
hash[8] = (hash[8] & 0x3f) | 0x80;
|
||||
return bytesToUuid(hash);
|
||||
}
|
||||
|
|
|
|||
514
tests/lifecycle-services.test.ts
Normal file
514
tests/lifecycle-services.test.ts
Normal file
|
|
@ -0,0 +1,514 @@
|
|||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { createApp } from "../src/app.js";
|
||||
import { openApiDocument } from "../src/api/openapi.js";
|
||||
import { env } from "../src/config/env.js";
|
||||
import { CatalogError } from "../src/modules/catalog/errors.js";
|
||||
import { buildLegacyMigrationPlan } from "../src/modules/catalog/legacy-migration.js";
|
||||
import { KnowledgeLifecycleReconciler } from "../src/modules/catalog/reconciler.js";
|
||||
import { CatalogRepository } from "../src/modules/catalog/repository.js";
|
||||
import { activateMigratedVersion, createCatalogVersion, rollbackMigratedBatch } from "../src/scripts/migrate-legacy-lifecycle.js";
|
||||
import { IngestService } from "../src/modules/ingest/service.js";
|
||||
import { RetrieveService } from "../src/modules/retrieve/service.js";
|
||||
import type { EmbeddingProvider } from "../src/modules/embeddings/provider.js";
|
||||
import type { VectorStoreClient } from "../src/modules/vectorstore/client.js";
|
||||
|
||||
function setEnvFlag(name: keyof typeof env, value: unknown) {
|
||||
(env as unknown as Record<string, unknown>)[name] = value;
|
||||
}
|
||||
|
||||
async function withEnvFlags(updates: Partial<Record<keyof typeof env, unknown>>, handler: () => Promise<void>) {
|
||||
const previous = new Map<keyof typeof env, unknown>();
|
||||
for (const [name, value] of Object.entries(updates) as Array<[keyof typeof env, unknown]>) {
|
||||
previous.set(name, env[name]);
|
||||
setEnvFlag(name, value);
|
||||
}
|
||||
try {
|
||||
await handler();
|
||||
} finally {
|
||||
for (const [name, value] of previous) {
|
||||
setEnvFlag(name, value);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function withTempMarkdown(content: string, handler: (filePath: string) => Promise<void>) {
|
||||
const dir = await mkdtemp(path.join(os.tmpdir(), "rag-lifecycle-test-"));
|
||||
try {
|
||||
const filePath = path.join(dir, "source.md");
|
||||
await writeFile(filePath, content, "utf8");
|
||||
await handler(filePath);
|
||||
} finally {
|
||||
await rm(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function fakeEmbeddingProvider(): EmbeddingProvider {
|
||||
return {
|
||||
providerName: "test-provider",
|
||||
modelName: "test-model",
|
||||
dimensions: 3,
|
||||
async embed(input: string[]) {
|
||||
return input.map(() => [0.1, 0.2, 0.3]);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
function fakeVectorStore() {
|
||||
const chunks: unknown[] = [];
|
||||
const vectorStore: Partial<VectorStoreClient> = {
|
||||
kind: "fake",
|
||||
async upsert(input) {
|
||||
chunks.push(...input);
|
||||
},
|
||||
async countVersionPoints() {
|
||||
return chunks.length;
|
||||
},
|
||||
async validateCollectionDimensions() {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
return { vectorStore: vectorStore as VectorStoreClient, chunks };
|
||||
}
|
||||
|
||||
function fakeTransactionPool(handler: (sql: string, params?: unknown[]) => Promise<{ rowCount: number; rows: unknown[] }>) {
|
||||
const client = {
|
||||
async query(sql: string, params?: unknown[]) {
|
||||
if (/^(BEGIN|COMMIT|ROLLBACK|SET CONSTRAINTS)/.test(sql.trim())) {
|
||||
return { rowCount: 0, rows: [] };
|
||||
}
|
||||
return handler(sql, params);
|
||||
},
|
||||
release() {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
return {
|
||||
async connect() {
|
||||
return client;
|
||||
},
|
||||
async query(sql: string, params?: unknown[]) {
|
||||
return handler(sql, params);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
test("activation failure leaves a newly indexed version ready instead of failed", async () => {
|
||||
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
|
||||
const { vectorStore } = fakeVectorStore();
|
||||
const calls: string[] = [];
|
||||
const catalog = {
|
||||
async beginAttempt() { return "attempt-1"; },
|
||||
async updateAttempt() { return undefined; },
|
||||
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async findReusableVersion() { return undefined; },
|
||||
async createPendingVersion() { return { versionId: "version-1", versionNumber: 1, state: "pending", previousVersionId: null }; },
|
||||
async markIndexing() { calls.push("indexing"); },
|
||||
async markReady() { calls.push("ready"); return { versionId: "version-1", versionNumber: 1, state: "ready", previousVersionId: null }; },
|
||||
async activateVersion() { throw new CatalogError("precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED"); },
|
||||
async markFailed() { calls.push("failed"); }
|
||||
};
|
||||
const service = new IngestService(fakeEmbeddingProvider(), vectorStore, catalog as never);
|
||||
await assert.rejects(
|
||||
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: "other" }),
|
||||
/precondition failed/
|
||||
);
|
||||
assert.deepEqual(calls, ["indexing", "ready"]);
|
||||
}));
|
||||
});
|
||||
|
||||
test("active no-op validates the expected active version precondition", async () => {
|
||||
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
|
||||
const { vectorStore, chunks } = fakeVectorStore();
|
||||
let preconditionChecked = false;
|
||||
const catalog = {
|
||||
async beginAttempt() { return "attempt-1"; },
|
||||
async updateAttempt() { return undefined; },
|
||||
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async findReusableVersion() { return { versionId: "active-1", versionNumber: 1, state: "active", previousVersionId: null, expectedDocumentCount: 1 }; },
|
||||
async assertActiveVersionPrecondition() {
|
||||
preconditionChecked = true;
|
||||
throw new CatalogError("Active version precondition failed", 409, "ACTIVE_VERSION_PRECONDITION_FAILED");
|
||||
},
|
||||
async markFailed() { throw new Error("markFailed must not run for no-op precondition errors"); }
|
||||
};
|
||||
const service = new IngestService(fakeEmbeddingProvider(), vectorStore, catalog as never);
|
||||
await assert.rejects(
|
||||
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: "wrong" }),
|
||||
/precondition failed/i
|
||||
);
|
||||
assert.equal(preconditionChecked, true);
|
||||
assert.equal(chunks.length, 0);
|
||||
}));
|
||||
});
|
||||
|
||||
test("/sources fails closed and /cleanup refuses deletion when lifecycle enforcement is enabled", async () => {
|
||||
setEnvFlag("knowledgeLifecycleEnforced", true);
|
||||
setEnvFlag("lifecycleAdminToken", "test-token");
|
||||
const server = createApp().listen(0);
|
||||
try {
|
||||
const address = server.address();
|
||||
assert.ok(address && typeof address === "object");
|
||||
const baseUrl = `http://127.0.0.1:${address.port}`;
|
||||
|
||||
const sources = await fetch(`${baseUrl}/sources`);
|
||||
assert.equal(sources.status, 503);
|
||||
|
||||
const cleanupWithoutAdmin = await fetch(`${baseUrl}/cleanup`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ scope: { sourceId: "src:default:folder:docs" } })
|
||||
});
|
||||
assert.equal(cleanupWithoutAdmin.status, 401);
|
||||
|
||||
const cleanup = await fetch(`${baseUrl}/cleanup`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json", Authorization: "Bearer test-token" },
|
||||
body: JSON.stringify({ scope: { sourceId: "src:default:folder:docs" } })
|
||||
});
|
||||
assert.equal(cleanup.status, 410);
|
||||
} finally {
|
||||
server.close();
|
||||
setEnvFlag("knowledgeLifecycleEnforced", false);
|
||||
setEnvFlag("lifecycleAdminToken", "");
|
||||
}
|
||||
});
|
||||
|
||||
test("retrieval fails closed when lifecycle is enforced without a catalog", async () => {
|
||||
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
|
||||
const service = new RetrieveService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore);
|
||||
await assert.rejects(
|
||||
service.retrieve("documental", "specific", "hello"),
|
||||
(error) => error instanceof CatalogError && error.statusCode === 503 && error.code === "CATALOG_UNAVAILABLE"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
test("repository state guards do not mark non-pending/indexing versions as failed", async () => {
|
||||
const sqlCalls: string[] = [];
|
||||
const pool = {
|
||||
async query(sql: string) {
|
||||
sqlCalls.push(sql);
|
||||
return sql.includes("UPDATE rag_source_versions") ? { rowCount: 0, rows: [] } : { rowCount: 1, rows: [] };
|
||||
}
|
||||
};
|
||||
const repository = new CatalogRepository(pool as never);
|
||||
await repository.markFailed("ready-version", "TEST", "must not change ready");
|
||||
assert.equal(sqlCalls.some((sql) => sql.includes("UPDATE rag_version_documents")), false);
|
||||
assert.match(sqlCalls[0], /state IN \('pending', 'indexing'\)/);
|
||||
});
|
||||
|
||||
test("reconciler reports invariant violations and recovers orphaned versions", async () => {
|
||||
const calls: string[] = [];
|
||||
const catalog = {
|
||||
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async resolveActiveVersions() { return [{ sourceId: "src:one", sourceVersionId: "v1", expectedPointCount: 2, sourceVersionNumber: 1, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }]; },
|
||||
async validateActiveInvariant() { return ["src:bad"]; },
|
||||
async listOrphanedIndexingCandidates() { return []; },
|
||||
async markNeedsReingest(sourceId: string) { calls.push(sourceId); }
|
||||
};
|
||||
const vectorStore = { async validateCollectionDimensions() { return undefined; }, async countVersionPoints() { return 1; } };
|
||||
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
|
||||
const status = await reconciler.runOnce();
|
||||
|
||||
assert.equal(status.ok, false);
|
||||
assert.deepEqual(status.inconsistentSources, ["src:one"]);
|
||||
assert.deepEqual(status.invariantViolations, ["src:bad"]);
|
||||
assert.equal(status.orphanedVersionsRecovered, 0);
|
||||
assert.deepEqual(calls, ["src:one"]);
|
||||
});
|
||||
|
||||
test("reconciler recovers stale complete indexing versions to ready", async () => {
|
||||
const calls: string[] = [];
|
||||
const catalog = {
|
||||
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async resolveActiveVersions() { return []; },
|
||||
async validateActiveInvariant() { return []; },
|
||||
async listOrphanedIndexingCandidates() {
|
||||
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
|
||||
},
|
||||
async withSourceTryLock(_sourceId: string, handler: () => Promise<unknown>) { calls.push("source-lock"); return handler(); },
|
||||
async withVersionTryLock(_versionId: string, handler: () => Promise<unknown>) { calls.push("version-lock"); return handler(); },
|
||||
async hasCompleteVersionDocuments() { return true; },
|
||||
async recoverVersionReady() { calls.push("ready"); return true; },
|
||||
async markFailed() { calls.push("failed"); }
|
||||
};
|
||||
const vectorStore = { async validateCollectionDimensions() { calls.push("dimensions"); }, async countVersionPoints() { calls.push("count"); return 2; } };
|
||||
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
|
||||
const status = await reconciler.runOnce();
|
||||
|
||||
assert.equal(status.ok, true);
|
||||
assert.equal(status.orphanedVersionsRecovered, 1);
|
||||
assert.equal(status.orphanedVersionsFailed, 0);
|
||||
assert.deepEqual(calls, ["source-lock", "version-lock", "dimensions", "count", "ready"]);
|
||||
});
|
||||
|
||||
test("reconciler marks stale partial indexing versions as failed", async () => {
|
||||
const calls: string[] = [];
|
||||
const catalog = {
|
||||
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async resolveActiveVersions() { return []; },
|
||||
async validateActiveInvariant() { return []; },
|
||||
async listOrphanedIndexingCandidates() {
|
||||
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
|
||||
},
|
||||
async withSourceTryLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async withVersionTryLock(_versionId: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async hasCompleteVersionDocuments() { return true; },
|
||||
async recoverVersionReady() { calls.push("ready"); return true; },
|
||||
async markFailed() { calls.push("failed"); }
|
||||
};
|
||||
const vectorStore = { async validateCollectionDimensions() { return undefined; }, async countVersionPoints() { return 1; } };
|
||||
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, vectorStore as never);
|
||||
const status = await reconciler.runOnce();
|
||||
|
||||
assert.equal(status.orphanedVersionsRecovered, 0);
|
||||
assert.equal(status.orphanedVersionsFailed, 1);
|
||||
assert.deepEqual(calls, ["failed"]);
|
||||
});
|
||||
|
||||
test("reconciler does not touch stale versions while a source lock is held", async () => {
|
||||
const calls: string[] = [];
|
||||
const catalog = {
|
||||
async withGlobalTryLock(_lockName: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async resolveActiveVersions() { return []; },
|
||||
async validateActiveInvariant() { return []; },
|
||||
async listOrphanedIndexingCandidates() {
|
||||
return [{ sourceId: "src:one", versionId: "v1", expectedDocumentCount: 1, expectedPointCount: 2, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
|
||||
},
|
||||
async withSourceTryLock() { return undefined; },
|
||||
async markFailed() { calls.push("failed"); }
|
||||
};
|
||||
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, fakeVectorStore().vectorStore);
|
||||
const status = await reconciler.runOnce();
|
||||
|
||||
assert.equal(status.orphanedVersionsRecovered, 0);
|
||||
assert.equal(status.orphanedVersionsFailed, 0);
|
||||
assert.deepEqual(calls, []);
|
||||
});
|
||||
|
||||
test("reconciler skips when another reconciler holds the advisory lock", async () => {
|
||||
const catalog = {
|
||||
async withGlobalTryLock() { return undefined; },
|
||||
async resolveActiveVersions() { throw new Error("must not run without the lock"); }
|
||||
};
|
||||
const reconciler = new KnowledgeLifecycleReconciler(catalog as never, fakeVectorStore().vectorStore);
|
||||
const status = await reconciler.runOnce();
|
||||
assert.equal(status.ok, true);
|
||||
assert.equal(status.inconsistentSources.length, 0);
|
||||
});
|
||||
|
||||
test("retrieval rejects active versions from a different Qdrant collection", async () => {
|
||||
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
|
||||
const catalog = {
|
||||
async resolveActiveVersions() {
|
||||
return [{ sourceId: "src:one", sourceVersionId: "v1", sourceVersionNumber: 1, expectedPointCount: 1, embeddingDimensions: 3, qdrantCollection: "other_collection" }];
|
||||
},
|
||||
async markNeedsReingest() { return undefined; },
|
||||
async withVersionSharedLocks(_versionIds: string[], handler: () => Promise<unknown>) { return handler(); }
|
||||
};
|
||||
const service = new RetrieveService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore, catalog as never);
|
||||
await assert.rejects(
|
||||
service.retrieve("documental", "specific", "hello"),
|
||||
(error) => error instanceof CatalogError && error.code === "SOURCE_VERSION_INCONSISTENT"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
test("retrieval holds shared locks while validating counts and searching Qdrant", async () => {
|
||||
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => {
|
||||
const calls: string[] = [];
|
||||
const catalog = {
|
||||
async resolveActiveVersions() {
|
||||
calls.push("resolve");
|
||||
return [{ sourceId: "src:one", sourceVersionId: "v1", sourceVersionNumber: 1, expectedPointCount: 1, embeddingDimensions: 3, qdrantCollection: env.qdrantCollection }];
|
||||
},
|
||||
async withVersionSharedLocks(_versionIds: string[], handler: () => Promise<unknown>) {
|
||||
calls.push("lock:start");
|
||||
const result = await handler();
|
||||
calls.push("lock:end");
|
||||
return result;
|
||||
},
|
||||
async markNeedsReingest() { calls.push("needs-reingest"); }
|
||||
};
|
||||
const vectorStore = {
|
||||
kind: "fake",
|
||||
async validateCollectionDimensions() { calls.push("dimensions"); },
|
||||
async countVersionPoints() { calls.push("count"); return 1; },
|
||||
async search() { calls.push("search"); return []; },
|
||||
async browseScope() { return []; }
|
||||
};
|
||||
const service = new RetrieveService(fakeEmbeddingProvider(), vectorStore as never, catalog as never);
|
||||
await service.retrieve("documental", "specific", "hello");
|
||||
assert.deepEqual(calls, ["resolve", "lock:start", "dimensions", "count", "search", "lock:end"]);
|
||||
});
|
||||
});
|
||||
|
||||
test("lifecycle ingest rejects vectors that do not match configured embedding dimensions before upsert", async () => {
|
||||
await withEnvFlags({ knowledgeLifecycleEnforced: true }, async () => withTempMarkdown("# Title\n\nUseful content", async (filePath) => {
|
||||
const calls: string[] = [];
|
||||
const provider: EmbeddingProvider = {
|
||||
providerName: "test-provider",
|
||||
modelName: "test-model",
|
||||
dimensions: 3,
|
||||
async embed(input: string[]) {
|
||||
return input.map(() => [0.1, 0.2]);
|
||||
}
|
||||
};
|
||||
const vectorStore = {
|
||||
kind: "fake",
|
||||
async upsert() { calls.push("upsert"); },
|
||||
async countVersionPoints() { return 0; }
|
||||
};
|
||||
const catalog = {
|
||||
async beginAttempt() { return "attempt-1"; },
|
||||
async updateAttempt() { return undefined; },
|
||||
async withSourceLock(_sourceId: string, handler: () => Promise<unknown>) { return handler(); },
|
||||
async findReusableVersion() { return undefined; },
|
||||
async createPendingVersion() { return { versionId: "version-1", versionNumber: 1, state: "pending", previousVersionId: null }; },
|
||||
async markIndexing() { calls.push("indexing"); },
|
||||
async markReady() { calls.push("ready"); return { versionId: "version-1", versionNumber: 1, state: "ready", previousVersionId: null }; },
|
||||
async markFailed() { calls.push("failed"); }
|
||||
};
|
||||
const service = new IngestService(provider, vectorStore as never, catalog as never);
|
||||
await assert.rejects(
|
||||
service.ingest({ sourceType: "file", sourceRef: "source.md", readPath: filePath, activate: true, expectedActiveVersionId: null }),
|
||||
(error) => error instanceof CatalogError && error.code === "EMBEDDING_DIMENSIONS_INVALID"
|
||||
);
|
||||
assert.deepEqual(calls, ["indexing", "failed"]);
|
||||
}));
|
||||
});
|
||||
|
||||
test("cleanup respects disabled ingest writes even on the legacy path", async () => {
|
||||
await withEnvFlags({ ingestWritesEnabled: false, knowledgeLifecycleEnforced: false }, async () => {
|
||||
const service = new IngestService(fakeEmbeddingProvider(), fakeVectorStore().vectorStore);
|
||||
await assert.rejects(
|
||||
service.cleanup({ sourceId: "src:default:folder:docs" }),
|
||||
(error) => error instanceof CatalogError && error.code === "INGEST_WRITES_DISABLED"
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
test("markPurged fails when the version is no longer in purging state", async () => {
|
||||
const pool = fakeTransactionPool(async (sql) => {
|
||||
if (sql.includes("UPDATE rag_source_versions")) {
|
||||
return { rowCount: 0, rows: [] };
|
||||
}
|
||||
return { rowCount: 1, rows: [] };
|
||||
});
|
||||
const repository = new CatalogRepository(pool as never);
|
||||
await assert.rejects(
|
||||
repository.markPurged("version-1"),
|
||||
(error) => error instanceof CatalogError && error.code === "INVALID_VERSION_STATE"
|
||||
);
|
||||
});
|
||||
|
||||
test("legacy migration apply reuses a catalog version created before payload update", async () => {
|
||||
const source = buildLegacyMigrationPlan([{ id: "p1", payload: {
|
||||
source_id: "src:default:file:source.md",
|
||||
source_type: "file",
|
||||
source_ref: "source.md",
|
||||
document_id: "doc:src:default:file:source.md:source.md",
|
||||
document_key: "source.md",
|
||||
chunk_id: "chk:doc:src:default:file:source.md:source.md:documental:0001",
|
||||
content: "content",
|
||||
embedding_provider: "openrouter",
|
||||
embedding_model: "qwen",
|
||||
embedding_dimensions: 3
|
||||
} }]).sources[0];
|
||||
const sqlCalls: string[] = [];
|
||||
const pool = fakeTransactionPool(async (sql) => {
|
||||
sqlCalls.push(sql);
|
||||
if (sql.includes("FROM rag_legacy_migration_items")) {
|
||||
return { rowCount: 1, rows: [{ version_id: "version-1", expected_point_count: 1 }] };
|
||||
}
|
||||
if (sql.includes("FROM rag_source_versions v")) {
|
||||
return { rowCount: 1, rows: [{ version_id: "version-1", version_number: "1", expected_point_count: "1", source_type: "file", source_ref: "source.md", qdrant_collection: env.qdrantCollection }] };
|
||||
}
|
||||
return { rowCount: 1, rows: [] };
|
||||
});
|
||||
|
||||
const result = await createCatalogVersion(pool as never, "batch-1", source, env.qdrantCollection);
|
||||
assert.deepEqual(result, { versionId: "version-1", versionNumber: 1 });
|
||||
assert.equal(sqlCalls.some((sql) => sql.includes("INSERT INTO rag_sources")), false);
|
||||
});
|
||||
|
||||
test("legacy migration activation is idempotent after payload update and prior activation", async () => {
|
||||
const sqlCalls: string[] = [];
|
||||
const pool = fakeTransactionPool(async (sql) => {
|
||||
sqlCalls.push(sql);
|
||||
if (sql.includes("SELECT v.state")) {
|
||||
return { rowCount: 1, rows: [{ state: "active", active_version_id: "version-1" }] };
|
||||
}
|
||||
return { rowCount: 1, rows: [] };
|
||||
});
|
||||
|
||||
await activateMigratedVersion(pool as never, "src:default:file:source.md", "version-1", 1);
|
||||
assert.equal(sqlCalls.some((sql) => sql.includes("SET state = 'active'")), false);
|
||||
});
|
||||
|
||||
test("legacy rollback prepares the whole batch before restoring and validates restored legacy counts before catalog cleanup", async () => {
|
||||
const calls: string[] = [];
|
||||
const pool = fakeTransactionPool(async (sql) => {
|
||||
if (sql.includes("SELECT state, snapshot_ref")) {
|
||||
calls.push("batch-lock");
|
||||
return { rowCount: 1, rows: [{ state: "verified", snapshot_ref: "snapshot-url" }] };
|
||||
}
|
||||
if (sql.includes("SELECT source_id, version_id, expected_point_count")) {
|
||||
calls.push("items-lock");
|
||||
return { rowCount: 1, rows: [{ source_id: "src:default:file:source.md", version_id: "version-1", expected_point_count: 1 }] };
|
||||
}
|
||||
if (sql.includes("SET active_version_id = NULL")) calls.push("active-null");
|
||||
if (sql.includes("SET state = 'purging'")) calls.push("purging");
|
||||
if (sql.includes("COUNT(*) AS count")) {
|
||||
calls.push("zero-active-check");
|
||||
return { rowCount: 1, rows: [{ count: "0" }] };
|
||||
}
|
||||
if (sql.includes("DELETE FROM rag_version_documents")) calls.push("delete-docs");
|
||||
if (sql.includes("UPDATE rag_legacy_migration_items SET version_id = NULL")) calls.push("items-rolled-back");
|
||||
return { rowCount: 1, rows: [] };
|
||||
});
|
||||
const qdrant = {
|
||||
async recoverSnapshot() { calls.push("restore-snapshot"); return true; },
|
||||
async scroll() {
|
||||
calls.push("verify-legacy-count");
|
||||
return { points: [{ id: "p1", payload: { source_id: "src:default:file:source.md" } }], next_page_offset: undefined };
|
||||
}
|
||||
};
|
||||
|
||||
const result = await rollbackMigratedBatch(pool as never, qdrant as never, "batch-1", "snapshot-url");
|
||||
assert.equal(result.ok, true);
|
||||
assert.ok(calls.indexOf("zero-active-check") < calls.indexOf("restore-snapshot"));
|
||||
assert.ok(calls.indexOf("verify-legacy-count") < calls.indexOf("delete-docs"));
|
||||
assert.ok(calls.includes("items-rolled-back"));
|
||||
});
|
||||
|
||||
test("OpenAPI documents /sources 503 responses", () => {
|
||||
const sources = openApiDocument.paths["/sources"].get.responses;
|
||||
assert.ok("503" in sources);
|
||||
});
|
||||
|
||||
test("legacy migration dry-run blocks weak document identity", () => {
|
||||
const plan = buildLegacyMigrationPlan([
|
||||
{
|
||||
id: "p1",
|
||||
payload: {
|
||||
source_id: "src:default:folder:docs",
|
||||
source_ref: "docs",
|
||||
source_type: "folder",
|
||||
document_id: "doc:src:default:folder:docs:readme.md",
|
||||
chunk_id: "chk:doc:src:default:folder:docs:readme.md:documental:0001",
|
||||
content: "content",
|
||||
embedding_provider: "openrouter",
|
||||
embedding_model: "qwen",
|
||||
embedding_dimensions: 3
|
||||
}
|
||||
}
|
||||
]);
|
||||
|
||||
assert.equal(plan.ok, false);
|
||||
assert.match(plan.blockedReasons.join("\n"), /weak document identity/);
|
||||
});
|
||||
61
tests/lifecycle-utils.test.ts
Normal file
61
tests/lifecycle-utils.test.ts
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import {
|
||||
buildMetadataHash,
|
||||
buildProcessingFingerprint,
|
||||
buildVersionedQdrantPointId,
|
||||
hashOrderedPairs,
|
||||
normalizeContentForHash,
|
||||
sha256Hex
|
||||
} from "../src/shared/utils/ids.js";
|
||||
import { CatalogError } from "../src/modules/catalog/errors.js";
|
||||
import { normalizeExpectedActiveVersion } from "../src/modules/catalog/lifecycle.js";
|
||||
|
||||
test("versioned Qdrant point IDs are deterministic UUIDv5 values", () => {
|
||||
const versionId = "123e4567-e89b-12d3-a456-426614174000";
|
||||
const chunkId = "chk:doc:src:default:folder:docs:readme.md:documental:0001";
|
||||
|
||||
const first = buildVersionedQdrantPointId(versionId, chunkId);
|
||||
const second = buildVersionedQdrantPointId(versionId, chunkId);
|
||||
const differentVersion = buildVersionedQdrantPointId("123e4567-e89b-12d3-a456-426614174001", chunkId);
|
||||
|
||||
assert.equal(first, second);
|
||||
assert.notEqual(first, differentVersion);
|
||||
assert.match(first, /^[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/);
|
||||
});
|
||||
|
||||
test("content hashing normalization is canonical", () => {
|
||||
const left = normalizeContentForHash("\uFEFFTitle\r\nBody \r\n\r\n");
|
||||
const right = normalizeContentForHash("Title\nBody\n");
|
||||
|
||||
assert.equal(left, "Title\nBody\n");
|
||||
assert.equal(sha256Hex(left), sha256Hex(right));
|
||||
});
|
||||
|
||||
test("ordered pair hashes are independent from input order", () => {
|
||||
const first = hashOrderedPairs([["b.md", "hash-b"], ["a.md", "hash-a"]]);
|
||||
const second = hashOrderedPairs([["a.md", "hash-a"], ["b.md", "hash-b"]]);
|
||||
|
||||
assert.equal(first, second);
|
||||
assert.equal(hashOrderedPairs([["a.md", null]]), null);
|
||||
});
|
||||
|
||||
test("processing and metadata fingerprints are stable with sorted keys", () => {
|
||||
assert.equal(
|
||||
buildProcessingFingerprint({ model: "qwen", provider: "openrouter" }),
|
||||
buildProcessingFingerprint({ provider: "openrouter", model: "qwen" })
|
||||
);
|
||||
assert.equal(
|
||||
buildMetadataHash(["b", "a"], { sourceRef: "docs" }),
|
||||
buildMetadataHash(["a", "b"], { sourceRef: "docs" })
|
||||
);
|
||||
});
|
||||
|
||||
test("activation precondition is mandatory when activation is requested", () => {
|
||||
assert.throws(
|
||||
() => normalizeExpectedActiveVersion(undefined, true),
|
||||
(error) => error instanceof CatalogError && error.statusCode === 400
|
||||
);
|
||||
assert.equal(normalizeExpectedActiveVersion(null, true), null);
|
||||
assert.equal(normalizeExpectedActiveVersion(undefined, false), undefined);
|
||||
});
|
||||
Loading…
Add table
Reference in a new issue