Compare commits
9 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
ad278dd680 | ||
|
|
e5ad4a5300 | ||
|
|
2e998a0a4e | ||
|
|
00eeb35923 | ||
|
|
41b91e65d9 | ||
|
|
b1dad3c6e2 | ||
|
|
aec7dc457c | ||
|
|
23f52832d1 | ||
|
|
efa248cfd4 |
@@ -1,2 +1,2 @@
|
||||
FROM semtech/mu-javascript-template:1.8.0
|
||||
LABEL maintainer="info@redpencil"
|
||||
FROM semtech/mu-javascript-template:1.9.1
|
||||
LABEL maintainer="info@redpencil"
|
||||
|
||||
107
README.md
107
README.md
@@ -7,6 +7,31 @@ writes a CSV. The LLM writes no SPARQL and sees no report data.
|
||||
Ad-hoc reports only. The scheduled reports stay in
|
||||
`loket-report-generation-service`.
|
||||
|
||||
## Two modes: refine, then execute
|
||||
|
||||
Every turn is classified first. One small LLM call (`src/llm/mode.js`)
|
||||
returns true/false: does the user's last message clearly say "run it now" or
|
||||
not. The answer picks one of two deterministic code paths.
|
||||
|
||||
**Refinement (Modus A).** The LLM proposes a spec and explains in Dutch what
|
||||
it will list and filter. It may call `lookup_values` to search code lists for
|
||||
candidate values, so its filter suggestions name real values instead of
|
||||
guesses. It never runs a report and never learns what a spec would match: the
|
||||
database it reads may differ from the one the report runs on, so a lookup is
|
||||
a suggestion, not a check. The user must confirm before anything executes.
|
||||
As long as the user has not given a clear command, the turn stays in this
|
||||
mode.
|
||||
|
||||
**Execution (Modus B).** The user has confirmed. No LLM runs. The last spec
|
||||
bijlage in the conversation is the agreed spec: the service reads it from the
|
||||
share, checks it against its profile and runs it as the caller. The answer is
|
||||
a fixed Dutch text with the CSV and the spec as bijlagen. No spec yet, a spec
|
||||
that no longer checks out, or a failed run each give a fixed Dutch failure
|
||||
message. Nothing is retried and no spec is rewritten.
|
||||
|
||||
Only refinement has tools: `list_profiles`, `describe_profile`,
|
||||
`validate_spec`, `read_spec` and `lookup_values`.
|
||||
|
||||
## How it works
|
||||
|
||||
Two documents drive everything:
|
||||
@@ -24,10 +49,16 @@ A **spec** is `rep:ReportSpec` + `sh:NodeShape`: `sh:property` selects the
|
||||
subjects, `rep:columns` says what the CSV holds. The two never interact. The
|
||||
namespace is `http://mu.semte.ch/vocabularies/reporting/` (`rep:`).
|
||||
|
||||
A filter tests only the end of its own path. To test a node in the middle of
|
||||
a path ("the mandate whose role is Burgemeester"), a filter or column carries
|
||||
`rep:where`: a condition written like a filter, from the row, that shares the
|
||||
nodes of every step it has in common with its host. On a column it drops
|
||||
values from the cell, never rows. The app's `report-spec-readme.md` has the
|
||||
details and an example.
|
||||
|
||||
The run: check → seed (`SELECT DISTINCT ?s`, paged) → columns (per group, per
|
||||
chunk of subjects) → assemble (dedup + `rep:collect`) → CSV → register file
|
||||
+ report resource. One run per question; the loop stops after `run_report`
|
||||
returns.
|
||||
+ report resource. One run per confirmed spec.
|
||||
|
||||
## One identity: mu-authorization's
|
||||
|
||||
@@ -41,40 +72,60 @@ the caller's.
|
||||
The tools the LLM calls return URIs, counts, statuses and column names —
|
||||
never a cell value.
|
||||
|
||||
## Two endpoints
|
||||
### The rule: no report data reaches the LLM
|
||||
|
||||
No report row, row count or run error ever reaches the LLM. A run starts
|
||||
after the turn's only LLM call (the mode check) and answers with fixed text.
|
||||
What the LLM does see: the conversation, the profiles, validator messages,
|
||||
spec files, and public reference data read under the service scope (inlined
|
||||
code lists and `lookup_values`, see below).
|
||||
|
||||
Keep it that way. Do not add a tool that previews, samples or counts what a
|
||||
spec would match, not even to catch a `rep:where` that hangs on the wrong
|
||||
node. That check stays on the profile: `validate_spec` says where each
|
||||
`rep:where` applies, and the proposal repeats it to the user.
|
||||
|
||||
### The service scope
|
||||
|
||||
The LLM's own reads during refinement (`describe_profile`'s code lists and
|
||||
`lookup_values`) run under a service scope, not the caller's session. mu's
|
||||
`query(q, { scope })` sends `mu-auth-scope`; the sparql-parser config grants
|
||||
that scope read access to `http://mu.semte.ch/graphs/public` only. So
|
||||
refinement reads only public code lists, regardless of who calls. Execution
|
||||
still runs as the caller, so the report contains what they may see, and the
|
||||
LLM sees none of it.
|
||||
|
||||
The scope URI is `SERVICE_SCOPE` (default
|
||||
`http://services.semantic.works/natural-language-report`) and must match the
|
||||
`with-scope` grant in the app's sparql-parser `config.lisp`. Do not set
|
||||
`DEFAULT_MU_AUTH_SCOPE` on the service: that would scope every query
|
||||
(including the run and the chat store) and break them.
|
||||
|
||||
## The endpoint
|
||||
|
||||
```
|
||||
POST /ask { question, history? } → { answer, toolCalls }
|
||||
POST /assistant/conversations/:id/turns { content } → 202 { id }
|
||||
```
|
||||
|
||||
`/ask` is synchronous: a Dutch question in, the agent loop runs inside the
|
||||
service (profiles, lookups, spec, validate, repair, run), and the answer in
|
||||
Dutch comes back with a trace of the tool calls. The CSV lands in
|
||||
`data/files/`. `history` carries the earlier turns for follow-ups; the
|
||||
conversation lives on the caller's side, the service stores nothing. One
|
||||
question makes one report; the loop stops after `run_report`.
|
||||
|
||||
`/assistant` is the chat. The turn module in
|
||||
This is the chat. The turn module in
|
||||
`src/chat/` is generic — it imports nothing from `src/llm/` or `src/runner/`
|
||||
and moves between services without edits — and `src/report-assistant.js` is
|
||||
this service's one `answer` hook. It records the question, answers `202` with
|
||||
the message id, then runs the same loop with the conversation's history. A
|
||||
report run becomes an interim message with a pending file (`as:Document` with
|
||||
no `as:url`); the URL is set when the run ends. The browser polls
|
||||
the message id, then classifies the turn and refines or executes with the
|
||||
conversation's history. The answer and its bijlagen are written when the turn
|
||||
ends. The browser polls
|
||||
`/chat-conversations/:id?include=messages.attachments` and finds the answer
|
||||
that way. Failure is a message, never a silent loader.
|
||||
|
||||
`/ask` needs `LLM_BASE_URL` set, and answers 503 without it. A full-org report
|
||||
run takes minutes.
|
||||
The chat needs `LLM_BASE_URL`; without it every turn answers with a failure
|
||||
message. A full-org report run takes minutes.
|
||||
|
||||
### The internal tools
|
||||
|
||||
The loop drives four tool handlers (`src/llm/tools.js`): `list_profiles`,
|
||||
`describe_profile`, `validate_spec`, `run_report`. They are machinery, not an
|
||||
external surface — the tool trace in the answer is the only thing a caller
|
||||
sees of them. All LLM-facing text (system prompt, profile menu, tool
|
||||
descriptions, validator errors) is English; the answer the user sees is Dutch.
|
||||
The refinement loop drives one tool set (`src/llm/tools.js`). The tools are
|
||||
machinery, not an external surface. All LLM-facing text (system prompts,
|
||||
profile menu, tool descriptions, validator errors) is English; the answer the
|
||||
user sees is Dutch.
|
||||
|
||||
## Environment
|
||||
|
||||
@@ -86,10 +137,9 @@ descriptions, validator errors) is English; the answer the user sees is Dutch.
|
||||
| `SUBJECT_CHUNK_SIZE` | `100` | subjects per column query (VALUES batch size); lower it when the auth layer rejects long queries |
|
||||
| `ROW_LIMIT` | `200000` | seed limit; failing the job beats truncating |
|
||||
| `MAX_PATH_DEPTH` | `8` | longest allowed spec path |
|
||||
| `RUN_TIMEOUT` | `60` | seconds of silence before `run_report` gives up. Every answered query (seed page, column batch) restarts the clock, so a big report only needs the window per batch, not for the whole run. The run itself is never cancelled: the file still lands in the chat. |
|
||||
| `INLINE_VALUES_MAX` | `50` | code lists up to this size are inlined in `describe_profile` |
|
||||
| `VALUES_TTL` | `3600` | seconds the inlined lists are cached |
|
||||
| `LLM_BASE_URL` | — | enables `/ask`; OpenAI-compatible (`https://ollama.com/v1`) |
|
||||
| `LLM_BASE_URL` | — | the chat's LLM; OpenAI-compatible (`https://ollama.com/v1`) |
|
||||
| `LLM_MODEL` | — | e.g. `gpt-oss:120b` |
|
||||
| `LLM_API_KEY` | — | bearer token, optional for local Ollama |
|
||||
| `SEED_PAGE_SIZE` | `5000` | seed page size |
|
||||
@@ -97,6 +147,7 @@ descriptions, validator errors) is English; the answer the user sees is Dutch.
|
||||
| `SHARE_DIR` | `/share` | where CSVs are written |
|
||||
| `CHAT_ASSISTANT_URI` | — | the `prov:SoftwareAgent` that signs assistant messages; found by lookup when unset |
|
||||
| `CHAT_HISTORY_LIMIT` | `20` | messages of history a turn reads |
|
||||
| `SERVICE_SCOPE` | `http://services.semantic.works/natural-language-report` | the mu-auth-scope for the LLM's own reads (code lists, `lookup_values`). Must match the `with-scope` grant in the app's sparql-parser config. |
|
||||
|
||||
Do not set `ALLOW_MU_AUTH_SUDO`; the template then refuses sudo queries.
|
||||
|
||||
@@ -107,7 +158,7 @@ Do not set `ALLOW_MU_AUTH_SUDO`; the template then refuses sudo queries.
|
||||
```yaml
|
||||
natural-language-report:
|
||||
build: ../natural-language-report-service/
|
||||
image: lblod/natural-language-report-service:0.1.0
|
||||
image: lblod/natural-language-report-service:0.2.0
|
||||
volumes:
|
||||
- ./data/files:/share
|
||||
- ./config/report-profiles/:/config/profiles/
|
||||
@@ -116,12 +167,12 @@ natural-language-report:
|
||||
`config/dispatcher/dispatcher.ex`:
|
||||
|
||||
```elixir
|
||||
match "/natural-language-reports/*path" do
|
||||
forward conn, path, "http://natural-language-report/"
|
||||
match "/assistant/*path" do
|
||||
forward conn, path, "http://natural-language-report/assistant/"
|
||||
end
|
||||
```
|
||||
|
||||
One route to the service root; `POST /ask` is the endpoint.
|
||||
One route: the chat's turn endpoint.
|
||||
First deploy: `docker compose build natural-language-report`, then
|
||||
`docker compose up -d natural-language-report dispatcher`.
|
||||
|
||||
|
||||
32
app.js
32
app.js
@@ -1,8 +1,5 @@
|
||||
import { app, errorHandler } from 'mu';
|
||||
import express from 'express';
|
||||
import { loadProfiles } from './src/runner/profile.js';
|
||||
import { sessionQuery, sessionUpdate, stripStoreError } from './src/db.js';
|
||||
import { ask } from './src/llm/loop.js';
|
||||
import { mountChat } from './src/chat/index.js';
|
||||
import { reportAssistant } from './src/report-assistant.js';
|
||||
|
||||
@@ -21,33 +18,8 @@ async function whenProfiles() {
|
||||
}
|
||||
}
|
||||
|
||||
// The chat. reportAssistant is the answer hook: it runs the LLM loop with the
|
||||
// conversation so far and turns a report run into an interim message with a
|
||||
// pending file.
|
||||
// The chat. reportAssistant is the answer hook: it refines the spec with the
|
||||
// LLM, or runs the agreed spec without it.
|
||||
mountChat(app, { path: '/assistant', answer: reportAssistant(whenProfiles) });
|
||||
|
||||
// POST /ask: one question in, the answer in Dutch and a trace of the tool
|
||||
// calls back. Follow-ups carry the earlier turns in `history`.
|
||||
// The template parses application/vnd.api+json only; this route parses the
|
||||
// plain JSON body itself.
|
||||
app.post('/ask', express.json(), async (req, res) => {
|
||||
const { question, history } = req.body || {};
|
||||
if (!question) return res.status(400).json({ error: 'no question in the request body' });
|
||||
if (!process.env.LLM_BASE_URL) return res.status(503).json({ error: 'no LLM configured (set LLM_BASE_URL)' });
|
||||
|
||||
try {
|
||||
const profiles = await whenProfiles();
|
||||
const turns = [
|
||||
...(Array.isArray(history) ? history : []),
|
||||
{ role: 'user', content: question },
|
||||
];
|
||||
const session = { query: sessionQuery, update: sessionUpdate };
|
||||
const { text, trace } = await ask(turns, profiles, session);
|
||||
res.json({ answer: text, toolCalls: trace });
|
||||
} catch (e) {
|
||||
console.error('[ask] failed:', e);
|
||||
res.status(502).json({ error: stripStoreError(e.message) });
|
||||
}
|
||||
});
|
||||
|
||||
app.use(errorHandler);
|
||||
4
package-lock.json
generated
4
package-lock.json
generated
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "natural-language-report-service",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "natural-language-report-service",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"express": "^5.2.1",
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "natural-language-report-service",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"description": "Turns a SHACL report spec into SPARQL queries, joins the results and writes a CSV. The LLM writes specs; this service runs them.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
81
src/chat/attachment-file.js
Normal file
81
src/chat/attachment-file.js
Normal file
@@ -0,0 +1,81 @@
|
||||
// A file made to sit on a message: written to the share, registered as a
|
||||
// logical nfo:FileDataObject over a physical share:// object, typed with
|
||||
// dct:type. The file service serves /files/<uuid>/download for it.
|
||||
import { writeFileSync } from 'fs';
|
||||
import { randomUUID } from 'crypto';
|
||||
import { sessionUpdate, sparqlEscapeUri, sparqlEscapeString,
|
||||
sparqlEscapeInt, sparqlEscapeDateTime } from '../db.js';
|
||||
import { ATTACHMENT_TYPES, SPEC_MEDIA_TYPE } from './vocab.js';
|
||||
|
||||
const SHARE_DIR = process.env.SHARE_DIR || '/share';
|
||||
const FILE_BASE = 'http://data.lblod.info/files/';
|
||||
const SHARE_BASE = 'share://';
|
||||
const UUID = 'http://mu.semte.ch/vocabularies/core/uuid';
|
||||
|
||||
const NFO = 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#';
|
||||
const FILE = `${NFO}FileDataObject`;
|
||||
const FILE_NAME = `${NFO}fileName`;
|
||||
const FILE_SIZE = `${NFO}fileSize`;
|
||||
const FILE_CREATED = `${NFO}fileCreated`;
|
||||
const FORMAT = 'http://purl.org/dc/terms/format';
|
||||
const RDF_TYPE = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type';
|
||||
const FILE_TYPE = 'http://purl.org/dc/terms/type';
|
||||
const DATA_SOURCE = 'http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource';
|
||||
|
||||
const file = sparqlEscapeUri;
|
||||
const str = sparqlEscapeString;
|
||||
|
||||
// storeSpecFile(update, content) → { uri, name }. One spec, one unique file:
|
||||
// spec bijlagen keep their name apart from the report's ttl-file. Every
|
||||
// spec file starts with "specificatie-", the read_spec tool opens only
|
||||
// those.
|
||||
export async function storeSpecFile(update, content) {
|
||||
const fileName = `specificatie-${randomUUID()}.ttl`;
|
||||
writeFileSync(`${SHARE_DIR}/${fileName}`, content, 'utf8');
|
||||
return registerFileObject(update, {
|
||||
fileName,
|
||||
format: SPEC_MEDIA_TYPE,
|
||||
size: Buffer.byteLength(content),
|
||||
type: ATTACHMENT_TYPES.spec,
|
||||
});
|
||||
}
|
||||
|
||||
// registerFileObject(update, { fileName, format, size, type }) → { uri, name }.
|
||||
// One logical file over one physical share:// file, both nfo:FileDataObject,
|
||||
// both carrying the dct:type when given.
|
||||
export async function registerFileObject(update, { fileName, format, size, type }) {
|
||||
const id = randomUUID();
|
||||
const physicalId = randomUUID();
|
||||
const logical = `${FILE_BASE}${id}`;
|
||||
const physical = `${SHARE_BASE}${fileName}`;
|
||||
const now = sparqlEscapeDateTime(new Date()) // full URI, xsd spelled out
|
||||
.replace('xsd:dateTime', '<http://www.w3.org/2001/XMLSchema#dateTime>');
|
||||
|
||||
const logicalPairs = [
|
||||
`${file(RDF_TYPE)} ${file(FILE)}`,
|
||||
`${file(UUID)} ${str(id)}`,
|
||||
`${file(FILE_NAME)} ${str(fileName)}`,
|
||||
`${file(FORMAT)} ${str(format)}`,
|
||||
`${file(FILE_SIZE)} ${sparqlEscapeInt(size)}`,
|
||||
`${file('http://purl.org/dc/terms/created')} ${now}`,
|
||||
...(type ? [`${file(FILE_TYPE)} ${file(type)}`] : []),
|
||||
];
|
||||
const physicalPairs = [
|
||||
`${file(RDF_TYPE)} ${file(FILE)}`,
|
||||
`${file(UUID)} ${str(physicalId)}`,
|
||||
`${file(FILE_NAME)} ${str(fileName)}`,
|
||||
`${file(FORMAT)} ${str(format)}`,
|
||||
`${file(FILE_SIZE)} ${sparqlEscapeInt(size)}`,
|
||||
`${file(FILE_CREATED)} ${now}`,
|
||||
`${file(DATA_SOURCE)} ${file(logical)}`,
|
||||
...(type ? [`${file(FILE_TYPE)} ${file(type)}`] : []),
|
||||
];
|
||||
|
||||
await (update || sessionUpdate)(`
|
||||
INSERT DATA {
|
||||
${file(logical)} ${logicalPairs.join(' ;\n ')} .
|
||||
${file(physical)} ${physicalPairs.join(' ;\n ')} .
|
||||
}`);
|
||||
|
||||
return { uri: logical, name: fileName };
|
||||
}
|
||||
@@ -4,7 +4,7 @@ import express from 'express';
|
||||
import { query as muQuery, update as muUpdate } from 'mu';
|
||||
import {
|
||||
readConversation, readHistory, findAssistant,
|
||||
writeMessage, setDocumentUrl, dropDocument,
|
||||
writeMessage,
|
||||
} from './store.js';
|
||||
|
||||
const TEXT = {
|
||||
@@ -58,13 +58,18 @@ export function mountChat(app, { path = '/assistant', answer }) {
|
||||
say: (text, attachments = []) => writeMessage({
|
||||
conversationUri: conversation.uri, content: text, maker: assistant, attachments,
|
||||
}),
|
||||
setUrl: (documentUri, url) => setDocumentUrl(documentUri, url, conversation.uri),
|
||||
drop: (documentUri) => dropDocument(documentUri),
|
||||
};
|
||||
|
||||
try {
|
||||
const text = await answer(turn);
|
||||
if (typeof text === 'string' && text.trim()) await turn.say(text.trim());
|
||||
// answer may return a string, or { text, attachments } for answers the
|
||||
// assistant writes with an attachment, such as the report spec.
|
||||
const result = await answer(turn);
|
||||
const text = typeof result === 'string' ? result : result?.text;
|
||||
const attachments
|
||||
= typeof result === 'string' ? [] : (result?.attachments ?? []);
|
||||
if (typeof text === 'string' && text.trim()) {
|
||||
await turn.say(text.trim(), attachments);
|
||||
}
|
||||
} catch (e) {
|
||||
console.error('[chat] turn failed:', e);
|
||||
try {
|
||||
|
||||
@@ -1,6 +1,12 @@
|
||||
// The chat's reads and writes. Nothing here knows the service it lives in.
|
||||
// Full URIs everywhere: the auth layer rejects a prefixed name with no
|
||||
// PREFIX line with an opaque 500.
|
||||
//
|
||||
// A bijlage is a real file: a logical nfo:FileDataObject the file service
|
||||
// serves at /files/<uuid>/download, over a physical share:// object with
|
||||
// nie:dataSource back to the logical one. Every attachment carries the
|
||||
// file's logical URI (see chat/attachment-file.js how those files are
|
||||
// made); nothing here writes files.
|
||||
import { randomUUID } from 'crypto';
|
||||
import { query as muQuery, update as muUpdate,
|
||||
sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime } from 'mu';
|
||||
@@ -20,14 +26,9 @@ const T = {
|
||||
created: 'http://purl.org/dc/terms/created',
|
||||
maker: 'http://xmlns.com/foaf/0.1/maker',
|
||||
attachment: 'https://www.w3.org/ns/activitystreams#attachment',
|
||||
document: 'https://www.w3.org/ns/activitystreams#Document',
|
||||
name: 'https://www.w3.org/ns/activitystreams#name',
|
||||
mediaType: 'https://www.w3.org/ns/activitystreams#mediaType',
|
||||
url: 'https://www.w3.org/ns/activitystreams#url',
|
||||
softwareAgent: 'http://www.w3.org/ns/prov#SoftwareAgent',
|
||||
};
|
||||
const MESSAGE_BASE = 'http://data.lblod.info/id/chat-messages/';
|
||||
const DOCUMENT_BASE = 'http://data.lblod.info/id/chat-documents/';
|
||||
|
||||
const u = sparqlEscapeUri;
|
||||
const s = sparqlEscapeString;
|
||||
@@ -53,19 +54,46 @@ export async function readConversation(id) {
|
||||
return { uri: b.conversation.value, id, title: b.title?.value, creator: b.creator?.value };
|
||||
}
|
||||
|
||||
// The last HISTORY_LIMIT messages, oldest first, as { role, content }.
|
||||
// The last HISTORY_LIMIT messages, oldest first, as
|
||||
// { role, content, attachments: [{ uri, name, mediaType, type }] }.
|
||||
export async function readHistory(conversationUri, assistantUri) {
|
||||
const json = await muQuery(`
|
||||
SELECT ?content ?maker ?created WHERE {
|
||||
SELECT ?message ?content ?maker ?created ?doc ?docName ?docFormat ?docType WHERE {
|
||||
?message ${u(T.hasContainer)} ${u(conversationUri)} ;
|
||||
${u(T.content)} ?content ;
|
||||
${u(T.maker)} ?maker ;
|
||||
${u(T.created)} ?created .
|
||||
} ORDER BY DESC(?created) LIMIT ${HISTORY_LIMIT}`);
|
||||
return bindings(json).reverse().map(b => ({
|
||||
role: b.maker.value === assistantUri ? 'assistant' : 'user',
|
||||
content: b.content.value,
|
||||
}));
|
||||
OPTIONAL {
|
||||
?message ${u(T.attachment)} ?doc .
|
||||
OPTIONAL { ?doc ${u('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName')} ?docName . }
|
||||
OPTIONAL { ?doc ${u('http://purl.org/dc/terms/format')} ?docFormat . }
|
||||
OPTIONAL { ?doc ${u('http://purl.org/dc/terms/type')} ?docType . }
|
||||
}
|
||||
} ORDER BY DESC(?created) LIMIT ${HISTORY_LIMIT * 4}`);
|
||||
const messages = [];
|
||||
const byMessage = new Map();
|
||||
for (const b of bindings(json).reverse()) {
|
||||
const uri = b.message.value;
|
||||
let message = byMessage.get(uri);
|
||||
if (!message) {
|
||||
message = {
|
||||
role: b.maker.value === assistantUri ? 'assistant' : 'user',
|
||||
content: b.content.value,
|
||||
attachments: [],
|
||||
};
|
||||
byMessage.set(uri, message);
|
||||
messages.push(message);
|
||||
}
|
||||
if (b.doc) {
|
||||
message.attachments.push({
|
||||
uri: b.doc.value,
|
||||
name: b.docName?.value,
|
||||
mediaType: b.docFormat?.value,
|
||||
type: b.docType?.value,
|
||||
});
|
||||
}
|
||||
}
|
||||
return messages.slice(-HISTORY_LIMIT);
|
||||
}
|
||||
|
||||
// The one assistant the caller can read. Throws unless there is exactly one.
|
||||
@@ -81,17 +109,14 @@ export async function findAssistant() {
|
||||
|
||||
// --- writes -------------------------------------------------------------------
|
||||
|
||||
// One message, both types, its attachments, and the conversation touched, in
|
||||
// One message, both types, its bijlagen, and the conversation touched, in
|
||||
// one update. `title` is only written when given (the first question names
|
||||
// an unnamed conversation). Returns { uri, id, documents: [{ uri, id }] }.
|
||||
// an unnamed conversation). Every attachment is the logical uri of a file
|
||||
// that already exists.
|
||||
export async function writeMessage({ conversationUri, content, maker, attachments = [], title }) {
|
||||
const id = randomUUID();
|
||||
const uri = `${MESSAGE_BASE}${id}`;
|
||||
const now = dt(new Date());
|
||||
const documents = attachments.map(a => {
|
||||
const id = randomUUID();
|
||||
return { ...a, id, uri: `${DOCUMENT_BASE}${id}` };
|
||||
});
|
||||
|
||||
// Predicate pairs joined with ';', never a trailing one: it breaks the Turtle.
|
||||
const pairs = [
|
||||
@@ -103,7 +128,10 @@ export async function writeMessage({ conversationUri, content, maker, attachment
|
||||
`${u(T.created)} ${now}`,
|
||||
];
|
||||
if (maker) pairs.push(`${u(T.maker)} ${u(maker)}`);
|
||||
for (const d of documents) pairs.push(`${u(T.attachment)} ${u(d.uri)}`);
|
||||
for (const attachment of attachments) {
|
||||
if (!attachment.uri) throw new Error('an attachment without a file uri');
|
||||
pairs.push(`${u(T.attachment)} ${u(attachment.uri)}`);
|
||||
}
|
||||
|
||||
const conversationPairs = [`${u(T.lastActivity)} ${now}`];
|
||||
if (title) conversationPairs.push(`${u(T.title)} ${s(title)}`);
|
||||
@@ -114,50 +142,11 @@ export async function writeMessage({ conversationUri, content, maker, attachment
|
||||
}
|
||||
INSERT {
|
||||
${u(uri)} ${pairs.join(' ;\n ')} .
|
||||
${documents.map(d => `
|
||||
${u(d.uri)} ${[
|
||||
`${u(T.type)} ${u(T.document)}`,
|
||||
`${u(T.uuid)} ${s(d.id)}`,
|
||||
`${u(T.name)} ${s(d.name)}`,
|
||||
`${u(T.mediaType)} ${s(d.mediaType || 'application/octet-stream')}`,
|
||||
...(d.url ? [`${u(T.url)} ${s(d.url)}`] : []),
|
||||
].join(' ;\n ')} .`).join('')}
|
||||
${u(conversationUri)} ${conversationPairs.join(' ;\n ')} .
|
||||
}
|
||||
WHERE {
|
||||
OPTIONAL { ${u(conversationUri)} ${u(T.lastActivity)} ?old . }
|
||||
}`);
|
||||
|
||||
return { uri, id, documents: documents.map(({ uri, id }) => ({ uri, id })) };
|
||||
}
|
||||
|
||||
// The file is ready.
|
||||
export async function setDocumentUrl(documentUri, url, conversationUri) {
|
||||
const now = dt(new Date());
|
||||
await muUpdate(`
|
||||
DELETE {
|
||||
${u(documentUri)} ${u(T.url)} ?oldUrl .
|
||||
${u(conversationUri)} ${u(T.lastActivity)} ?old .
|
||||
}
|
||||
INSERT {
|
||||
${u(documentUri)} ${u(T.url)} ${s(url)} .
|
||||
${u(conversationUri)} ${u(T.lastActivity)} ${now} .
|
||||
}
|
||||
WHERE {
|
||||
OPTIONAL { ${u(documentUri)} ${u(T.url)} ?oldUrl . }
|
||||
OPTIONAL { ${u(conversationUri)} ${u(T.lastActivity)} ?old . }
|
||||
}`);
|
||||
}
|
||||
|
||||
// The file failed; the card disappears, the caller says why in a message.
|
||||
export async function dropDocument(documentUri) {
|
||||
await muUpdate(`
|
||||
DELETE {
|
||||
?message ${u(T.attachment)} ${u(documentUri)} .
|
||||
${u(documentUri)} ?p ?o .
|
||||
}
|
||||
WHERE {
|
||||
${u(documentUri)} ?p ?o .
|
||||
OPTIONAL { ?message ${u(T.attachment)} ${u(documentUri)} . }
|
||||
}`);
|
||||
return { uri, id };
|
||||
}
|
||||
|
||||
8
src/chat/vocab.js
Normal file
8
src/chat/vocab.js
Normal file
@@ -0,0 +1,8 @@
|
||||
// The bijlage types of the report assistant. Match the
|
||||
// add-chat-attachment-types-codelist migration in the portal.
|
||||
export const ATTACHMENT_TYPES = {
|
||||
spec: 'http://lblod.data.gift/concepts/6693014e-7b7d-448c-bb24-80c9758521f3',
|
||||
result: 'http://lblod.data.gift/concepts/28d97698-e219-4a92-bdbd-7ab367a7524e',
|
||||
};
|
||||
export const SPEC_BASE_NAME = 'specificatie';
|
||||
export const SPEC_MEDIA_TYPE = 'text/turtle';
|
||||
17
src/db.js
17
src/db.js
@@ -13,7 +13,6 @@ import {
|
||||
sparqlEscapeBool,
|
||||
} from 'mu';
|
||||
|
||||
export const sessionQuery = muQuery;
|
||||
export const sessionUpdate = muUpdate;
|
||||
export {
|
||||
sparqlEscapeUri,
|
||||
@@ -24,7 +23,15 @@ export {
|
||||
sparqlEscapeBool,
|
||||
};
|
||||
|
||||
// The store's error text may name graphs or internals we do not pass on.
|
||||
export function stripStoreError(text) {
|
||||
return String(text).split('\n')[0].slice(0, 300);
|
||||
}
|
||||
// The service's own scope. Reads the LLM drives during refinement (code
|
||||
// lists, lookup_values) run under this scope, not the caller's session. The
|
||||
// sparql-parser config grants this scope read access to the public graph
|
||||
// only, so refinement reads only public data. Execution still runs as the
|
||||
// caller. Set to match the with-scope grant in config.lisp.
|
||||
export const SERVICE_SCOPE =
|
||||
process.env.SERVICE_SCOPE || 'http://services.semantic.works/natural-language-report';
|
||||
|
||||
// Wrap a mu query/update so it carries the service scope. mu's query/update
|
||||
// accept { scope } on mu-javascript-template >= 1.9.0.
|
||||
export function scopedQuery(q) { return muQuery(q, { scope: SERVICE_SCOPE }); }
|
||||
export function scopedUpdate(q) { return muUpdate(q, { scope: SERVICE_SCOPE }); }
|
||||
@@ -3,7 +3,6 @@
|
||||
// never has to search the short ones.
|
||||
|
||||
import { fieldsOf, entityForClass } from '../runner/profile.js';
|
||||
import { sessionQuery } from '../db.js';
|
||||
|
||||
const INLINE_VALUES_MAX = Number(process.env.INLINE_VALUES_MAX || 50);
|
||||
const VALUES_TTL = Number(process.env.VALUES_TTL || 3600) * 1000;
|
||||
@@ -11,26 +10,28 @@ const VALUES_TTL = Number(process.env.VALUES_TTL || 3600) * 1000;
|
||||
// In-memory cache: profile+field → { values, at }. Refreshed lazily when stale.
|
||||
const cache = new Map();
|
||||
|
||||
export async function codeListValues(profile) {
|
||||
// Returns a map: "shapeUri|predicate|inverse" → [{ uri, label }]
|
||||
// codeListValues(profile, queryFn) → map "shapeUri|predicate|inverse"
|
||||
// → [{ uri, label }]. queryFn runs under the service scope (public graph), so
|
||||
// code lists read the same regardless of the caller's rights.
|
||||
export async function codeListValues(profile, queryFn) {
|
||||
const out = {};
|
||||
for (const shape of profile.shapes) {
|
||||
for (const f of shape.fields) {
|
||||
if (!f.class) continue;
|
||||
const key = `${shape.uri}|${f.path}|${f.inverse ? 1 : 0}`;
|
||||
out[key] = await loadCodeList(profile, f, key);
|
||||
out[key] = await loadCodeList(f, key, queryFn);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function loadCodeList(profile, f, key) {
|
||||
async function loadCodeList(f, key, queryFn) {
|
||||
if (!f.class) return [];
|
||||
const hit = cache.get(key);
|
||||
if (hit && Date.now() - hit.at < VALUES_TTL) return hit.values;
|
||||
let values = [];
|
||||
try {
|
||||
values = await queryCodeList(f.class);
|
||||
values = await queryCodeList(f.class, queryFn);
|
||||
} catch (e) {
|
||||
// the read failed; report nothing, the LLM will use rep:anyOf with words
|
||||
values = [];
|
||||
@@ -39,12 +40,12 @@ async function loadCodeList(profile, f, key) {
|
||||
return values;
|
||||
}
|
||||
|
||||
async function queryCodeList(type) {
|
||||
async function queryCodeList(type, queryFn) {
|
||||
const q = `SELECT ?uri ?label WHERE {
|
||||
?uri a <${type}> .
|
||||
OPTIONAL { ?uri <http://www.w3.org/2004/02/skos/core#prefLabel> ?label . }
|
||||
} ORDER BY ?label LIMIT ${INLINE_VALUES_MAX + 1}`;
|
||||
const r = await sessionQuery(q);
|
||||
const r = await queryFn(q);
|
||||
const bindings = r.results.bindings;
|
||||
return bindings.slice(0, INLINE_VALUES_MAX).map(b => ({
|
||||
uri: b.uri.value,
|
||||
@@ -67,6 +68,7 @@ export function describeProfile(profile, codeLists = {}) {
|
||||
lines.push(' sh:property [ ... ] ; # filters, optional');
|
||||
lines.push(' rep:columns ( [ sh:path ( ... ) ; rdfs:label "..." ] ... ) .');
|
||||
lines.push('Every column ends on a value field. Use rep:self as sh:path for the subject URI.');
|
||||
lines.push('For the URI of a linked node, end the column on that link and add sh:nodeKind sh:IRI to the column.');
|
||||
lines.push('');
|
||||
lines.push('sh:targetClass is the full URI in brackets after the entity below.');
|
||||
lines.push('In sh:path, use the predicate URI shown after each field name, not the field name itself.');
|
||||
|
||||
117
src/llm/lookup.js
Normal file
117
src/llm/lookup.js
Normal file
@@ -0,0 +1,117 @@
|
||||
// Search a code list too long to inline: regex on str(?label), so
|
||||
// language-tagged labels match. Returns the matches, the total, and an
|
||||
// `exact` flag when a label equals the term (case aside).
|
||||
//
|
||||
// This is a suggest helper, not a spec check. It reads candidate values the
|
||||
// LLM can put in a filter; it never runs the spec and never sees report
|
||||
// rows. queryFn runs under the service scope (public graph only), so the
|
||||
// suggestions come from the same data set the profile describes, regardless
|
||||
// of who calls.
|
||||
|
||||
import { sparqlEscapeString } from '../db.js';
|
||||
|
||||
const LIMIT = 25;
|
||||
|
||||
// lookupValues(profile, term, fieldSpec, queryFn) →
|
||||
// { total, exact, matches } or { error }.
|
||||
// field is "shapeLabel.fieldLabel" as the LLM writes it, or a predicate URI.
|
||||
// We resolve it to the field's sh:class to know what type to search.
|
||||
export async function lookupValues(profile, term, fieldSpec, queryFn) {
|
||||
const type = resolveClass(profile, fieldSpec);
|
||||
if (!type) return { total: 0, exact: null, matches: [], error: `field "${fieldSpec}" not found in profile "${profile.title}"` };
|
||||
|
||||
// Search within the class first. A common word ("gemeente") also sits in
|
||||
// hundreds of unrelated labels; an unscoped scan drowns the code.
|
||||
let rows = await labelQuery(term, type, queryFn);
|
||||
// Code lists sometimes type their entries differently than the profile
|
||||
// says (MAR codes are ext:supervision/Nomenclature, the profile says
|
||||
// skos:Concept). Fall back to a scan over everything.
|
||||
if (!rows.length) rows = await labelQuery(term, null, queryFn);
|
||||
// Order: an exact label (case aside) first, then the rest as the store
|
||||
// ordered them. "Gent" must surface before "AGB Erfgoed Gent".
|
||||
const lower = term.toLowerCase();
|
||||
const exactRow = rows.find(b => b.label.toLowerCase() === lower);
|
||||
const rest = rows.filter(b => b !== exactRow);
|
||||
const ordered = exactRow ? [exactRow, ...rest] : rest;
|
||||
let matches = ordered.slice(0, LIMIT);
|
||||
if (!matches.length) {
|
||||
const byCode = await codeQuery(profile, type, term, queryFn);
|
||||
matches = byCode;
|
||||
}
|
||||
const exact = matches.find(m => m.label.toLowerCase() === term.toLowerCase());
|
||||
return {
|
||||
total: matches.length >= LIMIT ? `${LIMIT}+` : matches.length,
|
||||
exact: exact ? exact.uri : null,
|
||||
matches,
|
||||
};
|
||||
}
|
||||
|
||||
async function labelQuery(term, type, queryFn) {
|
||||
const q = `SELECT DISTINCT ?uri ?label WHERE {
|
||||
?uri a ${type ? `<${type}>` : '?anyType'} ;
|
||||
<http://www.w3.org/2004/02/skos/core#prefLabel> ?label .
|
||||
FILTER(REGEX(str(?label), ${sparqlEscapeString(escapeRegex(term))}, "i"))
|
||||
} ORDER BY ?label LIMIT ${LIMIT * 8}`;
|
||||
const r = await queryFn(q);
|
||||
return r.results.bindings.map(b => ({ label: b.label.value, uri: b.uri.value }));
|
||||
}
|
||||
|
||||
// MAR codes: the label says "MAR7300 - …" but the searchable code sits on
|
||||
// ext:supervision/nomenclatureCode. One extra query, only when the label
|
||||
// search came up empty.
|
||||
async function codeQuery(profile, type, term, queryFn) {
|
||||
const q = `SELECT ?uri ?label WHERE {
|
||||
?uri <http://mu.semte.ch/vocabularies/ext/supervision/nomenclatureCode> ?code ;
|
||||
<http://www.w3.org/2004/02/skos/core#prefLabel> ?label .
|
||||
FILTER(CONTAINS(LCASE(str(?code)), LCASE(${sparqlEscapeString(term)})))
|
||||
} ORDER BY ?label LIMIT ${LIMIT}`;
|
||||
try {
|
||||
const r = await queryFn(q);
|
||||
return r.results.bindings.map(b => ({ label: b.label.value, uri: b.uri.value }));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
function resolveClass(profile, fieldSpec) {
|
||||
if (fieldSpec.includes('/')) return null;
|
||||
// "shapeLabel.fieldLabel" → find the field carrying sh:class
|
||||
const dot = fieldSpec.lastIndexOf('.');
|
||||
if (dot < 0) {
|
||||
// bare predicate URI
|
||||
for (const shape of profile.shapes) {
|
||||
for (const f of shape.fields) {
|
||||
if (f.path === fieldSpec && f.class) return f.class;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
const shapeLabel = fieldSpec.slice(0, dot);
|
||||
const fieldLabel = fieldSpec.slice(dot + 1);
|
||||
const shape = profile.shapes.find(s => s.label === shapeLabel);
|
||||
if (!shape) return null;
|
||||
const f = shape.fields.find(x => x.name === fieldLabel);
|
||||
// sh:node marks a link, not a code list; the searchable type is the linked
|
||||
// shape's sh:targetClass ("melding.bestuurseenheid" → besluit:Bestuurseenheid).
|
||||
if (f?.class) return f.class;
|
||||
if (f?.node) {
|
||||
const linked = profile.shapes.find(s => s.uri === f.node);
|
||||
return linked?.targetClass || null;
|
||||
}
|
||||
// The model sometimes writes shape.field on a linked shape
|
||||
// ("bestuurseenheid.naam"); resolve it to that shape's target class.
|
||||
for (const s of profile.shapes) {
|
||||
if (s.label === shapeLabel || shortType(s.targetClass) === shapeLabel) {
|
||||
return s.targetClass;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function shortType(cls) {
|
||||
return cls.split('#').pop() || cls.split('/').pop();
|
||||
}
|
||||
|
||||
function escapeRegex(s) {
|
||||
return String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
}
|
||||
108
src/llm/loop.js
108
src/llm/loop.js
@@ -1,6 +1,12 @@
|
||||
// The agent loop: POST /v1/chat/completions with tool calls,
|
||||
// OpenAI-compatible (LLM_BASE_URL, LLM_MODEL, LLM_API_KEY). No provider
|
||||
// SDKs. Stops when the model answers without tool calls.
|
||||
//
|
||||
// Refinement only: the LLM proposes a spec, may look up candidate values in
|
||||
// code lists (under the service scope) and iterates with the user. It never
|
||||
// runs a report and never learns what a spec would match. The user must
|
||||
// confirm before anything executes; execution runs the agreed spec without
|
||||
// the LLM (../report-assistant.js).
|
||||
|
||||
import { buildTools } from './tools.js';
|
||||
|
||||
@@ -11,52 +17,107 @@ const LLM_API_KEY = process.env.LLM_API_KEY;
|
||||
// LOG_LLM=1 dumps every request to and response from the provider, verbatim.
|
||||
const LOG_LLM = /^(true|1)$/i.test(process.env.LOG_LLM || '');
|
||||
|
||||
const SYSTEM_PROMPT = `You write report specs. You never write SPARQL and you never see report data.
|
||||
// Refinement (Modus A). The LLM and the user iterate on a spec. The LLM may
|
||||
// look up candidate values in code lists (lookup_values, under the service
|
||||
// scope) to make concrete suggestions, but it may not run a report and never
|
||||
// learns how many subjects a spec would match. It ends with a proposal in
|
||||
// Dutch and asks the user to confirm.
|
||||
const REFINE_PROMPT = `You refine report specs with the user. You never write SPARQL and you never run a report.
|
||||
|
||||
The profile lists what can be asked. Compose paths by chaining fields.
|
||||
|
||||
Rules:
|
||||
- A column must end on a value, not on a link. Chain one more hop.
|
||||
- A column must end on a value, not on a link. Chain one more hop. Only when
|
||||
the user asks for the URI of a linked node itself, end the column on that
|
||||
link and add sh:nodeKind sh:IRI to the column.
|
||||
- When two entities share one sh:targetClass, the spec also needs rep:entity
|
||||
with that entity's URI; describe_profile shows it after each entity.
|
||||
- Filters go in sh:property. Columns go in rep:columns, in the order you want.
|
||||
- To match a topic rather than a value, list the Dutch words and compounds you
|
||||
would expect and put them all in rep:anyOf.
|
||||
- A filter tests only the end of its own path, and two filters never share a
|
||||
node. To test a node in the middle of a path (for example "the mandate
|
||||
whose role is Burgemeester"), give that filter or column a rep:where:
|
||||
rep:where [ sh:path ( <the same first steps> <more steps> ) ; <one condition> ].
|
||||
Write its path from the row, like a filter. The steps it has in common with
|
||||
its filter or column are the same nodes, and its condition tests where its
|
||||
own path ends. On a column, rep:where drops values from the cell, never
|
||||
rows. Several rep:where on one filter or column all apply.
|
||||
- validate_spec answers "ok" plus one line per rep:where saying where it
|
||||
applies. Say that in Dutch in your proposal, so the user can check it.
|
||||
- Only offer what validate_spec accepted. Before you suggest an alternative,
|
||||
write it and validate it; if it does not validate, say it cannot be done.
|
||||
- You cannot count, sort, take the first N, or compare two subjects. Say so.
|
||||
- The report's title is the dct:title you write in the spec. There is no other
|
||||
title input.
|
||||
- One question makes one report. Write one spec, validate it, run it, then stop.
|
||||
Do not write a second report for the same question.
|
||||
|
||||
Always prefix the spec with @prefix lines. Use the prefixes the profile declares.
|
||||
Write the spec as one Turtle block. Then call validate_spec. Fix what it says,
|
||||
up to three rounds. Then call run_report. After run_report returns, stop.
|
||||
You are in refinement mode. You propose a spec and explain in Dutch what it
|
||||
will list and filter. Use lookup_values to find concrete values (names, codes)
|
||||
for a filter, so your proposal names real values instead of guesses. That is
|
||||
the only database read you have, and it only suggests values: it never tells
|
||||
you whether a spec is good or how much it matches. The database you read may
|
||||
differ from the one the report runs on, so never treat a lookup as a check.
|
||||
Iterate with the user on the spec itself: paths, filters, columns. When the
|
||||
user wants an earlier spec changed, open it with read_spec and change what
|
||||
they asked for. Never call
|
||||
run_report; there is no such tool here. When the spec is ready, ask the user
|
||||
to confirm execution. Then stop.
|
||||
|
||||
The spec itself does not go in your text. validate_spec stores a spec that
|
||||
checks out as an attachment ("bijlage", a spec file) on your answer. So in
|
||||
your answer: explain in Dutch in full what the report will list and which
|
||||
filters it applies, tell the user the actual spec is in the bijlage and to
|
||||
look there, and ask the user to confirm execution. Do not print the Turtle
|
||||
yourself; the bijlage shows the validated version, so never rewrite it in
|
||||
the text either.
|
||||
|
||||
Always prefix the spec with @prefix lines. Use the prefixes the profile
|
||||
declares. Write the spec as one Turtle block. Then call validate_spec. Fix
|
||||
what it says, up to three rounds. Then propose and stop.
|
||||
|
||||
Reply to the user in Dutch. Earlier turns are context only. Answer the last one.`;
|
||||
|
||||
// ask(questionOrMessages, profiles, session) → { text, trace }
|
||||
// ask(questionOrMessages, profiles, session) → { text }
|
||||
// A string is one question; an array is the conversation so far,
|
||||
// [{ role, content }], newest last. The system prompt is always this file's
|
||||
// own; a system message in the input is dropped.
|
||||
export async function ask(questionOrMessages, profiles, session) {
|
||||
const { handlers } = buildTools(profiles, session);
|
||||
const trace = [];
|
||||
|
||||
const turns = typeof questionOrMessages === 'string'
|
||||
? [{ role: 'user', content: questionOrMessages }]
|
||||
: questionOrMessages.filter(m => m.role !== 'system');
|
||||
const messages = [{ role: 'system', content: SYSTEM_PROMPT }, ...turns];
|
||||
|
||||
return runLoop(withAttachmentSpecs(turns), profiles, session);
|
||||
}
|
||||
|
||||
// The history carries the spec as a bijlage. Paste one hint line per spec
|
||||
// under the message text, so the model knows a spec is there and opens it
|
||||
// with read_spec; the chat itself still keeps the Turtle out of the text.
|
||||
function withAttachmentSpecs(turns) {
|
||||
return turns.map(t => {
|
||||
const specs = (t.attachments || []).filter(a => a.mediaType === 'text/turtle' && a.uri);
|
||||
if (!specs.length) return t;
|
||||
return {
|
||||
...t,
|
||||
content: `${t.content}\n\n${specs.map(a =>
|
||||
`Bijlage "${a.name}", a spec file (read it with read_spec): ${a.uri}`).join('\n\n')}`,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
// runLoop drives the agent loop with the refinement prompt and tool set.
|
||||
async function runLoop(turns, profiles, session) {
|
||||
const { handlers } = buildTools(profiles, session);
|
||||
const messages = [{ role: 'system', content: REFINE_PROMPT }, ...turns];
|
||||
|
||||
for (let round = 0; round < MAX_ROUNDS; round++) {
|
||||
const reply = await chat(messages);
|
||||
const reply = await chat(messages, { tools: toolDefs() });
|
||||
messages.push(reply.message);
|
||||
const calls = reply.message.tool_calls || [];
|
||||
if (!calls.length) {
|
||||
return { text: reply.message.content || '', trace };
|
||||
return { text: reply.message.content || '' };
|
||||
}
|
||||
for (const call of calls) {
|
||||
const args = parseArgs(call.function.arguments);
|
||||
trace.push({ name: call.function.name, args });
|
||||
const fn = handlers.get(call.function.name);
|
||||
let resultText;
|
||||
if (!fn) {
|
||||
@@ -70,13 +131,16 @@ export async function ask(questionOrMessages, profiles, session) {
|
||||
}
|
||||
}
|
||||
messages.push({ role: 'tool', tool_call_id: call.id, content: resultText });
|
||||
trace[trace.length - 1].result = resultText;
|
||||
}
|
||||
}
|
||||
return { text: '(loop did not finish)', trace };
|
||||
return { text: '(loop did not finish)' };
|
||||
}
|
||||
|
||||
async function chat(messages) {
|
||||
// chat(messages, { tools, toolChoice }) → { message }
|
||||
// The one place that talks to the provider. tools null (the default) means a
|
||||
// plain completion (the mode classifier); runLoop passes the tool
|
||||
// definitions; toolChoice defaults to 'auto'.
|
||||
export async function chat(messages, { tools = null, toolChoice = 'auto' } = {}) {
|
||||
// Send only the fields every provider accepts back.
|
||||
const wire = messages.map(m => {
|
||||
const out = { role: m.role, content: m.content ?? null };
|
||||
@@ -93,12 +157,16 @@ async function chat(messages) {
|
||||
const body = {
|
||||
model: LLM_MODEL,
|
||||
messages: wire,
|
||||
tools: toolDefs(),
|
||||
tool_choice: 'auto',
|
||||
};
|
||||
if (tools) {
|
||||
body.tools = tools;
|
||||
body.tool_choice = toolChoice === 'auto' ? 'auto' : toolChoice;
|
||||
}
|
||||
if (LOG_LLM) {
|
||||
console.log(`[llm] POST ${LLM_BASE_URL}/chat/completions request:\n${
|
||||
JSON.stringify(body, null, 2)}`);
|
||||
} else {
|
||||
console.log('[llm] sending instructions to llm');
|
||||
}
|
||||
const res = await fetch(`${LLM_BASE_URL}/chat/completions`, {
|
||||
method: 'POST',
|
||||
|
||||
81
src/llm/mode.js
Normal file
81
src/llm/mode.js
Normal file
@@ -0,0 +1,81 @@
|
||||
// The mode classifier. One LLM call per turn: a guestimate of whether the
|
||||
// user wants to keep refining the query or has given a clear command to
|
||||
// execute it now. Returns true (execute) / false (refine). Deterministic
|
||||
// code branches on the answer; refine is the safe default when the call
|
||||
// fails or the answer does not parse.
|
||||
|
||||
import { chat } from './loop.js';
|
||||
|
||||
const MODE_PROMPT = `You read a conversation between a user and a report assistant.
|
||||
The assistant proposes report specs and asks the user to confirm before
|
||||
running them.
|
||||
|
||||
Decide whether the user's LAST message is a clear instruction to run the
|
||||
report now (for example "ja", "voer uit", "doe het", "maak het rapport",
|
||||
"go", "execute", "run it") or is still refining, asking, correcting or
|
||||
answering a question.
|
||||
|
||||
Rules:
|
||||
- Only return true when the user clearly tells the assistant to execute the
|
||||
agreed report right now. A short "ja" or "ok" after a proposal counts.
|
||||
- A new or amended question, a clarification, a "no", a question about the
|
||||
proposal, or the absence of an earlier proposal means false.
|
||||
- When in doubt, return false.
|
||||
|
||||
Reply with exactly one token: true or false. No other words, no punctuation,
|
||||
no markdown.`;
|
||||
|
||||
// classifyMode(turns) → boolean (true = execute now)
|
||||
// turns: [{ role, content }], newest last. A leading system message is
|
||||
// dropped; the assistant's own system prompt is never sent to the model.
|
||||
export async function classifyMode(turns) {
|
||||
const convo = (turns || [])
|
||||
.filter(m => m.role !== 'system')
|
||||
.map(m => ({ role: m.role === 'assistant' ? 'assistant' : 'user', content: m.content ?? '' }));
|
||||
if (!convo.length) return false;
|
||||
|
||||
// Keep the last few turns; the decision is about the latest message.
|
||||
const recent = convo.slice(-6);
|
||||
const messages = [{ role: 'system', content: MODE_PROMPT }, ...recent];
|
||||
let raw;
|
||||
try {
|
||||
const reply = await chat(messages, { tools: null });
|
||||
raw = firstText(reply.message);
|
||||
} catch (e) {
|
||||
// The classifier call failed (no LLM, provider down). Default to refine:
|
||||
// safer than running a report on a guess, and the refine loop will
|
||||
// surface the real error to the user.
|
||||
return false;
|
||||
}
|
||||
const text = stripMarkdown(raw).trim().toLowerCase();
|
||||
if (!text) return false;
|
||||
if (text.startsWith('true')) return true;
|
||||
if (text.startsWith('false')) return false;
|
||||
// A JSON {"execute": ...} shape some models emit.
|
||||
try {
|
||||
const m = text.match(/\{[\s\S]*\}/);
|
||||
if (m) {
|
||||
const v = JSON.parse(m[0]);
|
||||
if (typeof v === 'boolean') return v;
|
||||
if (v && typeof v.execute === 'boolean') return v.execute;
|
||||
if (v && typeof v.refine === 'boolean') return !v.refine;
|
||||
}
|
||||
} catch { /* ignore */ }
|
||||
return false;
|
||||
}
|
||||
|
||||
// Reasoning models may leave `content` empty and put the visible text in a
|
||||
// separate field (reasoning_content / thinking / reasoning), or may not
|
||||
// answer at all when the thinking burned the token budget. Read the visible
|
||||
// fields in order and take the first non-empty one.
|
||||
function firstText(message) {
|
||||
for (const key of ['content', 'reasoning_content', 'reasoning', 'thinking']) {
|
||||
const v = message?.[key];
|
||||
if (typeof v === 'string' && v.trim()) return v;
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function stripMarkdown(s) {
|
||||
return String(s).replace(/[*_`>#]/g, ' ');
|
||||
}
|
||||
@@ -1,39 +1,57 @@
|
||||
// The JSON schemas of the four tools. The descriptions are what the LLM
|
||||
// reads to decide which tool to call.
|
||||
// The JSON schemas of the tools. The descriptions are what the LLM reads to
|
||||
// decide which tool to call. Refinement only: list_profiles,
|
||||
// describe_profile, validate_spec, read_spec and lookup_values.
|
||||
|
||||
const COMMON = [
|
||||
{
|
||||
name: 'list_profiles',
|
||||
description: 'List the report profiles available, with their id and title.',
|
||||
inputSchema: { type: 'object', properties: {}, required: [] },
|
||||
},
|
||||
{
|
||||
name: 'describe_profile',
|
||||
description: 'Describe one profile: its entities, fields, relations and labels. Inline the values of short code lists.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { profile_id: { type: 'string', description: 'The profile URI returned by list_profiles' } },
|
||||
required: ['profile_id'],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'validate_spec',
|
||||
description: 'Validate a report spec (Turtle). Returns "ok", followed by one line per rep:where saying where it applies, or one clear sentence saying what is wrong and what to write instead. No database is touched.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { spec: { type: 'string', description: 'The spec as Turtle text' } },
|
||||
required: ['spec'],
|
||||
},
|
||||
},
|
||||
];
|
||||
|
||||
const LOOKUP_VALUES = {
|
||||
name: 'lookup_values',
|
||||
description: 'Search a code list for a term so you can suggest concrete values for a filter (for sh:hasValue or sh:in). Returns up to 25 matches with their URIs and labels, a total, and an exact flag when a label equals the term. Reads candidate values only; it never runs the spec and never tells you how many subjects a spec would match. Read-only, no report is created.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: {
|
||||
profile_id: { type: 'string', description: 'The profile URI returned by list_profiles' },
|
||||
field: { type: 'string', description: 'The field as "entityLabel.fieldLabel", e.g. "bestuurseenheid.naam"' },
|
||||
term: { type: 'string', description: 'The Dutch word or name to search for' },
|
||||
},
|
||||
required: ['profile_id', 'field', 'term'],
|
||||
},
|
||||
};
|
||||
|
||||
const READ_SPEC = {
|
||||
name: 'read_spec',
|
||||
description: 'Return the Turtle of a spec bijlage, a ttl file attached to an earlier message in this conversation. Use it to reuse the agreed spec, or to read back an earlier proposal before you change it. Read-only.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { file_uri: { type: 'string', description: 'The file URI of the spec bijlage, e.g. http://data.lblod.info/files/<uuid>' } },
|
||||
required: ['file_uri'],
|
||||
},
|
||||
};
|
||||
|
||||
export function toolSchemas() {
|
||||
return [
|
||||
{
|
||||
name: 'list_profiles',
|
||||
description: 'List the report profiles available, with their id and title.',
|
||||
inputSchema: { type: 'object', properties: {}, required: [] },
|
||||
},
|
||||
{
|
||||
name: 'describe_profile',
|
||||
description: 'Describe one profile: its entities, fields, relations and labels. Inline the values of short code lists.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { profile_id: { type: 'string', description: 'The profile URI returned by list_profiles' } },
|
||||
required: ['profile_id'],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'validate_spec',
|
||||
description: 'Validate a report spec (Turtle). Returns "ok" or one clear sentence saying what is wrong and what to write instead. No database is touched.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { spec: { type: 'string', description: 'The spec as Turtle text' } },
|
||||
required: ['spec'],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'run_report',
|
||||
description: 'Run a valid spec and return the report URI, the row count and the file URI. Runs as the caller, so it sees only what they may see. Waits as long as the run keeps answering; gives up after RUN_TIMEOUT seconds of silence. The report is named by the dct:title in the spec. Call this once per question, after validate_spec returns ok.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { spec: { type: 'string' } },
|
||||
required: ['spec'],
|
||||
},
|
||||
},
|
||||
];
|
||||
return [...COMMON, READ_SPEC, LOOKUP_VALUES];
|
||||
}
|
||||
125
src/llm/tools.js
125
src/llm/tools.js
@@ -1,23 +1,22 @@
|
||||
// The tools the LLM calls: list_profiles, describe_profile, validate_spec,
|
||||
// run_report. They return a report URI, a count, or an error we wrote —
|
||||
// never a cell value. The shape every tool returns is
|
||||
// { content: [{ type: 'text', text }] }, which the loop passes straight back
|
||||
// to the model.
|
||||
// The tools the LLM calls, refinement only: list_profiles, describe_profile,
|
||||
// validate_spec, lookup_values, read_spec. lookup_values searches a code list
|
||||
// under the service scope (public graph) and returns candidate values, so the
|
||||
// LLM can suggest concrete filter values. No report is created, no spec is
|
||||
// evaluated.
|
||||
//
|
||||
// The shape every tool returns is { content: [{ type: 'text', text }] },
|
||||
// which the loop passes straight back to the model.
|
||||
|
||||
import { checkSpec } from '../runner/check.js';
|
||||
import { checkSpec, whereNotes } from '../runner/check.js';
|
||||
import { parseSpec } from '../runner/spec.js';
|
||||
import { describeProfile, codeListValues } from './describe.js';
|
||||
import { run, slug } from '../runner/run.js';
|
||||
import { lookupValues } from './lookup.js';
|
||||
import { scopedQuery } from '../db.js';
|
||||
import { readFileSync } from 'fs';
|
||||
|
||||
const MAX_PATH_DEPTH = Number(process.env.MAX_PATH_DEPTH || 8);
|
||||
const RUN_TIMEOUT = Number(process.env.RUN_TIMEOUT || 60) * 1000;
|
||||
|
||||
// buildTools(profiles, session) → { handlers }
|
||||
// run_report waits while the run keeps answering: every query that comes
|
||||
// back (a seed page, a column batch) restarts the RUN_TIMEOUT clock, and
|
||||
// only that much silence gives up. The batch count is known upfront — the
|
||||
// subject list is complete after the seed — so a slow run with many small
|
||||
// batches finishes, however long it takes in total.
|
||||
export function buildTools(profiles, session) {
|
||||
const profileList = () => [...profiles.values()].map(p => ({ id: p.uri, title: p.title }));
|
||||
|
||||
@@ -26,7 +25,7 @@ export function buildTools(profiles, session) {
|
||||
async function describe_profile({ profile_id }) {
|
||||
const p = findProfile(profile_id);
|
||||
if (!p) return text(`no profile <${profile_id}>. Available: ${profileList().map(x => x.title).join(', ')}.`);
|
||||
const inlined = await codeListValues(p, profiles);
|
||||
const inlined = await codeListValues(p, scopedQuery);
|
||||
return text(describeProfile(p, inlined));
|
||||
}
|
||||
|
||||
@@ -37,77 +36,59 @@ export function buildTools(profiles, session) {
|
||||
const p = findProfile(parsed.profileUri);
|
||||
if (!p) return text(`no profile <${parsed.profileUri}>. Available: ${profileList().map(x => x.title).join(', ')}.`);
|
||||
const errors = checkSpec(parsed, p, MAX_PATH_DEPTH, profiles);
|
||||
return text(errors.length ? errors[0] : 'ok');
|
||||
if (errors.length) return text(errors[0]);
|
||||
// The spec that checks out becomes the attached proposal.
|
||||
session.onSpecValidated?.(spec);
|
||||
return text(['ok', ...whereNotes(parsed, p)].join('\n'));
|
||||
}
|
||||
|
||||
async function list_profiles() {
|
||||
return text(profileList().map(x => `${x.title} <${x.id}>`).join('\n'));
|
||||
}
|
||||
|
||||
async function run_report({ spec }) {
|
||||
let parsed;
|
||||
try { parsed = parseSpec(spec); }
|
||||
catch (e) { return text(`the spec did not parse: ${e.message}`); }
|
||||
const p = findProfile(parsed.profileUri);
|
||||
if (!p) return text(`no profile <${parsed.profileUri}>.`);
|
||||
const errors = checkSpec(parsed, p, MAX_PATH_DEPTH, profiles);
|
||||
if (errors.length) return text(`the spec is not valid: ${errors[0]}`);
|
||||
async function lookup_values({ profile_id, field, term }) {
|
||||
const p = findProfile(profile_id);
|
||||
if (!p) return text(`no profile <${profile_id}>. Available: ${profileList().map(x => x.title).join(', ')}.`);
|
||||
if (!term) return text('no term to search for.');
|
||||
try {
|
||||
// The spec's dct:title names the report.
|
||||
const name = parsed.title || 'report';
|
||||
await session.onReportStart?.({ title: name, fileName: `${slug(name)}.csv` });
|
||||
const waiter = newRunWaiter(RUN_TIMEOUT);
|
||||
const running = run(session.query, session.update, parsed, p, name, { spec }, waiter.tick);
|
||||
// The end callback also fires after the wait gave up: the run is not
|
||||
// cancelled, the file arrives when it arrives. The .catch keeps a late
|
||||
// failure from crashing the process.
|
||||
running
|
||||
.then((r) => session.onReportEnd?.(null, r), (e) => session.onReportEnd?.(e))
|
||||
.catch((e) => console.error('[reports] end callback failed:', e));
|
||||
const result = await waiter.finish(running);
|
||||
return text(`done. report <${result.reportUri}>, ${result.rowCount} rows, file <${result.fileUri}>.`);
|
||||
const out = await lookupValues(p, term, field, scopedQuery);
|
||||
if (out.error) return text(out.error);
|
||||
if (!out.matches.length) return text(`no value matches "${term}" in ${field}.`);
|
||||
const lines = out.matches.map(m => `${m.label} <${m.uri}>`);
|
||||
if (out.total === '25+') lines.push('(25+ matches, narrow the term)');
|
||||
if (out.exact) lines.push(`exact: <${out.exact}>`);
|
||||
return text(lines.join('\n'));
|
||||
} catch (e) {
|
||||
return text(`the report failed: ${String(e.message || e).split('\n')[0]}`);
|
||||
return text(`the lookup failed: ${String(e.message || e).split('\n')[0]}`);
|
||||
}
|
||||
}
|
||||
|
||||
const handlers = new Map([
|
||||
['list_profiles', list_profiles],
|
||||
['describe_profile', describe_profile],
|
||||
['validate_spec', validate_spec],
|
||||
['run_report', run_report],
|
||||
]);
|
||||
// A spec bijlage next to a message. The model opens it with read_spec:
|
||||
// only files this service made are readable, hte spec files under share.
|
||||
async function read_spec({ file_uri }) {
|
||||
const fileName = String(file_uri || '').split('/').pop();
|
||||
if (!fileName || !/^specificatie-[0-9a-f]{8}-[0-9a-f-]{27}\.ttl$/.test(fileName)) {
|
||||
return text('no such spec file.');
|
||||
}
|
||||
const shareDir = process.env.SHARE_DIR || '/share';
|
||||
try {
|
||||
return text(readFileSync(`${shareDir}/${fileName}`, 'utf8'));
|
||||
} catch (e) {
|
||||
return text(`the spec failed to load: ${String(e.message || e).split('\n')[0]}`);
|
||||
}
|
||||
}
|
||||
|
||||
const handlers = new Map(
|
||||
[
|
||||
['list_profiles', list_profiles],
|
||||
['describe_profile', describe_profile],
|
||||
['validate_spec', validate_spec],
|
||||
['read_spec', read_spec],
|
||||
['lookup_values', lookup_values],
|
||||
],
|
||||
);
|
||||
|
||||
return { handlers };
|
||||
}
|
||||
|
||||
function text(s) { return { content: [{ type: 'text', text: s }] }; }
|
||||
|
||||
// Waits for a report run while it makes progress. tick restarts the silence
|
||||
// clock; idleMs without a single tick rejects. The run is never cancelled:
|
||||
// finish's handlers only pass its own settlement on, so a run that answers
|
||||
// after the wait gave up still delivers through the end callback.
|
||||
function newRunWaiter(idleMs) {
|
||||
let deferred;
|
||||
const promise = new Promise((resolve, reject) => { deferred = { resolve, reject }; });
|
||||
let settled = false;
|
||||
let timer = setTimeout(fail, idleMs);
|
||||
function fail() {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
deferred.reject(new Error(`no query answered for ${idleMs / 1000}s`));
|
||||
}
|
||||
return {
|
||||
tick() {
|
||||
if (settled) return;
|
||||
clearTimeout(timer);
|
||||
timer = setTimeout(fail, idleMs);
|
||||
},
|
||||
finish(running) {
|
||||
running.then(
|
||||
(r) => { settled = true; clearTimeout(timer); deferred.resolve(r); },
|
||||
(e) => { settled = true; clearTimeout(timer); deferred.reject(e); });
|
||||
return promise;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -1,48 +1,129 @@
|
||||
// What this service answers a chat turn with: the LLM loop with the
|
||||
// conversation so far, and a report run becomes an interim message with a
|
||||
// pending file.
|
||||
// What this service answers a chat turn with. The service is synchronous
|
||||
// per turn: when the answer is written, every bijlage already exists.
|
||||
//
|
||||
// Execute turns skip the LLM loop. The mode classifier says the user wants
|
||||
// to run the report now; the agreed spec already sits in the conversation
|
||||
// as a bijlage, so it is read from disk and fed straight to the runner.
|
||||
// The answer is hard-coded: no extra prompt runs for it. When the run cannot
|
||||
// happen or fails, the answer is a hard-coded failure message: nothing is
|
||||
// retried and no spec is rewritten.
|
||||
import { classifyMode } from './llm/mode.js';
|
||||
import { ask } from './llm/loop.js';
|
||||
import { parseSpec } from './runner/spec.js';
|
||||
import { checkSpec } from './runner/check.js';
|
||||
import { run, slug } from './runner/run.js';
|
||||
import { readFileSync } from 'fs';
|
||||
import { storeSpecFile, registerFileObject } from './chat/attachment-file.js';
|
||||
import { SPEC_MEDIA_TYPE, ATTACHMENT_TYPES } from './chat/vocab.js';
|
||||
|
||||
const TEXT = {
|
||||
making: 'Ik maak het rapport. Het verschijnt hier zodra het klaar is.',
|
||||
failed: 'Het rapport is mislukt. Stel de vraag opnieuw, of anders geformuleerd.',
|
||||
executed: 'Het rapport is uitgevoerd. De bijlagen staan erbij.',
|
||||
noSpec: 'Er is nog geen voorstel om uit te voeren. Beschrijf eerst welk rapport je wilt.',
|
||||
invalid: 'Het voorstel is niet meer geldig. Vraag een nieuw voorstel.',
|
||||
failed: 'Het rapport kon niet uitgevoerd worden. Probeer het later opnieuw.',
|
||||
};
|
||||
|
||||
const MAX_PATH_DEPTH = Number(process.env.MAX_PATH_DEPTH || 8);
|
||||
const SHARE_DIR = process.env.SHARE_DIR || '/share';
|
||||
|
||||
// The chat mounts before the profiles are in; whenProfiles waits for the load.
|
||||
export function reportAssistant(whenProfiles) {
|
||||
return async function answer(turn) {
|
||||
// One pending file per turn: a second run_report overwrites it before the
|
||||
// first end callback fires.
|
||||
let pendingDocument = null;
|
||||
|
||||
const session = {
|
||||
query: turn.query,
|
||||
update: turn.update,
|
||||
async onReportStart({ fileName }) {
|
||||
const said = await turn.say(TEXT.making, [{ name: fileName, mediaType: 'text/csv' }]);
|
||||
pendingDocument = said.documents[0].uri;
|
||||
},
|
||||
async onReportEnd(error, result) {
|
||||
const documentUri = pendingDocument;
|
||||
pendingDocument = null;
|
||||
if (!documentUri) return;
|
||||
if (error) {
|
||||
await turn.drop(documentUri);
|
||||
if (turn.finished) await turn.say(TEXT.failed);
|
||||
return;
|
||||
}
|
||||
await turn.setUrl(documentUri, downloadHref(result.fileUri));
|
||||
},
|
||||
};
|
||||
|
||||
const turns = [...turn.history, { role: 'user', content: turn.content }];
|
||||
const profiles = await whenProfiles();
|
||||
const { text } = await ask([...turn.history, { role: 'user', content: turn.content }], profiles, session);
|
||||
return text;
|
||||
|
||||
if (await classifyMode(turns)) {
|
||||
try {
|
||||
return await runStoredSpec(turn, profiles);
|
||||
} catch (e) {
|
||||
console.error('[reports] execution failed:', e);
|
||||
return TEXT.failed;
|
||||
}
|
||||
}
|
||||
return llmAnswer(turn, turns, profiles);
|
||||
};
|
||||
}
|
||||
|
||||
// The file service serves /files/<uuid>/download; the logical file URI ends
|
||||
// in that uuid.
|
||||
function downloadHref(fileUri) {
|
||||
return `/files/${fileUri.split('/').pop()}/download`;
|
||||
// Execution: the last spec bijlage in the conversation is the agreed spec.
|
||||
// No prompt runs; the runner gets it straight away.
|
||||
async function runStoredSpec(turn, profiles) {
|
||||
const attachment = lastSpecAttachment(turn.history);
|
||||
const content = attachment && readSpecFile(attachment.name);
|
||||
if (!content) return TEXT.noSpec;
|
||||
|
||||
const parsed = parseSpec(content);
|
||||
const profile = profiles.get(parsed.profileUri);
|
||||
if (!profile) {
|
||||
console.error(`[reports] no profile <${parsed.profileUri}>`);
|
||||
return TEXT.invalid;
|
||||
}
|
||||
const errors = checkSpec(parsed, profile, MAX_PATH_DEPTH, profiles);
|
||||
if (errors.length) {
|
||||
console.error(`[reports] the agreed spec is not valid: ${errors[0]}`);
|
||||
return TEXT.invalid;
|
||||
}
|
||||
|
||||
// The conversation's title (named by the first question) names the file.
|
||||
// The report itself keeps the dct:title of the spec.
|
||||
const { fileUri } = await run(turn.query, turn.update, parsed, profile,
|
||||
parsed.title, {
|
||||
spec: content,
|
||||
fileType: ATTACHMENT_TYPES.result,
|
||||
fileName: `${slug(turn.conversation.title)}.csv`,
|
||||
});
|
||||
|
||||
// Same physical spec file, registered as a bijlage for this message.
|
||||
const specFile = await registerFileObject(turn.update, {
|
||||
fileName: attachment.name,
|
||||
format: SPEC_MEDIA_TYPE,
|
||||
size: Buffer.byteLength(content),
|
||||
type: ATTACHMENT_TYPES.spec,
|
||||
});
|
||||
|
||||
return {
|
||||
text: TEXT.executed,
|
||||
attachments: [
|
||||
{ uri: fileUri },
|
||||
{ ...specFile, mediaType: SPEC_MEDIA_TYPE },
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
function lastSpecAttachment(history) {
|
||||
for (const message of [...(history || [])].reverse()) {
|
||||
const specs = (message.attachments || []).filter(a =>
|
||||
a.mediaType === SPEC_MEDIA_TYPE
|
||||
&& /^specificatie-[0-9a-f-]+\.ttl$/.test(a.name || ''));
|
||||
if (specs.length) return specs[specs.length - 1];
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function readSpecFile(fileName) {
|
||||
try {
|
||||
return readFileSync(`${SHARE_DIR}/${fileName}`, 'utf8');
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// The LLM loop, refinement only. The spec that checked out goes on the
|
||||
// answer as a bijlage. The prompt never carries the Turtle.
|
||||
async function llmAnswer(turn, turns, profiles) {
|
||||
// The last spec that checked out; stored as a file for this turn's
|
||||
// final message.
|
||||
let spec = null;
|
||||
|
||||
const session = {
|
||||
onSpecValidated(validated) { spec = validated; },
|
||||
};
|
||||
|
||||
const { text } = await ask(turns, profiles, session);
|
||||
|
||||
const attachments = [];
|
||||
if (spec) {
|
||||
const stored = await storeSpecFile(turn.update, spec);
|
||||
attachments.push({ ...stored, mediaType: SPEC_MEDIA_TYPE });
|
||||
}
|
||||
return { text, attachments };
|
||||
}
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
// Values to rows. Subjects in stable order, dedup on term, collect per cell.
|
||||
// rep:self columns take the subject URI; a column with no values is an
|
||||
// empty cell, never a dropped row. rep:row expands one subject into several
|
||||
// lines; at most one column per spec may do that (checked in check.js).
|
||||
// empty cell, never a dropped row. A column without rep:collect joins its
|
||||
// values only when there is one; with several, each value gets its own row
|
||||
// (the other columns repeat theirs). rep:row forces that expansion; at most
|
||||
// one column per spec may do that explicitly (checked in check.js).
|
||||
|
||||
const SH = 'http://www.w3.org/ns/shacl#';
|
||||
const REP = 'http://mu.semte.ch/vocabularies/reporting/';
|
||||
@@ -26,7 +28,8 @@ export function assemble(subjects, values, spec) {
|
||||
});
|
||||
const lines = Math.max(1, ...cells.map(c => c.length));
|
||||
for (let r = 0; r < lines; r++) {
|
||||
// every cell except the expanding one keeps its value on each line
|
||||
// the expanding cell cycles its values, every other cell keeps its
|
||||
// value on each line
|
||||
rows.push(cells.map(c => c[Math.min(r, c.length - 1)]));
|
||||
}
|
||||
}
|
||||
@@ -36,9 +39,13 @@ export function assemble(subjects, values, spec) {
|
||||
function collect(terms, col) {
|
||||
switch (col.collect) {
|
||||
case SH + 'groupConcat':
|
||||
case null:
|
||||
case undefined:
|
||||
return [terms.map(t => t.value).join(col.separator ?? ',')];
|
||||
case undefined:
|
||||
case null:
|
||||
// no rep:collect: one value is a normal cell, several values each
|
||||
// get their own row instead of a joined string
|
||||
if (terms.length <= 1) return [terms.map(t => t.value).join(',')];
|
||||
return terms.map(t => t.value);
|
||||
case SH + 'min':
|
||||
return [extreme(terms, col, -1)];
|
||||
case SH + 'max':
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { fieldsOf } from './profile.js';
|
||||
import { NUMERIC_DATATYPES } from './assemble.js';
|
||||
import { sharedSteps } from './spec.js';
|
||||
|
||||
const SH = 'http://www.w3.org/ns/shacl#';
|
||||
const REP = 'http://mu.semte.ch/vocabularies/reporting/';
|
||||
@@ -57,6 +58,22 @@ function shapeList(profile) {
|
||||
return profile.shapes.map(s => s.label || s.targetClass).join(', ');
|
||||
}
|
||||
|
||||
// Every condition in the spec: the filters, and the rep:where conditions on
|
||||
// filters and columns, with the words that name each one in a message. A
|
||||
// rep:where also carries its host, the filter or column it hangs on.
|
||||
function conditions(spec) {
|
||||
const out = spec.filters.map((f, i) => ({ what: `filter ${i + 1}`, cond: f }));
|
||||
spec.filters.forEach((f, i) => f.where.forEach((w, j) => out.push({
|
||||
what: `rep:where ${j + 1} on filter ${i + 1}`, cond: w,
|
||||
host: { what: `filter ${i + 1}`, path: f.path },
|
||||
})));
|
||||
spec.columns.forEach(c => c.where.forEach((w, j) => out.push({
|
||||
what: `rep:where ${j + 1} on column "${c.label}"`, cond: w,
|
||||
host: { what: `column "${c.label}"`, path: c.path },
|
||||
})));
|
||||
return out;
|
||||
}
|
||||
|
||||
// The shape a spec starts from, after startEntity has approved it: rep:entity
|
||||
// wins, otherwise the one shape carrying the targetClass.
|
||||
function startShapeOf(spec, profile) {
|
||||
@@ -98,7 +115,7 @@ function pathsResolve(spec, profile, maxPathDepth) {
|
||||
const startShapeUri = startShapeOf(spec, profile);
|
||||
for (const col of spec.columns) {
|
||||
const err = walk(profile, startShapeUri, col.path, maxPathDepth,
|
||||
`column "${col.label}"`, true);
|
||||
`column "${col.label}"`, true, col.nodeKind === SH + 'IRI');
|
||||
if (err) return err;
|
||||
}
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
@@ -126,7 +143,7 @@ function collectMinMaxTyped(spec, profile) {
|
||||
: NUMERIC_DATATYPES.has(f.datatype) ? 'number'
|
||||
: null;
|
||||
if (!kind) {
|
||||
return `"${f.name || last.predicate}" is text, so ${name} does not apply. Use rep:row, or leave it out to join the values.`;
|
||||
return `"${f.name || last.predicate}" is text, so ${name} does not apply. Use rep:row (same as leaving rep:collect out: every value gets its own row).`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
@@ -147,20 +164,23 @@ function columnPure(spec) {
|
||||
for (const col of spec.columns) {
|
||||
if (Object.keys(col.constraints || {}).length) {
|
||||
const which = Object.keys(col.constraints)[0];
|
||||
return `a column carries ${which}. Constraints belong in sh:property.`;
|
||||
return `a column carries ${which}. Constraints belong in sh:property, or in rep:where for a condition on a step of this column.`;
|
||||
}
|
||||
if (col.nodeKind && col.nodeKind !== SH + 'IRI') {
|
||||
return `column "${col.label}" has sh:nodeKind <${col.nodeKind}>. A column takes only sh:nodeKind sh:IRI, to show the URI of the node its path ends on.`;
|
||||
}
|
||||
if (col.collect && ![SH + 'groupConcat', SH + 'min', SH + 'max', REP + 'row'].includes(col.collect)) {
|
||||
return `column "${col.label}" has an unknown rep:collect <${col.collect}>. Use sh:groupConcat (default), rep:row, sh:min or sh:max.`;
|
||||
return `column "${col.label}" has an unknown rep:collect <${col.collect}>. Use sh:groupConcat (joins the values), rep:row or nothing (each value gets its own row), sh:min or sh:max.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// No label inside a filter
|
||||
// No label inside a filter or rep:where
|
||||
function filterLabel(spec) {
|
||||
for (const filter of spec.filters) {
|
||||
if ((filter.constraints || {}).label) {
|
||||
return 'a filter carries rdfs:label. Move it to rep:columns to show it.';
|
||||
for (const { what, cond } of conditions(spec)) {
|
||||
if ((cond.constraints || {}).label) {
|
||||
return `${what} carries rdfs:label. Move it to rep:columns to show it.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
@@ -171,11 +191,11 @@ function filterLabel(spec) {
|
||||
function filterConstraint(spec) {
|
||||
const MEANINGFUL = ['minCount', 'maxCount', 'hasValue', 'in', 'minInclusive',
|
||||
'maxInclusive', 'minExclusive', 'maxExclusive', 'pattern', 'flags', 'anyOf'];
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
const c = filter.constraints || {};
|
||||
for (const { what, cond } of conditions(spec)) {
|
||||
const c = cond.constraints || {};
|
||||
if (!MEANINGFUL.some(k => c[k] !== undefined && c[k] !== null)) {
|
||||
const p = filter.path.map(h => h.predicate).join(' → ');
|
||||
return `filter ${i + 1} (path ${p}) has no condition. Add one: sh:hasValue for an exact value, sh:in for a list, rep:anyOf for words, or remove the filter.`;
|
||||
const p = cond.path.map(h => h.predicate).join(' → ');
|
||||
return `${what} (path ${p}) has no condition. Add one: sh:hasValue for an exact value, sh:in for a list, rep:anyOf for words, or remove the filter.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
@@ -183,8 +203,8 @@ function filterConstraint(spec) {
|
||||
|
||||
// sh:maxCount above 0 needs counting
|
||||
function maxCountZero(spec) {
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
const c = filter.constraints;
|
||||
for (const { cond } of conditions(spec)) {
|
||||
const c = cond.constraints;
|
||||
if (c.maxCount !== null && c.maxCount !== undefined && c.maxCount > 0) {
|
||||
return `sh:maxCount ${c.maxCount} needs counting, which this service cannot do. Use sh:maxCount 0 for "has none", or drop it.`;
|
||||
}
|
||||
@@ -197,8 +217,8 @@ function maxCountZero(spec) {
|
||||
|
||||
// rep:anyOf size limits
|
||||
function anyOfLimits(spec) {
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
const c = filter.constraints;
|
||||
for (const { cond } of conditions(spec)) {
|
||||
const c = cond.constraints;
|
||||
if (!c.anyOf) continue;
|
||||
if (c.anyOf.length > ANYOF_MAX_TERMS) {
|
||||
return `rep:anyOf has ${c.anyOf.length} terms. The limit is ${ANYOF_MAX_TERMS}. Narrow the words, or split the filter into two sh:property blocks.`;
|
||||
@@ -217,8 +237,8 @@ function anyOfLimits(spec) {
|
||||
|
||||
// sh:in must not mix URIs and literals
|
||||
function inHomogeneous(spec) {
|
||||
for (const filter of spec.filters) {
|
||||
const c = filter.constraints;
|
||||
for (const { cond } of conditions(spec)) {
|
||||
const c = cond.constraints;
|
||||
if (!c.in) continue;
|
||||
const uris = c.in.filter(t => t.type === 'uri').length;
|
||||
if (uris > 0 && uris < c.in.length) {
|
||||
@@ -231,8 +251,8 @@ function inHomogeneous(spec) {
|
||||
// Typed literals must parse
|
||||
function literalsParse(spec) {
|
||||
const constraintTerms = [];
|
||||
for (const filter of spec.filters) {
|
||||
const c = filter.constraints;
|
||||
for (const { cond } of conditions(spec)) {
|
||||
const c = cond.constraints;
|
||||
for (const key of ['hasValue', 'minInclusive', 'maxInclusive', 'minExclusive', 'maxExclusive']) {
|
||||
if (c[key]) constraintTerms.push(c[key]);
|
||||
}
|
||||
@@ -250,6 +270,81 @@ function literalsParse(spec) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// rep:where: walks from the row through the profile, shares at least one
|
||||
// step with its filter or column, and holds no rep:where of its own.
|
||||
function whereConditions(spec, profile, maxPathDepth) {
|
||||
const startShapeUri = startShapeOf(spec, profile);
|
||||
for (const { what, cond, host } of conditions(spec)) {
|
||||
if (!host) continue;
|
||||
if (cond.where.length) {
|
||||
return `${what} holds a rep:where of its own. Put every rep:where directly on the filter or column.`;
|
||||
}
|
||||
const err = walk(profile, startShapeUri, cond.path, maxPathDepth, what, false);
|
||||
if (err) return err;
|
||||
const k = sharedSteps(host.path, cond.path);
|
||||
if (!k) {
|
||||
return `${what} shares no step with ${host.what}. Start its path with the same steps as ${host.what}; a condition on the row itself goes in sh:property.`;
|
||||
}
|
||||
const c = cond.constraints;
|
||||
if (k === cond.path.length && (c.maxCount === 0 || c.minCount === 2)) {
|
||||
return `${what} ends on a step of ${host.what}, so ${c.maxCount === 0 ? 'sh:maxCount 0' : 'sh:minCount 2'} has nothing to count. Add the step that must be missing or repeated.`;
|
||||
}
|
||||
}
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
if (filter.where.length && filter.constraints.minCount === 2) {
|
||||
return `filter ${i + 1} combines sh:minCount 2 with rep:where. Use one of the two.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// A date or number condition compares values. On a link (an entity or a
|
||||
// code) it compares a URI and never matches, so its path must end on a field
|
||||
// with a datatype.
|
||||
function rangeOnValue(spec, profile) {
|
||||
const startShapeUri = startShapeOf(spec, profile);
|
||||
for (const { what, cond } of conditions(spec)) {
|
||||
const key = ['minInclusive', 'maxInclusive', 'minExclusive', 'maxExclusive']
|
||||
.find(k => cond.constraints[k]);
|
||||
if (!key || !cond.path.length) continue;
|
||||
const last = cond.path[cond.path.length - 1];
|
||||
const f = fieldsOf(profile, shapeAfter(profile, startShapeUri, cond.path))
|
||||
.find(x => x.path === last.predicate && !!x.inverse === !!last.inverse);
|
||||
if (!f || f.datatype) continue; // pathsResolve reports a missing step
|
||||
const target = f.node
|
||||
? `"${profile.shapes.find(s => s.uri === f.node)?.label || f.node}"`
|
||||
: `the code "${f.name}"`;
|
||||
const steps = fieldsOf(profile, f.node)
|
||||
.filter(x => x.datatype === XSD_DATE || x.datatype === XSD_DATETIME || NUMERIC_DATATYPES.has(x.datatype))
|
||||
.map(x => `${x.name} (${x.path})`);
|
||||
const add = steps.length ? `Add a step: ${steps.join(' or ')}.` : 'End the path on a date or number field.';
|
||||
return `${what} ends on ${target}, not on a value, so sh:${key} never matches. ${add}`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Where each rep:where applies, in the profile's words, so the proposal can
|
||||
// say it. Profile only, no database.
|
||||
export function whereNotes(spec, profile) {
|
||||
const startShapeUri = startShapeOf(spec, profile);
|
||||
return conditions(spec).filter(x => x.host).map(({ what, cond, host }) => {
|
||||
const k = sharedSteps(host.path, cond.path);
|
||||
return `${what} applies at ${nodeName(profile, startShapeUri, host.path.slice(0, k))}, after step ${k} of ${host.what}.`;
|
||||
});
|
||||
}
|
||||
|
||||
// The entity a path lands on, or the value field when it ends on one.
|
||||
function nodeName(profile, startShapeUri, hops) {
|
||||
let shapeUri = startShapeUri;
|
||||
for (const [i, hop] of hops.entries()) {
|
||||
const f = fieldsOf(profile, shapeUri).find(x => x.path === hop.predicate && !!x.inverse === !!hop.inverse);
|
||||
if (!f) return `step ${hops.length}`;
|
||||
if (!f.node) return i === hops.length - 1 ? `the value "${f.name}"` : `step ${hops.length}`;
|
||||
shapeUri = f.node;
|
||||
}
|
||||
return `"${profile.shapes.find(s => s.uri === shapeUri)?.label || shapeUri}"`;
|
||||
}
|
||||
|
||||
function parsesAsDate(value, withTime) {
|
||||
const d = new Date(value);
|
||||
if (Number.isNaN(d.getTime())) return false;
|
||||
@@ -270,6 +365,8 @@ const CHECKS = [
|
||||
['typed literal does not parse', literalsParse],
|
||||
['two rep:row columns', oneRowColumn],
|
||||
['path does not resolve', pathsResolve],
|
||||
['rep:where does not fit', whereConditions],
|
||||
['date or number condition on a link', rangeOnValue],
|
||||
['sh:min/sh:max on a non-number', collectMinMaxTyped],
|
||||
];
|
||||
|
||||
@@ -297,7 +394,7 @@ function shapeAfter(profile, startShapeUri, hops) {
|
||||
return shapeUri;
|
||||
}
|
||||
|
||||
function walk(profile, startShapeUri, hops, maxPathDepth, what, isColumn) {
|
||||
function walk(profile, startShapeUri, hops, maxPathDepth, what, isColumn, wantsIri = false) {
|
||||
if (hops.length > maxPathDepth) {
|
||||
return `${what} walks ${hops.length} hops. The limit is ${maxPathDepth}.`;
|
||||
}
|
||||
@@ -308,7 +405,7 @@ function walk(profile, startShapeUri, hops, maxPathDepth, what, isColumn) {
|
||||
const fields = fieldsOf(profile, shapeUri);
|
||||
const f = fields.find(x => x.path === hop.predicate && !!x.inverse === !!hop.inverse);
|
||||
if (!f) {
|
||||
const have = fields.map(x => (x.inverse ? `geen ${x.name} (${x.path}, omgekeerd)` : `${x.name} (${x.path})`)).join(', ');
|
||||
const have = fields.map(x => (x.inverse ? `${x.name} (${x.path}, inverse)` : `${x.name} (${x.path})`)).join(', ');
|
||||
const shapeLabel = profile.shapes.find(s => s.uri === shapeUri)?.label || shapeUri;
|
||||
return `at "${shapeLabel}" there is no "${hop.predicate}". It has: ${have || '(nothing)'}.`;
|
||||
}
|
||||
@@ -318,11 +415,14 @@ function walk(profile, startShapeUri, hops, maxPathDepth, what, isColumn) {
|
||||
}
|
||||
shapeUri = f.node;
|
||||
} else if (isColumn) {
|
||||
if (!f.datatype && !f.class) {
|
||||
if (wantsIri && f.datatype) {
|
||||
return `column "${colLabel(what)}" has sh:nodeKind sh:IRI but ends on the value "${f.name}", which is no URI. Drop sh:nodeKind.`;
|
||||
}
|
||||
if (!wantsIri && !f.datatype && !f.class) {
|
||||
const valueFields = fieldsOf(profile, f.node)
|
||||
.filter(x => x.datatype || x.class)
|
||||
.map(x => `${x.name} (${x.path})`);
|
||||
return `column "${colLabel(what)}" ends on a link. Add a hop: ${valueFields.join(' or ')}.`;
|
||||
return `column "${colLabel(what)}" ends on a link. Add a hop: ${valueFields.join(' or ')}. Only if the user asked for the URI of that node itself, add sh:nodeKind sh:IRI to the column instead.`;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import { sparqlEscapeUri } from '../db.js';
|
||||
import { wherePatterns } from './seed.js';
|
||||
|
||||
// Fetch column values for known subjects. Columns are grouped by their
|
||||
// shared non-leaf hops; two columns share a query only when every hop
|
||||
@@ -7,20 +8,13 @@ import { sparqlEscapeUri } from '../db.js';
|
||||
|
||||
const SUBJECT_CHUNK_SIZE = Math.max(1, Number(process.env.SUBJECT_CHUNK_SIZE) || 100);
|
||||
|
||||
export async function fetchColumns(sessionQuery, subjects, spec, values = new Map(), onProgress = null) {
|
||||
export async function fetchColumns(sessionQuery, subjects, spec, values = new Map()) {
|
||||
const groups = groupColumns(spec.columns);
|
||||
// The subject list is complete and the chunk size is fixed, so the number
|
||||
// of batch queries is known before the first one flies. onProgress fires
|
||||
// after every answered batch with { done, total }.
|
||||
const total = groups.length * Math.ceil(subjects.length / SUBJECT_CHUNK_SIZE);
|
||||
let done = 0;
|
||||
for (const group of groups) {
|
||||
for (let i = 0; i < subjects.length; i += SUBJECT_CHUNK_SIZE) {
|
||||
const chunk = subjects.slice(i, i + SUBJECT_CHUNK_SIZE);
|
||||
const result = await sessionQuery(groupQuery(group, chunk));
|
||||
collect(result, group, values);
|
||||
done += 1;
|
||||
onProgress?.({ done, total });
|
||||
}
|
||||
}
|
||||
return values;
|
||||
@@ -28,14 +22,17 @@ export async function fetchColumns(sessionQuery, subjects, spec, values = new Ma
|
||||
|
||||
// Group columns by their shared prefix (every hop before the leaf) and the
|
||||
// direction of the leaf. A column with an empty path (rep:self) joins no group.
|
||||
// A column with rep:where only shares a group with the same conditions.
|
||||
export function groupColumns(columns) {
|
||||
const groups = new Map();
|
||||
columns.forEach((col, index) => {
|
||||
if (!col.path.length) return; // rep:self
|
||||
const leaf = col.path[col.path.length - 1];
|
||||
const key = col.path.map(h => `${h.inverse ? '^' : ''}${h.predicate}`).join(' > ');
|
||||
const where = col.where || [];
|
||||
const key = col.path.map(h => `${h.inverse ? '^' : ''}${h.predicate}`).join(' > ')
|
||||
+ (where.length ? ` | ${JSON.stringify(where)}` : '');
|
||||
if (!groups.has(key)) {
|
||||
groups.set(key, { hops: col.path, columns: [] });
|
||||
groups.set(key, { hops: col.path, where, columns: [] });
|
||||
}
|
||||
groups.get(key).columns.push({ index, leaf: leaf.predicate });
|
||||
});
|
||||
@@ -56,6 +53,10 @@ export function groupQuery(group, chunk) {
|
||||
const leaf = group.hops[group.hops.length - 1];
|
||||
if (leaf.inverse) lines.push(`?v ${sparqlEscapeUri(leaf.predicate)} ${prev} .`);
|
||||
else lines.push(`${prev} ${sparqlEscapeUri(leaf.predicate)} ?v .`);
|
||||
// rep:where hangs on the column's own nodes: ?x<k-1> after k hops, ?v at
|
||||
// the end.
|
||||
const nodeVar = (k) => (k === group.hops.length ? '?v' : `?x${k - 1}`);
|
||||
lines.push(...wherePatterns(group.where, group.hops, nodeVar, 'c'));
|
||||
return `SELECT ?s ?v WHERE {\n VALUES ?s { ${subjects} }\n ${lines.join('\n ')}\n}`;
|
||||
}
|
||||
|
||||
|
||||
@@ -7,8 +7,8 @@ const REPORT_CLASS = process.env.REPORT_CLASS || 'http://lblod.data.gift/vocabul
|
||||
|
||||
// Register the CSV: a logical file (http://data.lblod.info/files/<uuid>) plus
|
||||
// a physical share:// file pointing back with nie:dataSource. The logical
|
||||
// URI is what you link to.
|
||||
export async function registerFile(sessionUpdateFn, fileName, filePath) {
|
||||
// URI is what you link to. When given, the file gets dct:type.
|
||||
export async function registerFile(sessionUpdateFn, fileName, filePath, { type } = {}) {
|
||||
const size = statSync(filePath).size;
|
||||
const uuid = randomUUID();
|
||||
const uuid2 = randomUUID();
|
||||
@@ -23,14 +23,16 @@ export async function registerFile(sessionUpdateFn, fileName, filePath) {
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName')} ${sparqlEscapeString(fileName)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/format')} ${sparqlEscapeString('text/csv')} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize')} ${sparqlEscapeInt(size)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/created')} ${sparqlEscapeDateTime(now)} .
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/created')} ${sparqlEscapeDateTime(now)}${type ? ` ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/type')} ${sparqlEscapeUri(type)}` : ''} .
|
||||
${sparqlEscapeUri(physicalUri)} a ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject')} ;
|
||||
${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(uuid2)} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName')} ${sparqlEscapeString(fileName)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/format')} ${sparqlEscapeString('text/csv')} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize')} ${sparqlEscapeInt(size)} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileCreated')} ${sparqlEscapeDateTime(now)} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource')} ${sparqlEscapeUri(logicalUri)} .
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource')} ${sparqlEscapeUri(logicalUri)}${type ? ` ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/type')} ${sparqlEscapeUri(type)}` : ''} .
|
||||
}`);
|
||||
|
||||
return logicalUri;
|
||||
|
||||
@@ -6,26 +6,22 @@ import { registerFile, registerReport } from './report.js';
|
||||
import { startShape } from './profile.js';
|
||||
|
||||
// Ties the steps together: seed → columns → assemble → CSV → register.
|
||||
// Returns { reportUri, fileUri, rowCount }. onProgress fires on every
|
||||
// answered query with what the phase knows — { subjects } per seed page,
|
||||
// { done, total } per column batch — so a waiter can tell a live run from
|
||||
// a stalled one.
|
||||
// Returns { reportUri, fileUri, rowCount }.
|
||||
|
||||
export async function run(query, update, parsed, profile, title, extra = {}, onProgress = null) {
|
||||
export async function run(query, update, parsed, profile, title, extra = {}) {
|
||||
const shape = startShape(profile, parsed);
|
||||
const subjects = await seed(query, parsed, shape,
|
||||
onProgress ? (n) => onProgress({ subjects: n }) : null);
|
||||
const subjects = await seed(query, parsed, shape);
|
||||
|
||||
const values = new Map();
|
||||
await fetchColumns(query, subjects, parsed, values, null, onProgress);
|
||||
await fetchColumns(query, subjects, parsed, values);
|
||||
|
||||
const rows = assemble(subjects, values, parsed);
|
||||
const fileName = `${slug(title)}.csv`;
|
||||
const fileName = extra.fileName || `${slug(title)}.csv`;
|
||||
const csv = rowsToCsv(rows);
|
||||
const filePath = writeCsv(fileName, csv);
|
||||
const specFile = extra.spec && writeSpec(fileName, extra.spec);
|
||||
|
||||
const fileUri = await registerFile(update, fileName, filePath);
|
||||
const fileUri = await registerFile(update, fileName, filePath, { type: extra.fileType });
|
||||
const reportUri = await registerReport(update, title, fileUri,
|
||||
specFile ? { ...extra, specFile } : extra);
|
||||
|
||||
|
||||
@@ -1,12 +1,11 @@
|
||||
import { sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime, sparqlEscapeDate } from '../db.js';
|
||||
import { sharedSteps } from './spec.js';
|
||||
|
||||
// Subject selection. One paged query per page; stops at ROW_LIMIT.
|
||||
// onPage(n) fires after every answered page with the subject count so far,
|
||||
// so a caller waiting on this run can tell a live one from a stalled one.
|
||||
const ROW_LIMIT = Number(process.env.ROW_LIMIT || 200000);
|
||||
const PAGE_SIZE = Number(process.env.SEED_PAGE_SIZE || 5000);
|
||||
|
||||
export async function seed(sessionQuery, spec, shape = null, onPage = null) {
|
||||
export async function seed(sessionQuery, spec, shape = null) {
|
||||
const subjects = [];
|
||||
let offset = 0;
|
||||
for (;;) {
|
||||
@@ -14,7 +13,6 @@ export async function seed(sessionQuery, spec, shape = null, onPage = null) {
|
||||
const result = await sessionQuery(query);
|
||||
const rows = result.results.bindings.map(b => b.s.value);
|
||||
subjects.push(...rows);
|
||||
onPage?.(subjects.length);
|
||||
if (subjects.length > ROW_LIMIT) {
|
||||
throw new Error(`the question matches more than ${ROW_LIMIT} subjects. Add a filter, for example a date range or a bestuurseenheid.`);
|
||||
}
|
||||
@@ -30,8 +28,7 @@ export function seedPageQuery(spec, limit, offset, shape = null) {
|
||||
parts.push(`?s a ${sparqlEscapeUri(targetClass)} .`);
|
||||
if (shape?.discriminator) parts.push(discriminatorPattern(shape.discriminator));
|
||||
spec.filters.forEach((filter, fi) => {
|
||||
parts.push(hopsPattern(filter.path, `f${fi}`, '?s'));
|
||||
parts.push(constraintsPattern(filter.constraints, `f${fi}`));
|
||||
parts.push(filterPattern(filter, `f${fi}`));
|
||||
});
|
||||
return `SELECT DISTINCT ?s WHERE {
|
||||
${parts.join('\n ')}
|
||||
@@ -69,9 +66,37 @@ export function hopsPattern(hops, varPrefix, startVar) {
|
||||
return patterns.join('\n ');
|
||||
}
|
||||
|
||||
export function constraintsPattern(c, varPrefix) {
|
||||
const parts = [];
|
||||
const v = `?${varPrefix}_v`;
|
||||
// One filter: its condition, plus its rep:where conditions on the nodes it
|
||||
// walks through. With sh:maxCount 0 the whole of it becomes "none such".
|
||||
function filterPattern(filter, varPrefix) {
|
||||
const nodeVar = (k) => (k === filter.path.length ? `?${varPrefix}_v` : `?${varPrefix}_${k}`);
|
||||
const wheres = wherePatterns(filter.where, filter.path, nodeVar, varPrefix);
|
||||
if (filter.constraints.maxCount === 0) {
|
||||
return `FILTER NOT EXISTS {\n ${[hopsPattern(filter.path, varPrefix, '?s'), ...wheres].join('\n ')}\n }`;
|
||||
}
|
||||
return [conditionPattern(filter.constraints, varPrefix, filter.path, '?s'), ...wheres].join('\n ');
|
||||
}
|
||||
|
||||
// The rep:where conditions of a filter or column. Each starts at the node
|
||||
// after the steps it shares with its host; nodeVar(k) names that node in the
|
||||
// host's own query.
|
||||
export function wherePatterns(where, hostHops, nodeVar, varPrefix) {
|
||||
return (where || []).map((w, wi) => {
|
||||
const k = sharedSteps(hostHops, w.path);
|
||||
return conditionPattern(w.constraints, `${varPrefix}_w${wi}`, w.path.slice(k), nodeVar(k));
|
||||
});
|
||||
}
|
||||
|
||||
// A condition: the hops from startVar and the test on where they end. A
|
||||
// filter starts at ?s; a rep:where starts at a node of its host and, with no
|
||||
// hops left, tests that node itself. sh:maxCount 0 means "none such", so its
|
||||
// hops only live inside the NOT EXISTS.
|
||||
export function conditionPattern(c, varPrefix, hops, startVar) {
|
||||
if (c.maxCount === 0) {
|
||||
return `FILTER NOT EXISTS { ${hopsPattern(hops, varPrefix + '_nex', startVar)} }`;
|
||||
}
|
||||
const v = hops.length ? `?${varPrefix}_v` : startVar;
|
||||
const parts = hops.length ? [hopsPattern(hops, varPrefix, startVar)] : [];
|
||||
|
||||
if (c.hasValue) parts.push(`FILTER(${v} = ${escapeTerm(c.hasValue)})`);
|
||||
if (c.in) parts.push(`FILTER(${v} IN (${c.in.map(escapeTerm).join(', ')}))`);
|
||||
@@ -91,19 +116,15 @@ export function constraintsPattern(c, varPrefix) {
|
||||
if (c.minCount === 2) {
|
||||
// two distinct leaf values on the same path
|
||||
const w = `?${varPrefix}_w`;
|
||||
parts.push(hopsPattern2(c, varPrefix, w));
|
||||
parts.push(hopsPattern2(hops, varPrefix, startVar, w));
|
||||
parts.push(`FILTER(${v} != ${w})`);
|
||||
}
|
||||
if (c.maxCount === 0) {
|
||||
parts.push(`FILTER NOT EXISTS { ${hopsPattern(c.path, varPrefix + '_nex', '?s')} }`);
|
||||
}
|
||||
return parts.join('\n ');
|
||||
}
|
||||
|
||||
function hopsPattern2(filter, varPrefix, wVar) {
|
||||
function hopsPattern2(hops, varPrefix, startVar, wVar) {
|
||||
// repeat the path to a second leaf variable
|
||||
const hops = filter.path.map(h => ({ ...h }));
|
||||
let prev = '?s';
|
||||
let prev = startVar;
|
||||
const patterns = [];
|
||||
for (let i = 0; i < hops.length; i++) {
|
||||
const hop = hops[i];
|
||||
|
||||
@@ -39,9 +39,27 @@ function parseFilter(store, p, { withLabel = false } = {}) {
|
||||
return {
|
||||
path,
|
||||
constraints: readConstraints(store, p, { withLabel }),
|
||||
where: readWhere(store, p),
|
||||
};
|
||||
}
|
||||
|
||||
// rep:where: conditions on the nodes a filter or column walks through. Each
|
||||
// is written like a filter, from the row; check.js refuses one nested in
|
||||
// another.
|
||||
function readWhere(store, node) {
|
||||
return store.getQuads(node, REP + 'where', null).map(q =>
|
||||
parseFilter(store, q.object, { withLabel: true }));
|
||||
}
|
||||
|
||||
// The number of steps two paths have in common from the row. A rep:where
|
||||
// shares those nodes with its filter or column.
|
||||
export function sharedSteps(a, b) {
|
||||
let k = 0;
|
||||
while (k < a.length && k < b.length
|
||||
&& a[k].predicate === b[k].predicate && !!a[k].inverse === !!b[k].inverse) k++;
|
||||
return k;
|
||||
}
|
||||
|
||||
function parseColumnList(store, listNode) {
|
||||
const columns = [];
|
||||
let cur = listNode;
|
||||
@@ -69,7 +87,10 @@ function parseColumn(store, col) {
|
||||
label: store.getQuads(col, RDFS_LABEL, null)[0]?.object.value || null,
|
||||
collect: store.getQuads(col, REP + 'collect', null)[0]?.object.value || null,
|
||||
separator: store.getQuads(col, SH + 'separator', null)[0]?.object.value || null,
|
||||
// sh:nodeKind sh:IRI: the column shows the URI of the node it ends on
|
||||
nodeKind: store.getQuads(col, SH + 'nodeKind', null)[0]?.object.value || null,
|
||||
constraints: readConstraints(store, col), // kept so check.js can refuse it
|
||||
where: readWhere(store, col),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user