Cleanup

Dockerfile

@@ -1,2 +1,2 @@
-FROM semtech/mu-javascript-template:1.8.0
-LABEL maintainer="info@redpencil"
+FROM semtech/mu-javascript-template:1.9.1
+LABEL maintainer="info@redpencil"

README.md

@@ -7,6 +7,31 @@ writes a CSV. The LLM writes no SPARQL and sees no report data.
 Ad-hoc reports only. The scheduled reports stay in
 `loket-report-generation-service`.
 
+## Two modes: refine, then execute
+
+Every turn is classified first. One small LLM call (`src/llm/mode.js`)
+returns true/false: does the user's last message clearly say "run it now" or
+not. The answer picks one of two deterministic code paths.
+
+**Refinement (Modus A).** The LLM proposes a spec and explains in Dutch what
+it will list and filter. It may call `lookup_values` to search code lists for
+candidate values, so its filter suggestions name real values instead of
+guesses. It never runs a report and never learns what a spec would match: the
+database it reads may differ from the one the report runs on, so a lookup is
+a suggestion, not a check. The user must confirm before anything executes.
+As long as the user has not given a clear command, the turn stays in this
+mode.
+
+**Execution (Modus B).** The user has confirmed. No LLM runs. The last spec
+bijlage in the conversation is the agreed spec: the service reads it from the
+share, checks it against its profile and runs it as the caller. The answer is
+a fixed Dutch text with the CSV and the spec as bijlagen. No spec yet, a spec
+that no longer checks out, or a failed run each give a fixed Dutch failure
+message. Nothing is retried and no spec is rewritten.
+
+Only refinement has tools: `list_profiles`, `describe_profile`,
+`validate_spec`, `read_spec` and `lookup_values`.
+
 ## How it works
 
 Two documents drive everything:
@@ -19,15 +44,22 @@ Two documents drive everything:
 A **profile** is a SHACL shapes graph: one `sh:NodeShape` per entity, fields
 as `sh:property`, code lists via `sh:class`, links via `sh:node`. The service
 ships no profile and knows no vocabulary; profiles live in the app repo.
+Profiles load at boot; a broken profile stops the service.
 
 A **spec** is `rep:ReportSpec` + `sh:NodeShape`: `sh:property` selects the
 subjects, `rep:columns` says what the CSV holds. The two never interact. The
 namespace is `http://mu.semte.ch/vocabularies/reporting/` (`rep:`).
 
+A filter tests only the end of its own path. To test a node in the middle of
+a path ("the mandate whose role is Burgemeester"), a filter or column carries
+`rep:where`: a condition written like a filter, from the row, that shares the
+nodes of every step it has in common with its host. On a column it drops
+values from the cell, never rows. The app's `report-spec-readme.md` has the
+details and an example.
+
 The run: check → seed (`SELECT DISTINCT ?s`, paged) → columns (per group, per
 chunk of subjects) → assemble (dedup + `rep:collect`) → CSV → register file
-+ report resource. One run per question; the loop stops after `run_report`
-returns.
++ report resource. One run per confirmed spec.
 
 ## One identity: mu-authorization's
 
@@ -41,40 +73,58 @@ the caller's.
 The tools the LLM calls return URIs, counts, statuses and column names —
 never a cell value.
 
-## Two endpoints
+### The rule: no report data reaches the LLM
+
+No report row, row count or run error ever reaches the LLM. A run starts
+after the turn's only LLM call (the mode check) and answers with fixed text.
+What the LLM does see: the conversation, the profiles, validator messages,
+spec files, and public reference data read under the service scope (inlined
+code lists and `lookup_values`, see below).
+
+Keep it that way. Do not add a tool that previews, samples or counts what a
+spec would match, not even to catch a `rep:where` that hangs on the wrong
+node. That check stays on the profile: `validate_spec` says where each
+`rep:where` applies, and the proposal repeats it to the user.
+
+### The service scope
+
+The LLM's own reads during refinement (`describe_profile`'s code lists and
+`lookup_values`) run under a service scope, not the caller's session. mu's
+`query(q, { scope })` sends `mu-auth-scope`; the sparql-parser config grants
+that scope read access to `http://mu.semte.ch/graphs/public` only. So
+refinement reads only public code lists, regardless of who calls. Execution
+still runs as the caller, so the report contains what they may see, and the
+LLM sees none of it.
+
+The scope URI is `SERVICE_SCOPE` (default
+`http://services.semantic.works/natural-language-report`) and must match the
+`with-scope` grant in the app's sparql-parser `config.lisp`. Do not set
+`DEFAULT_MU_AUTH_SCOPE` on the service: that would scope every query
+(including the run and the chat store) and break them.
+
+## The endpoint
 
 ```
-POST /ask                                     { question, history? } → { answer, toolCalls }
 POST /assistant/conversations/:id/turns      { content }           → 202 { id }
 ```
 
-`/ask` is synchronous: a Dutch question in, the agent loop runs inside the
-service (profiles, lookups, spec, validate, repair, run), and the answer in
-Dutch comes back with a trace of the tool calls. The CSV lands in
-`data/files/`. `history` carries the earlier turns for follow-ups; the
-conversation lives on the caller's side, the service stores nothing. One
-question makes one report; the loop stops after `run_report`.
-
-`/assistant` is the chat. The turn module in
-`src/chat/` is generic — it imports nothing from `src/llm/` or `src/runner/`
-and moves between services without edits — and `src/report-assistant.js` is
-this service's one `answer` hook. It records the question, answers `202` with
-the message id, then runs the same loop with the conversation's history. A
-report run becomes an interim message with a pending file (`as:Document` with
-no `as:url`); the URL is set when the run ends. The browser polls
+This is the chat. The route is in `app.js`: it records the question, answers
+`202` with the message id, then classifies the turn and refines or executes
+(`src/report-assistant.js`) with the conversation's history. The
+conversation, message and bijlage reads and writes are in `src/chat.js`. The
+answer and its bijlagen are written when the turn ends. The browser polls
 `/chat-conversations/:id?include=messages.attachments` and finds the answer
 that way. Failure is a message, never a silent loader.
 
-`/ask` needs `LLM_BASE_URL` set, and answers 503 without it. A full-org report
-run takes minutes.
+The chat needs `LLM_BASE_URL`; without it every turn answers with a failure
+message. A full-org report run takes minutes.
 
 ### The internal tools
 
-The loop drives four tool handlers (`src/llm/tools.js`): `list_profiles`,
-`describe_profile`, `validate_spec`, `run_report`. They are machinery, not an
-external surface — the tool trace in the answer is the only thing a caller
-sees of them. All LLM-facing text (system prompt, profile menu, tool
-descriptions, validator errors) is English; the answer the user sees is Dutch.
+The refinement loop drives one tool set (`src/llm/tools.js`). The tools are
+machinery, not an external surface. All LLM-facing text (system prompts,
+profile menu, tool descriptions, validator errors) is English; the answer the
+user sees is Dutch.
 
 ## Environment
 
@@ -86,10 +136,9 @@ descriptions, validator errors) is English; the answer the user sees is Dutch.
 | `SUBJECT_CHUNK_SIZE` | `100` | subjects per column query (VALUES batch size); lower it when the auth layer rejects long queries |
 | `ROW_LIMIT` | `200000` | seed limit; failing the job beats truncating |
 | `MAX_PATH_DEPTH` | `8` | longest allowed spec path |
-| `RUN_TIMEOUT` | `60` | seconds of silence before `run_report` gives up. Every answered query (seed page, column batch) restarts the clock, so a big report only needs the window per batch, not for the whole run. The run itself is never cancelled: the file still lands in the chat. |
 | `INLINE_VALUES_MAX` | `50` | code lists up to this size are inlined in `describe_profile` |
 | `VALUES_TTL` | `3600` | seconds the inlined lists are cached |
-| `LLM_BASE_URL` | — | enables `/ask`; OpenAI-compatible (`https://ollama.com/v1`) |
+| `LLM_BASE_URL` | — | the chat's LLM; OpenAI-compatible (`https://ollama.com/v1`) |
 | `LLM_MODEL` | — | e.g. `gpt-oss:120b` |
 | `LLM_API_KEY` | — | bearer token, optional for local Ollama |
 | `SEED_PAGE_SIZE` | `5000` | seed page size |
@@ -97,6 +146,7 @@ descriptions, validator errors) is English; the answer the user sees is Dutch.
 | `SHARE_DIR` | `/share` | where CSVs are written |
 | `CHAT_ASSISTANT_URI` | — | the `prov:SoftwareAgent` that signs assistant messages; found by lookup when unset |
 | `CHAT_HISTORY_LIMIT` | `20` | messages of history a turn reads |
+| `SERVICE_SCOPE` | `http://services.semantic.works/natural-language-report` | the mu-auth-scope for the LLM's own reads (code lists, `lookup_values`). Must match the `with-scope` grant in the app's sparql-parser config. |
 
 Do not set `ALLOW_MU_AUTH_SUDO`; the template then refuses sudo queries.
 
@@ -107,7 +157,7 @@ Do not set `ALLOW_MU_AUTH_SUDO`; the template then refuses sudo queries.
 ```yaml
 natural-language-report:
   build: ../natural-language-report-service/
-  image: lblod/natural-language-report-service:0.1.0
+  image: lblod/natural-language-report-service:0.2.0
   volumes:
     - ./data/files:/share
     - ./config/report-profiles/:/config/profiles/
@@ -116,12 +166,12 @@ natural-language-report:
 `config/dispatcher/dispatcher.ex`:
 
 ```elixir
-match "/natural-language-reports/*path" do
-  forward conn, path, "http://natural-language-report/"
+match "/assistant/*path" do
+  forward conn, path, "http://natural-language-report/assistant/"
 end
 ```
 
-One route to the service root; `POST /ask` is the endpoint.
+One route: the chat's turn endpoint.
 First deploy: `docker compose build natural-language-report`, then
 `docker compose up -d natural-language-report dispatcher`.
 

app.js

@@ -1,53 +1,66 @@
 import { app, errorHandler } from 'mu';
-import express from 'express';
+import bodyParser from 'body-parser';
 import { loadProfiles } from './src/runner/profile.js';
-import { sessionQuery, sessionUpdate, stripStoreError } from './src/db.js';
-import { ask } from './src/llm/loop.js';
-import { mountChat } from './src/chat/index.js';
-import { reportAssistant } from './src/report-assistant.js';
+import { readConversation, readHistory, findAssistant, writeMessage } from './src/chat.js';
+import { wantsExecution } from './src/llm/mode.js';
+import { executeSpec, refineSpec } from './src/report-assistant.js';
 
 const PROFILE_DIR = process.env.PROFILE_DIR || '/config/profiles';
+const CHAT_ASSISTANT_URI = process.env.CHAT_ASSISTANT_URI;
+const TITLE_LENGTH = 80;
+const FAILED = 'Er ging iets mis. Probeer het opnieuw.';
 
-// Profiles load at boot; handlers await the same promise. A failed load
-// answers an error per request instead of crashing boot.
-let profilesPromise = loadProfiles(PROFILE_DIR);
-profilesPromise.catch(e => console.error('[profiles] loading failed:', e.message));
+// Profiles load at boot. A broken profile stops the service.
+const profiles = loadProfiles(PROFILE_DIR);
 
-async function whenProfiles() {
+// One chat turn. Records the question, answers 202, then writes the
+// assistant's answer; the frontend polls the conversation for it. mu's
+// query/update run as the caller for the whole turn, also after the 202.
+// The template only parses application/vnd.api+json, the chat sends
+// application/json.
+app.post('/assistant/conversations/:id/turns', bodyParser.json(), async function(req, res) {
+  const content = String(req.body?.content ?? '').trim();
+  if (!content) return res.status(400).json({ error: 'content is required' });
+
+  let conversation, assistant, history, question;
   try {
-    return await profilesPromise;
+    // the access check: the caller's session decides what is readable
+    conversation = await readConversation(req.params.id);
+    if (!conversation) return res.status(404).json({ error: 'no such conversation' });
+    assistant = CHAT_ASSISTANT_URI || await findAssistant();
+    history = await readHistory(conversation.uri, assistant);
+    question = await writeMessage({
+      conversationUri: conversation.uri,
+      content,
+      maker: conversation.creator,
+      title: conversation.title ? undefined : content.slice(0, TITLE_LENGTH),
+    });
   } catch (e) {
-    throw new Error('profiles are not available: ' + e.message);
+    console.error('[chat] could not record the question:', e);
+    return res.status(500).json({ error: e.message });
   }
-}
 
-// The chat. reportAssistant is the answer hook: it runs the LLM loop with the
-// conversation so far and turns a report run into an interim message with a
-// pending file.
-mountChat(app, { path: '/assistant', answer: reportAssistant(whenProfiles) });
-
-// POST /ask: one question in, the answer in Dutch and a trace of the tool
-// calls back. Follow-ups carry the earlier turns in `history`.
-// The template parses application/vnd.api+json only; this route parses the
-// plain JSON body itself.
-app.post('/ask', express.json(), async (req, res) => {
-  const { question, history } = req.body || {};
-  if (!question) return res.status(400).json({ error: 'no question in the request body' });
-  if (!process.env.LLM_BASE_URL) return res.status(503).json({ error: 'no LLM configured (set LLM_BASE_URL)' });
+  res.status(202).json({ id: question.id });
 
   try {
-    const profiles = await whenProfiles();
-    const turns = [
-      ...(Array.isArray(history) ? history : []),
-      { role: 'user', content: question },
-    ];
-    const session = { query: sessionQuery, update: sessionUpdate };
-    const { text, trace } = await ask(turns, profiles, session);
-    res.json({ answer: text, toolCalls: trace });
+    const turns = [...history, { role: 'user', content }];
+    const answer = await wantsExecution(turns)
+      ? await executeSpec(history, conversation.title, profiles)
+      : await refineSpec(turns, profiles);
+    await writeMessage({
+      conversationUri: conversation.uri,
+      content: answer.text,
+      maker: assistant,
+      attachments: answer.attachments,
+    });
   } catch (e) {
-    console.error('[ask] failed:', e);
-    res.status(502).json({ error: stripStoreError(e.message) });
+    console.error('[chat] turn failed:', e);
+    try {
+      await writeMessage({ conversationUri: conversation.uri, content: FAILED, maker: assistant });
+    } catch (e2) {
+      console.error('[chat] could not write the failure message:', e2);
+    }
   }
 });
 
-app.use(errorHandler);
+app.use(errorHandler);

package-lock.json

@@ -1,15 +1,14 @@
 {
   "name": "natural-language-report-service",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "natural-language-report-service",
-      "version": "0.1.0",
+      "version": "0.2.0",
       "license": "MIT",
       "dependencies": {
-        "express": "^5.2.1",
         "n3": "^1.17.0"
       }
     },
@@ -25,19 +24,6 @@
         "node": ">=6.5"
       }
     },
-    "node_modules/accepts": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz",
-      "integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==",
-      "license": "MIT",
-      "dependencies": {
-        "mime-types": "^3.0.0",
-        "negotiator": "^1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
     "node_modules/base64-js": {
       "version": "1.5.1",
       "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
@@ -58,43 +44,6 @@
       ],
       "license": "MIT"
     },
-    "node_modules/body-parser": {
-      "version": "2.3.0",
-      "resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.3.0.tgz",
-      "integrity": "sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw==",
-      "license": "MIT",
-      "dependencies": {
-        "bytes": "^3.1.2",
-        "content-type": "^2.0.0",
-        "debug": "^4.4.3",
-        "http-errors": "^2.0.1",
-        "iconv-lite": "^0.7.2",
-        "on-finished": "^2.4.1",
-        "qs": "^6.15.2",
-        "raw-body": "^3.0.2",
-        "type-is": "^2.1.0"
-      },
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/body-parser/node_modules/content-type": {
-      "version": "2.1.0",
-      "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz",
-      "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
     "node_modules/buffer": {
       "version": "6.0.3",
       "resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz",
@@ -119,184 +68,6 @@
         "ieee754": "^1.2.1"
       }
     },
-    "node_modules/bytes": {
-      "version": "3.1.2",
-      "resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
-      "integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/call-bind-apply-helpers": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
-      "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0",
-        "function-bind": "^1.1.2"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/call-bound": {
-      "version": "1.0.4",
-      "resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz",
-      "integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bind-apply-helpers": "^1.0.2",
-        "get-intrinsic": "^1.3.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/content-disposition": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.1.0.tgz",
-      "integrity": "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/content-type": {
-      "version": "1.0.5",
-      "resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz",
-      "integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/cookie": {
-      "version": "0.7.2",
-      "resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz",
-      "integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/cookie-signature": {
-      "version": "1.2.2",
-      "resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz",
-      "integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=6.6.0"
-      }
-    },
-    "node_modules/debug": {
-      "version": "4.4.3",
-      "resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
-      "integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
-      "license": "MIT",
-      "dependencies": {
-        "ms": "^2.1.3"
-      },
-      "engines": {
-        "node": ">=6.0"
-      },
-      "peerDependenciesMeta": {
-        "supports-color": {
-          "optional": true
-        }
-      }
-    },
-    "node_modules/depd": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
-      "integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/dunder-proto": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
-      "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bind-apply-helpers": "^1.0.1",
-        "es-errors": "^1.3.0",
-        "gopd": "^1.2.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/ee-first": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
-      "integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==",
-      "license": "MIT"
-    },
-    "node_modules/encodeurl": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",
-      "integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/es-define-property": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
-      "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/es-errors": {
-      "version": "1.3.0",
-      "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz",
-      "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/es-object-atoms": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.2.tgz",
-      "integrity": "sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/escape-html": {
-      "version": "1.0.3",
-      "resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
-      "integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==",
-      "license": "MIT"
-    },
-    "node_modules/etag": {
-      "version": "1.8.1",
-      "resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
-      "integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
     "node_modules/event-target-shim": {
       "version": "5.0.1",
       "resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz",
@@ -315,206 +86,6 @@
         "node": ">=0.8.x"
       }
     },
-    "node_modules/express": {
-      "version": "5.2.1",
-      "resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz",
-      "integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==",
-      "license": "MIT",
-      "dependencies": {
-        "accepts": "^2.0.0",
-        "body-parser": "^2.2.1",
-        "content-disposition": "^1.0.0",
-        "content-type": "^1.0.5",
-        "cookie": "^0.7.1",
-        "cookie-signature": "^1.2.1",
-        "debug": "^4.4.0",
-        "depd": "^2.0.0",
-        "encodeurl": "^2.0.0",
-        "escape-html": "^1.0.3",
-        "etag": "^1.8.1",
-        "finalhandler": "^2.1.0",
-        "fresh": "^2.0.0",
-        "http-errors": "^2.0.0",
-        "merge-descriptors": "^2.0.0",
-        "mime-types": "^3.0.0",
-        "on-finished": "^2.4.1",
-        "once": "^1.4.0",
-        "parseurl": "^1.3.3",
-        "proxy-addr": "^2.0.7",
-        "qs": "^6.14.0",
-        "range-parser": "^1.2.1",
-        "router": "^2.2.0",
-        "send": "^1.1.0",
-        "serve-static": "^2.2.0",
-        "statuses": "^2.0.1",
-        "type-is": "^2.0.1",
-        "vary": "^1.1.2"
-      },
-      "engines": {
-        "node": ">= 18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/finalhandler": {
-      "version": "2.1.1",
-      "resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz",
-      "integrity": "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==",
-      "license": "MIT",
-      "dependencies": {
-        "debug": "^4.4.0",
-        "encodeurl": "^2.0.0",
-        "escape-html": "^1.0.3",
-        "on-finished": "^2.4.1",
-        "parseurl": "^1.3.3",
-        "statuses": "^2.0.1"
-      },
-      "engines": {
-        "node": ">= 18.0.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/forwarded": {
-      "version": "0.2.0",
-      "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
-      "integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/fresh": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz",
-      "integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/function-bind": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
-      "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
-      "license": "MIT",
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/get-intrinsic": {
-      "version": "1.3.0",
-      "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
-      "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bind-apply-helpers": "^1.0.2",
-        "es-define-property": "^1.0.1",
-        "es-errors": "^1.3.0",
-        "es-object-atoms": "^1.1.1",
-        "function-bind": "^1.1.2",
-        "get-proto": "^1.0.1",
-        "gopd": "^1.2.0",
-        "has-symbols": "^1.1.0",
-        "hasown": "^2.0.2",
-        "math-intrinsics": "^1.1.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/get-proto": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",
-      "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",
-      "license": "MIT",
-      "dependencies": {
-        "dunder-proto": "^1.0.1",
-        "es-object-atoms": "^1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/gopd": {
-      "version": "1.2.0",
-      "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
-      "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/has-symbols": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",
-      "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/hasown": {
-      "version": "2.0.4",
-      "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz",
-      "integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==",
-      "license": "MIT",
-      "dependencies": {
-        "function-bind": "^1.1.2"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/http-errors": {
-      "version": "2.0.1",
-      "resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",
-      "integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==",
-      "license": "MIT",
-      "dependencies": {
-        "depd": "~2.0.0",
-        "inherits": "~2.0.4",
-        "setprototypeof": "~1.2.0",
-        "statuses": "~2.0.2",
-        "toidentifier": "~1.0.1"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/iconv-lite": {
-      "version": "0.7.3",
-      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.3.tgz",
-      "integrity": "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==",
-      "license": "MIT",
-      "dependencies": {
-        "safer-buffer": ">= 2.1.2 < 3.0.0"
-      },
-      "engines": {
-        "node": ">=0.10.0"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
     "node_modules/ieee754": {
       "version": "1.2.1",
       "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
@@ -535,92 +106,6 @@
       ],
       "license": "BSD-3-Clause"
     },
-    "node_modules/inherits": {
-      "version": "2.0.4",
-      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
-      "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
-      "license": "ISC"
-    },
-    "node_modules/ipaddr.js": {
-      "version": "1.9.1",
-      "resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
-      "integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/is-promise": {
-      "version": "4.0.0",
-      "resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz",
-      "integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==",
-      "license": "MIT"
-    },
-    "node_modules/math-intrinsics": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
-      "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      }
-    },
-    "node_modules/media-typer": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.1.tgz",
-      "integrity": "sha512-yz3xRaG20c6/BOzvYoDaGtPmGscs7YivItZEEqe6GbwNfHuxu9YNmvnEkMzKldAGY4/80pRcQRZSEnhquk9XuQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/merge-descriptors": {
-      "version": "2.0.0",
-      "resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz",
-      "integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/sindresorhus"
-      }
-    },
-    "node_modules/mime-db": {
-      "version": "1.54.0",
-      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz",
-      "integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      }
-    },
-    "node_modules/mime-types": {
-      "version": "3.0.2",
-      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz",
-      "integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==",
-      "license": "MIT",
-      "dependencies": {
-        "mime-db": "^1.54.0"
-      },
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/ms": {
-      "version": "2.1.3",
-      "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
-      "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
-      "license": "MIT"
-    },
     "node_modules/n3": {
       "version": "1.26.0",
       "resolved": "https://registry.npmjs.org/n3/-/n3-1.26.0.tgz",
@@ -634,87 +119,6 @@
         "node": ">=12.0"
       }
     },
-    "node_modules/negotiator": {
-      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.1.0.tgz",
-      "integrity": "sha512-NMPBRMJgiQHjbd8phG3Vebdx4kZ1H121rbl5IkMqeOsahptB9BKo/d7oJ3zTXqTgagn2bWlNSXkh0QUGM31RYg==",
-      "license": "MIT",
-      "dependencies": {
-        "content-type": "^2.1.0"
-      },
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/negotiator/node_modules/content-type": {
-      "version": "2.1.0",
-      "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz",
-      "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/object-inspect": {
-      "version": "1.13.4",
-      "resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
-      "integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/on-finished": {
-      "version": "2.4.1",
-      "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",
-      "integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",
-      "license": "MIT",
-      "dependencies": {
-        "ee-first": "1.1.1"
-      },
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/once": {
-      "version": "1.4.0",
-      "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
-      "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
-      "license": "ISC",
-      "dependencies": {
-        "wrappy": "1"
-      }
-    },
-    "node_modules/parseurl": {
-      "version": "1.3.3",
-      "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
-      "integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/path-to-regexp": {
-      "version": "8.4.2",
-      "resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.4.2.tgz",
-      "integrity": "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA==",
-      "license": "MIT",
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
     "node_modules/process": {
       "version": "0.11.10",
       "resolved": "https://registry.npmjs.org/process/-/process-0.11.10.tgz",
@@ -724,63 +128,6 @@
         "node": ">= 0.6.0"
       }
     },
-    "node_modules/proxy-addr": {
-      "version": "2.0.7",
-      "resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
-      "integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==",
-      "license": "MIT",
-      "dependencies": {
-        "forwarded": "0.2.0",
-        "ipaddr.js": "1.9.1"
-      },
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
-    "node_modules/qs": {
-      "version": "6.15.3",
-      "resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz",
-      "integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==",
-      "license": "BSD-3-Clause",
-      "dependencies": {
-        "es-define-property": "^1.0.1",
-        "side-channel": "^1.1.1"
-      },
-      "engines": {
-        "node": ">=0.6"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/range-parser": {
-      "version": "1.3.0",
-      "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.3.0.tgz",
-      "integrity": "sha512-hek2mFQpPuI4E1BBKrSto+BU3e3x4xuarsbiwr3+lf7p44juvFMV0XFWQAP3xUyqXA4RrXLIoaSUGbSt056ZMw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.6"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/raw-body": {
-      "version": "3.0.2",
-      "resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.2.tgz",
-      "integrity": "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==",
-      "license": "MIT",
-      "dependencies": {
-        "bytes": "~3.1.2",
-        "http-errors": "~2.0.1",
-        "iconv-lite": "~0.7.0",
-        "unpipe": "~1.0.0"
-      },
-      "engines": {
-        "node": ">= 0.10"
-      }
-    },
     "node_modules/readable-stream": {
       "version": "4.7.0",
       "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz",
@@ -797,22 +144,6 @@
         "node": "^12.22.0 || ^14.17.0 || >=16.0.0"
       }
     },
-    "node_modules/router": {
-      "version": "2.2.0",
-      "resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz",
-      "integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==",
-      "license": "MIT",
-      "dependencies": {
-        "debug": "^4.4.0",
-        "depd": "^2.0.0",
-        "is-promise": "^4.0.0",
-        "parseurl": "^1.3.3",
-        "path-to-regexp": "^8.0.0"
-      },
-      "engines": {
-        "node": ">= 18"
-      }
-    },
     "node_modules/safe-buffer": {
       "version": "5.2.1",
       "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
@@ -833,144 +164,6 @@
       ],
       "license": "MIT"
     },
-    "node_modules/safer-buffer": {
-      "version": "2.1.2",
-      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
-      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
-      "license": "MIT"
-    },
-    "node_modules/send": {
-      "version": "1.2.1",
-      "resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz",
-      "integrity": "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==",
-      "license": "MIT",
-      "dependencies": {
-        "debug": "^4.4.3",
-        "encodeurl": "^2.0.0",
-        "escape-html": "^1.0.3",
-        "etag": "^1.8.1",
-        "fresh": "^2.0.0",
-        "http-errors": "^2.0.1",
-        "mime-types": "^3.0.2",
-        "ms": "^2.1.3",
-        "on-finished": "^2.4.1",
-        "range-parser": "^1.2.1",
-        "statuses": "^2.0.2"
-      },
-      "engines": {
-        "node": ">= 18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/serve-static": {
-      "version": "2.2.1",
-      "resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz",
-      "integrity": "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==",
-      "license": "MIT",
-      "dependencies": {
-        "encodeurl": "^2.0.0",
-        "escape-html": "^1.0.3",
-        "parseurl": "^1.3.3",
-        "send": "^1.2.0"
-      },
-      "engines": {
-        "node": ">= 18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/setprototypeof": {
-      "version": "1.2.0",
-      "resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",
-      "integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==",
-      "license": "ISC"
-    },
-    "node_modules/side-channel": {
-      "version": "1.1.1",
-      "resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz",
-      "integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0",
-        "object-inspect": "^1.13.4",
-        "side-channel-list": "^1.0.1",
-        "side-channel-map": "^1.0.1",
-        "side-channel-weakmap": "^1.0.2"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/side-channel-list": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",
-      "integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",
-      "license": "MIT",
-      "dependencies": {
-        "es-errors": "^1.3.0",
-        "object-inspect": "^1.13.4"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/side-channel-map": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz",
-      "integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bound": "^1.0.2",
-        "es-errors": "^1.3.0",
-        "get-intrinsic": "^1.2.5",
-        "object-inspect": "^1.13.3"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/side-channel-weakmap": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
-      "integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==",
-      "license": "MIT",
-      "dependencies": {
-        "call-bound": "^1.0.2",
-        "es-errors": "^1.3.0",
-        "get-intrinsic": "^1.2.5",
-        "object-inspect": "^1.13.3",
-        "side-channel-map": "^1.0.1"
-      },
-      "engines": {
-        "node": ">= 0.4"
-      },
-      "funding": {
-        "url": "https://github.com/sponsors/ljharb"
-      }
-    },
-    "node_modules/statuses": {
-      "version": "2.0.2",
-      "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
-      "integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
     "node_modules/string_decoder": {
       "version": "1.3.0",
       "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
@@ -979,70 +172,6 @@
       "dependencies": {
         "safe-buffer": "~5.2.0"
       }
-    },
-    "node_modules/toidentifier": {
-      "version": "1.0.1",
-      "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
-      "integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=0.6"
-      }
-    },
-    "node_modules/type-is": {
-      "version": "2.1.0",
-      "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.1.0.tgz",
-      "integrity": "sha512-faYHw0anBbc/kWF3zFTEnxSFOAGUX9GFbOBthvDdLsIlEoWOFOtS0zgCiQYwIskL9iGXZL3kAXD8OoZ4GmMATA==",
-      "license": "MIT",
-      "dependencies": {
-        "content-type": "^2.0.0",
-        "media-typer": "^1.1.0",
-        "mime-types": "^3.0.0"
-      },
-      "engines": {
-        "node": ">= 18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/type-is/node_modules/content-type": {
-      "version": "2.1.0",
-      "resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz",
-      "integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=18"
-      },
-      "funding": {
-        "type": "opencollective",
-        "url": "https://opencollective.com/express"
-      }
-    },
-    "node_modules/unpipe": {
-      "version": "1.0.0",
-      "resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
-      "integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/vary": {
-      "version": "1.1.2",
-      "resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
-      "integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==",
-      "license": "MIT",
-      "engines": {
-        "node": ">= 0.8"
-      }
-    },
-    "node_modules/wrappy": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
-      "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
-      "license": "ISC"
     }
   }
 }

package.json

@@ -1,6 +1,6 @@
 {
   "name": "natural-language-report-service",
-  "version": "0.1.0",
+  "version": "0.2.0",
   "description": "Turns a SHACL report spec into SPARQL queries, joins the results and writes a CSV. The LLM writes specs; this service runs them.",
   "license": "MIT",
   "type": "module",
@@ -9,7 +9,6 @@
     "start": "node app.js"
   },
   "dependencies": {
-    "express": "^5.2.1",
     "n3": "^1.17.0"
   }
 }

src/chat.js

@@ -0,0 +1,224 @@
+// The chat's reads and writes: conversations, messages and their bijlagen.
+// Full URIs everywhere: the auth layer rejects a prefixed name with no
+// PREFIX line with an opaque 500 (the xsd: types of mu's escape helpers are
+// fine).
+//
+// A bijlage is a real file: a logical nfo:FileDataObject the file service
+// serves at /files/<uuid>/download, over a physical share:// object with
+// nie:dataSource back to the logical one.
+import { readFileSync, writeFileSync, statSync } from 'fs';
+import { query, update, uuid,
+         sparqlEscapeUri, sparqlEscapeString, sparqlEscapeInt, sparqlEscapeDateTime } from 'mu';
+
+const HISTORY_LIMIT = Number(process.env.CHAT_HISTORY_LIMIT || 20);
+
+const T = {
+  type: 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type',
+  uuid: 'http://mu.semte.ch/vocabularies/core/uuid',
+  thread: 'http://rdfs.org/sioc/ns#Thread',
+  post: 'http://rdfs.org/sioc/ns#Post',
+  instantMessage: 'http://rdfs.org/sioc/types#InstantMessage',
+  hasContainer: 'http://rdfs.org/sioc/ns#has_container',
+  content: 'http://rdfs.org/sioc/ns#content',
+  lastActivity: 'http://rdfs.org/sioc/ns#last_activity_date',
+  title: 'http://purl.org/dc/terms/title',
+  created: 'http://purl.org/dc/terms/created',
+  maker: 'http://xmlns.com/foaf/0.1/maker',
+  attachment: 'https://www.w3.org/ns/activitystreams#attachment',
+  softwareAgent: 'http://www.w3.org/ns/prov#SoftwareAgent',
+  file: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject',
+  fileName: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName',
+  fileSize: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize',
+  fileCreated: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileCreated',
+  format: 'http://purl.org/dc/terms/format',
+  fileType: 'http://purl.org/dc/terms/type',
+  dataSource: 'http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource',
+};
+const MESSAGE_BASE = 'http://data.lblod.info/id/chat-messages/';
+const FILE_BASE = 'http://data.lblod.info/files/';
+const SHARE_DIR = process.env.SHARE_DIR || '/share';
+
+export const SPEC_MEDIA_TYPE = 'text/turtle';
+// The bijlage types. Match the add-chat-attachment-types-codelist migration
+// in the portal.
+export const SPEC_TYPE = 'http://lblod.data.gift/concepts/6693014e-7b7d-448c-bb24-80c9758521f3';
+export const RESULT_TYPE = 'http://lblod.data.gift/concepts/28d97698-e219-4a92-bdbd-7ab367a7524e';
+// Every spec file is called like this; readSpecFile opens nothing else.
+const SPEC_FILE_NAME = /^specificatie-[0-9a-f-]{36}\.ttl$/;
+
+const u = sparqlEscapeUri;
+const s = sparqlEscapeString;
+
+const bindings = (json) => json.results.bindings;
+
+// --- reads --------------------------------------------------------------------
+
+// The conversation by uuid, null when the caller may not read it.
+export async function readConversation(id) {
+  const json = await query(`
+    SELECT ?conversation ?title ?creator WHERE {
+      ?conversation ${u(T.type)} ${u(T.thread)} ;
+        ${u(T.uuid)} ${s(id)} .
+      OPTIONAL { ?conversation ${u(T.title)} ?title . }
+      OPTIONAL { ?conversation ${u(T.maker)} ?creator . }
+    } LIMIT 1`);
+  const b = bindings(json)[0];
+  if (!b) return null;
+  return { uri: b.conversation.value, id, title: b.title?.value, creator: b.creator?.value };
+}
+
+// The last HISTORY_LIMIT messages, oldest first, as
+// { role, content, attachments: [{ uri, name, mediaType, type }] }.
+export async function readHistory(conversationUri, assistantUri) {
+  const json = await query(`
+    SELECT ?message ?content ?maker ?created ?doc ?docName ?docFormat ?docType WHERE {
+      ?message ${u(T.hasContainer)} ${u(conversationUri)} ;
+        ${u(T.content)} ?content ;
+        ${u(T.maker)} ?maker ;
+        ${u(T.created)} ?created .
+      OPTIONAL {
+        ?message ${u(T.attachment)} ?doc .
+        OPTIONAL { ?doc ${u(T.fileName)} ?docName . }
+        OPTIONAL { ?doc ${u(T.format)} ?docFormat . }
+        OPTIONAL { ?doc ${u(T.fileType)} ?docType . }
+      }
+    } ORDER BY DESC(?created) LIMIT ${HISTORY_LIMIT * 4}`);
+  const messages = [];
+  const byMessage = new Map();
+  for (const b of bindings(json).reverse()) {
+    const uri = b.message.value;
+    let message = byMessage.get(uri);
+    if (!message) {
+      message = {
+        role: b.maker.value === assistantUri ? 'assistant' : 'user',
+        content: b.content.value,
+        attachments: [],
+      };
+      byMessage.set(uri, message);
+      messages.push(message);
+    }
+    if (b.doc) {
+      message.attachments.push({
+        uri: b.doc.value,
+        name: b.docName?.value,
+        mediaType: b.docFormat?.value,
+        type: b.docType?.value,
+      });
+    }
+  }
+  return messages.slice(-HISTORY_LIMIT);
+}
+
+// The one assistant the caller can read. Throws unless there is exactly one.
+export async function findAssistant() {
+  const json = await query(`
+    SELECT ?agent WHERE { ?agent ${u(T.type)} ${u(T.softwareAgent)} . } LIMIT 2`);
+  const rows = bindings(json);
+  if (rows.length !== 1) {
+    throw new Error(`expected one prov:SoftwareAgent, found ${rows.length}; seed one or set CHAT_ASSISTANT_URI`);
+  }
+  return rows[0].agent.value;
+}
+
+// --- writes -------------------------------------------------------------------
+
+// One message, both types, its bijlagen, and the conversation touched, in
+// one update. `title` is only written when given (the first question names
+// an unnamed conversation). attachments are the logical uris of files that
+// already exist.
+export async function writeMessage({ conversationUri, content, maker, attachments = [], title }) {
+  const id = uuid();
+  const uri = `${MESSAGE_BASE}${id}`;
+  const now = sparqlEscapeDateTime(new Date());
+
+  // Predicate pairs joined with ';', never a trailing one: it breaks the Turtle.
+  const pairs = [
+    `${u(T.type)} ${u(T.post)}`,
+    `${u(T.type)} ${u(T.instantMessage)}`,
+    `${u(T.uuid)} ${s(id)}`,
+    `${u(T.hasContainer)} ${u(conversationUri)}`,
+    `${u(T.content)} ${s(content)}`,
+    `${u(T.created)} ${now}`,
+  ];
+  if (maker) pairs.push(`${u(T.maker)} ${u(maker)}`);
+  for (const fileUri of attachments) {
+    pairs.push(`${u(T.attachment)} ${u(fileUri)}`);
+  }
+
+  const conversationPairs = [`${u(T.lastActivity)} ${now}`];
+  if (title) conversationPairs.push(`${u(T.title)} ${s(title)}`);
+
+  await update(`
+    DELETE {
+      ${u(conversationUri)} ${u(T.lastActivity)} ?old .
+    }
+    INSERT {
+      ${u(uri)} ${pairs.join(' ;\n        ')} .
+      ${u(conversationUri)} ${conversationPairs.join(' ;\n        ')} .
+    }
+    WHERE {
+      OPTIONAL { ${u(conversationUri)} ${u(T.lastActivity)} ?old . }
+    }`);
+
+  return { uri, id };
+}
+
+// --- files --------------------------------------------------------------------
+
+export function writeShareFile(fileName, content) {
+  writeFileSync(`${SHARE_DIR}/${fileName}`, content, 'utf8');
+}
+
+// storeSpecFile(spec) → the file uri of a new spec bijlage. A spec file
+// keeps one name for the logical and the physical file: readSpecFile finds
+// it by the name in the history.
+export async function storeSpecFile(spec) {
+  const fileName = `specificatie-${uuid()}.ttl`;
+  writeShareFile(fileName, spec);
+  return registerFile(fileName, SPEC_MEDIA_TYPE, SPEC_TYPE);
+}
+
+// readSpecFile(fileName) → the Turtle of a spec bijlage, null when there is
+// no such spec file.
+export function readSpecFile(fileName) {
+  if (!SPEC_FILE_NAME.test(fileName || '')) return null;
+  try {
+    return readFileSync(`${SHARE_DIR}/${fileName}`, 'utf8');
+  } catch {
+    return null;
+  }
+}
+
+// registerFile(shareName, format, type, name) → the logical file uri. One
+// logical file called `name` over the physical share file `shareName`, both
+// nfo:FileDataObject, both typed with dct:type. A share name must be
+// unique: two files on one share name overwrite each other.
+export async function registerFile(shareName, format, type, name = shareName) {
+  const id = uuid();
+  const physicalId = uuid();
+  const logical = `${FILE_BASE}${id}`;
+  const physical = `share://${shareName}`;
+  const size = statSync(`${SHARE_DIR}/${shareName}`).size;
+  const now = sparqlEscapeDateTime(new Date());
+
+  await update(`
+    INSERT DATA {
+      ${u(logical)} ${u(T.type)} ${u(T.file)} ;
+        ${u(T.uuid)} ${s(id)} ;
+        ${u(T.fileName)} ${s(name)} ;
+        ${u(T.format)} ${s(format)} ;
+        ${u(T.fileSize)} ${sparqlEscapeInt(size)} ;
+        ${u(T.created)} ${now} ;
+        ${u(T.fileType)} ${u(type)} .
+      ${u(physical)} ${u(T.type)} ${u(T.file)} ;
+        ${u(T.uuid)} ${s(physicalId)} ;
+        ${u(T.fileName)} ${s(shareName)} ;
+        ${u(T.format)} ${s(format)} ;
+        ${u(T.fileSize)} ${sparqlEscapeInt(size)} ;
+        ${u(T.fileCreated)} ${now} ;
+        ${u(T.dataSource)} ${u(logical)} ;
+        ${u(T.fileType)} ${u(type)} .
+    }`);
+
+  return logical;
+}

src/chat/index.js

@@ -1,81 +0,0 @@
-// The turn endpoint. Mount it with mountChat(app, { path, answer });
-// answer(turn) is the only thing the mounting service writes.
-import express from 'express';
-import { query as muQuery, update as muUpdate } from 'mu';
-import {
-  readConversation, readHistory, findAssistant,
-  writeMessage, setDocumentUrl, dropDocument,
-} from './store.js';
-
-const TEXT = {
-  failed: 'Er ging iets mis. Probeer het opnieuw.',
-};
-const TITLE_LENGTH = 80;
-
-export function mountChat(app, { path = '/assistant', answer }) {
-  if (typeof answer !== 'function') throw new Error('mountChat needs an answer(turn) function');
-
-  app.post(`${path}/conversations/:id/turns`, express.json(), async (req, res) => {
-    const content = String(req.body?.content ?? '').trim();
-    if (!content) return res.status(400).json({ error: 'content is required' });
-
-    // 1. the access check: the caller's session decides what is readable
-    let conversation;
-    try {
-      conversation = await readConversation(req.params.id);
-    } catch (e) {
-      console.error('[chat] conversation read failed:', e);
-      return res.status(500).json({ error: e.message });
-    }
-    if (!conversation) return res.status(404).json({ error: 'no such conversation' });
-
-    let assistant, history, userMessage;
-    try {
-      assistant = process.env.CHAT_ASSISTANT_URI || await findAssistant();
-      history = await readHistory(conversation.uri, assistant);
-      userMessage = await writeMessage({
-        conversationUri: conversation.uri,
-        content,
-        maker: conversation.creator,
-        title: conversation.title ? undefined : content.slice(0, TITLE_LENGTH),
-      });
-    } catch (e) {
-      console.error('[chat] could not record the question:', e);
-      return res.status(500).json({ error: e.message });
-    }
-
-    // 2. answer 202 now; the assistant's message comes when it comes. The
-    // continuation still runs as the caller.
-    res.status(202).json({ id: userMessage.id });
-
-    const turn = {
-      conversation: { uri: conversation.uri, id: conversation.id, title: conversation.title },
-      history,
-      content,
-      query: muQuery,
-      update: muUpdate,
-      finished: false,
-      say: (text, attachments = []) => writeMessage({
-        conversationUri: conversation.uri, content: text, maker: assistant, attachments,
-      }),
-      setUrl: (documentUri, url) => setDocumentUrl(documentUri, url, conversation.uri),
-      drop: (documentUri) => dropDocument(documentUri),
-    };
-
-    try {
-      const text = await answer(turn);
-      if (typeof text === 'string' && text.trim()) await turn.say(text.trim());
-    } catch (e) {
-      console.error('[chat] turn failed:', e);
-      try {
-        await turn.say(TEXT.failed);
-      } catch (e2) {
-        console.error('[chat] could not write the failure message:', e2);
-      }
-    } finally {
-      turn.finished = true;
-    }
-  });
-
-  console.log(`[chat] turns at POST ${path}/conversations/:id/turns`);
-}

src/chat/store.js

@@ -1,163 +0,0 @@
-// The chat's reads and writes. Nothing here knows the service it lives in.
-// Full URIs everywhere: the auth layer rejects a prefixed name with no
-// PREFIX line with an opaque 500.
-import { randomUUID } from 'crypto';
-import { query as muQuery, update as muUpdate,
-         sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime } from 'mu';
-
-const HISTORY_LIMIT = Number(process.env.CHAT_HISTORY_LIMIT || 20);
-
-const T = {
-  type: 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type',
-  uuid: 'http://mu.semte.ch/vocabularies/core/uuid',
-  thread: 'http://rdfs.org/sioc/ns#Thread',
-  post: 'http://rdfs.org/sioc/ns#Post',
-  instantMessage: 'http://rdfs.org/sioc/types#InstantMessage',
-  hasContainer: 'http://rdfs.org/sioc/ns#has_container',
-  content: 'http://rdfs.org/sioc/ns#content',
-  lastActivity: 'http://rdfs.org/sioc/ns#last_activity_date',
-  title: 'http://purl.org/dc/terms/title',
-  created: 'http://purl.org/dc/terms/created',
-  maker: 'http://xmlns.com/foaf/0.1/maker',
-  attachment: 'https://www.w3.org/ns/activitystreams#attachment',
-  document: 'https://www.w3.org/ns/activitystreams#Document',
-  name: 'https://www.w3.org/ns/activitystreams#name',
-  mediaType: 'https://www.w3.org/ns/activitystreams#mediaType',
-  url: 'https://www.w3.org/ns/activitystreams#url',
-  softwareAgent: 'http://www.w3.org/ns/prov#SoftwareAgent',
-};
-const MESSAGE_BASE = 'http://data.lblod.info/id/chat-messages/';
-const DOCUMENT_BASE = 'http://data.lblod.info/id/chat-documents/';
-
-const u = sparqlEscapeUri;
-const s = sparqlEscapeString;
-// Full URIs, so spell the dateTime type out.
-const dt = (value) => sparqlEscapeDateTime(value)
-  .replace('xsd:dateTime', '<http://www.w3.org/2001/XMLSchema#dateTime>');
-
-const bindings = (json) => json.results.bindings;
-
-// --- reads --------------------------------------------------------------------
-
-// The conversation by uuid, null when the caller may not read it.
-export async function readConversation(id) {
-  const json = await muQuery(`
-    SELECT ?conversation ?title ?creator WHERE {
-      ?conversation ${u(T.type)} ${u(T.thread)} ;
-        ${u(T.uuid)} ${s(id)} .
-      OPTIONAL { ?conversation ${u(T.title)} ?title . }
-      OPTIONAL { ?conversation ${u(T.maker)} ?creator . }
-    } LIMIT 1`);
-  const b = bindings(json)[0];
-  if (!b) return null;
-  return { uri: b.conversation.value, id, title: b.title?.value, creator: b.creator?.value };
-}
-
-// The last HISTORY_LIMIT messages, oldest first, as { role, content }.
-export async function readHistory(conversationUri, assistantUri) {
-  const json = await muQuery(`
-    SELECT ?content ?maker ?created WHERE {
-      ?message ${u(T.hasContainer)} ${u(conversationUri)} ;
-        ${u(T.content)} ?content ;
-        ${u(T.maker)} ?maker ;
-        ${u(T.created)} ?created .
-    } ORDER BY DESC(?created) LIMIT ${HISTORY_LIMIT}`);
-  return bindings(json).reverse().map(b => ({
-    role: b.maker.value === assistantUri ? 'assistant' : 'user',
-    content: b.content.value,
-  }));
-}
-
-// The one assistant the caller can read. Throws unless there is exactly one.
-export async function findAssistant() {
-  const json = await muQuery(`
-    SELECT ?agent WHERE { ?agent ${u(T.type)} ${u(T.softwareAgent)} . } LIMIT 2`);
-  const rows = bindings(json);
-  if (rows.length !== 1) {
-    throw new Error(`expected one prov:SoftwareAgent, found ${rows.length}; seed one or set CHAT_ASSISTANT_URI`);
-  }
-  return rows[0].agent.value;
-}
-
-// --- writes -------------------------------------------------------------------
-
-// One message, both types, its attachments, and the conversation touched, in
-// one update. `title` is only written when given (the first question names
-// an unnamed conversation). Returns { uri, id, documents: [{ uri, id }] }.
-export async function writeMessage({ conversationUri, content, maker, attachments = [], title }) {
-  const id = randomUUID();
-  const uri = `${MESSAGE_BASE}${id}`;
-  const now = dt(new Date());
-  const documents = attachments.map(a => {
-    const id = randomUUID();
-    return { ...a, id, uri: `${DOCUMENT_BASE}${id}` };
-  });
-
-  // Predicate pairs joined with ';', never a trailing one: it breaks the Turtle.
-  const pairs = [
-    `${u(T.type)} ${u(T.post)}`,
-    `${u(T.type)} ${u(T.instantMessage)}`,
-    `${u(T.uuid)} ${s(id)}`,
-    `${u(T.hasContainer)} ${u(conversationUri)}`,
-    `${u(T.content)} ${s(content)}`,
-    `${u(T.created)} ${now}`,
-  ];
-  if (maker) pairs.push(`${u(T.maker)} ${u(maker)}`);
-  for (const d of documents) pairs.push(`${u(T.attachment)} ${u(d.uri)}`);
-
-  const conversationPairs = [`${u(T.lastActivity)} ${now}`];
-  if (title) conversationPairs.push(`${u(T.title)} ${s(title)}`);
-
-  await muUpdate(`
-    DELETE {
-      ${u(conversationUri)} ${u(T.lastActivity)} ?old .
-    }
-    INSERT {
-      ${u(uri)} ${pairs.join(' ;\n        ')} .
-      ${documents.map(d => `
-      ${u(d.uri)} ${[
-        `${u(T.type)} ${u(T.document)}`,
-        `${u(T.uuid)} ${s(d.id)}`,
-        `${u(T.name)} ${s(d.name)}`,
-        `${u(T.mediaType)} ${s(d.mediaType || 'application/octet-stream')}`,
-        ...(d.url ? [`${u(T.url)} ${s(d.url)}`] : []),
-      ].join(' ;\n        ')} .`).join('')}
-      ${u(conversationUri)} ${conversationPairs.join(' ;\n        ')} .
-    }
-    WHERE {
-      OPTIONAL { ${u(conversationUri)} ${u(T.lastActivity)} ?old . }
-    }`);
-
-  return { uri, id, documents: documents.map(({ uri, id }) => ({ uri, id })) };
-}
-
-// The file is ready.
-export async function setDocumentUrl(documentUri, url, conversationUri) {
-  const now = dt(new Date());
-  await muUpdate(`
-    DELETE {
-      ${u(documentUri)} ${u(T.url)} ?oldUrl .
-      ${u(conversationUri)} ${u(T.lastActivity)} ?old .
-    }
-    INSERT {
-      ${u(documentUri)} ${u(T.url)} ${s(url)} .
-      ${u(conversationUri)} ${u(T.lastActivity)} ${now} .
-    }
-    WHERE {
-      OPTIONAL { ${u(documentUri)} ${u(T.url)} ?oldUrl . }
-      OPTIONAL { ${u(conversationUri)} ${u(T.lastActivity)} ?old . }
-    }`);
-}
-
-// The file failed; the card disappears, the caller says why in a message.
-export async function dropDocument(documentUri) {
-  await muUpdate(`
-    DELETE {
-      ?message ${u(T.attachment)} ${u(documentUri)} .
-      ${u(documentUri)} ?p ?o .
-    }
-    WHERE {
-      ${u(documentUri)} ?p ?o .
-      OPTIONAL { ?message ${u(T.attachment)} ${u(documentUri)} . }
-    }`);
-}

src/db.js

@@ -1,30 +0,0 @@
-// Thin re-export of mu's SPARQL helpers so the runner files share one import
-// site. mu's query/update attach the caller's session from the request
-// context, and mu-authorization decides graphs and visibility; the service
-// never touches an auth header.
-import {
-  query as muQuery,
-  update as muUpdate,
-  sparqlEscapeUri,
-  sparqlEscapeString,
-  sparqlEscapeInt,
-  sparqlEscapeDateTime,
-  sparqlEscapeDate,
-  sparqlEscapeBool,
-} from 'mu';
-
-export const sessionQuery = muQuery;
-export const sessionUpdate = muUpdate;
-export {
-  sparqlEscapeUri,
-  sparqlEscapeString,
-  sparqlEscapeInt,
-  sparqlEscapeDateTime,
-  sparqlEscapeDate,
-  sparqlEscapeBool,
-};
-
-// The store's error text may name graphs or internals we do not pass on.
-export function stripStoreError(text) {
-  return String(text).split('\n')[0].slice(0, 300);
-}

src/llm/describe.js

@@ -1,58 +1,9 @@
 // A profile as a plain text menu: entities, fields, code lists. Code lists up
-// to INLINE_VALUES_MAX are inlined and cached for VALUES_TTL, so the LLM
-// never has to search the short ones.
+// to INLINE_VALUES_MAX are inlined (and cached), so the LLM never has to
+// search the short ones.
+import { codeList, INLINE_VALUES_MAX } from './lookup.js';
 
-import { fieldsOf, entityForClass } from '../runner/profile.js';
-import { sessionQuery } from '../db.js';
-
-const INLINE_VALUES_MAX = Number(process.env.INLINE_VALUES_MAX || 50);
-const VALUES_TTL = Number(process.env.VALUES_TTL || 3600) * 1000;
-
-// In-memory cache: profile+field → { values, at }. Refreshed lazily when stale.
-const cache = new Map();
-
-export async function codeListValues(profile) {
-  // Returns a map: "shapeUri|predicate|inverse" → [{ uri, label }]
-  const out = {};
-  for (const shape of profile.shapes) {
-    for (const f of shape.fields) {
-      if (!f.class) continue;
-      const key = `${shape.uri}|${f.path}|${f.inverse ? 1 : 0}`;
-      out[key] = await loadCodeList(profile, f, key);
-    }
-  }
-  return out;
-}
-
-async function loadCodeList(profile, f, key) {
-  if (!f.class) return [];
-  const hit = cache.get(key);
-  if (hit && Date.now() - hit.at < VALUES_TTL) return hit.values;
-  let values = [];
-  try {
-    values = await queryCodeList(f.class);
-  } catch (e) {
-    // the read failed; report nothing, the LLM will use rep:anyOf with words
-    values = [];
-  }
-  cache.set(key, { values, at: Date.now() });
-  return values;
-}
-
-async function queryCodeList(type) {
-  const q = `SELECT ?uri ?label WHERE {
-    ?uri a <${type}> .
-    OPTIONAL { ?uri <http://www.w3.org/2004/02/skos/core#prefLabel> ?label . }
-  } ORDER BY ?label LIMIT ${INLINE_VALUES_MAX + 1}`;
-  const r = await sessionQuery(q);
-  const bindings = r.results.bindings;
-  return bindings.slice(0, INLINE_VALUES_MAX).map(b => ({
-    uri: b.uri.value,
-    label: b.label?.value || b.uri.value,
-  }));
-}
-
-export function describeProfile(profile, codeLists = {}) {
+export async function describeProfile(profile) {
   const lines = [];
   lines.push('A spec is Turtle, in exactly this shape (copy these prefixes):');
   lines.push('@prefix rep:  <http://mu.semte.ch/vocabularies/reporting/> .');
@@ -67,6 +18,7 @@ export function describeProfile(profile, codeLists = {}) {
   lines.push('  sh:property [ ... ] ;   # filters, optional');
   lines.push('  rep:columns ( [ sh:path ( ... ) ; rdfs:label "..." ] ... ) .');
   lines.push('Every column ends on a value field. Use rep:self as sh:path for the subject URI.');
+  lines.push('For the URI of a linked node, end the column on that link and add sh:nodeKind sh:IRI to the column.');
   lines.push('');
   lines.push('sh:targetClass is the full URI in brackets after the entity below.');
   lines.push('In sh:path, use the predicate URI shown after each field name, not the field name itself.');
@@ -78,13 +30,12 @@ export function describeProfile(profile, codeLists = {}) {
   for (const shape of profile.shapes) {
     lines.push(`${shape.label || shape.targetClass} — sh:targetClass <${shape.targetClass}> ; rep:entity <${shape.uri}>`);
     for (const f of shape.fields) {
-      const key = `${shape.uri}|${f.path}|${f.inverse ? 1 : 0}`;
       const kind = f.datatype ? datatypeName(f.datatype)
-        : f.class ? `→ ${conceptLabel(f.class, codeLists[key])}`
+        : f.class ? `→ ${conceptLabel(f.class, await codeList(f.class))}`
         : f.node ? `→ ${linkedLabel(profile, f.node)}`
         : '?';
       const dir = f.inverse ? ' (inverse)' : '';
-      const pathShown = typeof f.path === 'string' ? f.path : '?';
+      const pathShown = f.path || '?';
       lines.push(`  ${f.name || pathShown}  sh:path <${pathShown}>${dir}  ${kind}`);
     }
     lines.push('');
@@ -93,11 +44,10 @@ export function describeProfile(profile, codeLists = {}) {
 }
 
 function conceptLabel(type, values) {
-  const head = values.slice(0, INLINE_VALUES_MAX);
-  if (!head.length) return short(type);
-  const names = head.map(v => v.label).join(', ');
+  if (!values.length) return short(type);
+  const names = values.map(v => v.label).join(', ');
   const more = values.length >= INLINE_VALUES_MAX ? ', …' : '';
-  return `${short(type)} — ${head.length} waarden: ${names}${more}`;
+  return `${short(type)} — ${values.length} waarden: ${names}${more}`;
 }
 
 function linkedLabel(profile, nodeUri) {

src/llm/lookup.js

@@ -0,0 +1,137 @@
+// The LLM's own reads: code lists. They run under the service scope, not
+// the caller's session; the sparql-parser config grants that scope read
+// access to the public graph only, so they read the same for every caller.
+// Execution still runs as the caller.
+//
+// lookup_values is a suggest helper, not a spec check. It reads candidate
+// values the LLM can put in a filter; it never runs the spec and never sees
+// report rows.
+import { query, sparqlEscapeString } from 'mu';
+
+// Must match the with-scope grant in the app's config.lisp.
+const SERVICE_SCOPE = process.env.SERVICE_SCOPE || 'http://services.semantic.works/natural-language-report';
+export const INLINE_VALUES_MAX = Number(process.env.INLINE_VALUES_MAX || 50);
+const VALUES_TTL = Number(process.env.VALUES_TTL || 3600) * 1000;
+const LIMIT = 25;
+
+// class → { values, at }. Refreshed lazily when stale.
+const cache = new Map();
+
+function publicQuery(q) {
+  return query(q, { scope: SERVICE_SCOPE });
+}
+
+// codeList(type) → up to INLINE_VALUES_MAX [{ uri, label }], cached for
+// VALUES_TTL. Empty when the read fails; the LLM then uses rep:anyOf with
+// words.
+export async function codeList(type) {
+  const hit = cache.get(type);
+  if (hit && Date.now() - hit.at < VALUES_TTL) return hit.values;
+  let values = [];
+  try {
+    const r = await publicQuery(`SELECT ?uri ?label WHERE {
+    ?uri a <${type}> .
+    OPTIONAL { ?uri <http://www.w3.org/2004/02/skos/core#prefLabel> ?label . }
+  } ORDER BY ?label LIMIT ${INLINE_VALUES_MAX + 1}`);
+    values = r.results.bindings.slice(0, INLINE_VALUES_MAX).map(b => ({
+      uri: b.uri.value,
+      label: b.label?.value || b.uri.value,
+    }));
+  } catch (e) {
+    values = [];
+  }
+  cache.set(type, { values, at: Date.now() });
+  return values;
+}
+
+// lookupValues(profile, field, term) → the matches as text for the LLM: up
+// to LIMIT labels and URIs, an exact label (case aside) first. field is
+// "shapeLabel.fieldLabel" as the LLM writes it; it resolves to the class to
+// search.
+export async function lookupValues(profile, field, term) {
+  const type = resolveClass(profile, field);
+  if (!type) return `field "${field}" not found in profile "${profile.title}"`;
+
+  // Search within the class first. A common word ("gemeente") also sits in
+  // hundreds of unrelated labels; an unscoped scan drowns the code.
+  let rows = await labelQuery(term, type);
+  // Code lists sometimes type their entries differently than the profile
+  // says (MAR codes are ext:supervision/Nomenclature, the profile says
+  // skos:Concept). Fall back to a scan over everything.
+  if (!rows.length) rows = await labelQuery(term, null);
+  // "Gent" must surface before "AGB Erfgoed Gent".
+  const lower = term.toLowerCase();
+  const exactRow = rows.find(r => r.label.toLowerCase() === lower);
+  let matches = (exactRow ? [exactRow, ...rows.filter(r => r !== exactRow)] : rows).slice(0, LIMIT);
+  if (!matches.length) matches = await codeQuery(term);
+  if (!matches.length) return `no value matches "${term}" in ${field}.`;
+
+  const lines = matches.map(m => `${m.label}  <${m.uri}>`);
+  if (matches.length >= LIMIT) lines.push(`(${LIMIT}+ matches, narrow the term)`);
+  const exact = matches.find(m => m.label.toLowerCase() === lower);
+  if (exact) lines.push(`exact: <${exact.uri}>`);
+  return lines.join('\n');
+}
+
+// Regex on str(?label), so language-tagged labels match.
+async function labelQuery(term, type) {
+  const q = `SELECT DISTINCT ?uri ?label WHERE {
+    ?uri a ${type ? `<${type}>` : '?anyType'} ;
+         <http://www.w3.org/2004/02/skos/core#prefLabel> ?label .
+    FILTER(REGEX(str(?label), ${sparqlEscapeString(escapeRegex(term))}, "i"))
+  } ORDER BY ?label LIMIT ${LIMIT * 8}`;
+  const r = await publicQuery(q);
+  return r.results.bindings.map(b => ({ label: b.label.value, uri: b.uri.value }));
+}
+
+// MAR codes: the label says "MAR7300 - …" but the searchable code sits on
+// ext:supervision/nomenclatureCode. One extra query, only when the label
+// search came up empty.
+async function codeQuery(term) {
+  const q = `SELECT ?uri ?label WHERE {
+    ?uri <http://mu.semte.ch/vocabularies/ext/supervision/nomenclatureCode> ?code ;
+         <http://www.w3.org/2004/02/skos/core#prefLabel> ?label .
+    FILTER(CONTAINS(LCASE(str(?code)), LCASE(${sparqlEscapeString(term)})))
+  } ORDER BY ?label LIMIT ${LIMIT}`;
+  try {
+    const r = await publicQuery(q);
+    return r.results.bindings.map(b => ({ label: b.label.value, uri: b.uri.value }));
+  } catch {
+    return [];
+  }
+}
+
+// "shapeLabel.fieldLabel" → the class to search: the field's sh:class, or
+// for a link (sh:node) the linked shape's sh:targetClass
+// ("melding.bestuurseenheid" → besluit:Bestuurseenheid).
+function resolveClass(profile, field) {
+  if (field.includes('/')) return null;
+  const dot = field.lastIndexOf('.');
+  if (dot < 0) return null;
+  const shapeLabel = field.slice(0, dot);
+  const fieldLabel = field.slice(dot + 1);
+  const shape = profile.shapes.find(s => s.label === shapeLabel);
+  if (!shape) return null;
+  const f = shape.fields.find(x => x.name === fieldLabel);
+  if (f?.class) return f.class;
+  if (f?.node) {
+    const linked = profile.shapes.find(s => s.uri === f.node);
+    return linked?.targetClass || null;
+  }
+  // The model sometimes writes shape.field on a linked shape
+  // ("bestuurseenheid.naam"); resolve it to that shape's target class.
+  for (const s of profile.shapes) {
+    if (s.label === shapeLabel || shortType(s.targetClass) === shapeLabel) {
+      return s.targetClass;
+    }
+  }
+  return null;
+}
+
+function shortType(cls) {
+  return cls.split('#').pop() || cls.split('/').pop();
+}
+
+function escapeRegex(s) {
+  return String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}

src/llm/loop.js

@@ -1,8 +1,15 @@
 // The agent loop: POST /v1/chat/completions with tool calls,
 // OpenAI-compatible (LLM_BASE_URL, LLM_MODEL, LLM_API_KEY). No provider
 // SDKs. Stops when the model answers without tool calls.
+//
+// Refinement only: the LLM proposes a spec, may look up candidate values in
+// code lists (under the service scope) and iterates with the user. It never
+// runs a report and never learns what a spec would match. The user must
+// confirm before anything executes; execution runs the agreed spec without
+// the LLM (../report-assistant.js).
 
-import { buildTools } from './tools.js';
+import { TOOLS, runTool } from './tools.js';
+import { SPEC_MEDIA_TYPE } from '../chat.js';
 
 const MAX_ROUNDS = Number(process.env.MAX_ROUNDS || 12);
 const LLM_BASE_URL = process.env.LLM_BASE_URL;
@@ -11,72 +18,109 @@ const LLM_API_KEY = process.env.LLM_API_KEY;
 // LOG_LLM=1 dumps every request to and response from the provider, verbatim.
 const LOG_LLM = /^(true|1)$/i.test(process.env.LOG_LLM || '');
 
-const SYSTEM_PROMPT = `You write report specs. You never write SPARQL and you never see report data.
+// Refinement (Modus A). The LLM and the user iterate on a spec. The LLM may
+// look up candidate values in code lists (lookup_values, under the service
+// scope) to make concrete suggestions, but it may not run a report and never
+// learns how many subjects a spec would match. It ends with a proposal in
+// Dutch and asks the user to confirm.
+const REFINE_PROMPT = `You refine report specs with the user. You never write SPARQL and you never run a report.
 
 The profile lists what can be asked. Compose paths by chaining fields.
 
 Rules:
-- A column must end on a value, not on a link. Chain one more hop.
+- A column must end on a value, not on a link. Chain one more hop. Only when
+  the user asks for the URI of a linked node itself, end the column on that
+  link and add sh:nodeKind sh:IRI to the column.
 - When two entities share one sh:targetClass, the spec also needs rep:entity
   with that entity's URI; describe_profile shows it after each entity.
 - Filters go in sh:property. Columns go in rep:columns, in the order you want.
 - To match a topic rather than a value, list the Dutch words and compounds you
   would expect and put them all in rep:anyOf.
+- A filter tests only the end of its own path, and two filters never share a
+  node. To test a node in the middle of a path (for example "the mandate
+  whose role is Burgemeester"), give that filter or column a rep:where:
+  rep:where [ sh:path ( <the same first steps> <more steps> ) ; <one condition> ].
+  Write its path from the row, like a filter. The steps it has in common with
+  its filter or column are the same nodes, and its condition tests where its
+  own path ends. On a column, rep:where drops values from the cell, never
+  rows. Several rep:where on one filter or column all apply.
+- validate_spec answers "ok" plus one line per rep:where saying where it
+  applies. Say that in Dutch in your proposal, so the user can check it.
+- Only offer what validate_spec accepted. Before you suggest an alternative,
+  write it and validate it; if it does not validate, say it cannot be done.
 - You cannot count, sort, take the first N, or compare two subjects. Say so.
 - The report's title is the dct:title you write in the spec. There is no other
   title input.
-- One question makes one report. Write one spec, validate it, run it, then stop.
-  Do not write a second report for the same question.
 
-Always prefix the spec with @prefix lines. Use the prefixes the profile declares.
-Write the spec as one Turtle block. Then call validate_spec. Fix what it says,
-up to three rounds. Then call run_report. After run_report returns, stop.
+You are in refinement mode. You propose a spec and explain in Dutch what it
+will list and filter. Use lookup_values to find concrete values (names, codes)
+for a filter, so your proposal names real values instead of guesses. That is
+the only database read you have, and it only suggests values: it never tells
+you whether a spec is good or how much it matches. The database you read may
+differ from the one the report runs on, so never treat a lookup as a check.
+Iterate with the user on the spec itself: paths, filters, columns. When the
+user wants an earlier spec changed, open it with read_spec and change what
+they asked for. Never call
+run_report; there is no such tool here. When the spec is ready, ask the user
+to confirm execution. Then stop.
+
+The spec itself does not go in your text. validate_spec stores a spec that
+checks out as an attachment ("bijlage", a spec file) on your answer. So in
+your answer: explain in Dutch in full what the report will list and which
+filters it applies, tell the user the actual spec is in the bijlage and to
+look there, and ask the user to confirm execution. Do not print the Turtle
+yourself; the bijlage shows the validated version, so never rewrite it in
+the text either.
+
+Always prefix the spec with @prefix lines. Use the prefixes the profile
+declares. Write the spec as one Turtle block. Then call validate_spec. Fix
+what it says, up to three rounds. Then propose and stop.
 
 Reply to the user in Dutch. Earlier turns are context only. Answer the last one.`;
 
-// ask(questionOrMessages, profiles, session) → { text, trace }
-// A string is one question; an array is the conversation so far,
-// [{ role, content }], newest last. The system prompt is always this file's
-// own; a system message in the input is dropped.
-export async function ask(questionOrMessages, profiles, session) {
-  const { handlers } = buildTools(profiles, session);
-  const trace = [];
-
-  const turns = typeof questionOrMessages === 'string'
-    ? [{ role: 'user', content: questionOrMessages }]
-    : questionOrMessages.filter(m => m.role !== 'system');
-  const messages = [{ role: 'system', content: SYSTEM_PROMPT }, ...turns];
+// ask(turns, profiles) → { text, spec }. turns is the conversation so far,
+// [{ role, content, attachments }], newest last. spec is the last spec
+// validate_spec accepted in this turn, or null.
+export async function ask(turns, profiles) {
+  const messages = [{ role: 'system', content: REFINE_PROMPT }, ...turns.map(withSpecHint)];
+  let spec = null;
 
   for (let round = 0; round < MAX_ROUNDS; round++) {
-    const reply = await chat(messages);
-    messages.push(reply.message);
-    const calls = reply.message.tool_calls || [];
-    if (!calls.length) {
-      return { text: reply.message.content || '', trace };
-    }
+    const { message } = await chat(messages, TOOLS);
+    messages.push(message);
+    const calls = message.tool_calls || [];
+    if (!calls.length) return { text: message.content || '', spec };
+
     for (const call of calls) {
       const args = parseArgs(call.function.arguments);
-      trace.push({ name: call.function.name, args });
-      const fn = handlers.get(call.function.name);
-      let resultText;
-      if (!fn) {
-        resultText = `unknown tool "${call.function.name}"`;
-      } else {
-        try {
-          const out = await fn(args);
-          resultText = out.content?.[0]?.text ?? JSON.stringify(out);
-        } catch (e) {
-          resultText = `tool failed: ${String(e.message || e).split('\n')[0]}`;
-        }
+      let result;
+      try {
+        result = await runTool(call.function.name, args, profiles);
+      } catch (e) {
+        result = `tool failed: ${String(e.message || e).split('\n')[0]}`;
       }
-      messages.push({ role: 'tool', tool_call_id: call.id, content: resultText });
-      trace[trace.length - 1].result = resultText;
+      // validate_spec answers "ok" (plus notes) when the spec checks out;
+      // that spec becomes the bijlage of the answer.
+      if (call.function.name === 'validate_spec' && /^ok(\n|$)/.test(result)) spec = args.spec;
+      messages.push({ role: 'tool', tool_call_id: call.id, content: result });
     }
   }
-  return { text: '(loop did not finish)', trace };
+  return { text: 'Sorry, dit rapport is te moeilijk. Contacteer de developers.', spec };
 }
 
-async function chat(messages) {
+// The history carries the spec as a bijlage. Paste one hint line per spec
+// under the message text, so the model knows a spec is there and opens it
+// with read_spec; the chat itself still keeps the Turtle out of the text.
+function withSpecHint(turn) {
+  const specs = (turn.attachments || []).filter(a => a.mediaType === SPEC_MEDIA_TYPE);
+  if (!specs.length) return turn;
+  const hints = specs.map(a => `Bijlage "${a.name}", a spec file (read it with read_spec).`);
+  return { ...turn, content: `${turn.content}\n\n${hints.join('\n\n')}` };
+}
+
+// chat(messages, tools) → { message }. The one place that talks to the
+// provider. Without tools it is a plain completion (the mode check).
+export async function chat(messages, tools = null) {
   // Send only the fields every provider accepts back.
   const wire = messages.map(m => {
     const out = { role: m.role, content: m.content ?? null };
@@ -90,15 +134,16 @@ async function chat(messages) {
     if (m.role === 'tool' && m.tool_call_id) out.tool_call_id = m.tool_call_id;
     return out;
   });
-  const body = {
-    model: LLM_MODEL,
-    messages: wire,
-    tools: toolDefs(),
-    tool_choice: 'auto',
-  };
+  const body = { model: LLM_MODEL, messages: wire };
+  if (tools) {
+    body.tools = tools;
+    body.tool_choice = 'auto';
+  }
   if (LOG_LLM) {
     console.log(`[llm] POST ${LLM_BASE_URL}/chat/completions request:\n${
       JSON.stringify(body, null, 2)}`);
+  } else {
+    console.log('[llm] sending instructions to llm');
   }
   const res = await fetch(`${LLM_BASE_URL}/chat/completions`, {
     method: 'POST',
@@ -123,16 +168,3 @@ function parseArgs(s) {
   if (!s) return {};
   try { return JSON.parse(s); } catch { return {}; }
 }
-
-// The tool definitions sent to the model.
-import { toolSchemas } from './schemas.js';
-function toolDefs() {
-  return toolSchemas().map(t => ({
-    type: 'function',
-    function: {
-      name: t.name,
-      description: t.description,
-      parameters: t.inputSchema,
-    },
-  }));
-}

src/llm/mode.js

@@ -0,0 +1,56 @@
+// The mode classifier. One LLM call per turn: a guestimate of whether the
+// user wants to keep refining the query or has given a clear command to
+// execute it now. Returns true (execute) / false (refine). Deterministic
+// code branches on the answer; refine is the safe default when the call
+// fails or the answer does not parse.
+
+import { chat } from './loop.js';
+
+const MODE_PROMPT = `You read a conversation between a user and a report assistant.
+The assistant proposes report specs and asks the user to confirm before
+running them.
+
+Decide whether the user's LAST message is a clear instruction to run the
+report now (for example "ja", "voer uit", "doe het", "maak het rapport",
+"go", "execute", "run it") or is still refining, asking, correcting or
+answering a question.
+
+Rules:
+- Only return true when the user clearly tells the assistant to execute the
+  agreed report right now. A short "ja" or "ok" after a proposal counts.
+- A new or amended question, a clarification, a "no", a question about the
+  proposal, or the absence of an earlier proposal means false.
+- When in doubt, return false.
+
+Reply with exactly one token: true or false. No other words, no punctuation,
+no markdown.`;
+
+// wantsExecution(turns) → true when the user's last message says "run it
+// now". turns: [{ role, content }], newest last; the last few are enough.
+export async function wantsExecution(turns) {
+  let message;
+  try {
+    ({ message } = await chat([{ role: 'system', content: MODE_PROMPT }, ...turns.slice(-6)]));
+  } catch (e) {
+    // The call failed (no LLM, provider down). Refine: safer than running a
+    // report on a guess, and the refine loop will surface the real error.
+    return false;
+  }
+  return stripMarkdown(firstText(message)).trim().toLowerCase().startsWith('true');
+}
+
+// Reasoning models may leave `content` empty and put the visible text in a
+// separate field (reasoning_content / thinking / reasoning), or may not
+// answer at all when the thinking burned the token budget. Read the visible
+// fields in order and take the first non-empty one.
+function firstText(message) {
+  for (const key of ['content', 'reasoning_content', 'reasoning', 'thinking']) {
+    const v = message?.[key];
+    if (typeof v === 'string' && v.trim()) return v;
+  }
+  return '';
+}
+
+function stripMarkdown(s) {
+  return String(s).replace(/[*_`>#]/g, ' ');
+}

src/llm/schemas.js

@@ -1,39 +0,0 @@
-// The JSON schemas of the four tools. The descriptions are what the LLM
-// reads to decide which tool to call.
-
-export function toolSchemas() {
-  return [
-    {
-      name: 'list_profiles',
-      description: 'List the report profiles available, with their id and title.',
-      inputSchema: { type: 'object', properties: {}, required: [] },
-    },
-    {
-      name: 'describe_profile',
-      description: 'Describe one profile: its entities, fields, relations and labels. Inline the values of short code lists.',
-      inputSchema: {
-        type: 'object',
-        properties: { profile_id: { type: 'string', description: 'The profile URI returned by list_profiles' } },
-        required: ['profile_id'],
-      },
-    },
-    {
-      name: 'validate_spec',
-      description: 'Validate a report spec (Turtle). Returns "ok" or one clear sentence saying what is wrong and what to write instead. No database is touched.',
-      inputSchema: {
-        type: 'object',
-        properties: { spec: { type: 'string', description: 'The spec as Turtle text' } },
-        required: ['spec'],
-      },
-    },
-    {
-      name: 'run_report',
-      description: 'Run a valid spec and return the report URI, the row count and the file URI. Runs as the caller, so it sees only what they may see. Waits as long as the run keeps answering; gives up after RUN_TIMEOUT seconds of silence. The report is named by the dct:title in the spec. Call this once per question, after validate_spec returns ok.',
-      inputSchema: {
-        type: 'object',
-        properties: { spec: { type: 'string' } },
-        required: ['spec'],
-      },
-    },
-  ];
-}

src/llm/tools.js

@@ -1,113 +1,91 @@
-// The tools the LLM calls: list_profiles, describe_profile, validate_spec,
-// run_report. They return a report URI, a count, or an error we wrote —
-// never a cell value. The shape every tool returns is
-// { content: [{ type: 'text', text }] }, which the loop passes straight back
-// to the model.
-
-import { checkSpec } from '../runner/check.js';
+// The tools the LLM calls, refinement only: list_profiles,
+// describe_profile, validate_spec, read_spec and lookup_values. Each returns
+// plain text for the model. No report is created, no spec is evaluated;
+// the code-list reads run under the service scope (public graph).
 import { parseSpec } from '../runner/spec.js';
-import { describeProfile, codeListValues } from './describe.js';
-import { run, slug } from '../runner/run.js';
+import { checkSpec, whereNotes, profileError } from '../runner/check.js';
+import { describeProfile } from './describe.js';
+import { lookupValues } from './lookup.js';
+import { readSpecFile } from '../chat.js';
 
-const MAX_PATH_DEPTH = Number(process.env.MAX_PATH_DEPTH || 8);
-const RUN_TIMEOUT = Number(process.env.RUN_TIMEOUT || 60) * 1000;
+// The tool definitions sent to the model. The descriptions are what the LLM
+// reads to decide which tool to call.
+export const TOOLS = [
+  tool('list_profiles',
+    'List the report profiles available, with their id and title.',
+    {}),
+  tool('describe_profile',
+    'Describe one profile: its entities, fields, relations and labels. Inline the values of short code lists.',
+    { profile_id: 'The profile URI returned by list_profiles' }),
+  tool('validate_spec',
+    'Validate a report spec (Turtle). Returns "ok", followed by one line per rep:where saying where it applies, or one clear sentence saying what is wrong and what to write instead. No database is touched.',
+    { spec: 'The spec as Turtle text' }),
+  tool('read_spec',
+    'Return the Turtle of a spec bijlage, a ttl file attached to an earlier message in this conversation. Use it to reuse the agreed spec, or to read back an earlier proposal before you change it. Read-only.',
+    { file_name: 'The file name of the spec bijlage, e.g. specificatie-<uuid>.ttl' }),
+  tool('lookup_values',
+    'Search a code list for a term so you can suggest concrete values for a filter (for sh:hasValue or sh:in). Returns up to 25 matches with their URIs and labels, a total, and an exact flag when a label equals the term. Reads candidate values only; it never runs the spec and never tells you how many subjects a spec would match. Read-only, no report is created.',
+    {
+      profile_id: 'The profile URI returned by list_profiles',
+      field: 'The field as "entityLabel.fieldLabel", e.g. "bestuurseenheid.naam"',
+      term: 'The Dutch word or name to search for',
+    }),
+];
 
-// buildTools(profiles, session) → { handlers }
-// run_report waits while the run keeps answering: every query that comes
-// back (a seed page, a column batch) restarts the RUN_TIMEOUT clock, and
-// only that much silence gives up. The batch count is known upfront — the
-// subject list is complete after the seed — so a slow run with many small
-// batches finishes, however long it takes in total.
-export function buildTools(profiles, session) {
-  const profileList = () => [...profiles.values()].map(p => ({ id: p.uri, title: p.title }));
-
-  const findProfile = (id) => profiles.get(id);
-
-  async function describe_profile({ profile_id }) {
-    const p = findProfile(profile_id);
-    if (!p) return text(`no profile <${profile_id}>. Available: ${profileList().map(x => x.title).join(', ')}.`);
-    const inlined = await codeListValues(p, profiles);
-    return text(describeProfile(p, inlined));
-  }
-
-  async function validate_spec({ spec }) {
-    let parsed;
-    try { parsed = parseSpec(spec); }
-    catch (e) { return text(`the spec did not parse: ${e.message}`); }
-    const p = findProfile(parsed.profileUri);
-    if (!p) return text(`no profile <${parsed.profileUri}>. Available: ${profileList().map(x => x.title).join(', ')}.`);
-    const errors = checkSpec(parsed, p, MAX_PATH_DEPTH, profiles);
-    return text(errors.length ? errors[0] : 'ok');
-  }
-
-  async function list_profiles() {
-    return text(profileList().map(x => `${x.title}  <${x.id}>`).join('\n'));
-  }
-
-  async function run_report({ spec }) {
-    let parsed;
-    try { parsed = parseSpec(spec); }
-    catch (e) { return text(`the spec did not parse: ${e.message}`); }
-    const p = findProfile(parsed.profileUri);
-    if (!p) return text(`no profile <${parsed.profileUri}>.`);
-    const errors = checkSpec(parsed, p, MAX_PATH_DEPTH, profiles);
-    if (errors.length) return text(`the spec is not valid: ${errors[0]}`);
-    try {
-      // The spec's dct:title names the report.
-      const name = parsed.title || 'report';
-      await session.onReportStart?.({ title: name, fileName: `${slug(name)}.csv` });
-      const waiter = newRunWaiter(RUN_TIMEOUT);
-      const running = run(session.query, session.update, parsed, p, name, { spec }, waiter.tick);
-      // The end callback also fires after the wait gave up: the run is not
-      // cancelled, the file arrives when it arrives. The .catch keeps a late
-      // failure from crashing the process.
-      running
-        .then((r) => session.onReportEnd?.(null, r), (e) => session.onReportEnd?.(e))
-        .catch((e) => console.error('[reports] end callback failed:', e));
-      const result = await waiter.finish(running);
-      return text(`done. report <${result.reportUri}>, ${result.rowCount} rows, file <${result.fileUri}>.`);
-    } catch (e) {
-      return text(`the report failed: ${String(e.message || e).split('\n')[0]}`);
-    }
-  }
-
-  const handlers = new Map([
-    ['list_profiles', list_profiles],
-    ['describe_profile', describe_profile],
-    ['validate_spec', validate_spec],
-    ['run_report', run_report],
-  ]);
-
-  return { handlers };
-}
-
-function text(s) { return { content: [{ type: 'text', text: s }] }; }
-
-// Waits for a report run while it makes progress. tick restarts the silence
-// clock; idleMs without a single tick rejects. The run is never cancelled:
-// finish's handlers only pass its own settlement on, so a run that answers
-// after the wait gave up still delivers through the end callback.
-function newRunWaiter(idleMs) {
-  let deferred;
-  const promise = new Promise((resolve, reject) => { deferred = { resolve, reject }; });
-  let settled = false;
-  let timer = setTimeout(fail, idleMs);
-  function fail() {
-    if (settled) return;
-    settled = true;
-    deferred.reject(new Error(`no query answered for ${idleMs / 1000}s`));
+function tool(name, description, params) {
+  const properties = {};
+  for (const [key, text] of Object.entries(params)) {
+    properties[key] = { type: 'string', description: text };
   }
   return {
-    tick() {
-      if (settled) return;
-      clearTimeout(timer);
-      timer = setTimeout(fail, idleMs);
-    },
-    finish(running) {
-      running.then(
-        (r) => { settled = true; clearTimeout(timer); deferred.resolve(r); },
-        (e) => { settled = true; clearTimeout(timer); deferred.reject(e); });
-      return promise;
+    type: 'function',
+    function: {
+      name,
+      description,
+      parameters: { type: 'object', properties, required: Object.keys(params) },
     },
   };
-}
+}
+
+export async function runTool(name, args, profiles) {
+  switch (name) {
+    case 'list_profiles': return listProfiles(profiles);
+    case 'describe_profile': return describe(args, profiles);
+    case 'validate_spec': return validateSpec(args, profiles);
+    case 'read_spec': return readSpec(args);
+    case 'lookup_values': return lookup(args, profiles);
+    default: return `unknown tool "${name}"`;
+  }
+}
+
+function listProfiles(profiles) {
+  return [...profiles.values()].map(p => `${p.title}  <${p.uri}>`).join('\n');
+}
+
+async function describe({ profile_id }, profiles) {
+  return profileError(profiles, profile_id) || describeProfile(profiles.get(profile_id));
+}
+
+function validateSpec({ spec }, profiles) {
+  let parsed;
+  try {
+    parsed = parseSpec(spec);
+  } catch (e) {
+    return `the spec did not parse: ${e.message}`;
+  }
+  const error = checkSpec(parsed, profiles);
+  if (error) return error;
+  return ['ok', ...whereNotes(parsed, profiles.get(parsed.profileUri))].join('\n');
+}
+
+// Only spec files this service made are readable.
+function readSpec({ file_name }) {
+  return readSpecFile(file_name) || 'no such spec file.';
+}
+
+async function lookup({ profile_id, field, term }, profiles) {
+  const error = profileError(profiles, profile_id);
+  if (error) return error;
+  if (!term) return 'no term to search for.';
+  return lookupValues(profiles.get(profile_id), field, term);
+}

src/report-assistant.js

@@ -1,48 +1,71 @@
-// What this service answers a chat turn with: the LLM loop with the
-// conversation so far, and a report run becomes an interim message with a
-// pending file.
+// What the assistant answers a chat turn with: { text, attachments }. When
+// the answer is written, every bijlage already exists.
+//
+// Refine: the LLM proposes a spec; the last spec that checked out goes on
+// the answer as a bijlage. The prompt never carries the Turtle.
+//
+// Execute: no LLM runs. The last spec bijlage in the conversation is the
+// agreed spec; it is read from the share and fed straight to the runner.
+// The answer is fixed text. When the run cannot happen or fails, the answer
+// is a fixed failure message: nothing is retried and no spec is rewritten.
+import { uuid } from 'mu';
 import { ask } from './llm/loop.js';
+import { parseSpec } from './runner/spec.js';
+import { checkSpec } from './runner/check.js';
+import { run, registerReport, slug } from './runner/run.js';
+import { storeSpecFile, readSpecFile, writeShareFile, registerFile,
+         SPEC_MEDIA_TYPE, SPEC_TYPE, RESULT_TYPE } from './chat.js';
 
 const TEXT = {
-  making: 'Ik maak het rapport. Het verschijnt hier zodra het klaar is.',
-  failed: 'Het rapport is mislukt. Stel de vraag opnieuw, of anders geformuleerd.',
+  executed: 'Het rapport is uitgevoerd. De bijlagen staan erbij.',
+  noSpec: 'Er is nog geen voorstel om uit te voeren. Beschrijf eerst welk rapport je wilt.',
+  invalid: 'Het voorstel is niet meer geldig. Vraag een nieuw voorstel.',
+  failed: 'Het rapport kon niet uitgevoerd worden. Probeer het later opnieuw.',
+  noAnswer: 'Er kwam geen antwoord. Probeer het opnieuw.',
 };
 
-// The chat mounts before the profiles are in; whenProfiles waits for the load.
-export function reportAssistant(whenProfiles) {
-  return async function answer(turn) {
-    // One pending file per turn: a second run_report overwrites it before the
-    // first end callback fires.
-    let pendingDocument = null;
-
-    const session = {
-      query: turn.query,
-      update: turn.update,
-      async onReportStart({ fileName }) {
-        const said = await turn.say(TEXT.making, [{ name: fileName, mediaType: 'text/csv' }]);
-        pendingDocument = said.documents[0].uri;
-      },
-      async onReportEnd(error, result) {
-        const documentUri = pendingDocument;
-        pendingDocument = null;
-        if (!documentUri) return;
-        if (error) {
-          await turn.drop(documentUri);
-          if (turn.finished) await turn.say(TEXT.failed);
-          return;
-        }
-        await turn.setUrl(documentUri, downloadHref(result.fileUri));
-      },
-    };
-
-    const profiles = await whenProfiles();
-    const { text } = await ask([...turn.history, { role: 'user', content: turn.content }], profiles, session);
-    return text;
-  };
+// An empty answer (a reasoning model can end with no text) gets a fixed
+// text and no spec: only a spec the user saw explained may run.
+export async function refineSpec(turns, profiles) {
+  const { text, spec } = await ask(turns, profiles);
+  if (!text.trim()) return { text: TEXT.noAnswer };
+  const attachments = spec ? [await storeSpecFile(spec)] : [];
+  return { text: text.trim(), attachments };
 }
 
-// The file service serves /files/<uuid>/download; the logical file URI ends
-// in that uuid.
-function downloadHref(fileUri) {
-  return `/files/${fileUri.split('/').pop()}/download`;
+export async function executeSpec(history, conversationTitle, profiles) {
+  const attachment = lastSpecAttachment(history);
+  const content = attachment && readSpecFile(attachment.name);
+  if (!content) return { text: TEXT.noSpec };
+
+  try {
+    const spec = parseSpec(content);
+    const error = checkSpec(spec, profiles);
+    if (error) {
+      console.error(`[reports] the agreed spec is not valid: ${error}`);
+      return { text: TEXT.invalid };
+    }
+
+    // The conversation's title (named by the first question) names the
+    // file; the share name is unique, so a second run never overwrites the
+    // first. The report itself keeps the dct:title of the spec.
+    const shareName = `${uuid()}.csv`;
+    writeShareFile(shareName, await run(spec, profiles.get(spec.profileUri)));
+    const csvFile = await registerFile(shareName, 'text/csv', RESULT_TYPE, `${slug(conversationTitle)}.csv`);
+    await registerReport(spec.title, csvFile);
+
+    // Same physical spec file, registered as a bijlage for this message.
+    const specFile = await registerFile(attachment.name, SPEC_MEDIA_TYPE, SPEC_TYPE);
+
+    return { text: TEXT.executed, attachments: [csvFile, specFile] };
+  } catch (e) {
+    console.error('[reports] execution failed:', e);
+    return { text: TEXT.failed };
+  }
+}
+
+function lastSpecAttachment(history) {
+  const specs = history.flatMap(message => message.attachments)
+    .filter(a => a.mediaType === SPEC_MEDIA_TYPE && a.name?.startsWith('specificatie-'));
+  return specs.at(-1) || null;
 }

src/runner/assemble.js

@@ -1,7 +1,9 @@
 // Values to rows. Subjects in stable order, dedup on term, collect per cell.
 // rep:self columns take the subject URI; a column with no values is an
-// empty cell, never a dropped row. rep:row expands one subject into several
-// lines; at most one column per spec may do that (checked in check.js).
+// empty cell, never a dropped row. A column without rep:collect joins its
+// values only when there is one; with several, each value gets its own row
+// (the other columns repeat theirs). rep:row forces that expansion; at most
+// one column per spec may do that explicitly (checked in check.js).
 
 const SH = 'http://www.w3.org/ns/shacl#';
 const REP = 'http://mu.semte.ch/vocabularies/reporting/';
@@ -12,7 +14,6 @@ export const NUMERIC_DATATYPES = new Set([
   'nonNegativeInteger', 'positiveInteger', 'nonPositiveInteger', 'negativeInteger',
   'unsignedLong', 'unsignedInt', 'unsignedShort', 'unsignedByte',
 ].map(d => XSD + d));
-export const TEMPORAL_DATATYPES = new Set([XSD + 'date', XSD + 'dateTime']);
 
 export function assemble(subjects, values, spec) {
   const sorted = [...subjects].sort();
@@ -26,7 +27,8 @@ export function assemble(subjects, values, spec) {
     });
     const lines = Math.max(1, ...cells.map(c => c.length));
     for (let r = 0; r < lines; r++) {
-      // every cell except the expanding one keeps its value on each line
+      // the expanding cell cycles its values, every other cell keeps its
+      // value on each line
       rows.push(cells.map(c => c[Math.min(r, c.length - 1)]));
     }
   }
@@ -36,9 +38,13 @@ export function assemble(subjects, values, spec) {
 function collect(terms, col) {
   switch (col.collect) {
     case SH + 'groupConcat':
-    case null:
-    case undefined:
       return [terms.map(t => t.value).join(col.separator ?? ',')];
+    case undefined:
+    case null:
+      // no rep:collect: one value is a normal cell, several values each
+      // get their own row instead of a joined string
+      if (terms.length <= 1) return [terms.map(t => t.value).join(',')];
+      return terms.map(t => t.value);
     case SH + 'min':
       return [extreme(terms, col, -1)];
     case SH + 'max':
@@ -50,19 +56,18 @@ function collect(terms, col) {
   }
 }
 
-// min/max over values that share one datatype. Numbers compare as numbers,
-// dates and times as instants. Mixed datatypes throw: check.js should have
-// caught it, and a silently wrong cell is worse than a failed job.
+// min/max. Numbers compare as numbers. Dates and times compare as text:
+// xsd:date and xsd:dateTime are ISO strings, so text order is date order.
+// Numbers mixed with other values throw: check.js should have caught it,
+// and a silently wrong cell is worse than a failed job. The terms are SPARQL
+// JSON bindings, so the datatype is a plain string.
 function extreme(terms, col, direction) {
   if (!terms.length) return '';
-  const datatypes = [...new Set(terms.map(t => t.datatype?.value || ''))];
-  if (datatypes.length > 1) {
-    throw new Error(`column "${col.label}" asks for sh:${direction < 0 ? 'min' : 'max'} but its values carry ${datatypes.length} datatypes (${datatypes.join(', ')}).`);
+  const numbers = terms.filter(t => NUMERIC_DATATYPES.has(t.datatype)).length;
+  if (numbers && numbers < terms.length) {
+    throw new Error(`column "${col.label}" asks for sh:${direction < 0 ? 'min' : 'max'} but mixes numbers with other values.`);
   }
-  const dt = datatypes[0];
-  const key = NUMERIC_DATATYPES.has(dt) ? t => Number(t.value)
-    : TEMPORAL_DATATYPES.has(dt) ? t => new Date(t.value).getTime()
-    : t => t.value;
+  const key = numbers ? t => Number(t.value) : t => t.value;
   const cmp = (a, b) => a < b ? -1 : a > b ? 1 : 0;
   const best = terms.reduce((a, b) => (cmp(key(b), key(a)) * direction > 0 ? b : a));
   return best.value;

src/runner/check.js

@@ -1,5 +1,6 @@
-import { fieldsOf } from './profile.js';
+import { startShape, fieldsOf, field } from './profile.js';
 import { NUMERIC_DATATYPES } from './assemble.js';
+import { sharedSteps } from './spec.js';
 
 const SH = 'http://www.w3.org/ns/shacl#';
 const REP = 'http://mu.semte.ch/vocabularies/reporting/';
@@ -8,23 +9,18 @@ const XSD_DATE = XSD + 'date';
 const XSD_DATETIME = XSD + 'dateTime';
 
 // The validator. checkSpec runs the checks in order and returns the first
-// error as a one-element array (empty if ok). Every message names the thing
-// that is wrong in the profile's own words and says what to write instead.
-// No database call anywhere in this file.
+// error, or null when the spec is fine. Every message names the thing that
+// is wrong in the profile's own words and says what to write instead. No
+// database call anywhere in this file.
 
+const MAX_PATH_DEPTH = Number(process.env.MAX_PATH_DEPTH || 8);
 const ANYOF_MAX_TERMS = 24;
 const ANYOF_MAX_TERM_LENGTH = 100;
 
-// The spec names a profile we have. knownProfiles is the profile Map loaded
-// at boot; the check is skipped when it is omitted.
-function profileNamed(spec, knownProfiles) {
-  if (!spec.profileUri) {
-    return `no rep:profile named. Available: ${knownProfiles ? [...knownProfiles.values()].map(p => p.title).join(', ') : '(none)'} .`;
-  }
-  if (knownProfiles && !knownProfiles.has(spec.profileUri)) {
-    return `no profile <${spec.profileUri}>. Available: ${knownProfiles ? [...knownProfiles.values()].map(p => p.title).join(', ') : '(none)'}.`;
-  }
-  return null;
+// null when the profile exists, else what to write instead.
+export function profileError(profiles, uri) {
+  if (profiles.has(uri)) return null;
+  return `no profile <${uri}>. Available: ${[...profiles.values()].map(p => p.title).join(', ')}.`;
 }
 
 // The spec names an entity we have: by rep:entity, or by a targetClass only
@@ -57,14 +53,20 @@ function shapeList(profile) {
   return profile.shapes.map(s => s.label || s.targetClass).join(', ');
 }
 
-// The shape a spec starts from, after startEntity has approved it: rep:entity
-// wins, otherwise the one shape carrying the targetClass.
-function startShapeOf(spec, profile) {
-  if (spec.entity) {
-    return profile.shapes.find(s => s.uri === spec.entity)?.uri || null;
-  }
-  const matches = profile.shapes.filter(s => s.targetClass === spec.targetClass);
-  return matches.length === 1 ? matches[0].uri : null;
+// Every condition in the spec: the filters, and the rep:where conditions on
+// filters and columns, with the words that name each one in a message. A
+// rep:where also carries its host, the filter or column it hangs on.
+function conditions(spec) {
+  const out = spec.filters.map((f, i) => ({ what: `filter ${i + 1}`, cond: f }));
+  spec.filters.forEach((f, i) => f.where.forEach((w, j) => out.push({
+    what: `rep:where ${j + 1} on filter ${i + 1}`, cond: w,
+    host: { what: `filter ${i + 1}`, path: f.path },
+  })));
+  spec.columns.forEach(c => c.where.forEach((w, j) => out.push({
+    what: `rep:where ${j + 1} on column "${c.label}"`, cond: w,
+    host: { what: `column "${c.label}"`, path: c.path },
+  })));
+  return out;
 }
 
 // Columns exist
@@ -83,26 +85,26 @@ function columnLabel(spec) {
 
 // Labels are unique
 function columnLabelsUnique(spec) {
-  const seen = new Map();
+  const seen = new Set();
   for (const col of spec.columns) {
     if (seen.has(col.label)) {
       return `two columns are called "${col.label}". Labels must differ.`;
     }
-    seen.set(col.label, true);
+    seen.add(col.label);
   }
   return null;
 }
 
 // Every path resolves, columns end on a value, depth capped
-function pathsResolve(spec, profile, maxPathDepth) {
-  const startShapeUri = startShapeOf(spec, profile);
+function pathsResolve(spec, profile) {
+  const startShapeUri = startShape(profile, spec).uri;
   for (const col of spec.columns) {
-    const err = walk(profile, startShapeUri, col.path, maxPathDepth,
-      `column "${col.label}"`, true);
+    const err = walk(profile, startShapeUri, col.path,
+      `column "${col.label}"`, true, col.nodeKind === SH + 'IRI');
     if (err) return err;
   }
   for (const [i, filter] of spec.filters.entries()) {
-    const err = walk(profile, startShapeUri, filter.path, maxPathDepth,
+    const err = walk(profile, startShapeUri, filter.path,
       `filter ${i + 1}`, false);
     if (err) return err;
   }
@@ -111,7 +113,7 @@ function pathsResolve(spec, profile, maxPathDepth) {
 
 // sh:min/sh:max only on numbers and dates
 function collectMinMaxTyped(spec, profile) {
-  const startShapeUri = startShapeOf(spec, profile);
+  const startShapeUri = startShape(profile, spec).uri;
   for (const col of spec.columns) {
     if (col.collect !== SH + 'min' && col.collect !== SH + 'max') continue;
     const name = col.collect === SH + 'min' ? 'sh:min' : 'sh:max';
@@ -119,14 +121,13 @@ function collectMinMaxTyped(spec, profile) {
     const last = col.path[col.path.length - 1];
     const shapeUri = shapeAfter(profile, startShapeUri, col.path);
     if (!shapeUri) continue;   // pathsResolve already reported the real error
-    const f = fieldsOf(profile, shapeUri)
-      .find(x => x.path === last.predicate && !!x.inverse === !!last.inverse);
+    const f = field(profile, shapeUri, last);
     if (!f) continue;
     const kind = f.datatype === XSD_DATETIME || f.datatype === XSD_DATE ? 'date'
       : NUMERIC_DATATYPES.has(f.datatype) ? 'number'
       : null;
     if (!kind) {
-      return `"${f.name || last.predicate}" is text, so ${name} does not apply. Use rep:row, or leave it out to join the values.`;
+      return `"${f.name || last.predicate}" is text, so ${name} does not apply. Use rep:row (same as leaving rep:collect out: every value gets its own row).`;
     }
   }
   return null;
@@ -147,20 +148,23 @@ function columnPure(spec) {
   for (const col of spec.columns) {
     if (Object.keys(col.constraints || {}).length) {
       const which = Object.keys(col.constraints)[0];
-      return `a column carries ${which}. Constraints belong in sh:property.`;
+      return `a column carries ${which}. Constraints belong in sh:property, or in rep:where for a condition on a step of this column.`;
+    }
+    if (col.nodeKind && col.nodeKind !== SH + 'IRI') {
+      return `column "${col.label}" has sh:nodeKind <${col.nodeKind}>. A column takes only sh:nodeKind sh:IRI, to show the URI of the node its path ends on.`;
     }
     if (col.collect && ![SH + 'groupConcat', SH + 'min', SH + 'max', REP + 'row'].includes(col.collect)) {
-      return `column "${col.label}" has an unknown rep:collect <${col.collect}>. Use sh:groupConcat (default), rep:row, sh:min or sh:max.`;
+      return `column "${col.label}" has an unknown rep:collect <${col.collect}>. Use sh:groupConcat (joins the values), rep:row or nothing (each value gets its own row), sh:min or sh:max.`;
     }
   }
   return null;
 }
 
-// No label inside a filter
+// No label inside a filter or rep:where
 function filterLabel(spec) {
-  for (const filter of spec.filters) {
-    if ((filter.constraints || {}).label) {
-      return 'a filter carries rdfs:label. Move it to rep:columns to show it.';
+  for (const { what, cond } of conditions(spec)) {
+    if ((cond.constraints || {}).label) {
+      return `${what} carries rdfs:label. Move it to rep:columns to show it.`;
     }
   }
   return null;
@@ -171,11 +175,11 @@ function filterLabel(spec) {
 function filterConstraint(spec) {
   const MEANINGFUL = ['minCount', 'maxCount', 'hasValue', 'in', 'minInclusive',
     'maxInclusive', 'minExclusive', 'maxExclusive', 'pattern', 'flags', 'anyOf'];
-  for (const [i, filter] of spec.filters.entries()) {
-    const c = filter.constraints || {};
+  for (const { what, cond } of conditions(spec)) {
+    const c = cond.constraints || {};
     if (!MEANINGFUL.some(k => c[k] !== undefined && c[k] !== null)) {
-      const p = filter.path.map(h => h.predicate).join(' → ');
-      return `filter ${i + 1} (path ${p}) has no condition. Add one: sh:hasValue for an exact value, sh:in for a list, rep:anyOf for words, or remove the filter.`;
+      const p = cond.path.map(h => h.predicate).join(' → ');
+      return `${what} (path ${p}) has no condition. Add one: sh:hasValue for an exact value, sh:in for a list, rep:anyOf for words, or remove the filter.`;
     }
   }
   return null;
@@ -183,12 +187,12 @@ function filterConstraint(spec) {
 
 // sh:maxCount above 0 needs counting
 function maxCountZero(spec) {
-  for (const [i, filter] of spec.filters.entries()) {
-    const c = filter.constraints;
-    if (c.maxCount !== null && c.maxCount !== undefined && c.maxCount > 0) {
+  for (const { cond } of conditions(spec)) {
+    const c = cond.constraints;
+    if (c.maxCount > 0) {
       return `sh:maxCount ${c.maxCount} needs counting, which this service cannot do. Use sh:maxCount 0 for "has none", or drop it.`;
     }
-    if (c.minCount !== null && c.minCount !== undefined && c.minCount > 2) {
+    if (c.minCount > 2) {
       return `sh:minCount ${c.minCount} needs counting. Use sh:minCount 1 (at least one) or sh:minCount 2 (two distinct values); anything higher is beyond this service.`;
     }
   }
@@ -197,8 +201,8 @@ function maxCountZero(spec) {
 
 // rep:anyOf size limits
 function anyOfLimits(spec) {
-  for (const [i, filter] of spec.filters.entries()) {
-    const c = filter.constraints;
+  for (const { cond } of conditions(spec)) {
+    const c = cond.constraints;
     if (!c.anyOf) continue;
     if (c.anyOf.length > ANYOF_MAX_TERMS) {
       return `rep:anyOf has ${c.anyOf.length} terms. The limit is ${ANYOF_MAX_TERMS}. Narrow the words, or split the filter into two sh:property blocks.`;
@@ -217,8 +221,8 @@ function anyOfLimits(spec) {
 
 // sh:in must not mix URIs and literals
 function inHomogeneous(spec) {
-  for (const filter of spec.filters) {
-    const c = filter.constraints;
+  for (const { cond } of conditions(spec)) {
+    const c = cond.constraints;
     if (!c.in) continue;
     const uris = c.in.filter(t => t.type === 'uri').length;
     if (uris > 0 && uris < c.in.length) {
@@ -231,8 +235,8 @@ function inHomogeneous(spec) {
 // Typed literals must parse
 function literalsParse(spec) {
   const constraintTerms = [];
-  for (const filter of spec.filters) {
-    const c = filter.constraints;
+  for (const { cond } of conditions(spec)) {
+    const c = cond.constraints;
     for (const key of ['hasValue', 'minInclusive', 'maxInclusive', 'minExclusive', 'maxExclusive']) {
       if (c[key]) constraintTerms.push(c[key]);
     }
@@ -250,37 +254,117 @@ function literalsParse(spec) {
   return null;
 }
 
+// rep:where: walks from the row through the profile, shares at least one
+// step with its filter or column, and holds no rep:where of its own.
+function whereConditions(spec, profile) {
+  const startShapeUri = startShape(profile, spec).uri;
+  for (const { what, cond, host } of conditions(spec)) {
+    if (!host) continue;
+    if (cond.where.length) {
+      return `${what} holds a rep:where of its own. Put every rep:where directly on the filter or column.`;
+    }
+    const err = walk(profile, startShapeUri, cond.path, what, false);
+    if (err) return err;
+    const k = sharedSteps(host.path, cond.path);
+    if (!k) {
+      return `${what} shares no step with ${host.what}. Start its path with the same steps as ${host.what}; a condition on the row itself goes in sh:property.`;
+    }
+    const c = cond.constraints;
+    if (k === cond.path.length && (c.maxCount === 0 || c.minCount === 2)) {
+      return `${what} ends on a step of ${host.what}, so ${c.maxCount === 0 ? 'sh:maxCount 0' : 'sh:minCount 2'} has nothing to count. Add the step that must be missing or repeated.`;
+    }
+  }
+  for (const [i, filter] of spec.filters.entries()) {
+    if (filter.where.length && filter.constraints.minCount === 2) {
+      return `filter ${i + 1} combines sh:minCount 2 with rep:where. Use one of the two.`;
+    }
+  }
+  return null;
+}
+
+// A date or number condition compares values. On a link (an entity or a
+// code) it compares a URI and never matches, so its path must end on a field
+// with a datatype.
+function rangeOnValue(spec, profile) {
+  const startShapeUri = startShape(profile, spec).uri;
+  for (const { what, cond } of conditions(spec)) {
+    const key = ['minInclusive', 'maxInclusive', 'minExclusive', 'maxExclusive']
+      .find(k => cond.constraints[k]);
+    if (!key || !cond.path.length) continue;
+    const last = cond.path[cond.path.length - 1];
+    const f = field(profile, shapeAfter(profile, startShapeUri, cond.path), last);
+    if (!f || f.datatype) continue;   // pathsResolve reports a missing step
+    const target = f.node
+      ? `"${profile.shapes.find(s => s.uri === f.node)?.label || f.node}"`
+      : `the code "${f.name}"`;
+    const steps = fieldsOf(profile, f.node)
+      .filter(x => x.datatype === XSD_DATE || x.datatype === XSD_DATETIME || NUMERIC_DATATYPES.has(x.datatype))
+      .map(x => `${x.name} (${x.path})`);
+    const add = steps.length ? `Add a step: ${steps.join(' or ')}.` : 'End the path on a date or number field.';
+    return `${what} ends on ${target}, not on a value, so sh:${key} never matches. ${add}`;
+  }
+  return null;
+}
+
+// Where each rep:where applies, in the profile's words, so the proposal can
+// say it. Profile only, no database.
+export function whereNotes(spec, profile) {
+  const startShapeUri = startShape(profile, spec).uri;
+  return conditions(spec).filter(x => x.host).map(({ what, cond, host }) => {
+    const k = sharedSteps(host.path, cond.path);
+    return `${what} applies at ${nodeName(profile, startShapeUri, host.path.slice(0, k))}, after step ${k} of ${host.what}.`;
+  });
+}
+
+// The entity a path lands on, or the value field when it ends on one.
+function nodeName(profile, startShapeUri, hops) {
+  let shapeUri = startShapeUri;
+  for (const [i, hop] of hops.entries()) {
+    const f = field(profile, shapeUri, hop);
+    if (!f) return `step ${hops.length}`;
+    if (!f.node) return i === hops.length - 1 ? `the value "${f.name}"` : `step ${hops.length}`;
+    shapeUri = f.node;
+  }
+  return `"${profile.shapes.find(s => s.uri === shapeUri)?.label || shapeUri}"`;
+}
+
 function parsesAsDate(value, withTime) {
   const d = new Date(value);
   if (Number.isNaN(d.getTime())) return false;
   return withTime ? value.includes('T') : /^\d{4}-\d{2}-\d{2}$/.test(value);
 }
 
+// In this order: startEntity comes first, the checks after it rely on a
+// start shape.
 const CHECKS = [
-  ['no entity or unknown class', startEntity],
-  ['no columns', columnsPresent],
-  ['column without rdfs:label', columnLabel],
-  ['two columns with one label', columnLabelsUnique],
-  ['column carries constraints', columnPure],
-  ['filter carries rdfs:label', filterLabel],
-  ['filter without a condition', filterConstraint],
-  ['sh:maxCount above 0', maxCountZero],
-  ['rep:anyOf too big', anyOfLimits],
-  ['sh:in mixes URIs and literals', inHomogeneous],
-  ['typed literal does not parse', literalsParse],
-  ['two rep:row columns', oneRowColumn],
-  ['path does not resolve', pathsResolve],
-  ['sh:min/sh:max on a non-number', collectMinMaxTyped],
+  startEntity,
+  columnsPresent,
+  columnLabel,
+  columnLabelsUnique,
+  columnPure,
+  filterLabel,
+  filterConstraint,
+  maxCountZero,
+  anyOfLimits,
+  inHomogeneous,
+  literalsParse,
+  oneRowColumn,
+  pathsResolve,
+  whereConditions,
+  rangeOnValue,
+  collectMinMaxTyped,
 ];
 
-export function checkSpec(spec, profile, maxPathDepth, knownProfiles = null) {
-  const profileErr = profileNamed(spec, knownProfiles);
-  if (profileErr) return [profileErr];
-  for (const [name, check] of CHECKS) {
-    const err = check(spec, profile, maxPathDepth);
-    if (err) return [err];
+// checkSpec(spec, profiles) → the first error, or null.
+export function checkSpec(spec, profiles) {
+  const error = profileError(profiles, spec.profileUri);
+  if (error) return error;
+  const profile = profiles.get(spec.profileUri);
+  for (const check of CHECKS) {
+    const err = check(spec, profile);
+    if (err) return err;
   }
-  return [];
+  return null;
 }
 
 // The shape a path lands in: follow sh:node through every hop but the
@@ -289,17 +373,16 @@ export function checkSpec(spec, profile, maxPathDepth, knownProfiles = null) {
 function shapeAfter(profile, startShapeUri, hops) {
   let shapeUri = startShapeUri;
   for (let i = 0; i < hops.length - 1; i++) {
-    const fields = fieldsOf(profile, shapeUri);
-    const f = fields.find(x => x.path === hops[i].predicate && !!x.inverse === !!hops[i].inverse);
+    const f = field(profile, shapeUri, hops[i]);
     if (!f || !f.node) return null;
     shapeUri = f.node;
   }
   return shapeUri;
 }
 
-function walk(profile, startShapeUri, hops, maxPathDepth, what, isColumn) {
-  if (hops.length > maxPathDepth) {
-    return `${what} walks ${hops.length} hops. The limit is ${maxPathDepth}.`;
+function walk(profile, startShapeUri, hops, what, isColumn, wantsIri = false) {
+  if (hops.length > MAX_PATH_DEPTH) {
+    return `${what} walks ${hops.length} hops. The limit is ${MAX_PATH_DEPTH}.`;
   }
   if (!hops.length) return null;
   let shapeUri = startShapeUri;
@@ -308,7 +391,7 @@ function walk(profile, startShapeUri, hops, maxPathDepth, what, isColumn) {
     const fields = fieldsOf(profile, shapeUri);
     const f = fields.find(x => x.path === hop.predicate && !!x.inverse === !!hop.inverse);
     if (!f) {
-      const have = fields.map(x => (x.inverse ? `geen ${x.name} (${x.path}, omgekeerd)` : `${x.name} (${x.path})`)).join(', ');
+      const have = fields.map(x => (x.inverse ? `${x.name} (${x.path}, inverse)` : `${x.name} (${x.path})`)).join(', ');
       const shapeLabel = profile.shapes.find(s => s.uri === shapeUri)?.label || shapeUri;
       return `at "${shapeLabel}" there is no "${hop.predicate}". It has: ${have || '(nothing)'}.`;
     }
@@ -318,18 +401,16 @@ function walk(profile, startShapeUri, hops, maxPathDepth, what, isColumn) {
       }
       shapeUri = f.node;
     } else if (isColumn) {
-      if (!f.datatype && !f.class) {
+      if (wantsIri && f.datatype) {
+        return `${what} has sh:nodeKind sh:IRI but ends on the value "${f.name}", which is no URI. Drop sh:nodeKind.`;
+      }
+      if (!wantsIri && !f.datatype && !f.class) {
         const valueFields = fieldsOf(profile, f.node)
           .filter(x => x.datatype || x.class)
           .map(x => `${x.name} (${x.path})`);
-        return `column "${colLabel(what)}" ends on a link. Add a hop: ${valueFields.join(' or ')}.`;
+        return `${what} ends on a link. Add a hop: ${valueFields.join(' or ')}. Only if the user asked for the URI of that node itself, add sh:nodeKind sh:IRI to the column instead.`;
       }
     }
   }
   return null;
-}
-
-function colLabel(what) {
-  const m = what.match(/"([^"]+)"/);
-  return m ? m[1] : what;
 }

src/runner/columns.js

@@ -1,4 +1,5 @@
-import { sparqlEscapeUri } from '../db.js';
+import { query, sparqlEscapeUri } from 'mu';
+import { hopsPattern, wherePatterns } from './seed.js';
 
 // Fetch column values for known subjects. Columns are grouped by their
 // shared non-leaf hops; two columns share a query only when every hop
@@ -7,20 +8,15 @@ import { sparqlEscapeUri } from '../db.js';
 
 const SUBJECT_CHUNK_SIZE = Math.max(1, Number(process.env.SUBJECT_CHUNK_SIZE) || 100);
 
-export async function fetchColumns(sessionQuery, subjects, spec, values = new Map(), onProgress = null) {
+// fetchColumns(subjects, spec) → subject → column index → [term]
+export async function fetchColumns(subjects, spec) {
+  const values = new Map();
   const groups = groupColumns(spec.columns);
-  // The subject list is complete and the chunk size is fixed, so the number
-  // of batch queries is known before the first one flies. onProgress fires
-  // after every answered batch with { done, total }.
-  const total = groups.length * Math.ceil(subjects.length / SUBJECT_CHUNK_SIZE);
-  let done = 0;
   for (const group of groups) {
     for (let i = 0; i < subjects.length; i += SUBJECT_CHUNK_SIZE) {
       const chunk = subjects.slice(i, i + SUBJECT_CHUNK_SIZE);
-      const result = await sessionQuery(groupQuery(group, chunk));
+      const result = await query(groupQuery(group, chunk));
       collect(result, group, values);
-      done += 1;
-      onProgress?.({ done, total });
     }
   }
   return values;
@@ -28,14 +24,17 @@ export async function fetchColumns(sessionQuery, subjects, spec, values = new Ma
 
 // Group columns by their shared prefix (every hop before the leaf) and the
 // direction of the leaf. A column with an empty path (rep:self) joins no group.
+// A column with rep:where only shares a group with the same conditions.
 export function groupColumns(columns) {
   const groups = new Map();
   columns.forEach((col, index) => {
     if (!col.path.length) return;   // rep:self
     const leaf = col.path[col.path.length - 1];
-    const key = col.path.map(h => `${h.inverse ? '^' : ''}${h.predicate}`).join(' > ');
+    const where = col.where || [];
+    const key = col.path.map(h => `${h.inverse ? '^' : ''}${h.predicate}`).join(' > ')
+      + (where.length ? ` | ${JSON.stringify(where)}` : '');
     if (!groups.has(key)) {
-      groups.set(key, { hops: col.path, columns: [] });
+      groups.set(key, { hops: col.path, where, columns: [] });
     }
     groups.get(key).columns.push({ index, leaf: leaf.predicate });
   });
@@ -44,18 +43,12 @@ export function groupColumns(columns) {
 
 export function groupQuery(group, chunk) {
   const subjects = chunk.map(sparqlEscapeUri).join(' ');
-  const lines = [];
-  let prev = '?s';
   // every hop except the last gets its own variable; the leaf is ?v
-  group.hops.slice(0, -1).forEach((hop, i) => {
-    const v = `?x${i}`;
-    if (hop.inverse) lines.push(`${v} ${sparqlEscapeUri(hop.predicate)} ${prev} .`);
-    else lines.push(`${prev} ${sparqlEscapeUri(hop.predicate)} ${v} .`);
-    prev = v;
-  });
-  const leaf = group.hops[group.hops.length - 1];
-  if (leaf.inverse) lines.push(`?v ${sparqlEscapeUri(leaf.predicate)} ${prev} .`);
-  else lines.push(`${prev} ${sparqlEscapeUri(leaf.predicate)} ?v .`);
+  const lines = [hopsPattern(group.hops, 'x', '?s', '?v')];
+  // rep:where hangs on the column's own nodes: ?x_<k> after k hops, ?v at
+  // the end.
+  const nodeVar = (k) => (k === group.hops.length ? '?v' : `?x_${k}`);
+  lines.push(...wherePatterns(group.where, group.hops, nodeVar, 'c'));
   return `SELECT ?s ?v WHERE {\n  VALUES ?s { ${subjects} }\n  ${lines.join('\n  ')}\n}`;
 }
 

src/runner/csv.js

@@ -1,35 +0,0 @@
-import { mkdirSync, writeFileSync } from 'fs';
-
-// Rows to a CSV file. CSV_SEPARATOR is ';' by default. A cell is quoted
-// only when it holds the separator, a quote or a newline; a quote inside a
-// cell is doubled, per RFC 4180.
-
-const CSV_SEPARATOR = process.env.CSV_SEPARATOR || ';';
-const SHARE_DIR = process.env.SHARE_DIR || '/share';
-
-export function rowsToCsv(rows) {
-  return rows.map(row => row.map(escapeCell).join(CSV_SEPARATOR)).join('\n');
-}
-
-function escapeCell(cell) {
-  const s = String(cell ?? '');
-  return s.includes(CSV_SEPARATOR) || s.includes('"') || /[\n\r]/.test(s)
-    ? `"${s.replace(/"/g, '""')}"`
-    : s;
-}
-
-export function writeCsv(name, csv) {
-  mkdirSync(SHARE_DIR, { recursive: true });
-  const file = `${SHARE_DIR}/${name}`;
-  writeFileSync(file, csv, 'utf8');
-  return file;
-}
-
-// The spec of the run, next to its CSV (same base name, .ttl), so a report
-// can be debugged afterwards. The store keeps only the pointer.
-export function writeSpec(csvName, spec) {
-  mkdirSync(SHARE_DIR, { recursive: true });
-  const name = csvName.replace(/\.csv$/, '') + '.ttl';
-  writeFileSync(`${SHARE_DIR}/${name}`, spec, 'utf8');
-  return `share://${name}`;
-}

src/runner/profile.js

@@ -1,5 +1,5 @@
 import N3 from 'n3';
-import { readFileSync } from 'fs';
+import { readFileSync, readdirSync } from 'fs';
 
 // Parse a profile (a SHACL shapes graph) and answer questions about it.
 
@@ -8,13 +8,11 @@ const RDF = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#';
 const OWL_ONTOLOGY = 'http://www.w3.org/2002/07/owl#Ontology';
 const REP = 'http://mu.semte.ch/vocabularies/reporting/';
 
-export async function loadProfiles(dir) {
-  const fs = await import('fs');
-  const path = await import('path');
-  const files = fs.readdirSync(dir).filter(f => f.endsWith('.ttl')).sort();
+export function loadProfiles(dir) {
+  const files = readdirSync(dir).filter(f => f.endsWith('.ttl')).sort();
   const profiles = new Map();
   for (const file of files) {
-    const profile = await loadProfile(path.join(dir, file));
+    const profile = loadProfile(`${dir}/${file}`);
     profiles.set(profile.uri, profile);
     console.log(`[profile] "${profile.title}" (${file}): ${profile.shapes.length} entities, ` +
       `${profile.shapes.reduce((n, s) => n + s.fields.length, 0)} fields`);
@@ -23,29 +21,19 @@ export async function loadProfiles(dir) {
   return profiles;
 }
 
-export async function loadProfile(file) {
-  const parser = new N3.Parser();
-  const store = new N3.Store();
-  await new Promise((resolve, reject) => {
-    parser.parse(readFileSync(file, 'utf8'), (err, quad) => {
-      if (err) return reject(new Error(`${file}: ${err.message}`));
-      if (quad) store.addQuad(quad); else resolve();
-    });
-  });
+function loadProfile(file) {
+  let store;
+  try {
+    store = new N3.Store(new N3.Parser().parse(readFileSync(file, 'utf8')));
+  } catch (e) {
+    throw new Error(`${file}: ${e.message}`);
+  }
 
   const ontology = store.getQuads(null, RDF + 'type', OWL_ONTOLOGY)[0];
   if (!ontology) throw new Error(`${file}: no owl:Ontology. A profile file must name itself.`);
   const uri = ontology.subject.value;
   const title = store.getQuads(ontology.subject, 'http://purl.org/dc/terms/title', null)[0]?.object.value || uri;
 
-  const prefixQuads = store.getQuads(null, SH + 'declare', null);
-  const prefixes = {};
-  for (const q of prefixQuads) {
-    const p = store.getQuads(q.object, SH + 'prefix', null)[0]?.object.value;
-    const ns = store.getQuads(q.object, SH + 'namespace', null)[0]?.object.value;
-    if (p && ns) prefixes[p] = ns;
-  }
-
   const shapes = store.getQuads(null, RDF + 'type', SH + 'NodeShape').map(q => q.subject);
   if (!shapes.length) throw new Error(`${file}: no sh:NodeShape. A profile without entities is useless.`);
 
@@ -57,30 +45,26 @@ export async function loadProfile(file) {
     return { uri: shape.value, targetClass, label, fields, discriminator };
   });
 
-  return { uri, title, prefixes, shapes: shapeList, store };
+  return { uri, title, shapes: shapeList };
+}
+
+// A field's sh:path: one predicate, or [ sh:inversePath <predicate> ].
+function readPath(store, node) {
+  const pathNode = store.getQuads(node, SH + 'path', null)[0]?.object;
+  if (!pathNode) return { path: null, inverse: false };
+  if (pathNode.termType !== 'BlankNode') return { path: pathNode.value, inverse: false };
+  const inv = store.getQuads(pathNode, SH + 'inversePath', null)[0]?.object;
+  return inv ? { path: inv.value, inverse: true } : { path: null, inverse: false };
 }
 
 function parseField(store, p) {
-  let path = null, inverse = false;
-  const pathQuad = store.getQuads(p, SH + 'path', null)[0]?.object;
-  if (pathQuad) {
-    if (pathQuad.termType === 'BlankNode') {
-      const inv = store.getQuads(pathQuad, SH + 'inversePath', null)[0]?.object;
-      if (inv) { path = inv.value; inverse = true; }
-    } else {
-      path = pathQuad.value;
-    }
-  }
-  const one = (pred) => store.getQuads(p, pred, null)[0]?.object || null;
+  const one = (pred) => store.getQuads(p, pred, null)[0]?.object?.value || null;
   return {
-    path, inverse,
-    name: one(SH + 'name')?.value || null,
-    description: one(SH + 'description')?.value || null,
-    datatype: one(SH + 'datatype')?.value || null,
-    class: one(SH + 'class')?.value || null,
-    node: one(SH + 'node')?.value || null,
-    nodeKind: one(SH + 'nodeKind')?.value || null,
-    maxCount: one(SH + 'maxCount')?.value || null,
+    ...readPath(store, p),
+    name: one(SH + 'name'),
+    datatype: one(SH + 'datatype'),
+    class: one(SH + 'class'),
+    node: one(SH + 'node'),
   };
 }
 
@@ -89,16 +73,7 @@ function parseField(store, p) {
 function parseDiscriminator(store, shape) {
   const node = store.getQuads(shape, REP + 'discriminator', null)[0]?.object;
   if (!node) return null;
-  let path = null, inverse = false;
-  const pathQuad = store.getQuads(node, SH + 'path', null)[0]?.object;
-  if (pathQuad) {
-    if (pathQuad.termType === 'BlankNode') {
-      const inv = store.getQuads(pathQuad, SH + 'inversePath', null)[0]?.object;
-      if (inv) { path = inv.value; inverse = true; }
-    } else {
-      path = pathQuad.value;
-    }
-  }
+  const { path, inverse } = readPath(store, node);
   const one = (pred) => store.getQuads(node, pred, null)[0]?.object?.value || null;
   const minCount = one(SH + 'minCount');
   const maxCount = one(SH + 'maxCount');
@@ -110,11 +85,6 @@ function parseDiscriminator(store, shape) {
   };
 }
 
-export function entityForClass(profile, cls) {
-  const shape = profile.shapes.find(s => s.targetClass === cls);
-  return shape ? shape.uri : null;
-}
-
 // The shape a spec starts from. rep:entity wins; a targetClass only resolves
 // when exactly one shape carries it, so a class two entities share is a
 // validator error, not a silent first match.
@@ -132,10 +102,7 @@ export function fieldsOf(profile, shapeUri) {
   return shape(profile, shapeUri)?.fields || [];
 }
 
-export function field(profile, shapeUri, predicate, inverse) {
-  return fieldsOf(profile, shapeUri).find(f => f.path === predicate && !!f.inverse === !!inverse) || null;
+// The field a hop takes from a shape, or null.
+export function field(profile, shapeUri, hop) {
+  return fieldsOf(profile, shapeUri).find(f => f.path === hop.predicate && !!f.inverse === !!hop.inverse) || null;
 }
-
-export function prefixes(profile) {
-  return profile.prefixes;
-}

src/runner/report.js

@@ -1,61 +0,0 @@
-import { randomUUID } from 'crypto';
-import { statSync } from 'fs';
-import { sessionUpdate, sparqlEscapeUri, sparqlEscapeString, sparqlEscapeInt,
-         sparqlEscapeDateTime } from '../db.js';
-
-const REPORT_CLASS = process.env.REPORT_CLASS || 'http://lblod.data.gift/vocabularies/reporting/Report';
-
-// Register the CSV: a logical file (http://data.lblod.info/files/<uuid>) plus
-// a physical share:// file pointing back with nie:dataSource. The logical
-// URI is what you link to.
-export async function registerFile(sessionUpdateFn, fileName, filePath) {
-  const size = statSync(filePath).size;
-  const uuid = randomUUID();
-  const uuid2 = randomUUID();
-  const now = new Date();
-  const logicalUri = `http://data.lblod.info/files/${uuid}`;
-  const physicalUri = `share://${fileName}`;
-
-  await (sessionUpdateFn || sessionUpdate)(`
-    INSERT DATA {
-      ${sparqlEscapeUri(logicalUri)} a ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject')} ;
-        ${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(uuid)} ;
-        ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName')} ${sparqlEscapeString(fileName)} ;
-        ${sparqlEscapeUri('http://purl.org/dc/terms/format')} ${sparqlEscapeString('text/csv')} ;
-        ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize')} ${sparqlEscapeInt(size)} ;
-        ${sparqlEscapeUri('http://purl.org/dc/terms/created')} ${sparqlEscapeDateTime(now)} .
-      ${sparqlEscapeUri(physicalUri)} a ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject')} ;
-        ${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(uuid2)} ;
-        ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName')} ${sparqlEscapeString(fileName)} ;
-        ${sparqlEscapeUri('http://purl.org/dc/terms/format')} ${sparqlEscapeString('text/csv')} ;
-        ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize')} ${sparqlEscapeInt(size)} ;
-        ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileCreated')} ${sparqlEscapeDateTime(now)} ;
-        ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource')} ${sparqlEscapeUri(logicalUri)} .
-    }`);
-
-  return logicalUri;
-}
-
-// One report pointing at the logical file URI.
-export async function registerReport(sessionUpdateFn, title, fileUri, extra = {}) {
-  const uuid = randomUUID();
-  const now = new Date();
-  const reportUri = `http://data.lblod.info/id/reports/${uuid}`;
-
-  const extraTriples = [];
-  if (extra.question) extraTriples.push(`${sparqlEscapeUri(reportUri)} ${sparqlEscapeUri('http://mu.semte.ch/vocabularies/ext/question')} ${sparqlEscapeString(extra.question)} .`);
-  if (extra.specFile) extraTriples.push(`${sparqlEscapeUri(reportUri)} ${sparqlEscapeUri('http://mu.semte.ch/vocabularies/ext/specFile')} ${sparqlEscapeString(extra.specFile)} .`);
-  if (extra.creator) extraTriples.push(`${sparqlEscapeUri(reportUri)} ${sparqlEscapeUri('http://purl.org/dc/terms/creator')} ${sparqlEscapeUri(extra.creator)} .`);
-
-  await (sessionUpdateFn || sessionUpdate)(`
-    INSERT DATA {
-      ${sparqlEscapeUri(reportUri)} a ${sparqlEscapeUri(REPORT_CLASS)} ;
-        ${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(uuid)} ;
-        ${sparqlEscapeUri('http://purl.org/dc/terms/title')} ${sparqlEscapeString(title)} ;
-        ${sparqlEscapeUri('http://purl.org/dc/terms/created')} ${sparqlEscapeDateTime(now)} ;
-        ${sparqlEscapeUri('http://www.w3.org/ns/prov#generated')} ${sparqlEscapeUri(fileUri)} .
-      ${extraTriples.join('\n      ')}
-    }`);
-
-  return reportUri;
-}

src/runner/run.js

@@ -1,42 +1,49 @@
+import { update, uuid, sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime } from 'mu';
 import { seed } from './seed.js';
 import { fetchColumns } from './columns.js';
 import { assemble } from './assemble.js';
-import { rowsToCsv, writeCsv, writeSpec } from './csv.js';
-import { registerFile, registerReport } from './report.js';
 import { startShape } from './profile.js';
 
-// Ties the steps together: seed → columns → assemble → CSV → register.
-// Returns { reportUri, fileUri, rowCount }. onProgress fires on every
-// answered query with what the phase knows — { subjects } per seed page,
-// { done, total } per column batch — so a waiter can tell a live run from
-// a stalled one.
+const REPORT_CLASS = process.env.REPORT_CLASS || 'http://lblod.data.gift/vocabularies/reporting/Report';
+const CSV_SEPARATOR = process.env.CSV_SEPARATOR || ';';
 
-export async function run(query, update, parsed, profile, title, extra = {}, onProgress = null) {
-  const shape = startShape(profile, parsed);
-  const subjects = await seed(query, parsed, shape,
-    onProgress ? (n) => onProgress({ subjects: n }) : null);
-
-  const values = new Map();
-  await fetchColumns(query, subjects, parsed, values, null, onProgress);
-
-  const rows = assemble(subjects, values, parsed);
-  const fileName = `${slug(title)}.csv`;
-  const csv = rowsToCsv(rows);
-  const filePath = writeCsv(fileName, csv);
-  const specFile = extra.spec && writeSpec(fileName, extra.spec);
-
-  const fileUri = await registerFile(update, fileName, filePath);
-  const reportUri = await registerReport(update, title, fileUri,
-    specFile ? { ...extra, specFile } : extra);
-
-  return { reportUri, fileUri, rowCount: rows.length - 1, filePath };
+// run(spec, profile) → the CSV text. seed → columns → assemble → CSV.
+export async function run(spec, profile) {
+  const subjects = await seed(spec, startShape(profile, spec));
+  const values = await fetchColumns(subjects, spec);
+  const rows = assemble(subjects, values, spec);
+  return rows.map(row => row.map(csvCell).join(CSV_SEPARATOR)).join('\n');
+}
+
+// A cell is quoted only when it holds the separator, a quote or a newline;
+// a quote inside a cell is doubled, per RFC 4180.
+function csvCell(cell) {
+  const s = String(cell ?? '');
+  return s.includes(CSV_SEPARATOR) || s.includes('"') || /[\n\r]/.test(s)
+    ? `"${s.replace(/"/g, '""')}"`
+    : s;
+}
+
+// One report pointing at the logical file URI of its CSV.
+export async function registerReport(title, fileUri) {
+  const id = uuid();
+  const reportUri = `http://data.lblod.info/id/reports/${id}`;
+
+  await update(`
+    INSERT DATA {
+      ${sparqlEscapeUri(reportUri)} a ${sparqlEscapeUri(REPORT_CLASS)} ;
+        ${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(id)} ;
+        ${sparqlEscapeUri('http://purl.org/dc/terms/title')} ${sparqlEscapeString(title)} ;
+        ${sparqlEscapeUri('http://purl.org/dc/terms/created')} ${sparqlEscapeDateTime(new Date())} ;
+        ${sparqlEscapeUri('http://www.w3.org/ns/prov#generated')} ${sparqlEscapeUri(fileUri)} .
+    }`);
+
+  return reportUri;
 }
 
-// The file name must be known before the run starts, so the chat can show a
-// pending card named after the report.
 export function slug(title) {
   return (title || 'report').toLowerCase()
     .replace(/[^a-z0-9]+/g, '-')
     .replace(/^-+|-+$/g, '')
     .slice(0, 80) || 'report';
-}
+}

src/runner/seed.js

@@ -1,20 +1,17 @@
-import { sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime, sparqlEscapeDate } from '../db.js';
+import { query, sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime, sparqlEscapeDate } from 'mu';
+import { sharedSteps } from './spec.js';
 
 // Subject selection. One paged query per page; stops at ROW_LIMIT.
-// onPage(n) fires after every answered page with the subject count so far,
-// so a caller waiting on this run can tell a live one from a stalled one.
 const ROW_LIMIT = Number(process.env.ROW_LIMIT || 200000);
 const PAGE_SIZE = Number(process.env.SEED_PAGE_SIZE || 5000);
 
-export async function seed(sessionQuery, spec, shape = null, onPage = null) {
+export async function seed(spec, shape) {
   const subjects = [];
   let offset = 0;
   for (;;) {
-    const query = seedPageQuery(spec, PAGE_SIZE, offset, shape);
-    const result = await sessionQuery(query);
+    const result = await query(seedPageQuery(spec, PAGE_SIZE, offset, shape));
     const rows = result.results.bindings.map(b => b.s.value);
     subjects.push(...rows);
-    onPage?.(subjects.length);
     if (subjects.length > ROW_LIMIT) {
       throw new Error(`the question matches more than ${ROW_LIMIT} subjects. Add a filter, for example a date range or a bestuurseenheid.`);
     }
@@ -30,8 +27,7 @@ export function seedPageQuery(spec, limit, offset, shape = null) {
   parts.push(`?s a ${sparqlEscapeUri(targetClass)} .`);
   if (shape?.discriminator) parts.push(discriminatorPattern(shape.discriminator));
   spec.filters.forEach((filter, fi) => {
-    parts.push(hopsPattern(filter.path, `f${fi}`, '?s'));
-    parts.push(constraintsPattern(filter.constraints, `f${fi}`));
+    parts.push(filterPattern(filter, `f${fi}`));
   });
   return `SELECT DISTINCT ?s WHERE {
   ${parts.join('\n  ')}
@@ -52,13 +48,13 @@ function discriminatorPattern(d) {
 // Writes out every hop as its own triple pattern.
 // Forward:  ?s <p1> ?f0_1 . ?f0_1 <p2> ?f0_v
 // Inverse:  ?s ^<p1> → ?f0_1 <p1> ?s
-export function hopsPattern(hops, varPrefix, startVar) {
+export function hopsPattern(hops, varPrefix, startVar, lastVar = `?${varPrefix}_v`) {
   let prev = startVar;
   const patterns = [];
   for (let i = 0; i < hops.length; i++) {
     const hop = hops[i];
     const isLast = i === hops.length - 1;
-    const v = isLast ? `?${varPrefix}_v` : `?${varPrefix}_${i + 1}`;
+    const v = isLast ? lastVar : `?${varPrefix}_${i + 1}`;
     if (hop.inverse) {
       patterns.push(`${v} ${sparqlEscapeUri(hop.predicate)} ${prev} .`);
     } else {
@@ -69,9 +65,37 @@ export function hopsPattern(hops, varPrefix, startVar) {
   return patterns.join('\n  ');
 }
 
-export function constraintsPattern(c, varPrefix) {
-  const parts = [];
-  const v = `?${varPrefix}_v`;
+// One filter: its condition, plus its rep:where conditions on the nodes it
+// walks through. With sh:maxCount 0 the whole of it becomes "none such".
+function filterPattern(filter, varPrefix) {
+  const nodeVar = (k) => (k === filter.path.length ? `?${varPrefix}_v` : `?${varPrefix}_${k}`);
+  const wheres = wherePatterns(filter.where, filter.path, nodeVar, varPrefix);
+  if (filter.constraints.maxCount === 0) {
+    return `FILTER NOT EXISTS {\n    ${[hopsPattern(filter.path, varPrefix, '?s'), ...wheres].join('\n    ')}\n  }`;
+  }
+  return [conditionPattern(filter.constraints, varPrefix, filter.path, '?s'), ...wheres].join('\n  ');
+}
+
+// The rep:where conditions of a filter or column. Each starts at the node
+// after the steps it shares with its host; nodeVar(k) names that node in the
+// host's own query.
+export function wherePatterns(where, hostHops, nodeVar, varPrefix) {
+  return (where || []).map((w, wi) => {
+    const k = sharedSteps(hostHops, w.path);
+    return conditionPattern(w.constraints, `${varPrefix}_w${wi}`, w.path.slice(k), nodeVar(k));
+  });
+}
+
+// A condition: the hops from startVar and the test on where they end. A
+// filter starts at ?s; a rep:where starts at a node of its host and, with no
+// hops left, tests that node itself. sh:maxCount 0 means "none such", so its
+// hops only live inside the NOT EXISTS.
+export function conditionPattern(c, varPrefix, hops, startVar) {
+  if (c.maxCount === 0) {
+    return `FILTER NOT EXISTS { ${hopsPattern(hops, varPrefix + '_nex', startVar)} }`;
+  }
+  const v = hops.length ? `?${varPrefix}_v` : startVar;
+  const parts = hops.length ? [hopsPattern(hops, varPrefix, startVar)] : [];
 
   if (c.hasValue) parts.push(`FILTER(${v} = ${escapeTerm(c.hasValue)})`);
   if (c.in) parts.push(`FILTER(${v} IN (${c.in.map(escapeTerm).join(', ')}))`);
@@ -89,33 +113,14 @@ export function constraintsPattern(c, varPrefix) {
     parts.push(`FILTER(REGEX(str(${v}), ${sparqlEscapeString(alternation)}${flags}))`);
   }
   if (c.minCount === 2) {
-    // two distinct leaf values on the same path
+    // two distinct leaf values: the same path again, to a second leaf
     const w = `?${varPrefix}_w`;
-    parts.push(hopsPattern2(c, varPrefix, w));
+    parts.push(hopsPattern(hops, `${varPrefix}_2`, startVar, w));
     parts.push(`FILTER(${v} != ${w})`);
   }
-  if (c.maxCount === 0) {
-    parts.push(`FILTER NOT EXISTS { ${hopsPattern(c.path, varPrefix + '_nex', '?s')} }`);
-  }
   return parts.join('\n  ');
 }
 
-function hopsPattern2(filter, varPrefix, wVar) {
-  // repeat the path to a second leaf variable
-  const hops = filter.path.map(h => ({ ...h }));
-  let prev = '?s';
-  const patterns = [];
-  for (let i = 0; i < hops.length; i++) {
-    const hop = hops[i];
-    const isLast = i === hops.length - 1;
-    const v = isLast ? wVar : `?${varPrefix}_2_${i + 1}`;
-    if (hop.inverse) patterns.push(`${v} ${sparqlEscapeUri(hop.predicate)} ${prev} .`);
-    else patterns.push(`${prev} ${sparqlEscapeUri(hop.predicate)} ${v} .`);
-    prev = v;
-  }
-  return patterns.join('\n  ');
-}
-
 function escapeTerm(t) {
   if (t.type === 'uri') return sparqlEscapeUri(t.value);
   if (t.datatype === 'http://www.w3.org/2001/XMLSchema#dateTime')

src/runner/spec.js

@@ -23,35 +23,39 @@ export function parseSpec(turtle) {
   const profileUri = one(specNode, REP + 'profile')?.value || null;
   const title = one(specNode, 'http://purl.org/dc/terms/title')?.value || null;
 
-  const filters = store.getQuads(specNode, SH + 'property', null).map(q =>
-    parseFilter(store, q.object, { withLabel: true }));
+  const filters = store.getQuads(specNode, SH + 'property', null).map(q => parseFilter(store, q.object));
 
   const columnsQuad = one(specNode, REP + 'columns');
-  const columns = columnsQuad ? parseColumnList(store, columnsQuad) : [];
+  const columns = columnsQuad ? readRdfList(store, columnsQuad).map(col => parseColumn(store, col)) : [];
 
   return { uri: specNode.value, title, profileUri, targetClass, entity, filters, columns };
 }
 
-function parseFilter(store, p, { withLabel = false } = {}) {
+function parseFilter(store, p) {
   let path = [];
   const pathQuad = store.getQuads(p, SH + 'path', null)[0]?.object;
   if (pathQuad) path = parsePath(store, pathQuad);
   return {
     path,
-    constraints: readConstraints(store, p, { withLabel }),
+    constraints: readConstraints(store, p, { withLabel: true }),
+    where: readWhere(store, p),
   };
 }
 
-function parseColumnList(store, listNode) {
-  const columns = [];
-  let cur = listNode;
-  while (cur && cur.value !== RDF + 'nil') {
-    const item = store.getQuads(cur, RDF + 'first', null)[0]?.object;
-    if (item) columns.push(parseColumn(store, item));
-    cur = store.getQuads(cur, RDF + 'rest', null)[0]?.object;
-    if (!cur) throw new Error('rep:columns list is broken (no rdf:rest terminator)');
-  }
-  return columns;
+// rep:where: conditions on the nodes a filter or column walks through. Each
+// is written like a filter, from the row; check.js refuses one nested in
+// another.
+function readWhere(store, node) {
+  return store.getQuads(node, REP + 'where', null).map(q => parseFilter(store, q.object));
+}
+
+// The number of steps two paths have in common from the row. A rep:where
+// shares those nodes with its filter or column.
+export function sharedSteps(a, b) {
+  let k = 0;
+  while (k < a.length && k < b.length
+    && a[k].predicate === b[k].predicate && !!a[k].inverse === !!b[k].inverse) k++;
+  return k;
 }
 
 function parseColumn(store, col) {
@@ -69,7 +73,10 @@ function parseColumn(store, col) {
     label: store.getQuads(col, RDFS_LABEL, null)[0]?.object.value || null,
     collect: store.getQuads(col, REP + 'collect', null)[0]?.object.value || null,
     separator: store.getQuads(col, SH + 'separator', null)[0]?.object.value || null,
+    // sh:nodeKind sh:IRI: the column shows the URI of the node it ends on
+    nodeKind: store.getQuads(col, SH + 'nodeKind', null)[0]?.object.value || null,
     constraints: readConstraints(store, col),   // kept so check.js can refuse it
+    where: readWhere(store, col),
   };
 }
 
@@ -86,21 +93,12 @@ function parsePath(store, pathQuad) {
     return [{ predicate: pathQuad.value, inverse: false }];
   }
   // RDF list of predicates
-  const hops = [];
-  let cur = pathQuad;
-  while (cur && cur.value !== RDF + 'nil') {
-    const item = store.getQuads(cur, RDF + 'first', null)[0]?.object;
-    if (!item) throw new Error('sh:path list is broken');
-    if (item.termType === 'BlankNode') {
-      const inv = store.getQuads(item, SH + 'inversePath', null)[0]?.object;
-      if (!inv) throw new Error('sh:path list items must be URIs or [ sh:inversePath <predicate> ]');
-      hops.push({ predicate: inv.value, inverse: true });
-    } else {
-      hops.push({ predicate: item.value, inverse: false });
-    }
-    cur = store.getQuads(cur, RDF + 'rest', null)[0]?.object;
-  }
-  return hops;
+  return readRdfList(store, pathQuad).map(item => {
+    if (item.termType !== 'BlankNode') return { predicate: item.value, inverse: false };
+    const inv = store.getQuads(item, SH + 'inversePath', null)[0]?.object;
+    if (!inv) throw new Error('sh:path list items must be URIs or [ sh:inversePath <predicate> ]');
+    return { predicate: inv.value, inverse: true };
+  });
 }
 
 function readConstraints(store, p, { withLabel = false } = {}) {