felix.ruizdearcaute/natural-language-report-service
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

remove the tools

Browse Source
This commit is contained in:
felix
2026-09-27 14:36:20 +02:00
parent 78a7c42a37
commit 0802b75346
5 changed files with 34 additions and 205 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
README.md
View File

@@ -16,11 +16,9 @@ not. The answer picks one of two deterministic code paths.
**Refinement (Modus A).** The LLM proposes a spec and explains in Dutch what
it will list and filter. It may call `lookup_values` to search code lists for
candidate values, so its filter suggestions name real values instead of
guesses, `explore_data` to see the predicates, values and links of a target
class, and `sample_spec` to see a few rows a draft spec finds. All three read
the public graph only, under the service scope. The LLM never runs a report
and never learns what the report would hold: the report runs as the user, on
data the LLM may not see, so a sample is a hint, not a check. The user must confirm before anything executes.
guesses. It never runs a report and never learns what a spec would match: the
database it reads may differ from the one the report runs on, so a lookup is
a suggestion, not a check. The user must confirm before anything executes.
As long as the user has not given a clear command, the turn stays in this
mode.
@@ -32,8 +30,7 @@ that no longer checks out, or a failed run each give a fixed Dutch failure
message. Nothing is retried and no spec is rewritten.
Only refinement has tools: `list_profiles`, `describe_profile`,
`validate_spec`, `read_spec`, `lookup_values`, `explore_data` and
`sample_spec`.
`validate_spec`, `read_spec` and `lookup_values`.
## How it works
@@ -82,20 +79,19 @@ never a cell value.
No report row, row count or run error ever reaches the LLM. A run starts
after the turn's only LLM call (the mode check) and answers with fixed text.
What the LLM does see: the conversation, the profiles, validator messages,
spec files, and public data read under the service scope (inlined code
lists, `lookup_values`, `explore_data` and `sample_spec`, see below).
spec files, and public reference data read under the service scope (inlined
code lists and `lookup_values`, see below).
Keep it that way. Every database read the LLM drives goes through
`publicQuery` (the service scope), never the caller's session, and none of
them counts. `sample_spec` runs a draft spec's queries on the public graph
only; it is not the report. `validate_spec` still says where each
Keep it that way. Do not add a tool that previews, samples or counts what a
spec would match, not even to catch a `rep:where` that hangs on the wrong
node. That check stays on the profile: `validate_spec` says where each
`rep:where` applies, and the proposal repeats it to the user.
### The service scope
The LLM's own reads during refinement (`describe_profile`'s code lists,
`lookup_values`, `explore_data` and `sample_spec`) run under a service
scope, not the caller's session: `publicQuery` in `src/llm/explore.js`. mu's
The LLM's own reads during refinement (`describe_profile`'s code lists and
`lookup_values`) run under a service scope, not the caller's session:
`publicQuery` in `src/llm/explore.js`. mu's
`query(q, { scope })` sends `mu-auth-scope`; the sparql-parser config grants
that scope read access to `http://mu.semte.ch/graphs/public` only. So
refinement reads only the public graph, regardless of who calls. Execution
@@ -152,7 +148,7 @@ user sees is Dutch.
| `SHARE_DIR` | `/share` | where CSVs are written |
| `CHAT_ASSISTANT_URI` | `http://data.lblod.info/id/chat-agents/rapportassistent` | the `prov:SoftwareAgent` that signs assistant messages (the portal's chat-assistant migration seeds it) |
| `CHAT_HISTORY_LIMIT` | `20` | messages of history a turn reads |
| `SERVICE_SCOPE` | `http://services.semantic.works/natural-language-report` | the mu-auth-scope for the LLM's own reads (code lists, `lookup_values`, `explore_data`, `sample_spec`). Must match the `with-scope` grant in the app's sparql-parser config. |
| `SERVICE_SCOPE` | `http://services.semantic.works/natural-language-report` | the mu-auth-scope for the LLM's own reads (code lists, `lookup_values`). Must match the `with-scope` grant in the app's sparql-parser config. |
Do not set `ALLOW_MU_AUTH_SUDO`; the template then refuses sudo queries.

144
src/llm/explore.js
View File

@@ -1,30 +1,20 @@
// Everything the LLM reads from the database in refine mode. Every query
// goes through publicQuery: the service scope, not the caller's session.
// The sparql-parser config grants that scope read access to the public
// graph only, so these reads are the same for every caller. Nothing here
// counts, and execution still runs as the caller.
// - code lists: inlined in describe_profile, searched by lookup_values;
// - explore_data: the predicates, values and links of one target class;
// - sample_spec: the queries of a draft spec, for a few subjects.
// A suggestion, not a check: the report runs as the user and may find other
// data.
import { query, sparqlEscapeUri, sparqlEscapeString } from 'mu';
import { startShape } from '../runner/profile.js';
import { seedPageQuery } from '../runner/seed.js';
import { groupColumns, groupQuery, collect } from '../runner/columns.js';
import { assemble } from '../runner/assemble.js';
// Everything the LLM reads from the database in refine mode: the code
// lists, inlined in describe_profile and searched by lookup_values. Every
// query goes through publicQuery: the service scope, not the caller's
// session. The sparql-parser config grants that scope read access to the
// public graph only, so these reads are the same for every caller. Nothing
// here counts, and execution still runs as the caller.
//
// lookup_values is a suggest helper, not a spec check. It reads candidate
// values the LLM can put in a filter; it never runs the spec and never sees
// report rows.
import { query, sparqlEscapeString } from 'mu';
// Must match the with-scope grant in the app's config.lisp.
const SERVICE_SCOPE = process.env.SERVICE_SCOPE || 'http://services.semantic.works/natural-language-report';
export const INLINE_VALUES_MAX = Number(process.env.INLINE_VALUES_MAX || 50);
const VALUES_TTL = Number(process.env.VALUES_TTL || 3600) * 1000;
const LOOKUP_LIMIT = 25;
const SAMPLE_SIZE = 10;
const MAX_ROWS = 30;
const EXAMPLES = 3;
const MAX_VALUE_LENGTH = 60;
const RDF_TYPE = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type';
const XSD = 'http://www.w3.org/2001/XMLSchema#';
// class → { values, at }. Refreshed lazily when stale.
const cache = new Map();
@@ -147,115 +137,3 @@ function shortType(cls) {
function escapeRegex(s) {
return String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
}
// exploreClass(profile, cls) → text for the LLM. cls must be a
// sh:targetClass of the profile.
export async function exploreClass(profile, cls) {
const shapes = profile.shapes.filter(s => s.targetClass === cls);
if (!shapes.length) {
const classes = [...new Set(profile.shapes.map(s => s.targetClass).filter(Boolean))];
return `<${cls}> is no sh:targetClass in "${profile.title}". You can explore: ${classes.map(c => `<${c}>`).join(', ')}.`;
}
const sample = await publicQuery(`SELECT DISTINCT ?s WHERE {
?s a ${sparqlEscapeUri(cls)} .
} LIMIT ${SAMPLE_SIZE}`);
const subjects = sample.results.bindings.map(b => sparqlEscapeUri(b.s.value));
if (!subjects.length) {
return `no <${cls}> in the public data. That says nothing about the report: it runs as the user.`;
}
const outgoing = await publicQuery(`SELECT DISTINCT ?p ?o ?type WHERE {
VALUES ?s { ${subjects.join(' ')} }
?s ?p ?o .
OPTIONAL { ?o a ?type . }
}`);
const incoming = await publicQuery(`SELECT DISTINCT ?p ?type WHERE {
VALUES ?s { ${subjects.join(' ')} }
?x ?p ?s .
OPTIONAL { ?x a ?type . }
}`);
const lines = [`A sample of ${subjects.length} <${cls}> from the public data. It shows what the data looks like;
it is no check of a spec, and the report may run on other data.
Out = the instance points to it, In = something points to the instance.
→ / ← give the class of the linked node: explore it to follow the path.
A predicate "not in the profile" cannot be used in a spec.
`];
const out = new Map();
for (const b of outgoing.results.bindings) {
if (b.p.value === RDF_TYPE) continue;
const entry = out.get(b.p.value) || { examples: new Set(), types: new Set() };
if (entry.examples.size < EXAMPLES) entry.examples.add(showValue(b.o));
if (b.type) entry.types.add(`<${b.type.value}>`);
out.set(b.p.value, entry);
}
for (const [p, { examples, types }] of out) {
const linked = types.size ? ` → ${[...types].join(', ')}` : '';
lines.push(`Out <${p}> ${fieldName(shapes, p, false)}${linked} e.g. ${[...examples].join(' | ')}`);
}
const inc = new Map();
for (const b of incoming.results.bindings) {
const types = inc.get(b.p.value) || new Set();
if (b.type) types.add(`<${b.type.value}>`);
inc.set(b.p.value, types);
}
for (const [p, types] of inc) {
const linked = types.size ? ` ← ${[...types].join(', ')}` : '';
lines.push(`In <${p}> ${fieldName(shapes, p, true)}${linked}`);
}
return lines.join('\n');
}
// sampleSpec(spec, profile) → text for the LLM: the rows of up to
// SAMPLE_SIZE subjects the spec finds in the public data. The spec has
// passed checkSpec. Same query builders as the run, other identity.
export async function sampleSpec(spec, profile) {
const seed = await publicQuery(seedPageQuery(spec, SAMPLE_SIZE, 0, startShape(profile, spec)));
const subjects = seed.results.bindings.map(b => b.s.value);
if (!subjects.length) {
return 'the spec finds nothing in the public data. That says nothing about the report: it runs as the user, on data you cannot see.';
}
const values = new Map();
for (const group of groupColumns(spec.columns)) {
collect(await publicQuery(groupQuery(group, subjects)), group, values);
}
const [header, ...rows] = assemble(subjects, values, spec);
const table = [header.join(' | '), ...rows.slice(0, MAX_ROWS).map(row => row.map(showCell).join(' | '))];
return `A sample from the public data: rows for up to ${SAMPLE_SIZE} subjects the spec finds there.
The report runs as the user and may find other rows. Use it to see whether paths and filters
land where you expect, not to tell the user what the report will hold.
${table.join('\n')}`;
}
function showCell(cell) {
const text = String(cell ?? '');
return text.length > MAX_VALUE_LENGTH ? `${text.slice(0, MAX_VALUE_LENGTH)}…` : text;
}
// The profile field a predicate is, on any entity with this class.
function fieldName(shapes, predicate, inverse) {
const names = [];
for (const shape of shapes) {
const f = shape.fields.find(x => x.path === predicate && !!x.inverse === inverse);
if (f) names.push(shapes.length > 1 ? `"${f.name}" of ${shape.label}` : `"${f.name}"`);
}
return names.length ? `field ${names.join(', ')}${inverse ? ' (inverse)' : ''}` : 'not in the profile';
}
// A SPARQL JSON term as the LLM would write it in Turtle.
function showValue(term) {
if (term.type === 'uri') return `<${term.value}>`;
if (term.type === 'bnode') return '[blank node]';
const text = term.value.length > MAX_VALUE_LENGTH ? `${term.value.slice(0, MAX_VALUE_LENGTH)}…` : term.value;
if (term['xml:lang']) return `${JSON.stringify(text)}@${term['xml:lang']}`;
if (!term.datatype) return JSON.stringify(text);
const datatype = term.datatype.startsWith(XSD) ? term.datatype.replace(XSD, 'xsd:') : `<${term.datatype}>`;
return `${JSON.stringify(text)}^^${datatype}`;
}

24
src/llm/loop.js
View File

@@ -3,9 +3,8 @@
// SDKs. Stops when the model answers without tool calls.
//
// Refinement only: the LLM proposes a spec, may look up candidate values in
// code lists, explore the data and sample a draft spec (all on the public
// graph, under the service scope) and iterates with the user. It never
// runs a report and never learns what the report would hold. The user must
// code lists (under the service scope) and iterates with the user. It never
// runs a report and never learns what a spec would match. The user must
// confirm before anything executes; execution runs the agreed spec without
// the LLM (../report-assistant.js).
@@ -20,23 +19,10 @@ const LLM_API_KEY = process.env.LLM_API_KEY;
const LOG_LLM = /^(true|1)$/i.test(process.env.LOG_LLM || '');
// Refinement (Modus A). The LLM and the user iterate on a spec. The LLM may
// look up candidate values in code lists (lookup_values), explore the data
// of a target class (explore_data) and sample a draft spec (sample_spec),
// all on the public graph under the service scope, but it may not run a
// report and never learns how many subjects the report would match. It ends with a proposal in
// look up candidate values in code lists (lookup_values, under the service
// scope) to make concrete suggestions, but it may not run a report and never
// learns how many subjects a spec would match. It ends with a proposal in
// Dutch and asks the user to confirm.
// explore_data and sample_spec are switched off for now (see tools.js). To
// switch them back on, replace "That is the only database read ... as a
// check." in REFINE_PROMPT with:
// Use explore_data to see what the data of a target class holds: its
// predicates, example values and links; call it on a linked class to follow
// a path. A predicate that is not in the profile cannot go in a spec. Use
// sample_spec on a spec that validates to see a few of its rows, and fix
// paths or filters that do not land where you expect. All three read only
// public data. The report runs as the user, on data you may not see, so a
// lookup or a sample is a hint, never a count and never a promise: do not
// tell the user how many rows the report will have, or that it will be
// empty.
const REFINE_PROMPT = `You refine report specs with the user. You never write SPARQL and you never run a report.
The profile lists what can be asked. Compose paths by chaining fields.

39
src/llm/tools.js
View File

@@ -1,11 +1,11 @@
// The tools the LLM calls, refinement only: list_profiles,
// describe_profile, validate_spec, read_spec, lookup_values, explore_data
// and sample_spec. Each returns plain text for the model. No report is
// created; the database reads run under the service scope (public graph).
// describe_profile, validate_spec, read_spec and lookup_values. Each returns
// plain text for the model. No report is created, no spec is evaluated; the
// database reads run under the service scope (public graph).
import { parseSpec } from '../runner/spec.js';
import { checkSpec, whereNotes, profileError } from '../runner/check.js';
import { describeProfile } from './describe.js';
import { lookupValues, exploreClass, sampleSpec } from './explore.js';
import { lookupValues } from './explore.js';
import { readSpecFile } from '../chat.js';
// The tool definitions sent to the model. The descriptions are what the LLM
@@ -30,17 +30,6 @@ export const TOOLS = [
field: 'The field as "entityLabel.fieldLabel", e.g. "bestuurseenheid.naam"',
term: 'The Dutch word or name to search for',
}),
// Switched off for now: too much context for the LLM. To switch back on,
// uncomment these two, their cases in runTool, and the text in loop.js.
// tool('explore_data',
// 'Show a sample of the public data of one sh:targetClass of the profile: the predicates its instances use (outgoing and incoming), a few example values, the class of linked nodes, and which profile field each predicate is. Call it again on a linked class to follow a path. A predicate not in the profile cannot be used in a spec. A sample, not a check: it never runs a spec and counts nothing.',
// {
// profile_id: 'The profile URI returned by list_profiles',
// class: 'The sh:targetClass URI, as describe_profile shows it',
// }),
// tool('sample_spec',
// 'Run a spec that passes validate_spec on the public data and return the rows of up to 10 subjects it finds there. Use it to see whether its paths and filters land where you expect. Public data only: the report runs as the user and may find other rows, so an empty or short sample says nothing about the report. It does not attach the spec; validate_spec does.',
// { spec: 'The spec as Turtle text' }),
];
function tool(name, description, params) {
@@ -65,8 +54,6 @@ export async function runTool(name, args, profiles) {
case 'validate_spec': return validateSpec(args, profiles);
case 'read_spec': return readSpec(args);
case 'lookup_values': return lookup(args, profiles);
// case 'explore_data': return explore(args, profiles);
// case 'sample_spec': return sample(args, profiles);
default: return `unknown tool "${name}"`;
}
}
@@ -102,21 +89,3 @@ async function lookup({ profile_id, field, term }, profiles) {
if (!term) return 'no term to search for.';
return lookupValues(profiles.get(profile_id), field, term);
}
async function explore({ profile_id, class: cls }, profiles) {
const error = profileError(profiles, profile_id);
if (error) return error;
return exploreClass(profiles.get(profile_id), cls);
}
async function sample({ spec }, profiles) {
let parsed;
try {
parsed = parseSpec(spec);
} catch (e) {
return `the spec did not parse: ${e.message}`;
}
const error = checkSpec(parsed, profiles);
if (error) return error;
return sampleSpec(parsed, profiles.get(parsed.profileUri));
}

2
src/runner/columns.js
View File

@@ -56,7 +56,7 @@ export function groupQuery(group, chunk) {
return `SELECT DISTINCT ${vars.join(' ')} WHERE {\n VALUES ?s { ${subjects} }\n ${lines.join('\n ')}\n}`;
}
export function collect(result, group, values) {
function collect(result, group, values) {
for (const b of result.results.bindings) {
const chain = [...group.hops.slice(1).map((hop, i) => b[`x_${i + 1}`]), b.v];
let perColumn = values.get(b.s.value);