Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dac3e2854d | ||
|
|
1f80fdc815 | ||
|
|
538a2d949e | ||
|
|
b1fa1d1cd2 | ||
|
|
0802b75346 | ||
|
|
78a7c42a37 | ||
|
|
80097d3130 | ||
|
|
57292fa052 |
377
README.md
377
README.md
@@ -29,8 +29,8 @@ a fixed Dutch text with the CSV and the spec as bijlagen. No spec yet, a spec
|
||||
that no longer checks out, or a failed run each give a fixed Dutch failure
|
||||
message. Nothing is retried and no spec is rewritten.
|
||||
|
||||
Only refinement has tools: `list_profiles`, `describe_profile`,
|
||||
`validate_spec`, `read_spec` and `lookup_values`.
|
||||
Only refinement has tools: `describe_profile`, `validate_spec`, `read_spec`
|
||||
and `lookup_values`.
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -41,26 +41,330 @@ Two documents drive everything:
|
||||
| **profile** | the app | once, committed | `config/report-profiles/*.ttl`, mounted at `/config/profiles/` |
|
||||
| **spec** | the LLM | one per question | in the request, not stored |
|
||||
|
||||
A **profile** is a SHACL shapes graph: one `sh:NodeShape` per entity, fields
|
||||
as `sh:property`, code lists via `sh:class`, links via `sh:node`. The service
|
||||
ships no profile and knows no vocabulary; profiles live in the app repo.
|
||||
Profiles load at boot; a broken profile stops the service.
|
||||
|
||||
A **spec** is `rep:ReportSpec` + `sh:NodeShape`: `sh:property` selects the
|
||||
subjects, `rep:columns` says what the CSV holds. The two never interact. The
|
||||
namespace is `http://mu.semte.ch/vocabularies/reporting/` (`rep:`).
|
||||
|
||||
A filter tests only the end of its own path. To test a node in the middle of
|
||||
a path ("the mandate whose role is Burgemeester"), a filter or column carries
|
||||
`rep:where`: a condition written like a filter, from the row, that shares the
|
||||
nodes of every step it has in common with its host. On a column it drops
|
||||
values from the cell, never rows. The app's `report-spec-readme.md` has the
|
||||
details and an example.
|
||||
The profile says what the data looks like. The spec says what one report
|
||||
holds. Both are Turtle, and both borrow the words of SHACL, the W3C language
|
||||
to describe RDF data. Neither is used the way a SHACL tool uses it: see "How
|
||||
this relates to SHACL" below.
|
||||
|
||||
The run: check → seed (`SELECT DISTINCT ?s`, paged) → columns (per group, per
|
||||
chunk of subjects) → assemble (dedup + `rep:collect`) → CSV → register file
|
||||
chunk of subjects, with the nodes on each path) → assemble (pair by the
|
||||
nodes columns share, on every step; `rep:collect`) → CSV → register file
|
||||
+ report resource. One run per confirmed spec.
|
||||
|
||||
## The profile: a map of the data
|
||||
|
||||
A profile (using SHACL + some extensions) lists the kinds of things in the data (entities), what each one
|
||||
has (fields), and which fields lead to other entities (links). The app
|
||||
writes it; the service ships no profile and knows no vocabulary.
|
||||
|
||||
Three parts of the service read it:
|
||||
|
||||
- the validator checks every step of a spec's paths against it;
|
||||
- the LLM gets it as a menu (`describe_profile`), its only picture of the
|
||||
data;
|
||||
- the seed takes the class of the rows from it.
|
||||
|
||||
Nothing checks the data against a profile.
|
||||
|
||||
```turtle
|
||||
<http://data.lblod.info/id/report-profiles/mandaten> a owl:Ontology ;
|
||||
dct:title "Mandaten" .
|
||||
|
||||
ent:Mandataris a sh:NodeShape ;
|
||||
rdfs:label "mandataris" ;
|
||||
sh:targetClass mandaat:Mandataris ;
|
||||
sh:property [ sh:path mandaat:start ; sh:name "start" ; sh:datatype xsd:dateTime ] ;
|
||||
sh:property [ sh:path org:holds ; sh:name "bekleedt mandaat" ; sh:node ent:Mandaat ] ;
|
||||
sh:property [ sh:path mandaat:isBestuurlijkeAliasVan ; sh:name "persoon" ; sh:node ent:Persoon ] .
|
||||
```
|
||||
|
||||
| You write | It means |
|
||||
|---|---|
|
||||
| `owl:Ontology` + `dct:title` | the profile. A spec names this URI in `rep:profile`. |
|
||||
| `sh:NodeShape` | an entity. A spec names this URI in `rep:entity` when it must pick one. |
|
||||
| `rdfs:label` on the entity | its name in the menu and in error messages. Keep it unique in the profile. |
|
||||
| `sh:targetClass` | the `rdf:type` of the entity's subjects |
|
||||
| `sh:property` + `sh:path` | a field: one step, a predicate or `[ sh:inversePath p ]` |
|
||||
| `sh:name` | the field's name in the menu and in messages |
|
||||
| `sh:datatype` | a value field: a column may end here |
|
||||
| `sh:class` | a code-list field: a column may end here. Lists up to `INLINE_VALUES_MAX` values show in the menu. |
|
||||
| `sh:node` | a link to another entity: a spec may take this step |
|
||||
| `sh:or ( [ sh:node A ] [ sh:node B ] )` | a link that reaches A or B |
|
||||
|
||||
The service does not read `sh:description` (write it for people; the LLM
|
||||
never sees it), cardinality (every link counts as "many"), or a field path
|
||||
of more than one step.
|
||||
|
||||
At boot each profile must parse, name itself with `owl:Ontology` and hold at
|
||||
least one entity; a broken profile stops the service. A `sh:node` to an
|
||||
entity that does not exist only shows when a spec takes that step.
|
||||
|
||||
### A link to several kinds
|
||||
|
||||
When one step can reach entities of different kinds, say so with `sh:or`:
|
||||
|
||||
```turtle
|
||||
sh:property [ sh:path [ sh:inversePath besluit:bestuurt ] ; sh:name "bestuursorganen" ;
|
||||
sh:or ( [ sh:node ent:Bestuursorgaan ] [ sh:node ent:LeidinggevendBestuursorgaan ] ) ] .
|
||||
```
|
||||
|
||||
The validator follows all of them, and the next steps decide. A spec that
|
||||
wants one kind adds a `rep:where` on a step only that kind has. Write one
|
||||
field with `sh:or`, not two fields on one path: in SHACL, two fields mean a
|
||||
value must fit both. (The validator reads two fields like `sh:or`, but a
|
||||
SHACL reader would not.)
|
||||
|
||||
### Entities that share a class
|
||||
|
||||
A profile may hold several entities of one class. The portal's
|
||||
`mandaten.ttl` has four kinds of `besluit:Bestuursorgaan`: political and
|
||||
leidinggevend, each fixed (vast) and in time. A spec picks one with
|
||||
`rep:entity`; `rep:discriminator` tells the seed which subjects belong to
|
||||
it.
|
||||
|
||||
A discriminator is written like a spec filter (see "Which rows" below): a
|
||||
`sh:path` from the subject, of any length, and a test on where it ends.
|
||||
Several on one entity must all hold. `sh:and ( ... )`, `sh:or ( ... )` and
|
||||
`sh:not ...` combine them, and may nest.
|
||||
|
||||
```turtle
|
||||
# fixed, and its classification is in the list
|
||||
ent:LeidinggevendBestuursorgaan
|
||||
rep:discriminator [ sh:path generiek:isTijdspecialisatieVan ; sh:maxCount 0 ] ;
|
||||
rep:discriminator [ sh:path org:classification ; sh:in ent:LeidinggevendeOrgaanClassificaties ] .
|
||||
|
||||
# in time, and the classification of its fixed organ is not in the list
|
||||
ent:BestuursorgaanInTijd
|
||||
rep:discriminator [ sh:path generiek:isTijdspecialisatieVan ; sh:minCount 1 ] ;
|
||||
rep:discriminator [ sh:path ( generiek:isTijdspecialisatieVan org:classification ) ;
|
||||
sh:in ent:LeidinggevendeOrgaanClassificaties ; sh:maxCount 0 ] .
|
||||
```
|
||||
|
||||
A rule or a list can be a named node, written once and used in several
|
||||
places. `sh:not` on a named rule gives its exact opposite, so two entities
|
||||
split a class with no overlap and no gap:
|
||||
|
||||
```turtle
|
||||
ent:IsLeidinggevendOrgaan sh:path org:classification ;
|
||||
sh:in ent:LeidinggevendeOrgaanClassificaties .
|
||||
|
||||
ent:LeidinggevendBestuursorgaan rep:discriminator ent:IsLeidinggevendOrgaan .
|
||||
ent:Bestuursorgaan rep:discriminator [ sh:not ent:IsLeidinggevendOrgaan ] .
|
||||
```
|
||||
|
||||
Turtle cannot name a `( ... )` list, so write a named list out:
|
||||
`ent:X rdf:first <a> ; rdf:rest ( <b> <c> ) .` Do not give a named rule the
|
||||
type `sh:NodeShape`: it would become an entity.
|
||||
|
||||
Test what a thing is, not what hangs off it. The classification says what
|
||||
an organ is. "Has a period with a bestuursfunctie" needs no code list, but
|
||||
it trusts the data: an organ that lost its functie would move to the other
|
||||
entity, and no report could find it. `sh:or` of both catches the most.
|
||||
|
||||
The seed turns a plain discriminator into the same SPARQL as a filter.
|
||||
`sh:and`, `sh:or` and `sh:not` become one `FILTER(...)` over `EXISTS` tests;
|
||||
that form is not yet tested against mu-authorization or sparql-parser. A
|
||||
broken discriminator stops the service at boot and names the entity.
|
||||
|
||||
## The spec: one report
|
||||
|
||||
The LLM writes one spec per question. The user reads the proposal and
|
||||
confirms; the service then checks the spec again and runs it. A spec has two
|
||||
halves: which rows, and what the CSV shows about each row.
|
||||
|
||||
```turtle
|
||||
<http://data.lblod.info/id/report-specs/burgemeesters> a rep:ReportSpec , sh:NodeShape ;
|
||||
dct:title "Burgemeesters sinds 2025" ;
|
||||
rep:profile <http://data.lblod.info/id/report-profiles/mandaten> ;
|
||||
sh:targetClass mandaat:Mandataris ;
|
||||
sh:property [ sh:path ( org:holds org:role ) ;
|
||||
sh:hasValue <http://data.vlaanderen.be/id/concept/BestuursfunctieCode/5ab0e9b8a3b2ca7c5e000013> ] ;
|
||||
sh:property [ sh:path mandaat:start ;
|
||||
sh:minInclusive "2025-01-01T00:00:00"^^xsd:dateTime ] ;
|
||||
rep:columns (
|
||||
[ sh:path rep:self ; rdfs:label "mandataris" ]
|
||||
[ sh:path ( mandaat:isBestuurlijkeAliasVan foaf:givenName ) ; rdfs:label "voornaam" ]
|
||||
[ sh:path ( mandaat:isBestuurlijkeAliasVan foaf:familyName ) ; rdfs:label "achternaam" ]
|
||||
[ sh:path ( org:holds [ sh:inversePath org:hasPost ] generiek:isTijdspecialisatieVan
|
||||
besluit:bestuurt skos:prefLabel ) ; rdfs:label "gemeente" ]
|
||||
) .
|
||||
```
|
||||
|
||||
One row per mandataris whose mandate has the role Burgemeester and who
|
||||
started on or after 1 January 2025, with name and gemeente.
|
||||
|
||||
### Which rows
|
||||
|
||||
| You write | It means |
|
||||
|---|---|
|
||||
| `rep:ReportSpec` + `sh:NodeShape` | this is a spec |
|
||||
| `rep:profile` | the profile the spec was written against |
|
||||
| `sh:targetClass` | one row per subject of this class |
|
||||
| `rep:entity` | which entity, when more than one has this class |
|
||||
| `sh:property` | a filter: keep only the subjects that pass |
|
||||
|
||||
A filter has a `sh:path` from the row and a test. When the path reaches
|
||||
several values, the test passes if one of them passes.
|
||||
|
||||
| Test | Keeps the row when |
|
||||
|---|---|
|
||||
| `sh:hasValue x` | the path reaches x |
|
||||
| `sh:in ( x y )` | the path reaches x or y (all URIs, or all text) |
|
||||
| `sh:minInclusive`, `sh:maxInclusive`, `sh:minExclusive`, `sh:maxExclusive` | a date or number at the end lies in range; the path must end on a value field |
|
||||
| `sh:pattern` (+ `sh:flags`) | a value matches the regex |
|
||||
| `rep:anyOf ( "woord" ... )` | a value holds one of these words, upper or lower case; at most 24 words, no URIs |
|
||||
| `sh:minCount 1` | the path leads somewhere |
|
||||
| `sh:minCount 2` | the path reaches two different values |
|
||||
| `sh:maxCount 0` | the path leads nowhere |
|
||||
| `sh:maxCount 0` with `sh:in` or `sh:hasValue` | the path reaches none of these values |
|
||||
|
||||
The service cannot count: `sh:minCount` above 2 and `sh:maxCount` above 0 are
|
||||
refused.
|
||||
|
||||
### A test on the way: `rep:where`
|
||||
|
||||
A filter tests the end of its path. `rep:where` tests a node on the way.
|
||||
Write it like a filter, with its path from the row. The steps it shares with
|
||||
its column or filter are the same nodes; its test applies where its own path
|
||||
ends. On a filter it drops rows; on a column it drops values from the cell,
|
||||
never rows.
|
||||
|
||||
```turtle
|
||||
<http://data.lblod.info/id/report-specs/eenheden-burgemeester> a rep:ReportSpec , sh:NodeShape ;
|
||||
rep:profile <http://data.lblod.info/id/report-profiles/mandaten> ;
|
||||
sh:targetClass besluit:Bestuurseenheid ;
|
||||
rep:columns (
|
||||
[ sh:path skos:prefLabel ; rdfs:label "eenheid" ]
|
||||
[ sh:path ( [ sh:inversePath besluit:bestuurt ] [ sh:inversePath generiek:isTijdspecialisatieVan ]
|
||||
org:hasPost [ sh:inversePath org:holds ] mandaat:isBestuurlijkeAliasVan foaf:familyName ) ;
|
||||
rdfs:label "burgemeester" ;
|
||||
rep:where [ sh:path ( [ sh:inversePath besluit:bestuurt ] [ sh:inversePath generiek:isTijdspecialisatieVan ]
|
||||
org:hasPost org:role ) ;
|
||||
sh:hasValue <http://data.vlaanderen.be/id/concept/BestuursfunctieCode/5ab0e9b8a3b2ca7c5e000013> ] ]
|
||||
) .
|
||||
```
|
||||
|
||||
Per eenheid, the family name of its mayors. The `rep:where` shares three
|
||||
steps with the column, so it tests the mandate, not the organ or the
|
||||
person. `validate_spec` says so: `rep:where 1 on column "burgemeester"
|
||||
applies at "mandaat", after step 3`. The proposal repeats that line to the
|
||||
user.
|
||||
|
||||
A `rep:where` must share at least one step with its column or filter, and
|
||||
holds no `rep:where` of its own.
|
||||
|
||||
### What the CSV shows
|
||||
|
||||
`rep:columns` is a list; each item becomes a column, in that order.
|
||||
|
||||
| You write | It means |
|
||||
|---|---|
|
||||
| `sh:path ( ... )` | the steps from the row to the value |
|
||||
| `sh:path rep:self` | the row's own URI |
|
||||
| `rdfs:label` | the column header; required and unique |
|
||||
| `sh:nodeKind sh:IRI` | show the URI of the node the path ends on (only then may a path end on a link) |
|
||||
| no `rep:collect`, or `rep:collect rep:row` | each value gets its own row; at most one column may say `rep:row` |
|
||||
| `rep:collect sh:groupConcat` (+ `sh:separator`) | all values in one cell |
|
||||
| `rep:collect sh:min` / `sh:max` | the lowest / highest date or number |
|
||||
|
||||
A column holds no tests; those go in `sh:property`, or in the column's
|
||||
`rep:where`.
|
||||
|
||||
### How values become rows
|
||||
|
||||
Columns that walk the same steps (with the same `rep:where`) share the nodes
|
||||
on those steps, and their values pair by those nodes: one mayor's first and
|
||||
last name stay on one row. Where paths split and both sides have several
|
||||
values, every combination becomes a row. A missing value is an empty cell;
|
||||
identical rows show once. `sh:groupConcat`, `sh:min` and `sh:max` fold a
|
||||
column into one cell per node it shares with the other columns.
|
||||
|
||||
### What the validator checks
|
||||
|
||||
The validator (`src/runner/check.js`) runs on every proposal and again
|
||||
before every run. It returns the first problem, in words the LLM can act on.
|
||||
On top of the rules above, it checks that:
|
||||
|
||||
- the Turtle holds one `rep:ReportSpec`, no more: one answer is one report;
|
||||
- every step of every path exists in the profile;
|
||||
- no path is longer than `MAX_PATH_DEPTH` steps;
|
||||
- a date or number test ends on a value field, not on a link;
|
||||
- a class that several entities share comes with `rep:entity`.
|
||||
|
||||
## How this relates to SHACL
|
||||
|
||||
SHACL is the W3C language to check RDF data against shapes: a SHACL tool
|
||||
takes data and shapes, and reports which nodes fail. We borrow its words and
|
||||
its layout, so both documents are valid SHACL syntax and a SHACL tool can
|
||||
read them. We use them for other jobs.
|
||||
|
||||
### The profile: SHACL, with one extension
|
||||
|
||||
Read as SHACL, a profile says what good data looks like: nodes of this
|
||||
class, values of this datatype, links to nodes of that shape. We read the
|
||||
same statements as a map, and never check data with them. That is a use of
|
||||
SHACL, not a change to it. One thing does change the meaning:
|
||||
|
||||
- **`rep:discriminator` narrows `sh:targetClass`.** In SHACL, four shapes
|
||||
with `sh:targetClass besluit:Bestuursorgaan` mean that every organ must
|
||||
fit all four. Here each organ belongs only to the shapes whose
|
||||
discriminators it meets. A SHACL tool skips `rep:` terms, so to it the
|
||||
profile still says "all four". SHACL-AF has a tool for this job, custom
|
||||
targets (see below); we keep `sh:targetClass` so a spec can still name its
|
||||
class, and write the rule in the same terms as a spec filter.
|
||||
|
||||
Two smaller points:
|
||||
|
||||
- In SHACL, a node reached through `sh:node B` must pass B's field rules. We
|
||||
go one step further and treat it as "a B" for the next steps. If the data
|
||||
breaks that, the query finds nothing there; it never finds a wrong row.
|
||||
- Links run in circles (bestuursorgaan to orgaan in tijd and back). SHACL
|
||||
leaves such shapes to each tool. That only matters if someone checks data
|
||||
with a profile.
|
||||
|
||||
### The spec: SHACL syntax, our own meaning
|
||||
|
||||
A spec is valid SHACL syntax, but it does not mean what SHACL says. A SHACL
|
||||
tool reads a spec as a check and reports which subjects fail it. We read it
|
||||
as a query: the rows are the subjects that pass. For some terms the two
|
||||
readings agree; for others they do not.
|
||||
|
||||
| Term | In SHACL | In a spec | Same? |
|
||||
|---|---|---|---|
|
||||
| `sh:targetClass`, `sh:path` | which nodes; which values | same | yes |
|
||||
| `sh:hasValue` | one value equals | same | yes |
|
||||
| `sh:minCount 1` / `2` | at least 1 / 2 values | same | yes |
|
||||
| `sh:maxCount 0` | no values | same | yes |
|
||||
| `sh:in`, ranges, `sh:pattern` | every value passes, and a node without values passes too | one value passes | no |
|
||||
| `sh:maxCount 0` with a test | no values at all | no value that passes | no |
|
||||
| `rep:anyOf`, `rep:where` | skipped | tests of our own | added |
|
||||
| `rep:columns`, `rep:self`, `rep:collect`, `sh:nodeKind` and `rdfs:label` on a column | skipped | the CSV | added |
|
||||
|
||||
So a spec means the same in both readings only if it uses the "yes" terms
|
||||
alone; then its rows are exactly the subjects a SHACL tool finds good. Most
|
||||
specs use `sh:in` or a date range, so most specs do not.
|
||||
|
||||
Why "one value passes": reports ask for "organs with classification X" or
|
||||
"persons with a burgemeester mandate". With SHACL's `sh:in`, the first
|
||||
would also keep organs with no classification, and the second would drop a
|
||||
person who holds a second mandate too. SHACL can say "one value"
|
||||
(`sh:qualifiedValueShape` with `sh:qualifiedMinCount 1`), but the LLM writes
|
||||
the specs, and each extra construct is one more way to get it wrong.
|
||||
|
||||
The CSV half has no SHACL meaning at all: SHACL has no notion of output.
|
||||
|
||||
### SHACL-AF
|
||||
|
||||
SHACL Advanced Features is a W3C Working Group Note from 2017: a proposal,
|
||||
not a standard. It adds custom targets (`sh:SPARQLTarget`: pick nodes with a
|
||||
SPARQL query), values computed from the data, rules and functions. A later
|
||||
draft adds `sh:count`, `sh:sum`, `sh:min`, `sh:max`, `sh:groupConcat` and
|
||||
`sh:orderBy`. We borrow `sh:groupConcat`, `sh:min`, `sh:max` and
|
||||
`sh:separator` as values of `rep:collect`, and keep `sh:count`, `sh:sum` and
|
||||
`sh:orderBy` for when counting and sorting arrive. We do not use its custom
|
||||
targets: a `sh:SPARQLTarget` is a SPARQL string that the service could not
|
||||
check or combine with a spec's filters.
|
||||
|
||||
## One identity: mu-authorization's
|
||||
|
||||
All SPARQL goes through the template's own helpers (`mu`'s `query` and
|
||||
@@ -89,10 +393,11 @@ node. That check stays on the profile: `validate_spec` says where each
|
||||
### The service scope
|
||||
|
||||
The LLM's own reads during refinement (`describe_profile`'s code lists and
|
||||
`lookup_values`) run under a service scope, not the caller's session. mu's
|
||||
`lookup_values`) run under a service scope, not the caller's session:
|
||||
`publicQuery` in `src/llm/explore.js`. mu's
|
||||
`query(q, { scope })` sends `mu-auth-scope`; the sparql-parser config grants
|
||||
that scope read access to `http://mu.semte.ch/graphs/public` only. So
|
||||
refinement reads only public code lists, regardless of who calls. Execution
|
||||
refinement reads only the public graph, regardless of who calls. Execution
|
||||
still runs as the caller, so the report contains what they may see, and the
|
||||
LLM sees none of it.
|
||||
|
||||
@@ -106,6 +411,7 @@ The scope URI is `SERVICE_SCOPE` (default
|
||||
|
||||
```
|
||||
POST /assistant/conversations/:id/turns { content } → 202 { id }
|
||||
DELETE /assistant/conversations/:id → 204
|
||||
```
|
||||
|
||||
This is the chat. The route is in `app.js`: it records the question, answers
|
||||
@@ -119,6 +425,12 @@ that way. Failure is a message, never a silent loader.
|
||||
The chat needs `LLM_BASE_URL`; without it every turn answers with a failure
|
||||
message. A full-org report run takes minutes.
|
||||
|
||||
The delete is the trash bin in the chat's sidebar. It removes the
|
||||
conversation, its messages, their bijlagen (the triples and the files on
|
||||
the share) and the reports of those files. It runs as the caller: the auth
|
||||
layer keeps a user to their own conversations, and a conversation the caller
|
||||
cannot read is a 404.
|
||||
|
||||
### The internal tools
|
||||
|
||||
The refinement loop drives one tool set (`src/llm/tools.js`). The tools are
|
||||
@@ -126,6 +438,27 @@ machinery, not an external surface. All LLM-facing text (system prompts,
|
||||
profile menu, tool descriptions, validator errors) is English; the answer the
|
||||
user sees is Dutch.
|
||||
|
||||
### What the model sees
|
||||
|
||||
The system prompt ends with the profile list and, when there is a current
|
||||
spec, the menu of its profile (`src/llm/loop.js`). A turn that stays on that
|
||||
profile needs no `describe_profile` call; the model calls it only for
|
||||
another profile.
|
||||
|
||||
Each request to the model holds one spec in full: the newest.
|
||||
|
||||
- The history is text only. A spec bijlage shows as its file name, and a
|
||||
spec the model printed in its answer is cut out.
|
||||
- The current spec, the last spec bijlage in the conversation, comes in full
|
||||
under the user's new message. When the user refines, the model changes it;
|
||||
when the user asks for a report that lists another kind of thing per row,
|
||||
it starts a new spec.
|
||||
- Within a turn, each newer spec (a `validate_spec` call, or an older spec
|
||||
opened with `read_spec`) replaces the older one with "Left out: a newer
|
||||
spec follows."
|
||||
- The model opens an older spec with `read_spec` only when the user asks for
|
||||
it.
|
||||
|
||||
## Environment
|
||||
|
||||
| Var | Default | Meaning |
|
||||
@@ -144,7 +477,7 @@ user sees is Dutch.
|
||||
| `SEED_PAGE_SIZE` | `5000` | seed page size |
|
||||
| `SUBJECT_CHUNK_SIZE` | `100` | see above |
|
||||
| `SHARE_DIR` | `/share` | where CSVs are written |
|
||||
| `CHAT_ASSISTANT_URI` | — | the `prov:SoftwareAgent` that signs assistant messages; found by lookup when unset |
|
||||
| `CHAT_ASSISTANT_URI` | `http://data.lblod.info/id/chat-agents/rapportassistent` | the `prov:SoftwareAgent` that signs assistant messages (the portal's chat-assistant migration seeds it) |
|
||||
| `CHAT_HISTORY_LIMIT` | `20` | messages of history a turn reads |
|
||||
| `SERVICE_SCOPE` | `http://services.semantic.works/natural-language-report` | the mu-auth-scope for the LLM's own reads (code lists, `lookup_values`). Must match the `with-scope` grant in the app's sparql-parser config. |
|
||||
|
||||
@@ -200,4 +533,4 @@ non-admin callers. Opening this up means adding the type to their group first.
|
||||
## What it will not do
|
||||
|
||||
No counting, no sorting, no top-N, no comparing two subjects, one class per
|
||||
report, suggestions stop at the public graph.
|
||||
report, suggestions stop at the public graph.
|
||||
|
||||
32
app.js
32
app.js
@@ -1,12 +1,14 @@
|
||||
import { app, errorHandler } from 'mu';
|
||||
import bodyParser from 'body-parser';
|
||||
import { loadProfiles } from './src/runner/profile.js';
|
||||
import { readConversation, readHistory, findAssistant, writeMessage } from './src/chat.js';
|
||||
import { readConversation, readHistory, writeMessage, deleteConversation } from './src/chat.js';
|
||||
import { wantsExecution } from './src/llm/mode.js';
|
||||
import { executeSpec, refineSpec } from './src/report-assistant.js';
|
||||
|
||||
const PROFILE_DIR = process.env.PROFILE_DIR || '/config/profiles';
|
||||
const CHAT_ASSISTANT_URI = process.env.CHAT_ASSISTANT_URI;
|
||||
// The agent that signs the assistant's messages, seeded by the portal's
|
||||
// chat-assistant migration.
|
||||
const CHAT_ASSISTANT_URI = process.env.CHAT_ASSISTANT_URI || 'http://data.lblod.info/id/chat-agents/rapportassistent';
|
||||
const TITLE_LENGTH = 80;
|
||||
const FAILED = 'Er ging iets mis. Probeer het opnieuw.';
|
||||
|
||||
@@ -22,13 +24,12 @@ app.post('/assistant/conversations/:id/turns', bodyParser.json(), async function
|
||||
const content = String(req.body?.content ?? '').trim();
|
||||
if (!content) return res.status(400).json({ error: 'content is required' });
|
||||
|
||||
let conversation, assistant, history, question;
|
||||
let conversation, history, question;
|
||||
try {
|
||||
// the access check: the caller's session decides what is readable
|
||||
conversation = await readConversation(req.params.id);
|
||||
if (!conversation) return res.status(404).json({ error: 'no such conversation' });
|
||||
assistant = CHAT_ASSISTANT_URI || await findAssistant();
|
||||
history = await readHistory(conversation.uri, assistant);
|
||||
history = await readHistory(conversation.uri, CHAT_ASSISTANT_URI);
|
||||
question = await writeMessage({
|
||||
conversationUri: conversation.uri,
|
||||
content,
|
||||
@@ -50,17 +51,34 @@ app.post('/assistant/conversations/:id/turns', bodyParser.json(), async function
|
||||
await writeMessage({
|
||||
conversationUri: conversation.uri,
|
||||
content: answer.text,
|
||||
maker: assistant,
|
||||
maker: CHAT_ASSISTANT_URI,
|
||||
attachments: answer.attachments,
|
||||
});
|
||||
} catch (e) {
|
||||
console.error('[chat] turn failed:', e);
|
||||
try {
|
||||
await writeMessage({ conversationUri: conversation.uri, content: FAILED, maker: assistant });
|
||||
await writeMessage({ conversationUri: conversation.uri, content: FAILED, maker: CHAT_ASSISTANT_URI });
|
||||
} catch (e2) {
|
||||
console.error('[chat] could not write the failure message:', e2);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
// Delete a conversation with everything that hangs off it: the messages,
|
||||
// their bijlagen (triples and share files) and the reports. The read is the
|
||||
// access check: the caller's session decides what exists, so another user's
|
||||
// conversation is a 404.
|
||||
app.delete('/assistant/conversations/:id', async function(req, res) {
|
||||
try {
|
||||
const conversation = await readConversation(req.params.id);
|
||||
if (!conversation) return res.status(404).json({ error: 'no such conversation' });
|
||||
const removed = await deleteConversation(conversation.uri);
|
||||
console.log(`[chat] deleted conversation ${req.params.id}, ${removed} share file(s) removed`);
|
||||
res.status(204).send();
|
||||
} catch (e) {
|
||||
console.error('[chat] could not delete the conversation:', e);
|
||||
res.status(500).json({ error: e.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.use(errorHandler);
|
||||
|
||||
80
src/chat.js
80
src/chat.js
@@ -6,7 +6,7 @@
|
||||
// A bijlage is a real file: a logical nfo:FileDataObject the file service
|
||||
// serves at /files/<uuid>/download, over a physical share:// object with
|
||||
// nie:dataSource back to the logical one.
|
||||
import { readFileSync, writeFileSync, statSync } from 'fs';
|
||||
import { readFileSync, writeFileSync, statSync, unlinkSync } from 'fs';
|
||||
import { query, update, uuid,
|
||||
sparqlEscapeUri, sparqlEscapeString, sparqlEscapeInt, sparqlEscapeDateTime } from 'mu';
|
||||
|
||||
@@ -25,7 +25,6 @@ const T = {
|
||||
created: 'http://purl.org/dc/terms/created',
|
||||
maker: 'http://xmlns.com/foaf/0.1/maker',
|
||||
attachment: 'https://www.w3.org/ns/activitystreams#attachment',
|
||||
softwareAgent: 'http://www.w3.org/ns/prov#SoftwareAgent',
|
||||
file: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject',
|
||||
fileName: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName',
|
||||
fileSize: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize',
|
||||
@@ -33,6 +32,7 @@ const T = {
|
||||
format: 'http://purl.org/dc/terms/format',
|
||||
fileType: 'http://purl.org/dc/terms/type',
|
||||
dataSource: 'http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource',
|
||||
generated: 'http://www.w3.org/ns/prov#generated',
|
||||
};
|
||||
const MESSAGE_BASE = 'http://data.lblod.info/id/chat-messages/';
|
||||
const FILE_BASE = 'http://data.lblod.info/files/';
|
||||
@@ -109,17 +109,6 @@ export async function readHistory(conversationUri, assistantUri) {
|
||||
return messages.slice(-HISTORY_LIMIT);
|
||||
}
|
||||
|
||||
// The one assistant the caller can read. Throws unless there is exactly one.
|
||||
export async function findAssistant() {
|
||||
const json = await query(`
|
||||
SELECT ?agent WHERE { ?agent ${u(T.type)} ${u(T.softwareAgent)} . } LIMIT 2`);
|
||||
const rows = bindings(json);
|
||||
if (rows.length !== 1) {
|
||||
throw new Error(`expected one prov:SoftwareAgent, found ${rows.length}; seed one or set CHAT_ASSISTANT_URI`);
|
||||
}
|
||||
return rows[0].agent.value;
|
||||
}
|
||||
|
||||
// --- writes -------------------------------------------------------------------
|
||||
|
||||
// One message, both types, its bijlagen, and the conversation touched, in
|
||||
@@ -222,3 +211,68 @@ export async function registerFile(shareName, format, type, name = shareName) {
|
||||
|
||||
return logical;
|
||||
}
|
||||
|
||||
// --- deletes -------------------------------------------------------------------
|
||||
|
||||
// deleteConversation(uri) → the share files it removed (a number). Deletes
|
||||
// the conversation, its messages, their bijlagen (the logical file and the
|
||||
// physical one under it) and the reports of those files, then removes the
|
||||
// share files. Everything runs as the caller, so the auth layer keeps a
|
||||
// user to their own conversation.
|
||||
export async function deleteConversation(conversationUri) {
|
||||
// The bijlagen first: once the triples are gone their names cannot be
|
||||
// found back.
|
||||
const json = await query(`
|
||||
SELECT DISTINCT ?doc ?name WHERE {
|
||||
?message ${u(T.hasContainer)} ${u(conversationUri)} ;
|
||||
${u(T.attachment)} ?doc .
|
||||
OPTIONAL {
|
||||
?physical ${u(T.dataSource)} ?doc ;
|
||||
${u(T.fileName)} ?name .
|
||||
}
|
||||
}`);
|
||||
const rows = bindings(json);
|
||||
const docs = [...new Set(rows.map(b => b.doc.value))];
|
||||
const names = [...new Set(rows.filter(b => b.name).map(b => b.name.value))];
|
||||
|
||||
await update(`
|
||||
DELETE {
|
||||
?conversation ?cp ?co .
|
||||
} WHERE {
|
||||
VALUES ?conversation { ${u(conversationUri)} }
|
||||
?conversation ?cp ?co .
|
||||
}`);
|
||||
|
||||
await update(`
|
||||
DELETE {
|
||||
?message ?mp ?mo .
|
||||
} WHERE {
|
||||
?message ${u(T.hasContainer)} ${u(conversationUri)} ;
|
||||
?mp ?mo .
|
||||
}`);
|
||||
|
||||
if (docs.length) {
|
||||
await update(`
|
||||
DELETE {
|
||||
?doc ?dp ?do .
|
||||
?physical ?pp ?po .
|
||||
?report ?rp ?ro .
|
||||
} WHERE {
|
||||
VALUES ?doc { ${docs.map(u).join(' ')} }
|
||||
?doc ?dp ?do .
|
||||
OPTIONAL { ?physical ${u(T.dataSource)} ?doc ; ?pp ?po . }
|
||||
OPTIONAL { ?report ${u(T.generated)} ?doc ; ?rp ?ro . }
|
||||
}`);
|
||||
}
|
||||
|
||||
let removed = 0;
|
||||
for (const name of names) {
|
||||
try {
|
||||
unlinkSync(`${SHARE_DIR}/${name}`);
|
||||
removed++;
|
||||
} catch {
|
||||
// the file was already gone
|
||||
}
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
|
||||
@@ -1,38 +1,38 @@
|
||||
// A profile as a plain text menu: entities, fields, code lists. Code lists up
|
||||
// to INLINE_VALUES_MAX are inlined (and cached), so the LLM never has to
|
||||
// search the short ones.
|
||||
import { codeList, INLINE_VALUES_MAX } from './lookup.js';
|
||||
import { codeList, INLINE_VALUES_MAX } from './explore.js';
|
||||
|
||||
export async function describeProfile(profile) {
|
||||
const lines = [];
|
||||
lines.push('A spec is Turtle, in exactly this shape (copy these prefixes):');
|
||||
lines.push('@prefix rep: <http://mu.semte.ch/vocabularies/reporting/> .');
|
||||
lines.push('@prefix sh: <http://www.w3.org/ns/shacl#> .');
|
||||
lines.push('@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .');
|
||||
lines.push('@prefix dct: <http://purl.org/dc/terms/> .');
|
||||
lines.push('');
|
||||
lines.push('<http://data.lblod.info/id/report-specs/<your-id>> a rep:ReportSpec , sh:NodeShape ;');
|
||||
lines.push(' dct:title "..." ;');
|
||||
lines.push(` rep:profile <${profile.uri}> ;`);
|
||||
lines.push(' sh:targetClass <the rdf:type from the list below> ;');
|
||||
lines.push(' sh:property [ ... ] ; # filters, optional');
|
||||
lines.push(' rep:columns ( [ sh:path ( ... ) ; rdfs:label "..." ] ... ) .');
|
||||
lines.push('Every column ends on a value field. Use rep:self as sh:path for the subject URI.');
|
||||
lines.push('For the URI of a linked node, end the column on that link and add sh:nodeKind sh:IRI to the column.');
|
||||
lines.push('');
|
||||
lines.push('sh:targetClass is the full URI in brackets after the entity below.');
|
||||
lines.push('In sh:path, use the predicate URI shown after each field name, not the field name itself.');
|
||||
lines.push('An inverse hop (the field shows "(inverse)") goes in the list as [ sh:inversePath <predicate> ] .');
|
||||
lines.push('When two entities share one sh:targetClass, pick one with rep:entity <the URI after the entity>.');
|
||||
lines.push('A date filter is sh:minInclusive or sh:maxInclusive with "yyyy-mm-dd"^^xsd:dateTime');
|
||||
lines.push('(the xsd prefix above is not copied: add @prefix xsd: <http://www.w3.org/2001/XMLSchema#>).');
|
||||
lines.push('');
|
||||
const lines = [`A spec is Turtle, in exactly this shape (copy these prefixes):
|
||||
@prefix rep: <http://mu.semte.ch/vocabularies/reporting/> .
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||||
@prefix dct: <http://purl.org/dc/terms/> .
|
||||
|
||||
<http://data.lblod.info/id/report-specs/<your-id>> a rep:ReportSpec , sh:NodeShape ;
|
||||
dct:title "..." ;
|
||||
rep:profile <${profile.uri}> ;
|
||||
sh:targetClass <the rdf:type from the list below> ;
|
||||
sh:property [ ... ] ; # filters, optional
|
||||
rep:columns ( [ sh:path ( ... ) ; rdfs:label "..." ] ... ) .
|
||||
Every column ends on a value field. Use rep:self as sh:path for the subject URI.
|
||||
For the URI of a linked node, end the column on that link and add sh:nodeKind sh:IRI to the column.
|
||||
|
||||
sh:targetClass is the full URI in brackets after the entity below.
|
||||
In sh:path, use the predicate URI shown after each field name, not the field name itself.
|
||||
An inverse hop (the field shows "(inverse)") goes in the list as [ sh:inversePath <predicate> ] .
|
||||
When two entities share one sh:targetClass, pick one with rep:entity <the URI after the entity>.
|
||||
A field that links to several entities ("→ a or b") reaches all of them; the steps after it decide. To keep one kind, add a rep:where on a step only that kind has.
|
||||
A date filter is sh:minInclusive or sh:maxInclusive with "yyyy-mm-dd"^^xsd:dateTime
|
||||
(the xsd prefix above is not copied: add @prefix xsd: <http://www.w3.org/2001/XMLSchema#>).
|
||||
`];
|
||||
for (const shape of profile.shapes) {
|
||||
lines.push(`${shape.label || shape.targetClass} — sh:targetClass <${shape.targetClass}> ; rep:entity <${shape.uri}>`);
|
||||
for (const f of shape.fields) {
|
||||
const kind = f.datatype ? datatypeName(f.datatype)
|
||||
: f.class ? `→ ${conceptLabel(f.class, await codeList(f.class))}`
|
||||
: f.node ? `→ ${linkedLabel(profile, f.node)}`
|
||||
: f.nodes.length ? `→ ${f.nodes.map(n => linkedLabel(profile, n)).join(' or ')}`
|
||||
: '?';
|
||||
const dir = f.inverse ? ' (inverse)' : '';
|
||||
const pathShown = f.path || '?';
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
// The LLM's own reads: code lists. They run under the service scope, not
|
||||
// the caller's session; the sparql-parser config grants that scope read
|
||||
// access to the public graph only, so they read the same for every caller.
|
||||
// Execution still runs as the caller.
|
||||
// Everything the LLM reads from the database in refine mode: the code
|
||||
// lists, inlined in describe_profile and searched by lookup_values. Every
|
||||
// query goes through publicQuery: the service scope, not the caller's
|
||||
// session. The sparql-parser config grants that scope read access to the
|
||||
// public graph only, so these reads are the same for every caller. Nothing
|
||||
// here counts, and execution still runs as the caller.
|
||||
//
|
||||
// lookup_values is a suggest helper, not a spec check. It reads candidate
|
||||
// values the LLM can put in a filter; it never runs the spec and never sees
|
||||
@@ -12,7 +14,7 @@ import { query, sparqlEscapeString } from 'mu';
|
||||
const SERVICE_SCOPE = process.env.SERVICE_SCOPE || 'http://services.semantic.works/natural-language-report';
|
||||
export const INLINE_VALUES_MAX = Number(process.env.INLINE_VALUES_MAX || 50);
|
||||
const VALUES_TTL = Number(process.env.VALUES_TTL || 3600) * 1000;
|
||||
const LIMIT = 25;
|
||||
const LOOKUP_LIMIT = 25;
|
||||
|
||||
// class → { values, at }. Refreshed lazily when stale.
|
||||
const cache = new Map();
|
||||
@@ -45,7 +47,7 @@ export async function codeList(type) {
|
||||
}
|
||||
|
||||
// lookupValues(profile, field, term) → the matches as text for the LLM: up
|
||||
// to LIMIT labels and URIs, an exact label (case aside) first. field is
|
||||
// to LOOKUP_LIMIT labels and URIs, an exact label (case aside) first. field is
|
||||
// "shapeLabel.fieldLabel" as the LLM writes it; it resolves to the class to
|
||||
// search.
|
||||
export async function lookupValues(profile, field, term) {
|
||||
@@ -62,12 +64,12 @@ export async function lookupValues(profile, field, term) {
|
||||
// "Gent" must surface before "AGB Erfgoed Gent".
|
||||
const lower = term.toLowerCase();
|
||||
const exactRow = rows.find(r => r.label.toLowerCase() === lower);
|
||||
let matches = (exactRow ? [exactRow, ...rows.filter(r => r !== exactRow)] : rows).slice(0, LIMIT);
|
||||
let matches = (exactRow ? [exactRow, ...rows.filter(r => r !== exactRow)] : rows).slice(0, LOOKUP_LIMIT);
|
||||
if (!matches.length) matches = await codeQuery(term);
|
||||
if (!matches.length) return `no value matches "${term}" in ${field}.`;
|
||||
|
||||
const lines = matches.map(m => `${m.label} <${m.uri}>`);
|
||||
if (matches.length >= LIMIT) lines.push(`(${LIMIT}+ matches, narrow the term)`);
|
||||
if (matches.length >= LOOKUP_LIMIT) lines.push(`(${LOOKUP_LIMIT}+ matches, narrow the term)`);
|
||||
const exact = matches.find(m => m.label.toLowerCase() === lower);
|
||||
if (exact) lines.push(`exact: <${exact.uri}>`);
|
||||
return lines.join('\n');
|
||||
@@ -79,7 +81,7 @@ async function labelQuery(term, type) {
|
||||
?uri a ${type ? `<${type}>` : '?anyType'} ;
|
||||
<http://www.w3.org/2004/02/skos/core#prefLabel> ?label .
|
||||
FILTER(REGEX(str(?label), ${sparqlEscapeString(escapeRegex(term))}, "i"))
|
||||
} ORDER BY ?label LIMIT ${LIMIT * 8}`;
|
||||
} ORDER BY ?label LIMIT ${LOOKUP_LIMIT * 8}`;
|
||||
const r = await publicQuery(q);
|
||||
return r.results.bindings.map(b => ({ label: b.label.value, uri: b.uri.value }));
|
||||
}
|
||||
@@ -92,7 +94,7 @@ async function codeQuery(term) {
|
||||
?uri <http://mu.semte.ch/vocabularies/ext/supervision/nomenclatureCode> ?code ;
|
||||
<http://www.w3.org/2004/02/skos/core#prefLabel> ?label .
|
||||
FILTER(CONTAINS(LCASE(str(?code)), LCASE(${sparqlEscapeString(term)})))
|
||||
} ORDER BY ?label LIMIT ${LIMIT}`;
|
||||
} ORDER BY ?label LIMIT ${LOOKUP_LIMIT}`;
|
||||
try {
|
||||
const r = await publicQuery(q);
|
||||
return r.results.bindings.map(b => ({ label: b.label.value, uri: b.uri.value }));
|
||||
@@ -114,9 +116,10 @@ function resolveClass(profile, field) {
|
||||
if (!shape) return null;
|
||||
const f = shape.fields.find(x => x.name === fieldLabel);
|
||||
if (f?.class) return f.class;
|
||||
if (f?.node) {
|
||||
const linked = profile.shapes.find(s => s.uri === f.node);
|
||||
return linked?.targetClass || null;
|
||||
if (f?.nodes.length) {
|
||||
// sh:or of several entities: only when they share one class
|
||||
const classes = new Set(f.nodes.map(n => profile.shapes.find(s => s.uri === n)?.targetClass || null));
|
||||
return classes.size === 1 ? [...classes][0] : null;
|
||||
}
|
||||
// The model sometimes writes shape.field on a linked shape
|
||||
// ("bestuurseenheid.naam"); resolve it to that shape's target class.
|
||||
121
src/llm/loop.js
121
src/llm/loop.js
@@ -8,10 +8,14 @@
|
||||
// confirm before anything executes; execution runs the agreed spec without
|
||||
// the LLM (../report-assistant.js).
|
||||
|
||||
import { TOOLS, runTool } from './tools.js';
|
||||
import { SPEC_MEDIA_TYPE } from '../chat.js';
|
||||
import { TOOLS, runTool, listProfiles } from './tools.js';
|
||||
import { describeProfile } from './describe.js';
|
||||
import { parseSpec } from '../runner/spec.js';
|
||||
import { SPEC_MEDIA_TYPE, readSpecFile } from '../chat.js';
|
||||
|
||||
const MAX_ROUNDS = Number(process.env.MAX_ROUNDS || 12);
|
||||
// What an older spec becomes: only the newest spec goes to the model in full.
|
||||
const LEFT_OUT = 'Left out: a newer spec follows.';
|
||||
const LLM_BASE_URL = process.env.LLM_BASE_URL;
|
||||
const LLM_MODEL = process.env.LLM_MODEL;
|
||||
const LLM_API_KEY = process.env.LLM_API_KEY;
|
||||
@@ -25,7 +29,10 @@ const LOG_LLM = /^(true|1)$/i.test(process.env.LOG_LLM || '');
|
||||
// Dutch and asks the user to confirm.
|
||||
const REFINE_PROMPT = `You refine report specs with the user. You never write SPARQL and you never run a report.
|
||||
|
||||
The profile lists what can be asked. Compose paths by chaining fields.
|
||||
The profile lists what can be asked. Compose paths by chaining fields. The
|
||||
profiles are listed at the end of this prompt. When there is a current spec,
|
||||
the menu of its profile follows them: use it, and call describe_profile only
|
||||
for another profile.
|
||||
|
||||
Rules:
|
||||
- A column must end on a value, not on a link. Chain one more hop. Only when
|
||||
@@ -48,6 +55,9 @@ Rules:
|
||||
applies. Say that in Dutch in your proposal, so the user can check it.
|
||||
- Only offer what validate_spec accepted. Before you suggest an alternative,
|
||||
write it and validate it; if it does not validate, say it cannot be done.
|
||||
- One answer carries one spec, and one spec is one report on one profile. If
|
||||
the question needs two reports, say so, propose the first, and offer the
|
||||
second as the next step.
|
||||
- You cannot count, sort, take the first N, or compare two subjects. Say so.
|
||||
- The report's title is the dct:title you write in the spec. There is no other
|
||||
title input.
|
||||
@@ -58,9 +68,14 @@ for a filter, so your proposal names real values instead of guesses. That is
|
||||
the only database read you have, and it only suggests values: it never tells
|
||||
you whether a spec is good or how much it matches. The database you read may
|
||||
differ from the one the report runs on, so never treat a lookup as a check.
|
||||
Iterate with the user on the spec itself: paths, filters, columns. When the
|
||||
user wants an earlier spec changed, open it with read_spec and change what
|
||||
they asked for. Never call
|
||||
Iterate with the user on the spec itself: paths, filters, columns. The
|
||||
current spec, the last one in the conversation, comes with the user's
|
||||
message. When the user refines (other columns, filters or title), change the
|
||||
current spec. When the user asks for a different report, one that lists
|
||||
another kind of thing per row, start a new spec. Older specs show only as a
|
||||
bijlage name, and a spec you wrote before in this turn shows as "left out"
|
||||
once a newer one follows. Only when the user asks for an older spec, open it
|
||||
with read_spec and change that one. Never call
|
||||
run_report; there is no such tool here. When the spec is ready, ask the user
|
||||
to confirm execution. Then stop.
|
||||
|
||||
@@ -82,14 +97,17 @@ Reply to the user in Dutch. Earlier turns are context only. Answer the last one.
|
||||
// [{ role, content, attachments }], newest last. spec is the last spec
|
||||
// validate_spec accepted in this turn, or null.
|
||||
export async function ask(turns, profiles) {
|
||||
const messages = [{ role: 'system', content: REFINE_PROMPT }, ...turns.map(withSpecHint)];
|
||||
const current = currentSpec(turns);
|
||||
const messages = [{ role: 'system', content: await systemPrompt(current, profiles) },
|
||||
...withCurrentSpec(turns.map(withSpecHint), current)];
|
||||
let spec = null;
|
||||
|
||||
for (let round = 0; round < MAX_ROUNDS; round++) {
|
||||
const { message } = await chat(messages, TOOLS);
|
||||
const { message } = await chat(newestSpecOnly(messages), TOOLS);
|
||||
messages.push(message);
|
||||
const calls = message.tool_calls || [];
|
||||
if (!calls.length) return { text: message.content || '', spec };
|
||||
if (calls.some(c => c.function.name === 'validate_spec')) message.leftOut = { tool_calls: calls.map(leaveOutSpec) };
|
||||
|
||||
for (const call of calls) {
|
||||
const args = parseArgs(call.function.arguments);
|
||||
@@ -101,21 +119,92 @@ export async function ask(turns, profiles) {
|
||||
}
|
||||
// validate_spec answers "ok" (plus notes) when the spec checks out;
|
||||
// that spec becomes the bijlage of the answer.
|
||||
if (call.function.name === 'validate_spec' && /^ok(\n|$)/.test(result)) spec = args.spec;
|
||||
messages.push({ role: 'tool', tool_call_id: call.id, content: result });
|
||||
// One answer carries one spec: say so when this one replaces another.
|
||||
if (call.function.name === 'validate_spec' && /^ok(\n|$)/.test(result)) {
|
||||
if (spec && spec !== args.spec) {
|
||||
result += '\nThis spec replaces the one you validated before. Your answer carries only this one.';
|
||||
}
|
||||
spec = args.spec;
|
||||
}
|
||||
const reply = { role: 'tool', tool_call_id: call.id, content: result };
|
||||
if (call.function.name === 'read_spec' && result.includes('ReportSpec')) reply.leftOut = { content: LEFT_OUT };
|
||||
messages.push(reply);
|
||||
}
|
||||
}
|
||||
return { text: 'Sorry, dit rapport is te moeilijk. Contacteer de developers.', spec };
|
||||
}
|
||||
|
||||
// The history carries the spec as a bijlage. Paste one hint line per spec
|
||||
// under the message text, so the model knows a spec is there and opens it
|
||||
// with read_spec; the chat itself still keeps the Turtle out of the text.
|
||||
// The history carries the spec as a bijlage, never as text. Paste one hint
|
||||
// line per spec under the message text, so the model knows a spec is there
|
||||
// and can open it with read_spec when the user asks for it. A spec the model
|
||||
// printed in its answer anyway is cut from the text.
|
||||
function withSpecHint(turn) {
|
||||
const content = turn.role === 'assistant' ? withoutSpecText(turn.content) : turn.content;
|
||||
const specs = (turn.attachments || []).filter(a => a.mediaType === SPEC_MEDIA_TYPE);
|
||||
if (!specs.length) return turn;
|
||||
const hints = specs.map(a => `Bijlage "${a.name}", a spec file (read it with read_spec).`);
|
||||
return { ...turn, content: `${turn.content}\n\n${hints.join('\n\n')}` };
|
||||
if (!specs.length) return { ...turn, content };
|
||||
const hints = specs.map(a => `Bijlage "${a.name}", a spec file.`);
|
||||
return { ...turn, content: `${content}\n\n${hints.join('\n\n')}` };
|
||||
}
|
||||
|
||||
function withoutSpecText(text) {
|
||||
return text.replace(/```[\s\S]*?```/g, block => (block.includes('ReportSpec') ? '(spec left out)' : block));
|
||||
}
|
||||
|
||||
// The current spec: the last spec bijlage in the conversation, or null.
|
||||
function currentSpec(turns) {
|
||||
const name = turns.flatMap(t => t.attachments || [])
|
||||
.filter(a => a.mediaType === SPEC_MEDIA_TYPE).at(-1)?.name;
|
||||
const turtle = name && readSpecFile(name);
|
||||
return turtle ? { name, turtle } : null;
|
||||
}
|
||||
|
||||
// The system prompt also carries what the model would otherwise fetch at the
|
||||
// start of every turn: the profile list, and the menu of the current spec's
|
||||
// profile. Without a menu the model calls describe_profile itself.
|
||||
async function systemPrompt(current, profiles) {
|
||||
const parts = [REFINE_PROMPT, `The profiles:\n${listProfiles(profiles)}`];
|
||||
const uri = current && profileOf(current.turtle);
|
||||
if (profiles.has(uri)) {
|
||||
try {
|
||||
parts.push(`The menu of the current spec's profile <${uri}>:\n\n${await describeProfile(profiles.get(uri))}`);
|
||||
} catch (e) {
|
||||
console.error('[llm] could not describe the current profile:', e.message);
|
||||
}
|
||||
}
|
||||
return parts.join('\n\n');
|
||||
}
|
||||
|
||||
function profileOf(turtle) {
|
||||
try {
|
||||
return parseSpec(turtle).profileUri;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// The current spec goes in full under the user's new message: the one spec
|
||||
// the history holds as text.
|
||||
function withCurrentSpec(messages, current) {
|
||||
if (!current) return messages;
|
||||
const { name, turtle } = current;
|
||||
const last = messages.at(-1);
|
||||
return [...messages.slice(0, -1), {
|
||||
...last,
|
||||
content: `${last.content}\n\nThe current spec, bijlage "${name}":\n\n${turtle}`,
|
||||
leftOut: { content: `${last.content}\n\nThe current spec, bijlage "${name}": ${LEFT_OUT}` },
|
||||
}];
|
||||
}
|
||||
|
||||
// Only the newest spec goes to the model in full. A message that holds an
|
||||
// older one goes in its leftOut form; read_spec opens that spec again.
|
||||
function newestSpecOnly(messages) {
|
||||
const newest = messages.findLastIndex(m => m.leftOut);
|
||||
return messages.map((m, i) => (m.leftOut && i !== newest ? { ...m, ...m.leftOut } : m));
|
||||
}
|
||||
|
||||
function leaveOutSpec(call) {
|
||||
if (call.function.name !== 'validate_spec') return call;
|
||||
return { ...call, function: { ...call.function, arguments: JSON.stringify({ spec: LEFT_OUT }) } };
|
||||
}
|
||||
|
||||
// chat(messages, tools) → { message }. The one place that talks to the
|
||||
|
||||
@@ -1,32 +1,29 @@
|
||||
// The tools the LLM calls, refinement only: list_profiles,
|
||||
// describe_profile, validate_spec, read_spec and lookup_values. Each returns
|
||||
// plain text for the model. No report is created, no spec is evaluated;
|
||||
// the code-list reads run under the service scope (public graph).
|
||||
// The tools the LLM calls, refinement only: describe_profile,
|
||||
// validate_spec, read_spec and lookup_values. Each returns
|
||||
// plain text for the model. No report is created, no spec is evaluated; the
|
||||
// database reads run under the service scope (public graph).
|
||||
import { parseSpec } from '../runner/spec.js';
|
||||
import { checkSpec, whereNotes, profileError } from '../runner/check.js';
|
||||
import { describeProfile } from './describe.js';
|
||||
import { lookupValues } from './lookup.js';
|
||||
import { lookupValues } from './explore.js';
|
||||
import { readSpecFile } from '../chat.js';
|
||||
|
||||
// The tool definitions sent to the model. The descriptions are what the LLM
|
||||
// reads to decide which tool to call.
|
||||
export const TOOLS = [
|
||||
tool('list_profiles',
|
||||
'List the report profiles available, with their id and title.',
|
||||
{}),
|
||||
tool('describe_profile',
|
||||
'Describe one profile: its entities, fields, relations and labels. Inline the values of short code lists.',
|
||||
{ profile_id: 'The profile URI returned by list_profiles' }),
|
||||
{ profile_id: 'The profile URI from the profile list' }),
|
||||
tool('validate_spec',
|
||||
'Validate a report spec (Turtle). Returns "ok", followed by one line per rep:where saying where it applies, or one clear sentence saying what is wrong and what to write instead. No database is touched.',
|
||||
{ spec: 'The spec as Turtle text' }),
|
||||
tool('read_spec',
|
||||
'Return the Turtle of a spec bijlage, a ttl file attached to an earlier message in this conversation. Use it to reuse the agreed spec, or to read back an earlier proposal before you change it. Read-only.',
|
||||
'Return the Turtle of a spec bijlage, a ttl file attached to an earlier message in this conversation. The current spec already comes with the user\'s message; use this only when the user asks for an older spec. Read-only.',
|
||||
{ file_name: 'The file name of the spec bijlage, e.g. specificatie-<uuid>.ttl' }),
|
||||
tool('lookup_values',
|
||||
'Search a code list for a term so you can suggest concrete values for a filter (for sh:hasValue or sh:in). Returns up to 25 matches with their URIs and labels, a total, and an exact flag when a label equals the term. Reads candidate values only; it never runs the spec and never tells you how many subjects a spec would match. Read-only, no report is created.',
|
||||
{
|
||||
profile_id: 'The profile URI returned by list_profiles',
|
||||
profile_id: 'The profile URI from the profile list',
|
||||
field: 'The field as "entityLabel.fieldLabel", e.g. "bestuurseenheid.naam"',
|
||||
term: 'The Dutch word or name to search for',
|
||||
}),
|
||||
@@ -49,7 +46,6 @@ function tool(name, description, params) {
|
||||
|
||||
export async function runTool(name, args, profiles) {
|
||||
switch (name) {
|
||||
case 'list_profiles': return listProfiles(profiles);
|
||||
case 'describe_profile': return describe(args, profiles);
|
||||
case 'validate_spec': return validateSpec(args, profiles);
|
||||
case 'read_spec': return readSpec(args);
|
||||
@@ -58,7 +54,8 @@ export async function runTool(name, args, profiles) {
|
||||
}
|
||||
}
|
||||
|
||||
function listProfiles(profiles) {
|
||||
// The profile list, in the system prompt: one line per profile.
|
||||
export function listProfiles(profiles) {
|
||||
return [...profiles.values()].map(p => `${p.title} <${p.uri}>`).join('\n');
|
||||
}
|
||||
|
||||
|
||||
@@ -1,13 +1,18 @@
|
||||
// Values to rows. Subjects in stable order, dedup on term, collect per cell.
|
||||
// rep:self columns take the subject URI; a column with no values is an
|
||||
// empty cell, never a dropped row. A column without rep:collect joins its
|
||||
// values only when there is one; with several, each value gets its own row
|
||||
// (the other columns repeat theirs). rep:row forces that expansion; at most
|
||||
// one column per spec may do that explicitly (checked in check.js).
|
||||
import { sharedSteps } from './spec.js';
|
||||
|
||||
// Values to rows, paired by URI on every step. Two columns that walk the
|
||||
// same steps (with the same rep:where on them) share the nodes on those
|
||||
// steps, the way a rep:where shares nodes with its column. Per subject the
|
||||
// paths of the columns make a tree of the nodes found; every row is one way
|
||||
// down that tree. Columns through the same node stay on one row; branches
|
||||
// under one node multiply. A column with no value is an empty cell, never a
|
||||
// dropped row. sh:groupConcat, sh:min and sh:max fold a column into one cell
|
||||
// per node it shares with the other columns. rep:self takes the subject
|
||||
// URI. Subjects in stable order; identical rows appear once.
|
||||
|
||||
const SH = 'http://www.w3.org/ns/shacl#';
|
||||
const REP = 'http://mu.semte.ch/vocabularies/reporting/';
|
||||
const XSD = 'http://www.w3.org/2001/XMLSchema#';
|
||||
const FOLDING = [SH + 'groupConcat', SH + 'min', SH + 'max'];
|
||||
|
||||
export const NUMERIC_DATATYPES = new Set([
|
||||
'integer', 'decimal', 'double', 'float', 'long', 'int', 'short', 'byte',
|
||||
@@ -15,45 +20,98 @@ export const NUMERIC_DATATYPES = new Set([
|
||||
'unsignedLong', 'unsignedInt', 'unsignedShort', 'unsignedByte',
|
||||
].map(d => XSD + d));
|
||||
|
||||
// assemble(subjects, values, spec) → rows, the header first. values:
|
||||
// subject → column index → chains, one term per step (see columns.js).
|
||||
export function assemble(subjects, values, spec) {
|
||||
const sorted = [...subjects].sort();
|
||||
const tree = columnTree(spec.columns);
|
||||
const rows = [spec.columns.map(c => c.label)];
|
||||
for (const subject of sorted) {
|
||||
const perColumn = values.get(subject) || new Map();
|
||||
const cells = spec.columns.map((col, i) => {
|
||||
if (!col.path.length) return [subject]; // rep:self
|
||||
const list = dedup(perColumn.get(i) || []);
|
||||
return collect(list, col);
|
||||
});
|
||||
const lines = Math.max(1, ...cells.map(c => c.length));
|
||||
for (let r = 0; r < lines; r++) {
|
||||
// the expanding cell cycles its values, every other cell keeps its
|
||||
// value on each line
|
||||
rows.push(cells.map(c => c[Math.min(r, c.length - 1)]));
|
||||
for (const subject of [...subjects].sort()) {
|
||||
const top = { term: { value: subject }, below: new Map() };
|
||||
for (const [index, chains] of values.get(subject) || []) {
|
||||
for (const chain of chains) addChain(top, tree.paths[index], chain);
|
||||
}
|
||||
const seen = new Set();
|
||||
for (const cells of rowsAt(tree.root, top, tree, spec.columns)) {
|
||||
const row = spec.columns.map((col, i) => cells.get(i) ?? '');
|
||||
const key = JSON.stringify(row);
|
||||
if (!seen.has(key)) {
|
||||
seen.add(key);
|
||||
rows.push(row);
|
||||
}
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
function collect(terms, col) {
|
||||
switch (col.collect) {
|
||||
case SH + 'groupConcat':
|
||||
return [terms.map(t => t.value).join(col.separator ?? ',')];
|
||||
case undefined:
|
||||
case null:
|
||||
// no rep:collect: one value is a normal cell, several values each
|
||||
// get their own row instead of a joined string
|
||||
if (terms.length <= 1) return [terms.map(t => t.value).join(',')];
|
||||
return terms.map(t => t.value);
|
||||
case SH + 'min':
|
||||
return [extreme(terms, col, -1)];
|
||||
case SH + 'max':
|
||||
return [extreme(terms, col, 1)];
|
||||
case REP + 'row':
|
||||
return terms.length ? terms.map(t => t.value) : [''];
|
||||
default:
|
||||
throw new Error(`column "${col.label}" has an unknown rep:collect <${col.collect}>. Use sh:groupConcat, rep:row, sh:min or sh:max.`);
|
||||
// The steps of all columns as one tree of branches. Two columns share a
|
||||
// branch as long as they took the same steps with the same rep:where on
|
||||
// them. paths[i] lists the branches column i goes through; a branch knows
|
||||
// the columns that end on it and the columns that go through it.
|
||||
function columnTree(columns) {
|
||||
const newBranch = () => ({ next: new Map(), ends: [], columns: [] });
|
||||
const root = newBranch();
|
||||
const paths = columns.map((col, index) => {
|
||||
let branch = root;
|
||||
const branches = col.path.map((hop, k) => {
|
||||
const where = col.where.filter(w => sharedSteps(col.path, w.path) === k + 1);
|
||||
const key = `${hop.inverse ? '^' : ''}${hop.predicate} ${JSON.stringify(where)}`;
|
||||
if (!branch.next.has(key)) branch.next.set(key, newBranch());
|
||||
branch = branch.next.get(key);
|
||||
branch.columns.push(index);
|
||||
return branch;
|
||||
});
|
||||
branch.ends.push(index);
|
||||
return branches;
|
||||
});
|
||||
return { root, paths };
|
||||
}
|
||||
|
||||
// One path the store returned, subject to value, into the tree of nodes
|
||||
// found for that subject. Nodes are keyed by value: a node reached twice is
|
||||
// one node, a value found twice under one node is one value.
|
||||
function addChain(found, branches, chain) {
|
||||
branches.forEach((branch, k) => {
|
||||
if (!found.below.has(branch)) found.below.set(branch, new Map());
|
||||
const byValue = found.below.get(branch);
|
||||
if (!byValue.has(chain[k].value)) byValue.set(chain[k].value, { term: chain[k], below: new Map() });
|
||||
found = byValue.get(chain[k].value);
|
||||
});
|
||||
}
|
||||
|
||||
// The rows under one found node: its own cells, times the rows of each
|
||||
// branch below it. A branch where nothing was found adds empty cells.
|
||||
function rowsAt(branch, found, tree, columns) {
|
||||
let rows = [new Map(branch.ends.map(i => [i, found.term.value]))];
|
||||
for (const next of branch.next.values()) {
|
||||
const nodes = [...(found.below.get(next)?.values() || [])];
|
||||
const below = next.columns.every(i => FOLDING.includes(columns[i].collect))
|
||||
? [fold(next, nodes, tree, columns)]
|
||||
: nodes.length ? nodes.flatMap(node => rowsAt(next, node, tree, columns)) : [new Map()];
|
||||
rows = rows.flatMap(row => below.map(cells => new Map([...row, ...cells])));
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
// A branch where every column folds gives one set of cells: per column, all
|
||||
// its values in the branch, folded into one cell.
|
||||
function fold(branch, nodes, tree, columns) {
|
||||
const cells = new Map();
|
||||
for (const i of branch.columns) {
|
||||
const rest = tree.paths[i].slice(tree.paths[i].indexOf(branch) + 1);
|
||||
const terms = nodes.flatMap(node => termsBelow(node, rest));
|
||||
const unique = [...new Map(terms.map(t => [t.value, t])).values()];
|
||||
const col = columns[i];
|
||||
cells.set(i, col.collect === SH + 'groupConcat'
|
||||
? unique.map(t => t.value).join(col.separator ?? ',')
|
||||
: extreme(unique, col, col.collect === SH + 'min' ? -1 : 1));
|
||||
}
|
||||
return cells;
|
||||
}
|
||||
|
||||
function termsBelow(node, rest) {
|
||||
if (!rest.length) return [node.term];
|
||||
const byValue = node.below.get(rest[0]);
|
||||
return byValue ? [...byValue.values()].flatMap(next => termsBelow(next, rest.slice(1))) : [];
|
||||
}
|
||||
|
||||
// min/max. Numbers compare as numbers. Dates and times compare as text:
|
||||
@@ -72,17 +130,3 @@ function extreme(terms, col, direction) {
|
||||
const best = terms.reduce((a, b) => (cmp(key(b), key(a)) * direction > 0 ? b : a));
|
||||
return best.value;
|
||||
}
|
||||
|
||||
// Two terms are the same when the URIs match, or lexical form, datatype
|
||||
// and language tag all match.
|
||||
export function dedup(terms) {
|
||||
const seen = new Set();
|
||||
const out = [];
|
||||
for (const t of terms) {
|
||||
const key = t.termType === 'NamedNode'
|
||||
? `u:${t.value}`
|
||||
: `l:${t.value}|${t.datatype?.value || ''}|${t.language || ''}`;
|
||||
if (!seen.has(key)) { seen.add(key); out.push(t); }
|
||||
}
|
||||
return out;
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
import { startShape, fieldsOf, field } from './profile.js';
|
||||
import { startShape, fieldsOf } from './profile.js';
|
||||
import { NUMERIC_DATATYPES } from './assemble.js';
|
||||
import { sharedSteps } from './spec.js';
|
||||
|
||||
@@ -119,14 +119,10 @@ function collectMinMaxTyped(spec, profile) {
|
||||
const name = col.collect === SH + 'min' ? 'sh:min' : 'sh:max';
|
||||
if (!col.path.length) return `${name} on a rep:self column does not apply. A subject URI is not a number or date.`;
|
||||
const last = col.path[col.path.length - 1];
|
||||
const shapeUri = shapeAfter(profile, startShapeUri, col.path);
|
||||
if (!shapeUri) continue; // pathsResolve already reported the real error
|
||||
const f = field(profile, shapeUri, last);
|
||||
if (!f) continue;
|
||||
const kind = f.datatype === XSD_DATETIME || f.datatype === XSD_DATE ? 'date'
|
||||
: NUMERIC_DATATYPES.has(f.datatype) ? 'number'
|
||||
: null;
|
||||
if (!kind) {
|
||||
// no end field: pathsResolve already reported the real error
|
||||
const f = endFields(profile, startShapeUri, col.path)
|
||||
.find(x => x.datatype !== XSD_DATETIME && x.datatype !== XSD_DATE && !NUMERIC_DATATYPES.has(x.datatype));
|
||||
if (f) {
|
||||
return `"${f.name || last.predicate}" is text, so ${name} does not apply. Use rep:row (same as leaving rep:collect out: every value gets its own row).`;
|
||||
}
|
||||
}
|
||||
@@ -291,15 +287,15 @@ function rangeOnValue(spec, profile) {
|
||||
const key = ['minInclusive', 'maxInclusive', 'minExclusive', 'maxExclusive']
|
||||
.find(k => cond.constraints[k]);
|
||||
if (!key || !cond.path.length) continue;
|
||||
const last = cond.path[cond.path.length - 1];
|
||||
const f = field(profile, shapeAfter(profile, startShapeUri, cond.path), last);
|
||||
if (!f || f.datatype) continue; // pathsResolve reports a missing step
|
||||
const target = f.node
|
||||
? `"${profile.shapes.find(s => s.uri === f.node)?.label || f.node}"`
|
||||
// pathsResolve reports a missing step
|
||||
const f = endFields(profile, startShapeUri, cond.path).find(x => !x.datatype);
|
||||
if (!f) continue;
|
||||
const target = f.nodes.length
|
||||
? f.nodes.map(uri => shapeName(profile, uri)).join(' or ')
|
||||
: `the code "${f.name}"`;
|
||||
const steps = fieldsOf(profile, f.node)
|
||||
const steps = [...new Set(f.nodes.flatMap(uri => fieldsOf(profile, uri))
|
||||
.filter(x => x.datatype === XSD_DATE || x.datatype === XSD_DATETIME || NUMERIC_DATATYPES.has(x.datatype))
|
||||
.map(x => `${x.name} (${x.path})`);
|
||||
.map(x => `${x.name} (${x.path})`))];
|
||||
const add = steps.length ? `Add a step: ${steps.join(' or ')}.` : 'End the path on a date or number field.';
|
||||
return `${what} ends on ${target}, not on a value, so sh:${key} never matches. ${add}`;
|
||||
}
|
||||
@@ -316,15 +312,18 @@ export function whereNotes(spec, profile) {
|
||||
});
|
||||
}
|
||||
|
||||
// The entity a path lands on, or the value field when it ends on one.
|
||||
// The entity a path lands on, or the value field when it ends on one. A
|
||||
// step that leads to several entities names them all.
|
||||
function nodeName(profile, startShapeUri, hops) {
|
||||
let shapeUri = startShapeUri;
|
||||
for (const [i, hop] of hops.entries()) {
|
||||
const f = field(profile, shapeUri, hop);
|
||||
if (!f) return `step ${hops.length}`;
|
||||
if (!f.node) return i === hops.length - 1 ? `the value "${f.name}"` : `step ${hops.length}`;
|
||||
shapeUri = f.node;
|
||||
}
|
||||
if (!hops.length) return shapeName(profile, startShapeUri);
|
||||
const ends = endFields(profile, startShapeUri, hops);
|
||||
if (!ends.length) return `step ${hops.length}`;
|
||||
const value = ends.find(f => !f.nodes.length);
|
||||
if (value) return `the value "${value.name}"`;
|
||||
return [...new Set(ends.flatMap(f => f.nodes))].map(uri => shapeName(profile, uri)).join(' or ');
|
||||
}
|
||||
|
||||
function shapeName(profile, shapeUri) {
|
||||
return `"${profile.shapes.find(s => s.uri === shapeUri)?.label || shapeUri}"`;
|
||||
}
|
||||
|
||||
@@ -367,17 +366,24 @@ export function checkSpec(spec, profiles) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// The shape a path lands in: follow sh:node through every hop but the
|
||||
// last. walk() has already validated the path, so a gap here means the walk
|
||||
// reported the real error and this can bail.
|
||||
function shapeAfter(profile, startShapeUri, hops) {
|
||||
let shapeUri = startShapeUri;
|
||||
for (let i = 0; i < hops.length - 1; i++) {
|
||||
const f = field(profile, shapeUri, hops[i]);
|
||||
if (!f || !f.node) return null;
|
||||
shapeUri = f.node;
|
||||
// The fields a hop takes from any of the shapes a path has reached. A field
|
||||
// may link to several entities (sh:or of sh:node: an eenheid's political and
|
||||
// leidinggevende organen): the walk follows them all and the next steps
|
||||
// decide. The query follows predicates only, so this is what the data does.
|
||||
function hopFields(profile, shapeUris, hop) {
|
||||
return shapeUris.flatMap(uri => fieldsOf(profile, uri)
|
||||
.filter(f => f.path === hop.predicate && !!f.inverse === !!hop.inverse));
|
||||
}
|
||||
|
||||
// The fields a path can end on: follow sh:node through every hop but the
|
||||
// last. walk() has already validated the path, so none here means the walk
|
||||
// reported the real error and the caller can bail.
|
||||
function endFields(profile, startShapeUri, hops) {
|
||||
let shapeUris = [startShapeUri];
|
||||
for (const hop of hops.slice(0, -1)) {
|
||||
shapeUris = [...new Set(hopFields(profile, shapeUris, hop).flatMap(f => f.nodes))];
|
||||
}
|
||||
return shapeUri;
|
||||
return hopFields(profile, shapeUris, hops[hops.length - 1]);
|
||||
}
|
||||
|
||||
function walk(profile, startShapeUri, hops, what, isColumn, wantsIri = false) {
|
||||
@@ -385,30 +391,34 @@ function walk(profile, startShapeUri, hops, what, isColumn, wantsIri = false) {
|
||||
return `${what} walks ${hops.length} hops. The limit is ${MAX_PATH_DEPTH}.`;
|
||||
}
|
||||
if (!hops.length) return null;
|
||||
let shapeUri = startShapeUri;
|
||||
let shapeUris = [startShapeUri];
|
||||
for (let i = 0; i < hops.length; i++) {
|
||||
const hop = hops[i];
|
||||
const fields = fieldsOf(profile, shapeUri);
|
||||
const f = fields.find(x => x.path === hop.predicate && !!x.inverse === !!hop.inverse);
|
||||
if (!f) {
|
||||
const have = fields.map(x => (x.inverse ? `${x.name} (${x.path}, inverse)` : `${x.name} (${x.path})`)).join(', ');
|
||||
const shapeLabel = profile.shapes.find(s => s.uri === shapeUri)?.label || shapeUri;
|
||||
return `at "${shapeLabel}" there is no "${hop.predicate}". It has: ${have || '(nothing)'}.`;
|
||||
const found = hopFields(profile, shapeUris, hop);
|
||||
if (!found.length) {
|
||||
const have = [...new Set(shapeUris.flatMap(uri => fieldsOf(profile, uri))
|
||||
.map(x => (x.inverse ? `${x.name} (${x.path}, inverse)` : `${x.name} (${x.path})`)))].join(', ');
|
||||
const shapeLabel = shapeUris.map(uri => shapeName(profile, uri)).join(' or ');
|
||||
return `at ${shapeLabel} there is no "${hop.predicate}". It has: ${have || '(nothing)'}.`;
|
||||
}
|
||||
if (i < hops.length - 1) {
|
||||
if (!f.node) {
|
||||
const links = found.filter(f => f.nodes.length);
|
||||
if (!links.length) {
|
||||
return `${what} hop ${i + 1} <${hop.predicate}> holds a value, not a link. You cannot walk past it.`;
|
||||
}
|
||||
shapeUri = f.node;
|
||||
shapeUris = [...new Set(links.flatMap(f => f.nodes))];
|
||||
} else if (isColumn) {
|
||||
if (wantsIri && f.datatype) {
|
||||
return `${what} has sh:nodeKind sh:IRI but ends on the value "${f.name}", which is no URI. Drop sh:nodeKind.`;
|
||||
}
|
||||
if (!wantsIri && !f.datatype && !f.class) {
|
||||
const valueFields = fieldsOf(profile, f.node)
|
||||
.filter(x => x.datatype || x.class)
|
||||
.map(x => `${x.name} (${x.path})`);
|
||||
return `${what} ends on a link. Add a hop: ${valueFields.join(' or ')}. Only if the user asked for the URI of that node itself, add sh:nodeKind sh:IRI to the column instead.`;
|
||||
// every field the path can end on must suit the column
|
||||
for (const f of found) {
|
||||
if (wantsIri && f.datatype) {
|
||||
return `${what} has sh:nodeKind sh:IRI but ends on the value "${f.name}", which is no URI. Drop sh:nodeKind.`;
|
||||
}
|
||||
if (!wantsIri && !f.datatype && !f.class) {
|
||||
const valueFields = [...new Set(f.nodes.flatMap(uri => fieldsOf(profile, uri))
|
||||
.filter(x => x.datatype || x.class)
|
||||
.map(x => `${x.name} (${x.path})`))];
|
||||
return `${what} ends on a link. Add a hop: ${valueFields.join(' or ')}. Only if the user asked for the URI of that node itself, add sh:nodeKind sh:IRI to the column instead.`;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,11 +4,14 @@ import { hopsPattern, wherePatterns } from './seed.js';
|
||||
// Fetch column values for known subjects. Columns are grouped by their
|
||||
// shared non-leaf hops; two columns share a query only when every hop
|
||||
// before the leaf is identical, and the leaf is read the same way.
|
||||
// rep:self columns (empty path) are filled by assemble.js.
|
||||
// rep:self columns (empty path) are filled by assemble.js. Every query
|
||||
// returns the nodes on the path too, so assemble.js can pair columns by the
|
||||
// nodes they share.
|
||||
|
||||
const SUBJECT_CHUNK_SIZE = Math.max(1, Number(process.env.SUBJECT_CHUNK_SIZE) || 100);
|
||||
|
||||
// fetchColumns(subjects, spec) → subject → column index → [term]
|
||||
// fetchColumns(subjects, spec) → subject → column index → [chain], a chain
|
||||
// being the terms on the path, one per step, the value last.
|
||||
export async function fetchColumns(subjects, spec) {
|
||||
const values = new Map();
|
||||
const groups = groupColumns(spec.columns);
|
||||
@@ -49,17 +52,19 @@ export function groupQuery(group, chunk) {
|
||||
// the end.
|
||||
const nodeVar = (k) => (k === group.hops.length ? '?v' : `?x_${k}`);
|
||||
lines.push(...wherePatterns(group.where, group.hops, nodeVar, 'c'));
|
||||
return `SELECT ?s ?v WHERE {\n VALUES ?s { ${subjects} }\n ${lines.join('\n ')}\n}`;
|
||||
const vars = ['?s', ...group.hops.slice(1).map((hop, i) => `?x_${i + 1}`), '?v'];
|
||||
return `SELECT DISTINCT ${vars.join(' ')} WHERE {\n VALUES ?s { ${subjects} }\n ${lines.join('\n ')}\n}`;
|
||||
}
|
||||
|
||||
function collect(result, group, values) {
|
||||
for (const b of result.results.bindings) {
|
||||
const chain = [...group.hops.slice(1).map((hop, i) => b[`x_${i + 1}`]), b.v];
|
||||
let perColumn = values.get(b.s.value);
|
||||
if (!perColumn) { perColumn = new Map(); values.set(b.s.value, perColumn); }
|
||||
for (const col of group.columns) {
|
||||
let list = perColumn.get(col.index);
|
||||
if (!list) { list = []; perColumn.set(col.index, list); }
|
||||
list.push(b.v);
|
||||
list.push(chain);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import N3 from 'n3';
|
||||
import { readFileSync, readdirSync } from 'fs';
|
||||
import { parseFilter, readRdfList } from './spec.js';
|
||||
|
||||
// Parse a profile (a SHACL shapes graph) and answer questions about it.
|
||||
|
||||
@@ -41,8 +42,8 @@ function loadProfile(file) {
|
||||
const targetClass = store.getQuads(shape, SH + 'targetClass', null)[0]?.object.value || null;
|
||||
const label = store.getQuads(shape, 'http://www.w3.org/2000/01/rdf-schema#label', null)[0]?.object.value || null;
|
||||
const fields = store.getQuads(shape, SH + 'property', null).map(q => parseField(store, q.object));
|
||||
const discriminator = parseDiscriminator(store, shape);
|
||||
return { uri: shape.value, targetClass, label, fields, discriminator };
|
||||
const discriminators = parseDiscriminators(store, shape);
|
||||
return { uri: shape.value, targetClass, label, fields, discriminators };
|
||||
});
|
||||
|
||||
return { uri, title, shapes: shapeList };
|
||||
@@ -64,25 +65,49 @@ function parseField(store, p) {
|
||||
name: one(SH + 'name'),
|
||||
datatype: one(SH + 'datatype'),
|
||||
class: one(SH + 'class'),
|
||||
node: one(SH + 'node'),
|
||||
nodes: linkedShapes(store, p, one(SH + 'name')),
|
||||
};
|
||||
}
|
||||
|
||||
// rep:discriminator tells shapes that share a targetClass apart. sh:minCount 1
|
||||
// becomes a plain triple in the seed, sh:maxCount 0 a FILTER NOT EXISTS.
|
||||
function parseDiscriminator(store, shape) {
|
||||
const node = store.getQuads(shape, REP + 'discriminator', null)[0]?.object;
|
||||
if (!node) return null;
|
||||
const { path, inverse } = readPath(store, node);
|
||||
const one = (pred) => store.getQuads(node, pred, null)[0]?.object?.value || null;
|
||||
const minCount = one(SH + 'minCount');
|
||||
const maxCount = one(SH + 'maxCount');
|
||||
if (!path || (minCount === null && maxCount === null)) return null;
|
||||
return {
|
||||
path, inverse,
|
||||
minCount: minCount !== null ? Number(minCount) : null,
|
||||
maxCount: maxCount !== null ? Number(maxCount) : null,
|
||||
// The entities a field links to: its sh:node, or the sh:node of each item of
|
||||
// its sh:or (SHACL for "each value is one of these").
|
||||
function linkedShapes(store, p, name) {
|
||||
const node = store.getQuads(p, SH + 'node', null)[0]?.object;
|
||||
if (node) return [node.value];
|
||||
const or = store.getQuads(p, SH + 'or', null)[0]?.object;
|
||||
if (!or) return [];
|
||||
return readRdfList(store, or).map(item => {
|
||||
const n = store.getQuads(item, SH + 'node', null)[0]?.object;
|
||||
if (!n) throw new Error(`field "${name}": every sh:or item needs a sh:node.`);
|
||||
return n.value;
|
||||
});
|
||||
}
|
||||
|
||||
// rep:discriminator tells shapes that share a targetClass apart. A shape may
|
||||
// carry several; a subject meets them all.
|
||||
function parseDiscriminators(store, shape) {
|
||||
return store.getQuads(shape, REP + 'discriminator', null)
|
||||
.map(q => parseDiscriminator(store, q.object, shape.value));
|
||||
}
|
||||
|
||||
// One discriminator: a condition written like a spec filter (sh:path plus
|
||||
// sh:minCount, sh:maxCount, sh:in, sh:hasValue, ...), or sh:and / sh:or (a
|
||||
// list) or sh:not (one) over discriminators, as in SHACL.
|
||||
function parseDiscriminator(store, node, shapeUri) {
|
||||
const one = (pred) => store.getQuads(node, SH + pred, null)[0]?.object;
|
||||
const list = (pred) => {
|
||||
const items = readRdfList(store, one(pred)).map(x => parseDiscriminator(store, x, shapeUri));
|
||||
if (!items.length) throw new Error(`rep:discriminator on <${shapeUri}> has an empty sh:${pred}.`);
|
||||
return items;
|
||||
};
|
||||
if (one('and')) return { and: list('and') };
|
||||
if (one('or')) return { or: list('or') };
|
||||
if (one('not')) return { not: parseDiscriminator(store, one('not'), shapeUri) };
|
||||
const d = parseFilter(store, node);
|
||||
if (!d.path.length || !Object.keys(d.constraints).length) {
|
||||
throw new Error(`rep:discriminator on <${shapeUri}> needs a sh:path with a condition, or sh:and, sh:or or sh:not.`);
|
||||
}
|
||||
return d;
|
||||
}
|
||||
|
||||
// The shape a spec starts from. rep:entity wins; a targetClass only resolves
|
||||
@@ -101,8 +126,3 @@ export function shape(profile, shapeUri) {
|
||||
export function fieldsOf(profile, shapeUri) {
|
||||
return shape(profile, shapeUri)?.fields || [];
|
||||
}
|
||||
|
||||
// The field a hop takes from a shape, or null.
|
||||
export function field(profile, shapeUri, hop) {
|
||||
return fieldsOf(profile, shapeUri).find(f => f.path === hop.predicate && !!f.inverse === !!hop.inverse) || null;
|
||||
}
|
||||
|
||||
@@ -25,7 +25,9 @@ export function seedPageQuery(spec, limit, offset, shape = null) {
|
||||
const parts = [];
|
||||
const targetClass = spec.targetClass || shape?.targetClass;
|
||||
parts.push(`?s a ${sparqlEscapeUri(targetClass)} .`);
|
||||
if (shape?.discriminator) parts.push(discriminatorPattern(shape.discriminator));
|
||||
(shape?.discriminators || []).forEach((d, di) => {
|
||||
parts.push(discriminatorPattern(d, `disc${di}`));
|
||||
});
|
||||
spec.filters.forEach((filter, fi) => {
|
||||
parts.push(filterPattern(filter, `f${fi}`));
|
||||
});
|
||||
@@ -36,13 +38,20 @@ LIMIT ${limit} OFFSET ${offset}`;
|
||||
}
|
||||
|
||||
// A shape that shares its class with another says here what sets its
|
||||
// subjects apart: sh:minCount 1 is a plain triple, sh:maxCount 0 a NOT EXISTS.
|
||||
function discriminatorPattern(d) {
|
||||
const p = sparqlEscapeUri(d.path);
|
||||
const triple = d.inverse ? `?disc ${p} ?s .` : `?s ${p} ?disc .`;
|
||||
if (d.minCount !== null && d.minCount >= 1) return triple;
|
||||
if (d.maxCount === 0) return `FILTER NOT EXISTS { ${triple} }`;
|
||||
return null;
|
||||
// subjects apart. A plain discriminator is a filter; sh:and, sh:or and sh:not
|
||||
// become one FILTER over EXISTS tests, which see ?s from outside.
|
||||
function discriminatorPattern(d, varPrefix) {
|
||||
if (!d.and && !d.or && !d.not) return filterPattern(d, varPrefix);
|
||||
return `FILTER(${discriminatorTest(d, varPrefix)})`;
|
||||
}
|
||||
|
||||
function discriminatorTest(d, varPrefix) {
|
||||
if (d.and) return `(${d.and.map((x, i) => discriminatorTest(x, `${varPrefix}_${i}`)).join(' && ')})`;
|
||||
if (d.or) return `(${d.or.map((x, i) => discriminatorTest(x, `${varPrefix}_${i}`)).join(' || ')})`;
|
||||
if (d.not) return `!(${discriminatorTest(d.not, `${varPrefix}_n`)})`;
|
||||
const { maxCount, ...test } = d.constraints;
|
||||
if (maxCount === 0) return `NOT EXISTS { ${filterPattern({ ...d, constraints: test }, varPrefix)} }`;
|
||||
return `EXISTS { ${filterPattern(d, varPrefix)} }`;
|
||||
}
|
||||
|
||||
// Writes out every hop as its own triple pattern.
|
||||
@@ -66,13 +75,15 @@ export function hopsPattern(hops, varPrefix, startVar, lastVar = `?${varPrefix}_
|
||||
}
|
||||
|
||||
// One filter: its condition, plus its rep:where conditions on the nodes it
|
||||
// walks through. With sh:maxCount 0 the whole of it becomes "none such".
|
||||
// walks through. With sh:maxCount 0 the whole of it becomes "none such";
|
||||
// with a value test too (sh:in, sh:hasValue, ...), "none with such a value".
|
||||
function filterPattern(filter, varPrefix) {
|
||||
const { maxCount, ...test } = filter.constraints;
|
||||
if (maxCount === 0) {
|
||||
return `FILTER NOT EXISTS {\n ${filterPattern({ ...filter, constraints: test }, varPrefix)}\n }`;
|
||||
}
|
||||
const nodeVar = (k) => (k === filter.path.length ? `?${varPrefix}_v` : `?${varPrefix}_${k}`);
|
||||
const wheres = wherePatterns(filter.where, filter.path, nodeVar, varPrefix);
|
||||
if (filter.constraints.maxCount === 0) {
|
||||
return `FILTER NOT EXISTS {\n ${[hopsPattern(filter.path, varPrefix, '?s'), ...wheres].join('\n ')}\n }`;
|
||||
}
|
||||
return [conditionPattern(filter.constraints, varPrefix, filter.path, '?s'), ...wheres].join('\n ');
|
||||
}
|
||||
|
||||
@@ -89,10 +100,11 @@ export function wherePatterns(where, hostHops, nodeVar, varPrefix) {
|
||||
// A condition: the hops from startVar and the test on where they end. A
|
||||
// filter starts at ?s; a rep:where starts at a node of its host and, with no
|
||||
// hops left, tests that node itself. sh:maxCount 0 means "none such", so its
|
||||
// hops only live inside the NOT EXISTS.
|
||||
// hops and any value test only live inside the NOT EXISTS.
|
||||
export function conditionPattern(c, varPrefix, hops, startVar) {
|
||||
if (c.maxCount === 0) {
|
||||
return `FILTER NOT EXISTS { ${hopsPattern(hops, varPrefix + '_nex', startVar)} }`;
|
||||
const { maxCount, ...test } = c;
|
||||
return `FILTER NOT EXISTS { ${conditionPattern(test, varPrefix + '_nex', hops, startVar)} }`;
|
||||
}
|
||||
const v = hops.length ? `?${varPrefix}_v` : startVar;
|
||||
const parts = hops.length ? [hopsPattern(hops, varPrefix, startVar)] : [];
|
||||
|
||||
@@ -13,8 +13,13 @@ export function parseSpec(turtle) {
|
||||
const quads = parser.parse(turtle);
|
||||
const store = new N3.Store(quads);
|
||||
|
||||
const specNode = store.getQuads(null, RDF + 'type', REP + 'ReportSpec')[0]?.subject;
|
||||
if (!specNode) throw new Error('spec has no rep:ReportSpec');
|
||||
const specNodes = store.getQuads(null, RDF + 'type', REP + 'ReportSpec').map(q => q.subject);
|
||||
if (!specNodes.length) throw new Error('spec has no rep:ReportSpec');
|
||||
// One spec is one report. A second one would pass unchecked and never run.
|
||||
if (specNodes.length > 1) {
|
||||
throw new Error(`one spec per answer. This Turtle has ${specNodes.length} rep:ReportSpec. Keep one, and offer the other as the next report.`);
|
||||
}
|
||||
const specNode = specNodes[0];
|
||||
|
||||
const one = (s, pred) => store.getQuads(s, pred, null)[0]?.object || null;
|
||||
|
||||
@@ -31,7 +36,7 @@ export function parseSpec(turtle) {
|
||||
return { uri: specNode.value, title, profileUri, targetClass, entity, filters, columns };
|
||||
}
|
||||
|
||||
function parseFilter(store, p) {
|
||||
export function parseFilter(store, p) {
|
||||
let path = [];
|
||||
const pathQuad = store.getQuads(p, SH + 'path', null)[0]?.object;
|
||||
if (pathQuad) path = parsePath(store, pathQuad);
|
||||
@@ -134,7 +139,7 @@ function readConstraints(store, p, { withLabel = false } = {}) {
|
||||
return constraints;
|
||||
}
|
||||
|
||||
function readRdfList(store, node) {
|
||||
export function readRdfList(store, node) {
|
||||
const items = [];
|
||||
let cur = node;
|
||||
while (cur && cur.value !== RDF + 'nil') {
|
||||
|
||||
Reference in New Issue
Block a user