Compare commits
18 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
dac3e2854d | ||
|
|
1f80fdc815 | ||
|
|
538a2d949e | ||
|
|
b1fa1d1cd2 | ||
|
|
0802b75346 | ||
|
|
78a7c42a37 | ||
|
|
80097d3130 | ||
|
|
57292fa052 | ||
|
|
9b0b8a85c9 | ||
|
|
ad278dd680 | ||
|
|
e5ad4a5300 | ||
|
|
2e998a0a4e | ||
|
|
00eeb35923 | ||
|
|
41b91e65d9 | ||
|
|
b1dad3c6e2 | ||
|
|
aec7dc457c | ||
|
|
23f52832d1 | ||
|
|
efa248cfd4 |
@@ -1,2 +1,2 @@
|
||||
FROM semtech/mu-javascript-template:1.8.0
|
||||
LABEL maintainer="info@redpencil"
|
||||
FROM semtech/mu-javascript-template:1.9.1
|
||||
LABEL maintainer="info@redpencil"
|
||||
|
||||
465
README.md
465
README.md
@@ -7,6 +7,31 @@ writes a CSV. The LLM writes no SPARQL and sees no report data.
|
||||
Ad-hoc reports only. The scheduled reports stay in
|
||||
`loket-report-generation-service`.
|
||||
|
||||
## Two modes: refine, then execute
|
||||
|
||||
Every turn is classified first. One small LLM call (`src/llm/mode.js`)
|
||||
returns true/false: does the user's last message clearly say "run it now" or
|
||||
not. The answer picks one of two deterministic code paths.
|
||||
|
||||
**Refinement (Modus A).** The LLM proposes a spec and explains in Dutch what
|
||||
it will list and filter. It may call `lookup_values` to search code lists for
|
||||
candidate values, so its filter suggestions name real values instead of
|
||||
guesses. It never runs a report and never learns what a spec would match: the
|
||||
database it reads may differ from the one the report runs on, so a lookup is
|
||||
a suggestion, not a check. The user must confirm before anything executes.
|
||||
As long as the user has not given a clear command, the turn stays in this
|
||||
mode.
|
||||
|
||||
**Execution (Modus B).** The user has confirmed. No LLM runs. The last spec
|
||||
bijlage in the conversation is the agreed spec: the service reads it from the
|
||||
share, checks it against its profile and runs it as the caller. The answer is
|
||||
a fixed Dutch text with the CSV and the spec as bijlagen. No spec yet, a spec
|
||||
that no longer checks out, or a failed run each give a fixed Dutch failure
|
||||
message. Nothing is retried and no spec is rewritten.
|
||||
|
||||
Only refinement has tools: `describe_profile`, `validate_spec`, `read_spec`
|
||||
and `lookup_values`.
|
||||
|
||||
## How it works
|
||||
|
||||
Two documents drive everything:
|
||||
@@ -16,18 +41,329 @@ Two documents drive everything:
|
||||
| **profile** | the app | once, committed | `config/report-profiles/*.ttl`, mounted at `/config/profiles/` |
|
||||
| **spec** | the LLM | one per question | in the request, not stored |
|
||||
|
||||
A **profile** is a SHACL shapes graph: one `sh:NodeShape` per entity, fields
|
||||
as `sh:property`, code lists via `sh:class`, links via `sh:node`. The service
|
||||
ships no profile and knows no vocabulary; profiles live in the app repo.
|
||||
|
||||
A **spec** is `rep:ReportSpec` + `sh:NodeShape`: `sh:property` selects the
|
||||
subjects, `rep:columns` says what the CSV holds. The two never interact. The
|
||||
namespace is `http://mu.semte.ch/vocabularies/reporting/` (`rep:`).
|
||||
The profile says what the data looks like. The spec says what one report
|
||||
holds. Both are Turtle, and both borrow the words of SHACL, the W3C language
|
||||
to describe RDF data. Neither is used the way a SHACL tool uses it: see "How
|
||||
this relates to SHACL" below.
|
||||
|
||||
The run: check → seed (`SELECT DISTINCT ?s`, paged) → columns (per group, per
|
||||
chunk of subjects) → assemble (dedup + `rep:collect`) → CSV → register file
|
||||
+ report resource. One run per question; the loop stops after `run_report`
|
||||
returns.
|
||||
chunk of subjects, with the nodes on each path) → assemble (pair by the
|
||||
nodes columns share, on every step; `rep:collect`) → CSV → register file
|
||||
+ report resource. One run per confirmed spec.
|
||||
|
||||
## The profile: a map of the data
|
||||
|
||||
A profile (using SHACL + some extensions) lists the kinds of things in the data (entities), what each one
|
||||
has (fields), and which fields lead to other entities (links). The app
|
||||
writes it; the service ships no profile and knows no vocabulary.
|
||||
|
||||
Three parts of the service read it:
|
||||
|
||||
- the validator checks every step of a spec's paths against it;
|
||||
- the LLM gets it as a menu (`describe_profile`), its only picture of the
|
||||
data;
|
||||
- the seed takes the class of the rows from it.
|
||||
|
||||
Nothing checks the data against a profile.
|
||||
|
||||
```turtle
|
||||
<http://data.lblod.info/id/report-profiles/mandaten> a owl:Ontology ;
|
||||
dct:title "Mandaten" .
|
||||
|
||||
ent:Mandataris a sh:NodeShape ;
|
||||
rdfs:label "mandataris" ;
|
||||
sh:targetClass mandaat:Mandataris ;
|
||||
sh:property [ sh:path mandaat:start ; sh:name "start" ; sh:datatype xsd:dateTime ] ;
|
||||
sh:property [ sh:path org:holds ; sh:name "bekleedt mandaat" ; sh:node ent:Mandaat ] ;
|
||||
sh:property [ sh:path mandaat:isBestuurlijkeAliasVan ; sh:name "persoon" ; sh:node ent:Persoon ] .
|
||||
```
|
||||
|
||||
| You write | It means |
|
||||
|---|---|
|
||||
| `owl:Ontology` + `dct:title` | the profile. A spec names this URI in `rep:profile`. |
|
||||
| `sh:NodeShape` | an entity. A spec names this URI in `rep:entity` when it must pick one. |
|
||||
| `rdfs:label` on the entity | its name in the menu and in error messages. Keep it unique in the profile. |
|
||||
| `sh:targetClass` | the `rdf:type` of the entity's subjects |
|
||||
| `sh:property` + `sh:path` | a field: one step, a predicate or `[ sh:inversePath p ]` |
|
||||
| `sh:name` | the field's name in the menu and in messages |
|
||||
| `sh:datatype` | a value field: a column may end here |
|
||||
| `sh:class` | a code-list field: a column may end here. Lists up to `INLINE_VALUES_MAX` values show in the menu. |
|
||||
| `sh:node` | a link to another entity: a spec may take this step |
|
||||
| `sh:or ( [ sh:node A ] [ sh:node B ] )` | a link that reaches A or B |
|
||||
|
||||
The service does not read `sh:description` (write it for people; the LLM
|
||||
never sees it), cardinality (every link counts as "many"), or a field path
|
||||
of more than one step.
|
||||
|
||||
At boot each profile must parse, name itself with `owl:Ontology` and hold at
|
||||
least one entity; a broken profile stops the service. A `sh:node` to an
|
||||
entity that does not exist only shows when a spec takes that step.
|
||||
|
||||
### A link to several kinds
|
||||
|
||||
When one step can reach entities of different kinds, say so with `sh:or`:
|
||||
|
||||
```turtle
|
||||
sh:property [ sh:path [ sh:inversePath besluit:bestuurt ] ; sh:name "bestuursorganen" ;
|
||||
sh:or ( [ sh:node ent:Bestuursorgaan ] [ sh:node ent:LeidinggevendBestuursorgaan ] ) ] .
|
||||
```
|
||||
|
||||
The validator follows all of them, and the next steps decide. A spec that
|
||||
wants one kind adds a `rep:where` on a step only that kind has. Write one
|
||||
field with `sh:or`, not two fields on one path: in SHACL, two fields mean a
|
||||
value must fit both. (The validator reads two fields like `sh:or`, but a
|
||||
SHACL reader would not.)
|
||||
|
||||
### Entities that share a class
|
||||
|
||||
A profile may hold several entities of one class. The portal's
|
||||
`mandaten.ttl` has four kinds of `besluit:Bestuursorgaan`: political and
|
||||
leidinggevend, each fixed (vast) and in time. A spec picks one with
|
||||
`rep:entity`; `rep:discriminator` tells the seed which subjects belong to
|
||||
it.
|
||||
|
||||
A discriminator is written like a spec filter (see "Which rows" below): a
|
||||
`sh:path` from the subject, of any length, and a test on where it ends.
|
||||
Several on one entity must all hold. `sh:and ( ... )`, `sh:or ( ... )` and
|
||||
`sh:not ...` combine them, and may nest.
|
||||
|
||||
```turtle
|
||||
# fixed, and its classification is in the list
|
||||
ent:LeidinggevendBestuursorgaan
|
||||
rep:discriminator [ sh:path generiek:isTijdspecialisatieVan ; sh:maxCount 0 ] ;
|
||||
rep:discriminator [ sh:path org:classification ; sh:in ent:LeidinggevendeOrgaanClassificaties ] .
|
||||
|
||||
# in time, and the classification of its fixed organ is not in the list
|
||||
ent:BestuursorgaanInTijd
|
||||
rep:discriminator [ sh:path generiek:isTijdspecialisatieVan ; sh:minCount 1 ] ;
|
||||
rep:discriminator [ sh:path ( generiek:isTijdspecialisatieVan org:classification ) ;
|
||||
sh:in ent:LeidinggevendeOrgaanClassificaties ; sh:maxCount 0 ] .
|
||||
```
|
||||
|
||||
A rule or a list can be a named node, written once and used in several
|
||||
places. `sh:not` on a named rule gives its exact opposite, so two entities
|
||||
split a class with no overlap and no gap:
|
||||
|
||||
```turtle
|
||||
ent:IsLeidinggevendOrgaan sh:path org:classification ;
|
||||
sh:in ent:LeidinggevendeOrgaanClassificaties .
|
||||
|
||||
ent:LeidinggevendBestuursorgaan rep:discriminator ent:IsLeidinggevendOrgaan .
|
||||
ent:Bestuursorgaan rep:discriminator [ sh:not ent:IsLeidinggevendOrgaan ] .
|
||||
```
|
||||
|
||||
Turtle cannot name a `( ... )` list, so write a named list out:
|
||||
`ent:X rdf:first <a> ; rdf:rest ( <b> <c> ) .` Do not give a named rule the
|
||||
type `sh:NodeShape`: it would become an entity.
|
||||
|
||||
Test what a thing is, not what hangs off it. The classification says what
|
||||
an organ is. "Has a period with a bestuursfunctie" needs no code list, but
|
||||
it trusts the data: an organ that lost its functie would move to the other
|
||||
entity, and no report could find it. `sh:or` of both catches the most.
|
||||
|
||||
The seed turns a plain discriminator into the same SPARQL as a filter.
|
||||
`sh:and`, `sh:or` and `sh:not` become one `FILTER(...)` over `EXISTS` tests;
|
||||
that form is not yet tested against mu-authorization or sparql-parser. A
|
||||
broken discriminator stops the service at boot and names the entity.
|
||||
|
||||
## The spec: one report
|
||||
|
||||
The LLM writes one spec per question. The user reads the proposal and
|
||||
confirms; the service then checks the spec again and runs it. A spec has two
|
||||
halves: which rows, and what the CSV shows about each row.
|
||||
|
||||
```turtle
|
||||
<http://data.lblod.info/id/report-specs/burgemeesters> a rep:ReportSpec , sh:NodeShape ;
|
||||
dct:title "Burgemeesters sinds 2025" ;
|
||||
rep:profile <http://data.lblod.info/id/report-profiles/mandaten> ;
|
||||
sh:targetClass mandaat:Mandataris ;
|
||||
sh:property [ sh:path ( org:holds org:role ) ;
|
||||
sh:hasValue <http://data.vlaanderen.be/id/concept/BestuursfunctieCode/5ab0e9b8a3b2ca7c5e000013> ] ;
|
||||
sh:property [ sh:path mandaat:start ;
|
||||
sh:minInclusive "2025-01-01T00:00:00"^^xsd:dateTime ] ;
|
||||
rep:columns (
|
||||
[ sh:path rep:self ; rdfs:label "mandataris" ]
|
||||
[ sh:path ( mandaat:isBestuurlijkeAliasVan foaf:givenName ) ; rdfs:label "voornaam" ]
|
||||
[ sh:path ( mandaat:isBestuurlijkeAliasVan foaf:familyName ) ; rdfs:label "achternaam" ]
|
||||
[ sh:path ( org:holds [ sh:inversePath org:hasPost ] generiek:isTijdspecialisatieVan
|
||||
besluit:bestuurt skos:prefLabel ) ; rdfs:label "gemeente" ]
|
||||
) .
|
||||
```
|
||||
|
||||
One row per mandataris whose mandate has the role Burgemeester and who
|
||||
started on or after 1 January 2025, with name and gemeente.
|
||||
|
||||
### Which rows
|
||||
|
||||
| You write | It means |
|
||||
|---|---|
|
||||
| `rep:ReportSpec` + `sh:NodeShape` | this is a spec |
|
||||
| `rep:profile` | the profile the spec was written against |
|
||||
| `sh:targetClass` | one row per subject of this class |
|
||||
| `rep:entity` | which entity, when more than one has this class |
|
||||
| `sh:property` | a filter: keep only the subjects that pass |
|
||||
|
||||
A filter has a `sh:path` from the row and a test. When the path reaches
|
||||
several values, the test passes if one of them passes.
|
||||
|
||||
| Test | Keeps the row when |
|
||||
|---|---|
|
||||
| `sh:hasValue x` | the path reaches x |
|
||||
| `sh:in ( x y )` | the path reaches x or y (all URIs, or all text) |
|
||||
| `sh:minInclusive`, `sh:maxInclusive`, `sh:minExclusive`, `sh:maxExclusive` | a date or number at the end lies in range; the path must end on a value field |
|
||||
| `sh:pattern` (+ `sh:flags`) | a value matches the regex |
|
||||
| `rep:anyOf ( "woord" ... )` | a value holds one of these words, upper or lower case; at most 24 words, no URIs |
|
||||
| `sh:minCount 1` | the path leads somewhere |
|
||||
| `sh:minCount 2` | the path reaches two different values |
|
||||
| `sh:maxCount 0` | the path leads nowhere |
|
||||
| `sh:maxCount 0` with `sh:in` or `sh:hasValue` | the path reaches none of these values |
|
||||
|
||||
The service cannot count: `sh:minCount` above 2 and `sh:maxCount` above 0 are
|
||||
refused.
|
||||
|
||||
### A test on the way: `rep:where`
|
||||
|
||||
A filter tests the end of its path. `rep:where` tests a node on the way.
|
||||
Write it like a filter, with its path from the row. The steps it shares with
|
||||
its column or filter are the same nodes; its test applies where its own path
|
||||
ends. On a filter it drops rows; on a column it drops values from the cell,
|
||||
never rows.
|
||||
|
||||
```turtle
|
||||
<http://data.lblod.info/id/report-specs/eenheden-burgemeester> a rep:ReportSpec , sh:NodeShape ;
|
||||
rep:profile <http://data.lblod.info/id/report-profiles/mandaten> ;
|
||||
sh:targetClass besluit:Bestuurseenheid ;
|
||||
rep:columns (
|
||||
[ sh:path skos:prefLabel ; rdfs:label "eenheid" ]
|
||||
[ sh:path ( [ sh:inversePath besluit:bestuurt ] [ sh:inversePath generiek:isTijdspecialisatieVan ]
|
||||
org:hasPost [ sh:inversePath org:holds ] mandaat:isBestuurlijkeAliasVan foaf:familyName ) ;
|
||||
rdfs:label "burgemeester" ;
|
||||
rep:where [ sh:path ( [ sh:inversePath besluit:bestuurt ] [ sh:inversePath generiek:isTijdspecialisatieVan ]
|
||||
org:hasPost org:role ) ;
|
||||
sh:hasValue <http://data.vlaanderen.be/id/concept/BestuursfunctieCode/5ab0e9b8a3b2ca7c5e000013> ] ]
|
||||
) .
|
||||
```
|
||||
|
||||
Per eenheid, the family name of its mayors. The `rep:where` shares three
|
||||
steps with the column, so it tests the mandate, not the organ or the
|
||||
person. `validate_spec` says so: `rep:where 1 on column "burgemeester"
|
||||
applies at "mandaat", after step 3`. The proposal repeats that line to the
|
||||
user.
|
||||
|
||||
A `rep:where` must share at least one step with its column or filter, and
|
||||
holds no `rep:where` of its own.
|
||||
|
||||
### What the CSV shows
|
||||
|
||||
`rep:columns` is a list; each item becomes a column, in that order.
|
||||
|
||||
| You write | It means |
|
||||
|---|---|
|
||||
| `sh:path ( ... )` | the steps from the row to the value |
|
||||
| `sh:path rep:self` | the row's own URI |
|
||||
| `rdfs:label` | the column header; required and unique |
|
||||
| `sh:nodeKind sh:IRI` | show the URI of the node the path ends on (only then may a path end on a link) |
|
||||
| no `rep:collect`, or `rep:collect rep:row` | each value gets its own row; at most one column may say `rep:row` |
|
||||
| `rep:collect sh:groupConcat` (+ `sh:separator`) | all values in one cell |
|
||||
| `rep:collect sh:min` / `sh:max` | the lowest / highest date or number |
|
||||
|
||||
A column holds no tests; those go in `sh:property`, or in the column's
|
||||
`rep:where`.
|
||||
|
||||
### How values become rows
|
||||
|
||||
Columns that walk the same steps (with the same `rep:where`) share the nodes
|
||||
on those steps, and their values pair by those nodes: one mayor's first and
|
||||
last name stay on one row. Where paths split and both sides have several
|
||||
values, every combination becomes a row. A missing value is an empty cell;
|
||||
identical rows show once. `sh:groupConcat`, `sh:min` and `sh:max` fold a
|
||||
column into one cell per node it shares with the other columns.
|
||||
|
||||
### What the validator checks
|
||||
|
||||
The validator (`src/runner/check.js`) runs on every proposal and again
|
||||
before every run. It returns the first problem, in words the LLM can act on.
|
||||
On top of the rules above, it checks that:
|
||||
|
||||
- the Turtle holds one `rep:ReportSpec`, no more: one answer is one report;
|
||||
- every step of every path exists in the profile;
|
||||
- no path is longer than `MAX_PATH_DEPTH` steps;
|
||||
- a date or number test ends on a value field, not on a link;
|
||||
- a class that several entities share comes with `rep:entity`.
|
||||
|
||||
## How this relates to SHACL
|
||||
|
||||
SHACL is the W3C language to check RDF data against shapes: a SHACL tool
|
||||
takes data and shapes, and reports which nodes fail. We borrow its words and
|
||||
its layout, so both documents are valid SHACL syntax and a SHACL tool can
|
||||
read them. We use them for other jobs.
|
||||
|
||||
### The profile: SHACL, with one extension
|
||||
|
||||
Read as SHACL, a profile says what good data looks like: nodes of this
|
||||
class, values of this datatype, links to nodes of that shape. We read the
|
||||
same statements as a map, and never check data with them. That is a use of
|
||||
SHACL, not a change to it. One thing does change the meaning:
|
||||
|
||||
- **`rep:discriminator` narrows `sh:targetClass`.** In SHACL, four shapes
|
||||
with `sh:targetClass besluit:Bestuursorgaan` mean that every organ must
|
||||
fit all four. Here each organ belongs only to the shapes whose
|
||||
discriminators it meets. A SHACL tool skips `rep:` terms, so to it the
|
||||
profile still says "all four". SHACL-AF has a tool for this job, custom
|
||||
targets (see below); we keep `sh:targetClass` so a spec can still name its
|
||||
class, and write the rule in the same terms as a spec filter.
|
||||
|
||||
Two smaller points:
|
||||
|
||||
- In SHACL, a node reached through `sh:node B` must pass B's field rules. We
|
||||
go one step further and treat it as "a B" for the next steps. If the data
|
||||
breaks that, the query finds nothing there; it never finds a wrong row.
|
||||
- Links run in circles (bestuursorgaan to orgaan in tijd and back). SHACL
|
||||
leaves such shapes to each tool. That only matters if someone checks data
|
||||
with a profile.
|
||||
|
||||
### The spec: SHACL syntax, our own meaning
|
||||
|
||||
A spec is valid SHACL syntax, but it does not mean what SHACL says. A SHACL
|
||||
tool reads a spec as a check and reports which subjects fail it. We read it
|
||||
as a query: the rows are the subjects that pass. For some terms the two
|
||||
readings agree; for others they do not.
|
||||
|
||||
| Term | In SHACL | In a spec | Same? |
|
||||
|---|---|---|---|
|
||||
| `sh:targetClass`, `sh:path` | which nodes; which values | same | yes |
|
||||
| `sh:hasValue` | one value equals | same | yes |
|
||||
| `sh:minCount 1` / `2` | at least 1 / 2 values | same | yes |
|
||||
| `sh:maxCount 0` | no values | same | yes |
|
||||
| `sh:in`, ranges, `sh:pattern` | every value passes, and a node without values passes too | one value passes | no |
|
||||
| `sh:maxCount 0` with a test | no values at all | no value that passes | no |
|
||||
| `rep:anyOf`, `rep:where` | skipped | tests of our own | added |
|
||||
| `rep:columns`, `rep:self`, `rep:collect`, `sh:nodeKind` and `rdfs:label` on a column | skipped | the CSV | added |
|
||||
|
||||
So a spec means the same in both readings only if it uses the "yes" terms
|
||||
alone; then its rows are exactly the subjects a SHACL tool finds good. Most
|
||||
specs use `sh:in` or a date range, so most specs do not.
|
||||
|
||||
Why "one value passes": reports ask for "organs with classification X" or
|
||||
"persons with a burgemeester mandate". With SHACL's `sh:in`, the first
|
||||
would also keep organs with no classification, and the second would drop a
|
||||
person who holds a second mandate too. SHACL can say "one value"
|
||||
(`sh:qualifiedValueShape` with `sh:qualifiedMinCount 1`), but the LLM writes
|
||||
the specs, and each extra construct is one more way to get it wrong.
|
||||
|
||||
The CSV half has no SHACL meaning at all: SHACL has no notion of output.
|
||||
|
||||
### SHACL-AF
|
||||
|
||||
SHACL Advanced Features is a W3C Working Group Note from 2017: a proposal,
|
||||
not a standard. It adds custom targets (`sh:SPARQLTarget`: pick nodes with a
|
||||
SPARQL query), values computed from the data, rules and functions. A later
|
||||
draft adds `sh:count`, `sh:sum`, `sh:min`, `sh:max`, `sh:groupConcat` and
|
||||
`sh:orderBy`. We borrow `sh:groupConcat`, `sh:min`, `sh:max` and
|
||||
`sh:separator` as values of `rep:collect`, and keep `sh:count`, `sh:sum` and
|
||||
`sh:orderBy` for when counting and sorting arrive. We do not use its custom
|
||||
targets: a `sh:SPARQLTarget` is a SPARQL string that the service could not
|
||||
check or combine with a spec's filters.
|
||||
|
||||
## One identity: mu-authorization's
|
||||
|
||||
@@ -41,40 +377,87 @@ the caller's.
|
||||
The tools the LLM calls return URIs, counts, statuses and column names —
|
||||
never a cell value.
|
||||
|
||||
## Two endpoints
|
||||
### The rule: no report data reaches the LLM
|
||||
|
||||
No report row, row count or run error ever reaches the LLM. A run starts
|
||||
after the turn's only LLM call (the mode check) and answers with fixed text.
|
||||
What the LLM does see: the conversation, the profiles, validator messages,
|
||||
spec files, and public reference data read under the service scope (inlined
|
||||
code lists and `lookup_values`, see below).
|
||||
|
||||
Keep it that way. Do not add a tool that previews, samples or counts what a
|
||||
spec would match, not even to catch a `rep:where` that hangs on the wrong
|
||||
node. That check stays on the profile: `validate_spec` says where each
|
||||
`rep:where` applies, and the proposal repeats it to the user.
|
||||
|
||||
### The service scope
|
||||
|
||||
The LLM's own reads during refinement (`describe_profile`'s code lists and
|
||||
`lookup_values`) run under a service scope, not the caller's session:
|
||||
`publicQuery` in `src/llm/explore.js`. mu's
|
||||
`query(q, { scope })` sends `mu-auth-scope`; the sparql-parser config grants
|
||||
that scope read access to `http://mu.semte.ch/graphs/public` only. So
|
||||
refinement reads only the public graph, regardless of who calls. Execution
|
||||
still runs as the caller, so the report contains what they may see, and the
|
||||
LLM sees none of it.
|
||||
|
||||
The scope URI is `SERVICE_SCOPE` (default
|
||||
`http://services.semantic.works/natural-language-report`) and must match the
|
||||
`with-scope` grant in the app's sparql-parser `config.lisp`. Do not set
|
||||
`DEFAULT_MU_AUTH_SCOPE` on the service: that would scope every query
|
||||
(including the run and the chat store) and break them.
|
||||
|
||||
## The endpoint
|
||||
|
||||
```
|
||||
POST /ask { question, history? } → { answer, toolCalls }
|
||||
POST /assistant/conversations/:id/turns { content } → 202 { id }
|
||||
DELETE /assistant/conversations/:id → 204
|
||||
```
|
||||
|
||||
`/ask` is synchronous: a Dutch question in, the agent loop runs inside the
|
||||
service (profiles, lookups, spec, validate, repair, run), and the answer in
|
||||
Dutch comes back with a trace of the tool calls. The CSV lands in
|
||||
`data/files/`. `history` carries the earlier turns for follow-ups; the
|
||||
conversation lives on the caller's side, the service stores nothing. One
|
||||
question makes one report; the loop stops after `run_report`.
|
||||
|
||||
`/assistant` is the chat. The turn module in
|
||||
`src/chat/` is generic — it imports nothing from `src/llm/` or `src/runner/`
|
||||
and moves between services without edits — and `src/report-assistant.js` is
|
||||
this service's one `answer` hook. It records the question, answers `202` with
|
||||
the message id, then runs the same loop with the conversation's history. A
|
||||
report run becomes an interim message with a pending file (`as:Document` with
|
||||
no `as:url`); the URL is set when the run ends. The browser polls
|
||||
This is the chat. The route is in `app.js`: it records the question, answers
|
||||
`202` with the message id, then classifies the turn and refines or executes
|
||||
(`src/report-assistant.js`) with the conversation's history. The
|
||||
conversation, message and bijlage reads and writes are in `src/chat.js`. The
|
||||
answer and its bijlagen are written when the turn ends. The browser polls
|
||||
`/chat-conversations/:id?include=messages.attachments` and finds the answer
|
||||
that way. Failure is a message, never a silent loader.
|
||||
|
||||
`/ask` needs `LLM_BASE_URL` set, and answers 503 without it. A full-org report
|
||||
run takes minutes.
|
||||
The chat needs `LLM_BASE_URL`; without it every turn answers with a failure
|
||||
message. A full-org report run takes minutes.
|
||||
|
||||
The delete is the trash bin in the chat's sidebar. It removes the
|
||||
conversation, its messages, their bijlagen (the triples and the files on
|
||||
the share) and the reports of those files. It runs as the caller: the auth
|
||||
layer keeps a user to their own conversations, and a conversation the caller
|
||||
cannot read is a 404.
|
||||
|
||||
### The internal tools
|
||||
|
||||
The loop drives four tool handlers (`src/llm/tools.js`): `list_profiles`,
|
||||
`describe_profile`, `validate_spec`, `run_report`. They are machinery, not an
|
||||
external surface — the tool trace in the answer is the only thing a caller
|
||||
sees of them. All LLM-facing text (system prompt, profile menu, tool
|
||||
descriptions, validator errors) is English; the answer the user sees is Dutch.
|
||||
The refinement loop drives one tool set (`src/llm/tools.js`). The tools are
|
||||
machinery, not an external surface. All LLM-facing text (system prompts,
|
||||
profile menu, tool descriptions, validator errors) is English; the answer the
|
||||
user sees is Dutch.
|
||||
|
||||
### What the model sees
|
||||
|
||||
The system prompt ends with the profile list and, when there is a current
|
||||
spec, the menu of its profile (`src/llm/loop.js`). A turn that stays on that
|
||||
profile needs no `describe_profile` call; the model calls it only for
|
||||
another profile.
|
||||
|
||||
Each request to the model holds one spec in full: the newest.
|
||||
|
||||
- The history is text only. A spec bijlage shows as its file name, and a
|
||||
spec the model printed in its answer is cut out.
|
||||
- The current spec, the last spec bijlage in the conversation, comes in full
|
||||
under the user's new message. When the user refines, the model changes it;
|
||||
when the user asks for a report that lists another kind of thing per row,
|
||||
it starts a new spec.
|
||||
- Within a turn, each newer spec (a `validate_spec` call, or an older spec
|
||||
opened with `read_spec`) replaces the older one with "Left out: a newer
|
||||
spec follows."
|
||||
- The model opens an older spec with `read_spec` only when the user asks for
|
||||
it.
|
||||
|
||||
## Environment
|
||||
|
||||
@@ -86,17 +469,17 @@ descriptions, validator errors) is English; the answer the user sees is Dutch.
|
||||
| `SUBJECT_CHUNK_SIZE` | `100` | subjects per column query (VALUES batch size); lower it when the auth layer rejects long queries |
|
||||
| `ROW_LIMIT` | `200000` | seed limit; failing the job beats truncating |
|
||||
| `MAX_PATH_DEPTH` | `8` | longest allowed spec path |
|
||||
| `RUN_TIMEOUT` | `60` | seconds of silence before `run_report` gives up. Every answered query (seed page, column batch) restarts the clock, so a big report only needs the window per batch, not for the whole run. The run itself is never cancelled: the file still lands in the chat. |
|
||||
| `INLINE_VALUES_MAX` | `50` | code lists up to this size are inlined in `describe_profile` |
|
||||
| `VALUES_TTL` | `3600` | seconds the inlined lists are cached |
|
||||
| `LLM_BASE_URL` | — | enables `/ask`; OpenAI-compatible (`https://ollama.com/v1`) |
|
||||
| `LLM_BASE_URL` | — | the chat's LLM; OpenAI-compatible (`https://ollama.com/v1`) |
|
||||
| `LLM_MODEL` | — | e.g. `gpt-oss:120b` |
|
||||
| `LLM_API_KEY` | — | bearer token, optional for local Ollama |
|
||||
| `SEED_PAGE_SIZE` | `5000` | seed page size |
|
||||
| `SUBJECT_CHUNK_SIZE` | `100` | see above |
|
||||
| `SHARE_DIR` | `/share` | where CSVs are written |
|
||||
| `CHAT_ASSISTANT_URI` | — | the `prov:SoftwareAgent` that signs assistant messages; found by lookup when unset |
|
||||
| `CHAT_ASSISTANT_URI` | `http://data.lblod.info/id/chat-agents/rapportassistent` | the `prov:SoftwareAgent` that signs assistant messages (the portal's chat-assistant migration seeds it) |
|
||||
| `CHAT_HISTORY_LIMIT` | `20` | messages of history a turn reads |
|
||||
| `SERVICE_SCOPE` | `http://services.semantic.works/natural-language-report` | the mu-auth-scope for the LLM's own reads (code lists, `lookup_values`). Must match the `with-scope` grant in the app's sparql-parser config. |
|
||||
|
||||
Do not set `ALLOW_MU_AUTH_SUDO`; the template then refuses sudo queries.
|
||||
|
||||
@@ -107,7 +490,7 @@ Do not set `ALLOW_MU_AUTH_SUDO`; the template then refuses sudo queries.
|
||||
```yaml
|
||||
natural-language-report:
|
||||
build: ../natural-language-report-service/
|
||||
image: lblod/natural-language-report-service:0.1.0
|
||||
image: lblod/natural-language-report-service:0.2.0
|
||||
volumes:
|
||||
- ./data/files:/share
|
||||
- ./config/report-profiles/:/config/profiles/
|
||||
@@ -116,12 +499,12 @@ natural-language-report:
|
||||
`config/dispatcher/dispatcher.ex`:
|
||||
|
||||
```elixir
|
||||
match "/natural-language-reports/*path" do
|
||||
forward conn, path, "http://natural-language-report/"
|
||||
match "/assistant/*path" do
|
||||
forward conn, path, "http://natural-language-report/assistant/"
|
||||
end
|
||||
```
|
||||
|
||||
One route to the service root; `POST /ask` is the endpoint.
|
||||
One route: the chat's turn endpoint.
|
||||
First deploy: `docker compose build natural-language-report`, then
|
||||
`docker compose up -d natural-language-report dispatcher`.
|
||||
|
||||
@@ -150,4 +533,4 @@ non-admin callers. Opening this up means adding the type to their group first.
|
||||
## What it will not do
|
||||
|
||||
No counting, no sorting, no top-N, no comparing two subjects, one class per
|
||||
report, suggestions stop at the public graph.
|
||||
report, suggestions stop at the public graph.
|
||||
|
||||
105
app.js
105
app.js
@@ -1,53 +1,84 @@
|
||||
import { app, errorHandler } from 'mu';
|
||||
import express from 'express';
|
||||
import bodyParser from 'body-parser';
|
||||
import { loadProfiles } from './src/runner/profile.js';
|
||||
import { sessionQuery, sessionUpdate, stripStoreError } from './src/db.js';
|
||||
import { ask } from './src/llm/loop.js';
|
||||
import { mountChat } from './src/chat/index.js';
|
||||
import { reportAssistant } from './src/report-assistant.js';
|
||||
import { readConversation, readHistory, writeMessage, deleteConversation } from './src/chat.js';
|
||||
import { wantsExecution } from './src/llm/mode.js';
|
||||
import { executeSpec, refineSpec } from './src/report-assistant.js';
|
||||
|
||||
const PROFILE_DIR = process.env.PROFILE_DIR || '/config/profiles';
|
||||
// The agent that signs the assistant's messages, seeded by the portal's
|
||||
// chat-assistant migration.
|
||||
const CHAT_ASSISTANT_URI = process.env.CHAT_ASSISTANT_URI || 'http://data.lblod.info/id/chat-agents/rapportassistent';
|
||||
const TITLE_LENGTH = 80;
|
||||
const FAILED = 'Er ging iets mis. Probeer het opnieuw.';
|
||||
|
||||
// Profiles load at boot; handlers await the same promise. A failed load
|
||||
// answers an error per request instead of crashing boot.
|
||||
let profilesPromise = loadProfiles(PROFILE_DIR);
|
||||
profilesPromise.catch(e => console.error('[profiles] loading failed:', e.message));
|
||||
// Profiles load at boot. A broken profile stops the service.
|
||||
const profiles = loadProfiles(PROFILE_DIR);
|
||||
|
||||
async function whenProfiles() {
|
||||
// One chat turn. Records the question, answers 202, then writes the
|
||||
// assistant's answer; the frontend polls the conversation for it. mu's
|
||||
// query/update run as the caller for the whole turn, also after the 202.
|
||||
// The template only parses application/vnd.api+json, the chat sends
|
||||
// application/json.
|
||||
app.post('/assistant/conversations/:id/turns', bodyParser.json(), async function(req, res) {
|
||||
const content = String(req.body?.content ?? '').trim();
|
||||
if (!content) return res.status(400).json({ error: 'content is required' });
|
||||
|
||||
let conversation, history, question;
|
||||
try {
|
||||
return await profilesPromise;
|
||||
// the access check: the caller's session decides what is readable
|
||||
conversation = await readConversation(req.params.id);
|
||||
if (!conversation) return res.status(404).json({ error: 'no such conversation' });
|
||||
history = await readHistory(conversation.uri, CHAT_ASSISTANT_URI);
|
||||
question = await writeMessage({
|
||||
conversationUri: conversation.uri,
|
||||
content,
|
||||
maker: conversation.creator,
|
||||
title: conversation.title ? undefined : content.slice(0, TITLE_LENGTH),
|
||||
});
|
||||
} catch (e) {
|
||||
throw new Error('profiles are not available: ' + e.message);
|
||||
console.error('[chat] could not record the question:', e);
|
||||
return res.status(500).json({ error: e.message });
|
||||
}
|
||||
}
|
||||
|
||||
// The chat. reportAssistant is the answer hook: it runs the LLM loop with the
|
||||
// conversation so far and turns a report run into an interim message with a
|
||||
// pending file.
|
||||
mountChat(app, { path: '/assistant', answer: reportAssistant(whenProfiles) });
|
||||
|
||||
// POST /ask: one question in, the answer in Dutch and a trace of the tool
|
||||
// calls back. Follow-ups carry the earlier turns in `history`.
|
||||
// The template parses application/vnd.api+json only; this route parses the
|
||||
// plain JSON body itself.
|
||||
app.post('/ask', express.json(), async (req, res) => {
|
||||
const { question, history } = req.body || {};
|
||||
if (!question) return res.status(400).json({ error: 'no question in the request body' });
|
||||
if (!process.env.LLM_BASE_URL) return res.status(503).json({ error: 'no LLM configured (set LLM_BASE_URL)' });
|
||||
res.status(202).json({ id: question.id });
|
||||
|
||||
try {
|
||||
const profiles = await whenProfiles();
|
||||
const turns = [
|
||||
...(Array.isArray(history) ? history : []),
|
||||
{ role: 'user', content: question },
|
||||
];
|
||||
const session = { query: sessionQuery, update: sessionUpdate };
|
||||
const { text, trace } = await ask(turns, profiles, session);
|
||||
res.json({ answer: text, toolCalls: trace });
|
||||
const turns = [...history, { role: 'user', content }];
|
||||
const answer = await wantsExecution(turns)
|
||||
? await executeSpec(history, conversation.title, profiles)
|
||||
: await refineSpec(turns, profiles);
|
||||
await writeMessage({
|
||||
conversationUri: conversation.uri,
|
||||
content: answer.text,
|
||||
maker: CHAT_ASSISTANT_URI,
|
||||
attachments: answer.attachments,
|
||||
});
|
||||
} catch (e) {
|
||||
console.error('[ask] failed:', e);
|
||||
res.status(502).json({ error: stripStoreError(e.message) });
|
||||
console.error('[chat] turn failed:', e);
|
||||
try {
|
||||
await writeMessage({ conversationUri: conversation.uri, content: FAILED, maker: CHAT_ASSISTANT_URI });
|
||||
} catch (e2) {
|
||||
console.error('[chat] could not write the failure message:', e2);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
app.use(errorHandler);
|
||||
// Delete a conversation with everything that hangs off it: the messages,
|
||||
// their bijlagen (triples and share files) and the reports. The read is the
|
||||
// access check: the caller's session decides what exists, so another user's
|
||||
// conversation is a 404.
|
||||
app.delete('/assistant/conversations/:id', async function(req, res) {
|
||||
try {
|
||||
const conversation = await readConversation(req.params.id);
|
||||
if (!conversation) return res.status(404).json({ error: 'no such conversation' });
|
||||
const removed = await deleteConversation(conversation.uri);
|
||||
console.log(`[chat] deleted conversation ${req.params.id}, ${removed} share file(s) removed`);
|
||||
res.status(204).send();
|
||||
} catch (e) {
|
||||
console.error('[chat] could not delete the conversation:', e);
|
||||
res.status(500).json({ error: e.message });
|
||||
}
|
||||
});
|
||||
|
||||
app.use(errorHandler);
|
||||
|
||||
875
package-lock.json
generated
875
package-lock.json
generated
@@ -1,15 +1,14 @@
|
||||
{
|
||||
"name": "natural-language-report-service",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "natural-language-report-service",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"express": "^5.2.1",
|
||||
"n3": "^1.17.0"
|
||||
}
|
||||
},
|
||||
@@ -25,19 +24,6 @@
|
||||
"node": ">=6.5"
|
||||
}
|
||||
},
|
||||
"node_modules/accepts": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/accepts/-/accepts-2.0.0.tgz",
|
||||
"integrity": "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"mime-types": "^3.0.0",
|
||||
"negotiator": "^1.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/base64-js": {
|
||||
"version": "1.5.1",
|
||||
"resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz",
|
||||
@@ -58,43 +44,6 @@
|
||||
],
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/body-parser": {
|
||||
"version": "2.3.0",
|
||||
"resolved": "https://registry.npmjs.org/body-parser/-/body-parser-2.3.0.tgz",
|
||||
"integrity": "sha512-2cGmJupaNgg+QUwVLAucDuWuoMZ6EX9iHDRswZ5lsNYEmwPaRknMPCLZz07yTzVq/83p4o/wzbDZbBrTvGGTIw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"bytes": "^3.1.2",
|
||||
"content-type": "^2.0.0",
|
||||
"debug": "^4.4.3",
|
||||
"http-errors": "^2.0.1",
|
||||
"iconv-lite": "^0.7.2",
|
||||
"on-finished": "^2.4.1",
|
||||
"qs": "^6.15.2",
|
||||
"raw-body": "^3.0.2",
|
||||
"type-is": "^2.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/body-parser/node_modules/content-type": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz",
|
||||
"integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/buffer": {
|
||||
"version": "6.0.3",
|
||||
"resolved": "https://registry.npmjs.org/buffer/-/buffer-6.0.3.tgz",
|
||||
@@ -119,184 +68,6 @@
|
||||
"ieee754": "^1.2.1"
|
||||
}
|
||||
},
|
||||
"node_modules/bytes": {
|
||||
"version": "3.1.2",
|
||||
"resolved": "https://registry.npmjs.org/bytes/-/bytes-3.1.2.tgz",
|
||||
"integrity": "sha512-/Nf7TyzTx6S3yRJObOAV7956r8cr2+Oj8AC5dt8wSP3BQAoeX58NoHyCU8P8zGkNXStjTSi6fzO6F0pBdcYbEg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/call-bind-apply-helpers": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
|
||||
"integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"es-errors": "^1.3.0",
|
||||
"function-bind": "^1.1.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/call-bound": {
|
||||
"version": "1.0.4",
|
||||
"resolved": "https://registry.npmjs.org/call-bound/-/call-bound-1.0.4.tgz",
|
||||
"integrity": "sha512-+ys997U96po4Kx/ABpBCqhA9EuxJaQWDQg7295H4hBphv3IZg0boBKuwYpt4YXp6MZ5AmZQnU/tyMTlRpaSejg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"call-bind-apply-helpers": "^1.0.2",
|
||||
"get-intrinsic": "^1.3.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/content-disposition": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-1.1.0.tgz",
|
||||
"integrity": "sha512-5jRCH9Z/+DRP7rkvY83B+yGIGX96OYdJmzngqnw2SBSxqCFPd0w2km3s5iawpGX8krnwSGmF0FW5Nhr0Hfai3g==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/content-type": {
|
||||
"version": "1.0.5",
|
||||
"resolved": "https://registry.npmjs.org/content-type/-/content-type-1.0.5.tgz",
|
||||
"integrity": "sha512-nTjqfcBFEipKdXCv4YDQWCfmcLZKm81ldF0pAopTvyrFGVbcR6P/VAAd5G7N+0tTr8QqiU0tFadD6FK4NtJwOA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/cookie": {
|
||||
"version": "0.7.2",
|
||||
"resolved": "https://registry.npmjs.org/cookie/-/cookie-0.7.2.tgz",
|
||||
"integrity": "sha512-yki5XnKuf750l50uGTllt6kKILY4nQ1eNIQatoXEByZ5dWgnKqbnqmTrBE5B4N7lrMJKQ2ytWMiTO2o0v6Ew/w==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/cookie-signature": {
|
||||
"version": "1.2.2",
|
||||
"resolved": "https://registry.npmjs.org/cookie-signature/-/cookie-signature-1.2.2.tgz",
|
||||
"integrity": "sha512-D76uU73ulSXrD1UXF4KE2TMxVVwhsnCgfAyTg9k8P6KGZjlXKrOLe4dJQKI3Bxi5wjesZoFXJWElNWBjPZMbhg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=6.6.0"
|
||||
}
|
||||
},
|
||||
"node_modules/debug": {
|
||||
"version": "4.4.3",
|
||||
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
|
||||
"integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"ms": "^2.1.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=6.0"
|
||||
},
|
||||
"peerDependenciesMeta": {
|
||||
"supports-color": {
|
||||
"optional": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/depd": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz",
|
||||
"integrity": "sha512-g7nH6P6dyDioJogAAGprGpCtVImJhpPk/roCzdb3fIh61/s/nPsfR6onyMwkCAR/OlC3yBC0lESvUoQEAssIrw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/dunder-proto": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
|
||||
"integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"call-bind-apply-helpers": "^1.0.1",
|
||||
"es-errors": "^1.3.0",
|
||||
"gopd": "^1.2.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/ee-first": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
|
||||
"integrity": "sha512-WMwm9LhRUo+WUaRN+vRuETqG89IgZphVSNkdFgeb6sS/E4OrDIN7t48CAewSHXc6C8lefD8KKfr5vY61brQlow==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/encodeurl": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/encodeurl/-/encodeurl-2.0.0.tgz",
|
||||
"integrity": "sha512-Q0n9HRi4m6JuGIV1eFlmvJB7ZEVxu93IrMyiMsGC0lrMJMWzRgx6WGquyfQgZVb31vhGgXnfmPNNXmxnOkRBrg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/es-define-property": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz",
|
||||
"integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/es-errors": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz",
|
||||
"integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/es-object-atoms": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.2.tgz",
|
||||
"integrity": "sha512-HWcBoN6NileqtSydK2FqHbS/LoDd2pqrnQHLyJzBj4kOp/ky2MWMN694xOfkK8/SnUsW2DH7EfyVlydKCsm1Zw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"es-errors": "^1.3.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/escape-html": {
|
||||
"version": "1.0.3",
|
||||
"resolved": "https://registry.npmjs.org/escape-html/-/escape-html-1.0.3.tgz",
|
||||
"integrity": "sha512-NiSupZ4OeuGwr68lGIeym/ksIZMJodUGOSCZ/FSnTxcrekbvqrgdUxlJOMpijaKZVjAJrWrGs/6Jy8OMuyj9ow==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/etag": {
|
||||
"version": "1.8.1",
|
||||
"resolved": "https://registry.npmjs.org/etag/-/etag-1.8.1.tgz",
|
||||
"integrity": "sha512-aIL5Fx7mawVa300al2BnEE4iNvo1qETxLrPI/o05L7z6go7fCw1J6EQmbK4FmJ2AS7kgVF/KEZWufBfdClMcPg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/event-target-shim": {
|
||||
"version": "5.0.1",
|
||||
"resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz",
|
||||
@@ -315,206 +86,6 @@
|
||||
"node": ">=0.8.x"
|
||||
}
|
||||
},
|
||||
"node_modules/express": {
|
||||
"version": "5.2.1",
|
||||
"resolved": "https://registry.npmjs.org/express/-/express-5.2.1.tgz",
|
||||
"integrity": "sha512-hIS4idWWai69NezIdRt2xFVofaF4j+6INOpJlVOLDO8zXGpUVEVzIYk12UUi2JzjEzWL3IOAxcTubgz9Po0yXw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"accepts": "^2.0.0",
|
||||
"body-parser": "^2.2.1",
|
||||
"content-disposition": "^1.0.0",
|
||||
"content-type": "^1.0.5",
|
||||
"cookie": "^0.7.1",
|
||||
"cookie-signature": "^1.2.1",
|
||||
"debug": "^4.4.0",
|
||||
"depd": "^2.0.0",
|
||||
"encodeurl": "^2.0.0",
|
||||
"escape-html": "^1.0.3",
|
||||
"etag": "^1.8.1",
|
||||
"finalhandler": "^2.1.0",
|
||||
"fresh": "^2.0.0",
|
||||
"http-errors": "^2.0.0",
|
||||
"merge-descriptors": "^2.0.0",
|
||||
"mime-types": "^3.0.0",
|
||||
"on-finished": "^2.4.1",
|
||||
"once": "^1.4.0",
|
||||
"parseurl": "^1.3.3",
|
||||
"proxy-addr": "^2.0.7",
|
||||
"qs": "^6.14.0",
|
||||
"range-parser": "^1.2.1",
|
||||
"router": "^2.2.0",
|
||||
"send": "^1.1.0",
|
||||
"serve-static": "^2.2.0",
|
||||
"statuses": "^2.0.1",
|
||||
"type-is": "^2.0.1",
|
||||
"vary": "^1.1.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/finalhandler": {
|
||||
"version": "2.1.1",
|
||||
"resolved": "https://registry.npmjs.org/finalhandler/-/finalhandler-2.1.1.tgz",
|
||||
"integrity": "sha512-S8KoZgRZN+a5rNwqTxlZZePjT/4cnm0ROV70LedRHZ0p8u9fRID0hJUZQpkKLzro8LfmC8sx23bY6tVNxv8pQA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"debug": "^4.4.0",
|
||||
"encodeurl": "^2.0.0",
|
||||
"escape-html": "^1.0.3",
|
||||
"on-finished": "^2.4.1",
|
||||
"parseurl": "^1.3.3",
|
||||
"statuses": "^2.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 18.0.0"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/forwarded": {
|
||||
"version": "0.2.0",
|
||||
"resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz",
|
||||
"integrity": "sha512-buRG0fpBtRHSTCOASe6hD258tEubFoRLb4ZNA6NxMVHNw2gOcwHo9wyablzMzOA5z9xA9L1KNjk/Nt6MT9aYow==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/fresh": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/fresh/-/fresh-2.0.0.tgz",
|
||||
"integrity": "sha512-Rx/WycZ60HOaqLKAi6cHRKKI7zxWbJ31MhntmtwMoaTeF7XFH9hhBp8vITaMidfljRQ6eYWCKkaTK+ykVJHP2A==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/function-bind": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz",
|
||||
"integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/get-intrinsic": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz",
|
||||
"integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"call-bind-apply-helpers": "^1.0.2",
|
||||
"es-define-property": "^1.0.1",
|
||||
"es-errors": "^1.3.0",
|
||||
"es-object-atoms": "^1.1.1",
|
||||
"function-bind": "^1.1.2",
|
||||
"get-proto": "^1.0.1",
|
||||
"gopd": "^1.2.0",
|
||||
"has-symbols": "^1.1.0",
|
||||
"hasown": "^2.0.2",
|
||||
"math-intrinsics": "^1.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/get-proto": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz",
|
||||
"integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"dunder-proto": "^1.0.1",
|
||||
"es-object-atoms": "^1.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/gopd": {
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
|
||||
"integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/has-symbols": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz",
|
||||
"integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/hasown": {
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.4.tgz",
|
||||
"integrity": "sha512-T2UbfbBEF32wiepXIsMlTW9+dDYC6wMh/t/vYA4tuOMKqWz/n3vr1NFSxQiyP+zk2mXsoMA/i/7qV6LKut1t1A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"function-bind": "^1.1.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/http-errors": {
|
||||
"version": "2.0.1",
|
||||
"resolved": "https://registry.npmjs.org/http-errors/-/http-errors-2.0.1.tgz",
|
||||
"integrity": "sha512-4FbRdAX+bSdmo4AUFuS0WNiPz8NgFt+r8ThgNWmlrjQjt1Q7ZR9+zTlce2859x4KSXrwIsaeTqDoKQmtP8pLmQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"depd": "~2.0.0",
|
||||
"inherits": "~2.0.4",
|
||||
"setprototypeof": "~1.2.0",
|
||||
"statuses": "~2.0.2",
|
||||
"toidentifier": "~1.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/iconv-lite": {
|
||||
"version": "0.7.3",
|
||||
"resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.3.tgz",
|
||||
"integrity": "sha512-IKXpvIzjnC9XTAUbVBcMfGS0EPaIXtW6v+zr+RRp+hqULEpo0owZax6wyRwPOJbWbzjYspQwusTsfVr0ifh4uQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"safer-buffer": ">= 2.1.2 < 3.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/ieee754": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz",
|
||||
@@ -535,92 +106,6 @@
|
||||
],
|
||||
"license": "BSD-3-Clause"
|
||||
},
|
||||
"node_modules/inherits": {
|
||||
"version": "2.0.4",
|
||||
"resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz",
|
||||
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/ipaddr.js": {
|
||||
"version": "1.9.1",
|
||||
"resolved": "https://registry.npmjs.org/ipaddr.js/-/ipaddr.js-1.9.1.tgz",
|
||||
"integrity": "sha512-0KI/607xoxSToH7GjN1FfSbLoU0+btTicjsQSWQlh/hZykN8KpmMf7uYwPW3R+akZ6R/w18ZlXSHBYXiYUPO3g==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.10"
|
||||
}
|
||||
},
|
||||
"node_modules/is-promise": {
|
||||
"version": "4.0.0",
|
||||
"resolved": "https://registry.npmjs.org/is-promise/-/is-promise-4.0.0.tgz",
|
||||
"integrity": "sha512-hvpoI6korhJMnej285dSg6nu1+e6uxs7zG3BYAm5byqDsgJNWwxzM6z6iZiAgQR4TJ30JmBTOwqZUw3WlyH3AQ==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/math-intrinsics": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz",
|
||||
"integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
}
|
||||
},
|
||||
"node_modules/media-typer": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/media-typer/-/media-typer-1.1.1.tgz",
|
||||
"integrity": "sha512-yz3xRaG20c6/BOzvYoDaGtPmGscs7YivItZEEqe6GbwNfHuxu9YNmvnEkMzKldAGY4/80pRcQRZSEnhquk9XuQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/merge-descriptors": {
|
||||
"version": "2.0.0",
|
||||
"resolved": "https://registry.npmjs.org/merge-descriptors/-/merge-descriptors-2.0.0.tgz",
|
||||
"integrity": "sha512-Snk314V5ayFLhp3fkUREub6WtjBfPdCPY1Ln8/8munuLuiYhsABgBVWsozAG+MWMbVEvcdcpbi9R7ww22l9Q3g==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/mime-db": {
|
||||
"version": "1.54.0",
|
||||
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.54.0.tgz",
|
||||
"integrity": "sha512-aU5EJuIN2WDemCcAp2vFBfp/m4EAhWJnUNSSw0ixs7/kXbd6Pg64EmwJkNdFhB8aWt1sH2CTXrLxo/iAGV3oPQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/mime-types": {
|
||||
"version": "3.0.2",
|
||||
"resolved": "https://registry.npmjs.org/mime-types/-/mime-types-3.0.2.tgz",
|
||||
"integrity": "sha512-Lbgzdk0h4juoQ9fCKXW4by0UJqj+nOOrI9MJ1sSj4nI8aI2eo1qmvQEie4VD1glsS250n15LsWsYtCugiStS5A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"mime-db": "^1.54.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/ms": {
|
||||
"version": "2.1.3",
|
||||
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
|
||||
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/n3": {
|
||||
"version": "1.26.0",
|
||||
"resolved": "https://registry.npmjs.org/n3/-/n3-1.26.0.tgz",
|
||||
@@ -634,87 +119,6 @@
|
||||
"node": ">=12.0"
|
||||
}
|
||||
},
|
||||
"node_modules/negotiator": {
|
||||
"version": "1.1.0",
|
||||
"resolved": "https://registry.npmjs.org/negotiator/-/negotiator-1.1.0.tgz",
|
||||
"integrity": "sha512-NMPBRMJgiQHjbd8phG3Vebdx4kZ1H121rbl5IkMqeOsahptB9BKo/d7oJ3zTXqTgagn2bWlNSXkh0QUGM31RYg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"content-type": "^2.1.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/negotiator/node_modules/content-type": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz",
|
||||
"integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/object-inspect": {
|
||||
"version": "1.13.4",
|
||||
"resolved": "https://registry.npmjs.org/object-inspect/-/object-inspect-1.13.4.tgz",
|
||||
"integrity": "sha512-W67iLl4J2EXEGTbfeHCffrjDfitvLANg0UlX3wFUUSTx92KXRFegMHUVgSqE+wvhAbi4WqjGg9czysTV2Epbew==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/on-finished": {
|
||||
"version": "2.4.1",
|
||||
"resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",
|
||||
"integrity": "sha512-oVlzkg3ENAhCk2zdv7IJwd/QUD4z2RxRwpkcGY8psCVcCYZNq4wYnVWALHM+brtuJjePWiYF/ClmuDr8Ch5+kg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"ee-first": "1.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/once": {
|
||||
"version": "1.4.0",
|
||||
"resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
|
||||
"integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==",
|
||||
"license": "ISC",
|
||||
"dependencies": {
|
||||
"wrappy": "1"
|
||||
}
|
||||
},
|
||||
"node_modules/parseurl": {
|
||||
"version": "1.3.3",
|
||||
"resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
|
||||
"integrity": "sha512-CiyeOxFT/JZyN5m0z9PfXw4SCBJ6Sygz1Dpl0wqjlhDEGGBP1GnsUVEL0p63hoG1fcj3fHynXi9NYO4nWOL+qQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/path-to-regexp": {
|
||||
"version": "8.4.2",
|
||||
"resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-8.4.2.tgz",
|
||||
"integrity": "sha512-qRcuIdP69NPm4qbACK+aDogI5CBDMi1jKe0ry5rSQJz8JVLsC7jV8XpiJjGRLLol3N+R5ihGYcrPLTno6pAdBA==",
|
||||
"license": "MIT",
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/process": {
|
||||
"version": "0.11.10",
|
||||
"resolved": "https://registry.npmjs.org/process/-/process-0.11.10.tgz",
|
||||
@@ -724,63 +128,6 @@
|
||||
"node": ">= 0.6.0"
|
||||
}
|
||||
},
|
||||
"node_modules/proxy-addr": {
|
||||
"version": "2.0.7",
|
||||
"resolved": "https://registry.npmjs.org/proxy-addr/-/proxy-addr-2.0.7.tgz",
|
||||
"integrity": "sha512-llQsMLSUDUPT44jdrU/O37qlnifitDP+ZwrmmZcoSKyLKvtZxpyV0n2/bD/N4tBAAZ/gJEdZU7KMraoK1+XYAg==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"forwarded": "0.2.0",
|
||||
"ipaddr.js": "1.9.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.10"
|
||||
}
|
||||
},
|
||||
"node_modules/qs": {
|
||||
"version": "6.15.3",
|
||||
"resolved": "https://registry.npmjs.org/qs/-/qs-6.15.3.tgz",
|
||||
"integrity": "sha512-O9gl3zCl5h5blw1KGUzQKhA5oUXSl8rwUIM5o0S3nCXMliSvy5Dzx7/DJcI+SwgICv+IneSZwhBh1oSyEHA71A==",
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"es-define-property": "^1.0.1",
|
||||
"side-channel": "^1.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=0.6"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/range-parser": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.3.0.tgz",
|
||||
"integrity": "sha512-hek2mFQpPuI4E1BBKrSto+BU3e3x4xuarsbiwr3+lf7p44juvFMV0XFWQAP3xUyqXA4RrXLIoaSUGbSt056ZMw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.6"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/raw-body": {
|
||||
"version": "3.0.2",
|
||||
"resolved": "https://registry.npmjs.org/raw-body/-/raw-body-3.0.2.tgz",
|
||||
"integrity": "sha512-K5zQjDllxWkf7Z5xJdV0/B0WTNqx6vxG70zJE4N0kBs4LovmEYWJzQGxC9bS9RAKu3bgM40lrd5zoLJ12MQ5BA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"bytes": "~3.1.2",
|
||||
"http-errors": "~2.0.1",
|
||||
"iconv-lite": "~0.7.0",
|
||||
"unpipe": "~1.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.10"
|
||||
}
|
||||
},
|
||||
"node_modules/readable-stream": {
|
||||
"version": "4.7.0",
|
||||
"resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-4.7.0.tgz",
|
||||
@@ -797,22 +144,6 @@
|
||||
"node": "^12.22.0 || ^14.17.0 || >=16.0.0"
|
||||
}
|
||||
},
|
||||
"node_modules/router": {
|
||||
"version": "2.2.0",
|
||||
"resolved": "https://registry.npmjs.org/router/-/router-2.2.0.tgz",
|
||||
"integrity": "sha512-nLTrUKm2UyiL7rlhapu/Zl45FwNgkZGaCpZbIHajDYgwlJCOzLSk+cIPAnsEqV955GjILJnKbdQC1nVPz+gAYQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"debug": "^4.4.0",
|
||||
"depd": "^2.0.0",
|
||||
"is-promise": "^4.0.0",
|
||||
"parseurl": "^1.3.3",
|
||||
"path-to-regexp": "^8.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 18"
|
||||
}
|
||||
},
|
||||
"node_modules/safe-buffer": {
|
||||
"version": "5.2.1",
|
||||
"resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz",
|
||||
@@ -833,144 +164,6 @@
|
||||
],
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/safer-buffer": {
|
||||
"version": "2.1.2",
|
||||
"resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
|
||||
"integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==",
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/send": {
|
||||
"version": "1.2.1",
|
||||
"resolved": "https://registry.npmjs.org/send/-/send-1.2.1.tgz",
|
||||
"integrity": "sha512-1gnZf7DFcoIcajTjTwjwuDjzuz4PPcY2StKPlsGAQ1+YH20IRVrBaXSWmdjowTJ6u8Rc01PoYOGHXfP1mYcZNQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"debug": "^4.4.3",
|
||||
"encodeurl": "^2.0.0",
|
||||
"escape-html": "^1.0.3",
|
||||
"etag": "^1.8.1",
|
||||
"fresh": "^2.0.0",
|
||||
"http-errors": "^2.0.1",
|
||||
"mime-types": "^3.0.2",
|
||||
"ms": "^2.1.3",
|
||||
"on-finished": "^2.4.1",
|
||||
"range-parser": "^1.2.1",
|
||||
"statuses": "^2.0.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/serve-static": {
|
||||
"version": "2.2.1",
|
||||
"resolved": "https://registry.npmjs.org/serve-static/-/serve-static-2.2.1.tgz",
|
||||
"integrity": "sha512-xRXBn0pPqQTVQiC8wyQrKs2MOlX24zQ0POGaj0kultvoOCstBQM5yvOhAVSUwOMjQtTvsPWoNCHfPGwaaQJhTw==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"encodeurl": "^2.0.0",
|
||||
"escape-html": "^1.0.3",
|
||||
"parseurl": "^1.3.3",
|
||||
"send": "^1.2.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/setprototypeof": {
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/setprototypeof/-/setprototypeof-1.2.0.tgz",
|
||||
"integrity": "sha512-E5LDX7Wrp85Kil5bhZv46j8jOeboKq5JMmYM3gVGdGH8xFpPWXUMsNrlODCrkoxMEeNi/XZIwuRvY4XNwYMJpw==",
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/side-channel": {
|
||||
"version": "1.1.1",
|
||||
"resolved": "https://registry.npmjs.org/side-channel/-/side-channel-1.1.1.tgz",
|
||||
"integrity": "sha512-6x6dK6zJdpTzF4sQeNYxwtvBzf6Eg4GtlesS94HOvTudUeyK2WXAaIfmDgsyslYrRBeFIlsi54AYsFGUuhmvrQ==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"es-errors": "^1.3.0",
|
||||
"object-inspect": "^1.13.4",
|
||||
"side-channel-list": "^1.0.1",
|
||||
"side-channel-map": "^1.0.1",
|
||||
"side-channel-weakmap": "^1.0.2"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/side-channel-list": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/side-channel-list/-/side-channel-list-1.0.1.tgz",
|
||||
"integrity": "sha512-mjn/0bi/oUURjc5Xl7IaWi/OJJJumuoJFQJfDDyO46+hBWsfaVM65TBHq2eoZBhzl9EchxOijpkbRC8SVBQU0w==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"es-errors": "^1.3.0",
|
||||
"object-inspect": "^1.13.4"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/side-channel-map": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/side-channel-map/-/side-channel-map-1.0.1.tgz",
|
||||
"integrity": "sha512-VCjCNfgMsby3tTdo02nbjtM/ewra6jPHmpThenkTYh8pG9ucZ/1P8So4u4FGBek/BjpOVsDCMoLA/iuBKIFXRA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"call-bound": "^1.0.2",
|
||||
"es-errors": "^1.3.0",
|
||||
"get-intrinsic": "^1.2.5",
|
||||
"object-inspect": "^1.13.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/side-channel-weakmap": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/side-channel-weakmap/-/side-channel-weakmap-1.0.2.tgz",
|
||||
"integrity": "sha512-WPS/HvHQTYnHisLo9McqBHOJk2FkHO/tlpvldyrnem4aeQp4hai3gythswg6p01oSoTl58rcpiFAjF2br2Ak2A==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"call-bound": "^1.0.2",
|
||||
"es-errors": "^1.3.0",
|
||||
"get-intrinsic": "^1.2.5",
|
||||
"object-inspect": "^1.13.3",
|
||||
"side-channel-map": "^1.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 0.4"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/ljharb"
|
||||
}
|
||||
},
|
||||
"node_modules/statuses": {
|
||||
"version": "2.0.2",
|
||||
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz",
|
||||
"integrity": "sha512-DvEy55V3DB7uknRo+4iOGT5fP1slR8wQohVdknigZPMpMstaKJQWhwiYBACJE3Ul2pTnATihhBYnRhZQHGBiRw==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/string_decoder": {
|
||||
"version": "1.3.0",
|
||||
"resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz",
|
||||
@@ -979,70 +172,6 @@
|
||||
"dependencies": {
|
||||
"safe-buffer": "~5.2.0"
|
||||
}
|
||||
},
|
||||
"node_modules/toidentifier": {
|
||||
"version": "1.0.1",
|
||||
"resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",
|
||||
"integrity": "sha512-o5sSPKEkg/DIQNmH43V0/uerLrpzVedkUh8tGNvaeXpfpuwjKenlSox/2O/BTlZUtEe+JG7s5YhEz608PlAHRA==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=0.6"
|
||||
}
|
||||
},
|
||||
"node_modules/type-is": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/type-is/-/type-is-2.1.0.tgz",
|
||||
"integrity": "sha512-faYHw0anBbc/kWF3zFTEnxSFOAGUX9GFbOBthvDdLsIlEoWOFOtS0zgCiQYwIskL9iGXZL3kAXD8OoZ4GmMATA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"content-type": "^2.0.0",
|
||||
"media-typer": "^1.1.0",
|
||||
"mime-types": "^3.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/type-is/node_modules/content-type": {
|
||||
"version": "2.1.0",
|
||||
"resolved": "https://registry.npmjs.org/content-type/-/content-type-2.1.0.tgz",
|
||||
"integrity": "sha512-mj7UPXE0jaqaOsukNZRUEfEi2AcL7C/vwmwcHV0O97eO1E1pxBZuyjlZrx5seTaNBg1U6+o35wpa35Qfcc+7ag==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">=18"
|
||||
},
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/express"
|
||||
}
|
||||
},
|
||||
"node_modules/unpipe": {
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/unpipe/-/unpipe-1.0.0.tgz",
|
||||
"integrity": "sha512-pjy2bYhSsufwWlKwPc+l3cN7+wuJlK6uz0YdJEOlQDbl6jo/YlPi4mb8agUkVC8BF7V8NuzeyPNqRksA3hztKQ==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/vary": {
|
||||
"version": "1.1.2",
|
||||
"resolved": "https://registry.npmjs.org/vary/-/vary-1.1.2.tgz",
|
||||
"integrity": "sha512-BNGbWLfd0eUPabhkXUVm0j8uuvREyTh5ovRa/dyow/BqAbZJyC+5fU+IzQOzmAKzYqYRAISoRhdQr3eIZ/PXqg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 0.8"
|
||||
}
|
||||
},
|
||||
"node_modules/wrappy": {
|
||||
"version": "1.0.2",
|
||||
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
|
||||
"integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==",
|
||||
"license": "ISC"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "natural-language-report-service",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"description": "Turns a SHACL report spec into SPARQL queries, joins the results and writes a CSV. The LLM writes specs; this service runs them.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
@@ -9,7 +9,6 @@
|
||||
"start": "node app.js"
|
||||
},
|
||||
"dependencies": {
|
||||
"express": "^5.2.1",
|
||||
"n3": "^1.17.0"
|
||||
}
|
||||
}
|
||||
|
||||
278
src/chat.js
Normal file
278
src/chat.js
Normal file
@@ -0,0 +1,278 @@
|
||||
// The chat's reads and writes: conversations, messages and their bijlagen.
|
||||
// Full URIs everywhere: the auth layer rejects a prefixed name with no
|
||||
// PREFIX line with an opaque 500 (the xsd: types of mu's escape helpers are
|
||||
// fine).
|
||||
//
|
||||
// A bijlage is a real file: a logical nfo:FileDataObject the file service
|
||||
// serves at /files/<uuid>/download, over a physical share:// object with
|
||||
// nie:dataSource back to the logical one.
|
||||
import { readFileSync, writeFileSync, statSync, unlinkSync } from 'fs';
|
||||
import { query, update, uuid,
|
||||
sparqlEscapeUri, sparqlEscapeString, sparqlEscapeInt, sparqlEscapeDateTime } from 'mu';
|
||||
|
||||
const HISTORY_LIMIT = Number(process.env.CHAT_HISTORY_LIMIT || 20);
|
||||
|
||||
const T = {
|
||||
type: 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type',
|
||||
uuid: 'http://mu.semte.ch/vocabularies/core/uuid',
|
||||
thread: 'http://rdfs.org/sioc/ns#Thread',
|
||||
post: 'http://rdfs.org/sioc/ns#Post',
|
||||
instantMessage: 'http://rdfs.org/sioc/types#InstantMessage',
|
||||
hasContainer: 'http://rdfs.org/sioc/ns#has_container',
|
||||
content: 'http://rdfs.org/sioc/ns#content',
|
||||
lastActivity: 'http://rdfs.org/sioc/ns#last_activity_date',
|
||||
title: 'http://purl.org/dc/terms/title',
|
||||
created: 'http://purl.org/dc/terms/created',
|
||||
maker: 'http://xmlns.com/foaf/0.1/maker',
|
||||
attachment: 'https://www.w3.org/ns/activitystreams#attachment',
|
||||
file: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject',
|
||||
fileName: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName',
|
||||
fileSize: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize',
|
||||
fileCreated: 'http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileCreated',
|
||||
format: 'http://purl.org/dc/terms/format',
|
||||
fileType: 'http://purl.org/dc/terms/type',
|
||||
dataSource: 'http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource',
|
||||
generated: 'http://www.w3.org/ns/prov#generated',
|
||||
};
|
||||
const MESSAGE_BASE = 'http://data.lblod.info/id/chat-messages/';
|
||||
const FILE_BASE = 'http://data.lblod.info/files/';
|
||||
const SHARE_DIR = process.env.SHARE_DIR || '/share';
|
||||
|
||||
export const SPEC_MEDIA_TYPE = 'text/turtle';
|
||||
// The bijlage types. Match the add-chat-attachment-types-codelist migration
|
||||
// in the portal.
|
||||
export const SPEC_TYPE = 'http://lblod.data.gift/concepts/6693014e-7b7d-448c-bb24-80c9758521f3';
|
||||
export const RESULT_TYPE = 'http://lblod.data.gift/concepts/28d97698-e219-4a92-bdbd-7ab367a7524e';
|
||||
// Every spec file is called like this; readSpecFile opens nothing else.
|
||||
const SPEC_FILE_NAME = /^specificatie-[0-9a-f-]{36}\.ttl$/;
|
||||
|
||||
const u = sparqlEscapeUri;
|
||||
const s = sparqlEscapeString;
|
||||
|
||||
const bindings = (json) => json.results.bindings;
|
||||
|
||||
// --- reads --------------------------------------------------------------------
|
||||
|
||||
// The conversation by uuid, null when the caller may not read it.
|
||||
export async function readConversation(id) {
|
||||
const json = await query(`
|
||||
SELECT ?conversation ?title ?creator WHERE {
|
||||
?conversation ${u(T.type)} ${u(T.thread)} ;
|
||||
${u(T.uuid)} ${s(id)} .
|
||||
OPTIONAL { ?conversation ${u(T.title)} ?title . }
|
||||
OPTIONAL { ?conversation ${u(T.maker)} ?creator . }
|
||||
} LIMIT 1`);
|
||||
const b = bindings(json)[0];
|
||||
if (!b) return null;
|
||||
return { uri: b.conversation.value, id, title: b.title?.value, creator: b.creator?.value };
|
||||
}
|
||||
|
||||
// The last HISTORY_LIMIT messages, oldest first, as
|
||||
// { role, content, attachments: [{ uri, name, mediaType, type }] }.
|
||||
export async function readHistory(conversationUri, assistantUri) {
|
||||
const json = await query(`
|
||||
SELECT ?message ?content ?maker ?created ?doc ?docName ?docFormat ?docType WHERE {
|
||||
?message ${u(T.hasContainer)} ${u(conversationUri)} ;
|
||||
${u(T.content)} ?content ;
|
||||
${u(T.maker)} ?maker ;
|
||||
${u(T.created)} ?created .
|
||||
OPTIONAL {
|
||||
?message ${u(T.attachment)} ?doc .
|
||||
OPTIONAL { ?doc ${u(T.fileName)} ?docName . }
|
||||
OPTIONAL { ?doc ${u(T.format)} ?docFormat . }
|
||||
OPTIONAL { ?doc ${u(T.fileType)} ?docType . }
|
||||
}
|
||||
} ORDER BY DESC(?created) LIMIT ${HISTORY_LIMIT * 4}`);
|
||||
const messages = [];
|
||||
const byMessage = new Map();
|
||||
for (const b of bindings(json).reverse()) {
|
||||
const uri = b.message.value;
|
||||
let message = byMessage.get(uri);
|
||||
if (!message) {
|
||||
message = {
|
||||
role: b.maker.value === assistantUri ? 'assistant' : 'user',
|
||||
content: b.content.value,
|
||||
attachments: [],
|
||||
};
|
||||
byMessage.set(uri, message);
|
||||
messages.push(message);
|
||||
}
|
||||
if (b.doc) {
|
||||
message.attachments.push({
|
||||
uri: b.doc.value,
|
||||
name: b.docName?.value,
|
||||
mediaType: b.docFormat?.value,
|
||||
type: b.docType?.value,
|
||||
});
|
||||
}
|
||||
}
|
||||
return messages.slice(-HISTORY_LIMIT);
|
||||
}
|
||||
|
||||
// --- writes -------------------------------------------------------------------
|
||||
|
||||
// One message, both types, its bijlagen, and the conversation touched, in
|
||||
// one update. `title` is only written when given (the first question names
|
||||
// an unnamed conversation). attachments are the logical uris of files that
|
||||
// already exist.
|
||||
export async function writeMessage({ conversationUri, content, maker, attachments = [], title }) {
|
||||
const id = uuid();
|
||||
const uri = `${MESSAGE_BASE}${id}`;
|
||||
const now = sparqlEscapeDateTime(new Date());
|
||||
|
||||
// Predicate pairs joined with ';', never a trailing one: it breaks the Turtle.
|
||||
const pairs = [
|
||||
`${u(T.type)} ${u(T.post)}`,
|
||||
`${u(T.type)} ${u(T.instantMessage)}`,
|
||||
`${u(T.uuid)} ${s(id)}`,
|
||||
`${u(T.hasContainer)} ${u(conversationUri)}`,
|
||||
`${u(T.content)} ${s(content)}`,
|
||||
`${u(T.created)} ${now}`,
|
||||
];
|
||||
if (maker) pairs.push(`${u(T.maker)} ${u(maker)}`);
|
||||
for (const fileUri of attachments) {
|
||||
pairs.push(`${u(T.attachment)} ${u(fileUri)}`);
|
||||
}
|
||||
|
||||
const conversationPairs = [`${u(T.lastActivity)} ${now}`];
|
||||
if (title) conversationPairs.push(`${u(T.title)} ${s(title)}`);
|
||||
|
||||
await update(`
|
||||
DELETE {
|
||||
${u(conversationUri)} ${u(T.lastActivity)} ?old .
|
||||
}
|
||||
INSERT {
|
||||
${u(uri)} ${pairs.join(' ;\n ')} .
|
||||
${u(conversationUri)} ${conversationPairs.join(' ;\n ')} .
|
||||
}
|
||||
WHERE {
|
||||
OPTIONAL { ${u(conversationUri)} ${u(T.lastActivity)} ?old . }
|
||||
}`);
|
||||
|
||||
return { uri, id };
|
||||
}
|
||||
|
||||
// --- files --------------------------------------------------------------------
|
||||
|
||||
export function writeShareFile(fileName, content) {
|
||||
writeFileSync(`${SHARE_DIR}/${fileName}`, content, 'utf8');
|
||||
}
|
||||
|
||||
// storeSpecFile(spec) → the file uri of a new spec bijlage. A spec file
|
||||
// keeps one name for the logical and the physical file: readSpecFile finds
|
||||
// it by the name in the history.
|
||||
export async function storeSpecFile(spec) {
|
||||
const fileName = `specificatie-${uuid()}.ttl`;
|
||||
writeShareFile(fileName, spec);
|
||||
return registerFile(fileName, SPEC_MEDIA_TYPE, SPEC_TYPE);
|
||||
}
|
||||
|
||||
// readSpecFile(fileName) → the Turtle of a spec bijlage, null when there is
|
||||
// no such spec file.
|
||||
export function readSpecFile(fileName) {
|
||||
if (!SPEC_FILE_NAME.test(fileName || '')) return null;
|
||||
try {
|
||||
return readFileSync(`${SHARE_DIR}/${fileName}`, 'utf8');
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// registerFile(shareName, format, type, name) → the logical file uri. One
|
||||
// logical file called `name` over the physical share file `shareName`, both
|
||||
// nfo:FileDataObject, both typed with dct:type. A share name must be
|
||||
// unique: two files on one share name overwrite each other.
|
||||
export async function registerFile(shareName, format, type, name = shareName) {
|
||||
const id = uuid();
|
||||
const physicalId = uuid();
|
||||
const logical = `${FILE_BASE}${id}`;
|
||||
const physical = `share://${shareName}`;
|
||||
const size = statSync(`${SHARE_DIR}/${shareName}`).size;
|
||||
const now = sparqlEscapeDateTime(new Date());
|
||||
|
||||
await update(`
|
||||
INSERT DATA {
|
||||
${u(logical)} ${u(T.type)} ${u(T.file)} ;
|
||||
${u(T.uuid)} ${s(id)} ;
|
||||
${u(T.fileName)} ${s(name)} ;
|
||||
${u(T.format)} ${s(format)} ;
|
||||
${u(T.fileSize)} ${sparqlEscapeInt(size)} ;
|
||||
${u(T.created)} ${now} ;
|
||||
${u(T.fileType)} ${u(type)} .
|
||||
${u(physical)} ${u(T.type)} ${u(T.file)} ;
|
||||
${u(T.uuid)} ${s(physicalId)} ;
|
||||
${u(T.fileName)} ${s(shareName)} ;
|
||||
${u(T.format)} ${s(format)} ;
|
||||
${u(T.fileSize)} ${sparqlEscapeInt(size)} ;
|
||||
${u(T.fileCreated)} ${now} ;
|
||||
${u(T.dataSource)} ${u(logical)} ;
|
||||
${u(T.fileType)} ${u(type)} .
|
||||
}`);
|
||||
|
||||
return logical;
|
||||
}
|
||||
|
||||
// --- deletes -------------------------------------------------------------------
|
||||
|
||||
// deleteConversation(uri) → the share files it removed (a number). Deletes
|
||||
// the conversation, its messages, their bijlagen (the logical file and the
|
||||
// physical one under it) and the reports of those files, then removes the
|
||||
// share files. Everything runs as the caller, so the auth layer keeps a
|
||||
// user to their own conversation.
|
||||
export async function deleteConversation(conversationUri) {
|
||||
// The bijlagen first: once the triples are gone their names cannot be
|
||||
// found back.
|
||||
const json = await query(`
|
||||
SELECT DISTINCT ?doc ?name WHERE {
|
||||
?message ${u(T.hasContainer)} ${u(conversationUri)} ;
|
||||
${u(T.attachment)} ?doc .
|
||||
OPTIONAL {
|
||||
?physical ${u(T.dataSource)} ?doc ;
|
||||
${u(T.fileName)} ?name .
|
||||
}
|
||||
}`);
|
||||
const rows = bindings(json);
|
||||
const docs = [...new Set(rows.map(b => b.doc.value))];
|
||||
const names = [...new Set(rows.filter(b => b.name).map(b => b.name.value))];
|
||||
|
||||
await update(`
|
||||
DELETE {
|
||||
?conversation ?cp ?co .
|
||||
} WHERE {
|
||||
VALUES ?conversation { ${u(conversationUri)} }
|
||||
?conversation ?cp ?co .
|
||||
}`);
|
||||
|
||||
await update(`
|
||||
DELETE {
|
||||
?message ?mp ?mo .
|
||||
} WHERE {
|
||||
?message ${u(T.hasContainer)} ${u(conversationUri)} ;
|
||||
?mp ?mo .
|
||||
}`);
|
||||
|
||||
if (docs.length) {
|
||||
await update(`
|
||||
DELETE {
|
||||
?doc ?dp ?do .
|
||||
?physical ?pp ?po .
|
||||
?report ?rp ?ro .
|
||||
} WHERE {
|
||||
VALUES ?doc { ${docs.map(u).join(' ')} }
|
||||
?doc ?dp ?do .
|
||||
OPTIONAL { ?physical ${u(T.dataSource)} ?doc ; ?pp ?po . }
|
||||
OPTIONAL { ?report ${u(T.generated)} ?doc ; ?rp ?ro . }
|
||||
}`);
|
||||
}
|
||||
|
||||
let removed = 0;
|
||||
for (const name of names) {
|
||||
try {
|
||||
unlinkSync(`${SHARE_DIR}/${name}`);
|
||||
removed++;
|
||||
} catch {
|
||||
// the file was already gone
|
||||
}
|
||||
}
|
||||
return removed;
|
||||
}
|
||||
@@ -1,81 +0,0 @@
|
||||
// The turn endpoint. Mount it with mountChat(app, { path, answer });
|
||||
// answer(turn) is the only thing the mounting service writes.
|
||||
import express from 'express';
|
||||
import { query as muQuery, update as muUpdate } from 'mu';
|
||||
import {
|
||||
readConversation, readHistory, findAssistant,
|
||||
writeMessage, setDocumentUrl, dropDocument,
|
||||
} from './store.js';
|
||||
|
||||
const TEXT = {
|
||||
failed: 'Er ging iets mis. Probeer het opnieuw.',
|
||||
};
|
||||
const TITLE_LENGTH = 80;
|
||||
|
||||
export function mountChat(app, { path = '/assistant', answer }) {
|
||||
if (typeof answer !== 'function') throw new Error('mountChat needs an answer(turn) function');
|
||||
|
||||
app.post(`${path}/conversations/:id/turns`, express.json(), async (req, res) => {
|
||||
const content = String(req.body?.content ?? '').trim();
|
||||
if (!content) return res.status(400).json({ error: 'content is required' });
|
||||
|
||||
// 1. the access check: the caller's session decides what is readable
|
||||
let conversation;
|
||||
try {
|
||||
conversation = await readConversation(req.params.id);
|
||||
} catch (e) {
|
||||
console.error('[chat] conversation read failed:', e);
|
||||
return res.status(500).json({ error: e.message });
|
||||
}
|
||||
if (!conversation) return res.status(404).json({ error: 'no such conversation' });
|
||||
|
||||
let assistant, history, userMessage;
|
||||
try {
|
||||
assistant = process.env.CHAT_ASSISTANT_URI || await findAssistant();
|
||||
history = await readHistory(conversation.uri, assistant);
|
||||
userMessage = await writeMessage({
|
||||
conversationUri: conversation.uri,
|
||||
content,
|
||||
maker: conversation.creator,
|
||||
title: conversation.title ? undefined : content.slice(0, TITLE_LENGTH),
|
||||
});
|
||||
} catch (e) {
|
||||
console.error('[chat] could not record the question:', e);
|
||||
return res.status(500).json({ error: e.message });
|
||||
}
|
||||
|
||||
// 2. answer 202 now; the assistant's message comes when it comes. The
|
||||
// continuation still runs as the caller.
|
||||
res.status(202).json({ id: userMessage.id });
|
||||
|
||||
const turn = {
|
||||
conversation: { uri: conversation.uri, id: conversation.id, title: conversation.title },
|
||||
history,
|
||||
content,
|
||||
query: muQuery,
|
||||
update: muUpdate,
|
||||
finished: false,
|
||||
say: (text, attachments = []) => writeMessage({
|
||||
conversationUri: conversation.uri, content: text, maker: assistant, attachments,
|
||||
}),
|
||||
setUrl: (documentUri, url) => setDocumentUrl(documentUri, url, conversation.uri),
|
||||
drop: (documentUri) => dropDocument(documentUri),
|
||||
};
|
||||
|
||||
try {
|
||||
const text = await answer(turn);
|
||||
if (typeof text === 'string' && text.trim()) await turn.say(text.trim());
|
||||
} catch (e) {
|
||||
console.error('[chat] turn failed:', e);
|
||||
try {
|
||||
await turn.say(TEXT.failed);
|
||||
} catch (e2) {
|
||||
console.error('[chat] could not write the failure message:', e2);
|
||||
}
|
||||
} finally {
|
||||
turn.finished = true;
|
||||
}
|
||||
});
|
||||
|
||||
console.log(`[chat] turns at POST ${path}/conversations/:id/turns`);
|
||||
}
|
||||
@@ -1,163 +0,0 @@
|
||||
// The chat's reads and writes. Nothing here knows the service it lives in.
|
||||
// Full URIs everywhere: the auth layer rejects a prefixed name with no
|
||||
// PREFIX line with an opaque 500.
|
||||
import { randomUUID } from 'crypto';
|
||||
import { query as muQuery, update as muUpdate,
|
||||
sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime } from 'mu';
|
||||
|
||||
const HISTORY_LIMIT = Number(process.env.CHAT_HISTORY_LIMIT || 20);
|
||||
|
||||
const T = {
|
||||
type: 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type',
|
||||
uuid: 'http://mu.semte.ch/vocabularies/core/uuid',
|
||||
thread: 'http://rdfs.org/sioc/ns#Thread',
|
||||
post: 'http://rdfs.org/sioc/ns#Post',
|
||||
instantMessage: 'http://rdfs.org/sioc/types#InstantMessage',
|
||||
hasContainer: 'http://rdfs.org/sioc/ns#has_container',
|
||||
content: 'http://rdfs.org/sioc/ns#content',
|
||||
lastActivity: 'http://rdfs.org/sioc/ns#last_activity_date',
|
||||
title: 'http://purl.org/dc/terms/title',
|
||||
created: 'http://purl.org/dc/terms/created',
|
||||
maker: 'http://xmlns.com/foaf/0.1/maker',
|
||||
attachment: 'https://www.w3.org/ns/activitystreams#attachment',
|
||||
document: 'https://www.w3.org/ns/activitystreams#Document',
|
||||
name: 'https://www.w3.org/ns/activitystreams#name',
|
||||
mediaType: 'https://www.w3.org/ns/activitystreams#mediaType',
|
||||
url: 'https://www.w3.org/ns/activitystreams#url',
|
||||
softwareAgent: 'http://www.w3.org/ns/prov#SoftwareAgent',
|
||||
};
|
||||
const MESSAGE_BASE = 'http://data.lblod.info/id/chat-messages/';
|
||||
const DOCUMENT_BASE = 'http://data.lblod.info/id/chat-documents/';
|
||||
|
||||
const u = sparqlEscapeUri;
|
||||
const s = sparqlEscapeString;
|
||||
// Full URIs, so spell the dateTime type out.
|
||||
const dt = (value) => sparqlEscapeDateTime(value)
|
||||
.replace('xsd:dateTime', '<http://www.w3.org/2001/XMLSchema#dateTime>');
|
||||
|
||||
const bindings = (json) => json.results.bindings;
|
||||
|
||||
// --- reads --------------------------------------------------------------------
|
||||
|
||||
// The conversation by uuid, null when the caller may not read it.
|
||||
export async function readConversation(id) {
|
||||
const json = await muQuery(`
|
||||
SELECT ?conversation ?title ?creator WHERE {
|
||||
?conversation ${u(T.type)} ${u(T.thread)} ;
|
||||
${u(T.uuid)} ${s(id)} .
|
||||
OPTIONAL { ?conversation ${u(T.title)} ?title . }
|
||||
OPTIONAL { ?conversation ${u(T.maker)} ?creator . }
|
||||
} LIMIT 1`);
|
||||
const b = bindings(json)[0];
|
||||
if (!b) return null;
|
||||
return { uri: b.conversation.value, id, title: b.title?.value, creator: b.creator?.value };
|
||||
}
|
||||
|
||||
// The last HISTORY_LIMIT messages, oldest first, as { role, content }.
|
||||
export async function readHistory(conversationUri, assistantUri) {
|
||||
const json = await muQuery(`
|
||||
SELECT ?content ?maker ?created WHERE {
|
||||
?message ${u(T.hasContainer)} ${u(conversationUri)} ;
|
||||
${u(T.content)} ?content ;
|
||||
${u(T.maker)} ?maker ;
|
||||
${u(T.created)} ?created .
|
||||
} ORDER BY DESC(?created) LIMIT ${HISTORY_LIMIT}`);
|
||||
return bindings(json).reverse().map(b => ({
|
||||
role: b.maker.value === assistantUri ? 'assistant' : 'user',
|
||||
content: b.content.value,
|
||||
}));
|
||||
}
|
||||
|
||||
// The one assistant the caller can read. Throws unless there is exactly one.
|
||||
export async function findAssistant() {
|
||||
const json = await muQuery(`
|
||||
SELECT ?agent WHERE { ?agent ${u(T.type)} ${u(T.softwareAgent)} . } LIMIT 2`);
|
||||
const rows = bindings(json);
|
||||
if (rows.length !== 1) {
|
||||
throw new Error(`expected one prov:SoftwareAgent, found ${rows.length}; seed one or set CHAT_ASSISTANT_URI`);
|
||||
}
|
||||
return rows[0].agent.value;
|
||||
}
|
||||
|
||||
// --- writes -------------------------------------------------------------------
|
||||
|
||||
// One message, both types, its attachments, and the conversation touched, in
|
||||
// one update. `title` is only written when given (the first question names
|
||||
// an unnamed conversation). Returns { uri, id, documents: [{ uri, id }] }.
|
||||
export async function writeMessage({ conversationUri, content, maker, attachments = [], title }) {
|
||||
const id = randomUUID();
|
||||
const uri = `${MESSAGE_BASE}${id}`;
|
||||
const now = dt(new Date());
|
||||
const documents = attachments.map(a => {
|
||||
const id = randomUUID();
|
||||
return { ...a, id, uri: `${DOCUMENT_BASE}${id}` };
|
||||
});
|
||||
|
||||
// Predicate pairs joined with ';', never a trailing one: it breaks the Turtle.
|
||||
const pairs = [
|
||||
`${u(T.type)} ${u(T.post)}`,
|
||||
`${u(T.type)} ${u(T.instantMessage)}`,
|
||||
`${u(T.uuid)} ${s(id)}`,
|
||||
`${u(T.hasContainer)} ${u(conversationUri)}`,
|
||||
`${u(T.content)} ${s(content)}`,
|
||||
`${u(T.created)} ${now}`,
|
||||
];
|
||||
if (maker) pairs.push(`${u(T.maker)} ${u(maker)}`);
|
||||
for (const d of documents) pairs.push(`${u(T.attachment)} ${u(d.uri)}`);
|
||||
|
||||
const conversationPairs = [`${u(T.lastActivity)} ${now}`];
|
||||
if (title) conversationPairs.push(`${u(T.title)} ${s(title)}`);
|
||||
|
||||
await muUpdate(`
|
||||
DELETE {
|
||||
${u(conversationUri)} ${u(T.lastActivity)} ?old .
|
||||
}
|
||||
INSERT {
|
||||
${u(uri)} ${pairs.join(' ;\n ')} .
|
||||
${documents.map(d => `
|
||||
${u(d.uri)} ${[
|
||||
`${u(T.type)} ${u(T.document)}`,
|
||||
`${u(T.uuid)} ${s(d.id)}`,
|
||||
`${u(T.name)} ${s(d.name)}`,
|
||||
`${u(T.mediaType)} ${s(d.mediaType || 'application/octet-stream')}`,
|
||||
...(d.url ? [`${u(T.url)} ${s(d.url)}`] : []),
|
||||
].join(' ;\n ')} .`).join('')}
|
||||
${u(conversationUri)} ${conversationPairs.join(' ;\n ')} .
|
||||
}
|
||||
WHERE {
|
||||
OPTIONAL { ${u(conversationUri)} ${u(T.lastActivity)} ?old . }
|
||||
}`);
|
||||
|
||||
return { uri, id, documents: documents.map(({ uri, id }) => ({ uri, id })) };
|
||||
}
|
||||
|
||||
// The file is ready.
|
||||
export async function setDocumentUrl(documentUri, url, conversationUri) {
|
||||
const now = dt(new Date());
|
||||
await muUpdate(`
|
||||
DELETE {
|
||||
${u(documentUri)} ${u(T.url)} ?oldUrl .
|
||||
${u(conversationUri)} ${u(T.lastActivity)} ?old .
|
||||
}
|
||||
INSERT {
|
||||
${u(documentUri)} ${u(T.url)} ${s(url)} .
|
||||
${u(conversationUri)} ${u(T.lastActivity)} ${now} .
|
||||
}
|
||||
WHERE {
|
||||
OPTIONAL { ${u(documentUri)} ${u(T.url)} ?oldUrl . }
|
||||
OPTIONAL { ${u(conversationUri)} ${u(T.lastActivity)} ?old . }
|
||||
}`);
|
||||
}
|
||||
|
||||
// The file failed; the card disappears, the caller says why in a message.
|
||||
export async function dropDocument(documentUri) {
|
||||
await muUpdate(`
|
||||
DELETE {
|
||||
?message ${u(T.attachment)} ${u(documentUri)} .
|
||||
${u(documentUri)} ?p ?o .
|
||||
}
|
||||
WHERE {
|
||||
${u(documentUri)} ?p ?o .
|
||||
OPTIONAL { ?message ${u(T.attachment)} ${u(documentUri)} . }
|
||||
}`);
|
||||
}
|
||||
30
src/db.js
30
src/db.js
@@ -1,30 +0,0 @@
|
||||
// Thin re-export of mu's SPARQL helpers so the runner files share one import
|
||||
// site. mu's query/update attach the caller's session from the request
|
||||
// context, and mu-authorization decides graphs and visibility; the service
|
||||
// never touches an auth header.
|
||||
import {
|
||||
query as muQuery,
|
||||
update as muUpdate,
|
||||
sparqlEscapeUri,
|
||||
sparqlEscapeString,
|
||||
sparqlEscapeInt,
|
||||
sparqlEscapeDateTime,
|
||||
sparqlEscapeDate,
|
||||
sparqlEscapeBool,
|
||||
} from 'mu';
|
||||
|
||||
export const sessionQuery = muQuery;
|
||||
export const sessionUpdate = muUpdate;
|
||||
export {
|
||||
sparqlEscapeUri,
|
||||
sparqlEscapeString,
|
||||
sparqlEscapeInt,
|
||||
sparqlEscapeDateTime,
|
||||
sparqlEscapeDate,
|
||||
sparqlEscapeBool,
|
||||
};
|
||||
|
||||
// The store's error text may name graphs or internals we do not pass on.
|
||||
export function stripStoreError(text) {
|
||||
return String(text).split('\n')[0].slice(0, 300);
|
||||
}
|
||||
@@ -1,90 +1,41 @@
|
||||
// A profile as a plain text menu: entities, fields, code lists. Code lists up
|
||||
// to INLINE_VALUES_MAX are inlined and cached for VALUES_TTL, so the LLM
|
||||
// never has to search the short ones.
|
||||
// to INLINE_VALUES_MAX are inlined (and cached), so the LLM never has to
|
||||
// search the short ones.
|
||||
import { codeList, INLINE_VALUES_MAX } from './explore.js';
|
||||
|
||||
import { fieldsOf, entityForClass } from '../runner/profile.js';
|
||||
import { sessionQuery } from '../db.js';
|
||||
export async function describeProfile(profile) {
|
||||
const lines = [`A spec is Turtle, in exactly this shape (copy these prefixes):
|
||||
@prefix rep: <http://mu.semte.ch/vocabularies/reporting/> .
|
||||
@prefix sh: <http://www.w3.org/ns/shacl#> .
|
||||
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
|
||||
@prefix dct: <http://purl.org/dc/terms/> .
|
||||
|
||||
const INLINE_VALUES_MAX = Number(process.env.INLINE_VALUES_MAX || 50);
|
||||
const VALUES_TTL = Number(process.env.VALUES_TTL || 3600) * 1000;
|
||||
<http://data.lblod.info/id/report-specs/<your-id>> a rep:ReportSpec , sh:NodeShape ;
|
||||
dct:title "..." ;
|
||||
rep:profile <${profile.uri}> ;
|
||||
sh:targetClass <the rdf:type from the list below> ;
|
||||
sh:property [ ... ] ; # filters, optional
|
||||
rep:columns ( [ sh:path ( ... ) ; rdfs:label "..." ] ... ) .
|
||||
Every column ends on a value field. Use rep:self as sh:path for the subject URI.
|
||||
For the URI of a linked node, end the column on that link and add sh:nodeKind sh:IRI to the column.
|
||||
|
||||
// In-memory cache: profile+field → { values, at }. Refreshed lazily when stale.
|
||||
const cache = new Map();
|
||||
|
||||
export async function codeListValues(profile) {
|
||||
// Returns a map: "shapeUri|predicate|inverse" → [{ uri, label }]
|
||||
const out = {};
|
||||
for (const shape of profile.shapes) {
|
||||
for (const f of shape.fields) {
|
||||
if (!f.class) continue;
|
||||
const key = `${shape.uri}|${f.path}|${f.inverse ? 1 : 0}`;
|
||||
out[key] = await loadCodeList(profile, f, key);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function loadCodeList(profile, f, key) {
|
||||
if (!f.class) return [];
|
||||
const hit = cache.get(key);
|
||||
if (hit && Date.now() - hit.at < VALUES_TTL) return hit.values;
|
||||
let values = [];
|
||||
try {
|
||||
values = await queryCodeList(f.class);
|
||||
} catch (e) {
|
||||
// the read failed; report nothing, the LLM will use rep:anyOf with words
|
||||
values = [];
|
||||
}
|
||||
cache.set(key, { values, at: Date.now() });
|
||||
return values;
|
||||
}
|
||||
|
||||
async function queryCodeList(type) {
|
||||
const q = `SELECT ?uri ?label WHERE {
|
||||
?uri a <${type}> .
|
||||
OPTIONAL { ?uri <http://www.w3.org/2004/02/skos/core#prefLabel> ?label . }
|
||||
} ORDER BY ?label LIMIT ${INLINE_VALUES_MAX + 1}`;
|
||||
const r = await sessionQuery(q);
|
||||
const bindings = r.results.bindings;
|
||||
return bindings.slice(0, INLINE_VALUES_MAX).map(b => ({
|
||||
uri: b.uri.value,
|
||||
label: b.label?.value || b.uri.value,
|
||||
}));
|
||||
}
|
||||
|
||||
export function describeProfile(profile, codeLists = {}) {
|
||||
const lines = [];
|
||||
lines.push('A spec is Turtle, in exactly this shape (copy these prefixes):');
|
||||
lines.push('@prefix rep: <http://mu.semte.ch/vocabularies/reporting/> .');
|
||||
lines.push('@prefix sh: <http://www.w3.org/ns/shacl#> .');
|
||||
lines.push('@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .');
|
||||
lines.push('@prefix dct: <http://purl.org/dc/terms/> .');
|
||||
lines.push('');
|
||||
lines.push('<http://data.lblod.info/id/report-specs/<your-id>> a rep:ReportSpec , sh:NodeShape ;');
|
||||
lines.push(' dct:title "..." ;');
|
||||
lines.push(` rep:profile <${profile.uri}> ;`);
|
||||
lines.push(' sh:targetClass <the rdf:type from the list below> ;');
|
||||
lines.push(' sh:property [ ... ] ; # filters, optional');
|
||||
lines.push(' rep:columns ( [ sh:path ( ... ) ; rdfs:label "..." ] ... ) .');
|
||||
lines.push('Every column ends on a value field. Use rep:self as sh:path for the subject URI.');
|
||||
lines.push('');
|
||||
lines.push('sh:targetClass is the full URI in brackets after the entity below.');
|
||||
lines.push('In sh:path, use the predicate URI shown after each field name, not the field name itself.');
|
||||
lines.push('An inverse hop (the field shows "(inverse)") goes in the list as [ sh:inversePath <predicate> ] .');
|
||||
lines.push('When two entities share one sh:targetClass, pick one with rep:entity <the URI after the entity>.');
|
||||
lines.push('A date filter is sh:minInclusive or sh:maxInclusive with "yyyy-mm-dd"^^xsd:dateTime');
|
||||
lines.push('(the xsd prefix above is not copied: add @prefix xsd: <http://www.w3.org/2001/XMLSchema#>).');
|
||||
lines.push('');
|
||||
sh:targetClass is the full URI in brackets after the entity below.
|
||||
In sh:path, use the predicate URI shown after each field name, not the field name itself.
|
||||
An inverse hop (the field shows "(inverse)") goes in the list as [ sh:inversePath <predicate> ] .
|
||||
When two entities share one sh:targetClass, pick one with rep:entity <the URI after the entity>.
|
||||
A field that links to several entities ("→ a or b") reaches all of them; the steps after it decide. To keep one kind, add a rep:where on a step only that kind has.
|
||||
A date filter is sh:minInclusive or sh:maxInclusive with "yyyy-mm-dd"^^xsd:dateTime
|
||||
(the xsd prefix above is not copied: add @prefix xsd: <http://www.w3.org/2001/XMLSchema#>).
|
||||
`];
|
||||
for (const shape of profile.shapes) {
|
||||
lines.push(`${shape.label || shape.targetClass} — sh:targetClass <${shape.targetClass}> ; rep:entity <${shape.uri}>`);
|
||||
for (const f of shape.fields) {
|
||||
const key = `${shape.uri}|${f.path}|${f.inverse ? 1 : 0}`;
|
||||
const kind = f.datatype ? datatypeName(f.datatype)
|
||||
: f.class ? `→ ${conceptLabel(f.class, codeLists[key])}`
|
||||
: f.node ? `→ ${linkedLabel(profile, f.node)}`
|
||||
: f.class ? `→ ${conceptLabel(f.class, await codeList(f.class))}`
|
||||
: f.nodes.length ? `→ ${f.nodes.map(n => linkedLabel(profile, n)).join(' or ')}`
|
||||
: '?';
|
||||
const dir = f.inverse ? ' (inverse)' : '';
|
||||
const pathShown = typeof f.path === 'string' ? f.path : '?';
|
||||
const pathShown = f.path || '?';
|
||||
lines.push(` ${f.name || pathShown} sh:path <${pathShown}>${dir} ${kind}`);
|
||||
}
|
||||
lines.push('');
|
||||
@@ -93,11 +44,10 @@ export function describeProfile(profile, codeLists = {}) {
|
||||
}
|
||||
|
||||
function conceptLabel(type, values) {
|
||||
const head = values.slice(0, INLINE_VALUES_MAX);
|
||||
if (!head.length) return short(type);
|
||||
const names = head.map(v => v.label).join(', ');
|
||||
if (!values.length) return short(type);
|
||||
const names = values.map(v => v.label).join(', ');
|
||||
const more = values.length >= INLINE_VALUES_MAX ? ', …' : '';
|
||||
return `${short(type)} — ${head.length} waarden: ${names}${more}`;
|
||||
return `${short(type)} — ${values.length} waarden: ${names}${more}`;
|
||||
}
|
||||
|
||||
function linkedLabel(profile, nodeUri) {
|
||||
|
||||
140
src/llm/explore.js
Normal file
140
src/llm/explore.js
Normal file
@@ -0,0 +1,140 @@
|
||||
// Everything the LLM reads from the database in refine mode: the code
|
||||
// lists, inlined in describe_profile and searched by lookup_values. Every
|
||||
// query goes through publicQuery: the service scope, not the caller's
|
||||
// session. The sparql-parser config grants that scope read access to the
|
||||
// public graph only, so these reads are the same for every caller. Nothing
|
||||
// here counts, and execution still runs as the caller.
|
||||
//
|
||||
// lookup_values is a suggest helper, not a spec check. It reads candidate
|
||||
// values the LLM can put in a filter; it never runs the spec and never sees
|
||||
// report rows.
|
||||
import { query, sparqlEscapeString } from 'mu';
|
||||
|
||||
// Must match the with-scope grant in the app's config.lisp.
|
||||
const SERVICE_SCOPE = process.env.SERVICE_SCOPE || 'http://services.semantic.works/natural-language-report';
|
||||
export const INLINE_VALUES_MAX = Number(process.env.INLINE_VALUES_MAX || 50);
|
||||
const VALUES_TTL = Number(process.env.VALUES_TTL || 3600) * 1000;
|
||||
const LOOKUP_LIMIT = 25;
|
||||
|
||||
// class → { values, at }. Refreshed lazily when stale.
|
||||
const cache = new Map();
|
||||
|
||||
function publicQuery(q) {
|
||||
return query(q, { scope: SERVICE_SCOPE });
|
||||
}
|
||||
|
||||
// codeList(type) → up to INLINE_VALUES_MAX [{ uri, label }], cached for
|
||||
// VALUES_TTL. Empty when the read fails; the LLM then uses rep:anyOf with
|
||||
// words.
|
||||
export async function codeList(type) {
|
||||
const hit = cache.get(type);
|
||||
if (hit && Date.now() - hit.at < VALUES_TTL) return hit.values;
|
||||
let values = [];
|
||||
try {
|
||||
const r = await publicQuery(`SELECT ?uri ?label WHERE {
|
||||
?uri a <${type}> .
|
||||
OPTIONAL { ?uri <http://www.w3.org/2004/02/skos/core#prefLabel> ?label . }
|
||||
} ORDER BY ?label LIMIT ${INLINE_VALUES_MAX + 1}`);
|
||||
values = r.results.bindings.slice(0, INLINE_VALUES_MAX).map(b => ({
|
||||
uri: b.uri.value,
|
||||
label: b.label?.value || b.uri.value,
|
||||
}));
|
||||
} catch (e) {
|
||||
values = [];
|
||||
}
|
||||
cache.set(type, { values, at: Date.now() });
|
||||
return values;
|
||||
}
|
||||
|
||||
// lookupValues(profile, field, term) → the matches as text for the LLM: up
|
||||
// to LOOKUP_LIMIT labels and URIs, an exact label (case aside) first. field is
|
||||
// "shapeLabel.fieldLabel" as the LLM writes it; it resolves to the class to
|
||||
// search.
|
||||
export async function lookupValues(profile, field, term) {
|
||||
const type = resolveClass(profile, field);
|
||||
if (!type) return `field "${field}" not found in profile "${profile.title}"`;
|
||||
|
||||
// Search within the class first. A common word ("gemeente") also sits in
|
||||
// hundreds of unrelated labels; an unscoped scan drowns the code.
|
||||
let rows = await labelQuery(term, type);
|
||||
// Code lists sometimes type their entries differently than the profile
|
||||
// says (MAR codes are ext:supervision/Nomenclature, the profile says
|
||||
// skos:Concept). Fall back to a scan over everything.
|
||||
if (!rows.length) rows = await labelQuery(term, null);
|
||||
// "Gent" must surface before "AGB Erfgoed Gent".
|
||||
const lower = term.toLowerCase();
|
||||
const exactRow = rows.find(r => r.label.toLowerCase() === lower);
|
||||
let matches = (exactRow ? [exactRow, ...rows.filter(r => r !== exactRow)] : rows).slice(0, LOOKUP_LIMIT);
|
||||
if (!matches.length) matches = await codeQuery(term);
|
||||
if (!matches.length) return `no value matches "${term}" in ${field}.`;
|
||||
|
||||
const lines = matches.map(m => `${m.label} <${m.uri}>`);
|
||||
if (matches.length >= LOOKUP_LIMIT) lines.push(`(${LOOKUP_LIMIT}+ matches, narrow the term)`);
|
||||
const exact = matches.find(m => m.label.toLowerCase() === lower);
|
||||
if (exact) lines.push(`exact: <${exact.uri}>`);
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
// Regex on str(?label), so language-tagged labels match.
|
||||
async function labelQuery(term, type) {
|
||||
const q = `SELECT DISTINCT ?uri ?label WHERE {
|
||||
?uri a ${type ? `<${type}>` : '?anyType'} ;
|
||||
<http://www.w3.org/2004/02/skos/core#prefLabel> ?label .
|
||||
FILTER(REGEX(str(?label), ${sparqlEscapeString(escapeRegex(term))}, "i"))
|
||||
} ORDER BY ?label LIMIT ${LOOKUP_LIMIT * 8}`;
|
||||
const r = await publicQuery(q);
|
||||
return r.results.bindings.map(b => ({ label: b.label.value, uri: b.uri.value }));
|
||||
}
|
||||
|
||||
// MAR codes: the label says "MAR7300 - …" but the searchable code sits on
|
||||
// ext:supervision/nomenclatureCode. One extra query, only when the label
|
||||
// search came up empty.
|
||||
async function codeQuery(term) {
|
||||
const q = `SELECT ?uri ?label WHERE {
|
||||
?uri <http://mu.semte.ch/vocabularies/ext/supervision/nomenclatureCode> ?code ;
|
||||
<http://www.w3.org/2004/02/skos/core#prefLabel> ?label .
|
||||
FILTER(CONTAINS(LCASE(str(?code)), LCASE(${sparqlEscapeString(term)})))
|
||||
} ORDER BY ?label LIMIT ${LOOKUP_LIMIT}`;
|
||||
try {
|
||||
const r = await publicQuery(q);
|
||||
return r.results.bindings.map(b => ({ label: b.label.value, uri: b.uri.value }));
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
// "shapeLabel.fieldLabel" → the class to search: the field's sh:class, or
|
||||
// for a link (sh:node) the linked shape's sh:targetClass
|
||||
// ("melding.bestuurseenheid" → besluit:Bestuurseenheid).
|
||||
function resolveClass(profile, field) {
|
||||
if (field.includes('/')) return null;
|
||||
const dot = field.lastIndexOf('.');
|
||||
if (dot < 0) return null;
|
||||
const shapeLabel = field.slice(0, dot);
|
||||
const fieldLabel = field.slice(dot + 1);
|
||||
const shape = profile.shapes.find(s => s.label === shapeLabel);
|
||||
if (!shape) return null;
|
||||
const f = shape.fields.find(x => x.name === fieldLabel);
|
||||
if (f?.class) return f.class;
|
||||
if (f?.nodes.length) {
|
||||
// sh:or of several entities: only when they share one class
|
||||
const classes = new Set(f.nodes.map(n => profile.shapes.find(s => s.uri === n)?.targetClass || null));
|
||||
return classes.size === 1 ? [...classes][0] : null;
|
||||
}
|
||||
// The model sometimes writes shape.field on a linked shape
|
||||
// ("bestuurseenheid.naam"); resolve it to that shape's target class.
|
||||
for (const s of profile.shapes) {
|
||||
if (s.label === shapeLabel || shortType(s.targetClass) === shapeLabel) {
|
||||
return s.targetClass;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function shortType(cls) {
|
||||
return cls.split('#').pop() || cls.split('/').pop();
|
||||
}
|
||||
|
||||
function escapeRegex(s) {
|
||||
return String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
}
|
||||
245
src/llm/loop.js
245
src/llm/loop.js
@@ -1,82 +1,215 @@
|
||||
// The agent loop: POST /v1/chat/completions with tool calls,
|
||||
// OpenAI-compatible (LLM_BASE_URL, LLM_MODEL, LLM_API_KEY). No provider
|
||||
// SDKs. Stops when the model answers without tool calls.
|
||||
//
|
||||
// Refinement only: the LLM proposes a spec, may look up candidate values in
|
||||
// code lists (under the service scope) and iterates with the user. It never
|
||||
// runs a report and never learns what a spec would match. The user must
|
||||
// confirm before anything executes; execution runs the agreed spec without
|
||||
// the LLM (../report-assistant.js).
|
||||
|
||||
import { buildTools } from './tools.js';
|
||||
import { TOOLS, runTool, listProfiles } from './tools.js';
|
||||
import { describeProfile } from './describe.js';
|
||||
import { parseSpec } from '../runner/spec.js';
|
||||
import { SPEC_MEDIA_TYPE, readSpecFile } from '../chat.js';
|
||||
|
||||
const MAX_ROUNDS = Number(process.env.MAX_ROUNDS || 12);
|
||||
// What an older spec becomes: only the newest spec goes to the model in full.
|
||||
const LEFT_OUT = 'Left out: a newer spec follows.';
|
||||
const LLM_BASE_URL = process.env.LLM_BASE_URL;
|
||||
const LLM_MODEL = process.env.LLM_MODEL;
|
||||
const LLM_API_KEY = process.env.LLM_API_KEY;
|
||||
// LOG_LLM=1 dumps every request to and response from the provider, verbatim.
|
||||
const LOG_LLM = /^(true|1)$/i.test(process.env.LOG_LLM || '');
|
||||
|
||||
const SYSTEM_PROMPT = `You write report specs. You never write SPARQL and you never see report data.
|
||||
// Refinement (Modus A). The LLM and the user iterate on a spec. The LLM may
|
||||
// look up candidate values in code lists (lookup_values, under the service
|
||||
// scope) to make concrete suggestions, but it may not run a report and never
|
||||
// learns how many subjects a spec would match. It ends with a proposal in
|
||||
// Dutch and asks the user to confirm.
|
||||
const REFINE_PROMPT = `You refine report specs with the user. You never write SPARQL and you never run a report.
|
||||
|
||||
The profile lists what can be asked. Compose paths by chaining fields.
|
||||
The profile lists what can be asked. Compose paths by chaining fields. The
|
||||
profiles are listed at the end of this prompt. When there is a current spec,
|
||||
the menu of its profile follows them: use it, and call describe_profile only
|
||||
for another profile.
|
||||
|
||||
Rules:
|
||||
- A column must end on a value, not on a link. Chain one more hop.
|
||||
- A column must end on a value, not on a link. Chain one more hop. Only when
|
||||
the user asks for the URI of a linked node itself, end the column on that
|
||||
link and add sh:nodeKind sh:IRI to the column.
|
||||
- When two entities share one sh:targetClass, the spec also needs rep:entity
|
||||
with that entity's URI; describe_profile shows it after each entity.
|
||||
- Filters go in sh:property. Columns go in rep:columns, in the order you want.
|
||||
- To match a topic rather than a value, list the Dutch words and compounds you
|
||||
would expect and put them all in rep:anyOf.
|
||||
- A filter tests only the end of its own path, and two filters never share a
|
||||
node. To test a node in the middle of a path (for example "the mandate
|
||||
whose role is Burgemeester"), give that filter or column a rep:where:
|
||||
rep:where [ sh:path ( <the same first steps> <more steps> ) ; <one condition> ].
|
||||
Write its path from the row, like a filter. The steps it has in common with
|
||||
its filter or column are the same nodes, and its condition tests where its
|
||||
own path ends. On a column, rep:where drops values from the cell, never
|
||||
rows. Several rep:where on one filter or column all apply.
|
||||
- validate_spec answers "ok" plus one line per rep:where saying where it
|
||||
applies. Say that in Dutch in your proposal, so the user can check it.
|
||||
- Only offer what validate_spec accepted. Before you suggest an alternative,
|
||||
write it and validate it; if it does not validate, say it cannot be done.
|
||||
- One answer carries one spec, and one spec is one report on one profile. If
|
||||
the question needs two reports, say so, propose the first, and offer the
|
||||
second as the next step.
|
||||
- You cannot count, sort, take the first N, or compare two subjects. Say so.
|
||||
- The report's title is the dct:title you write in the spec. There is no other
|
||||
title input.
|
||||
- One question makes one report. Write one spec, validate it, run it, then stop.
|
||||
Do not write a second report for the same question.
|
||||
|
||||
Always prefix the spec with @prefix lines. Use the prefixes the profile declares.
|
||||
Write the spec as one Turtle block. Then call validate_spec. Fix what it says,
|
||||
up to three rounds. Then call run_report. After run_report returns, stop.
|
||||
You are in refinement mode. You propose a spec and explain in Dutch what it
|
||||
will list and filter. Use lookup_values to find concrete values (names, codes)
|
||||
for a filter, so your proposal names real values instead of guesses. That is
|
||||
the only database read you have, and it only suggests values: it never tells
|
||||
you whether a spec is good or how much it matches. The database you read may
|
||||
differ from the one the report runs on, so never treat a lookup as a check.
|
||||
Iterate with the user on the spec itself: paths, filters, columns. The
|
||||
current spec, the last one in the conversation, comes with the user's
|
||||
message. When the user refines (other columns, filters or title), change the
|
||||
current spec. When the user asks for a different report, one that lists
|
||||
another kind of thing per row, start a new spec. Older specs show only as a
|
||||
bijlage name, and a spec you wrote before in this turn shows as "left out"
|
||||
once a newer one follows. Only when the user asks for an older spec, open it
|
||||
with read_spec and change that one. Never call
|
||||
run_report; there is no such tool here. When the spec is ready, ask the user
|
||||
to confirm execution. Then stop.
|
||||
|
||||
The spec itself does not go in your text. validate_spec stores a spec that
|
||||
checks out as an attachment ("bijlage", a spec file) on your answer. So in
|
||||
your answer: explain in Dutch in full what the report will list and which
|
||||
filters it applies, tell the user the actual spec is in the bijlage and to
|
||||
look there, and ask the user to confirm execution. Do not print the Turtle
|
||||
yourself; the bijlage shows the validated version, so never rewrite it in
|
||||
the text either.
|
||||
|
||||
Always prefix the spec with @prefix lines. Use the prefixes the profile
|
||||
declares. Write the spec as one Turtle block. Then call validate_spec. Fix
|
||||
what it says, up to three rounds. Then propose and stop.
|
||||
|
||||
Reply to the user in Dutch. Earlier turns are context only. Answer the last one.`;
|
||||
|
||||
// ask(questionOrMessages, profiles, session) → { text, trace }
|
||||
// A string is one question; an array is the conversation so far,
|
||||
// [{ role, content }], newest last. The system prompt is always this file's
|
||||
// own; a system message in the input is dropped.
|
||||
export async function ask(questionOrMessages, profiles, session) {
|
||||
const { handlers } = buildTools(profiles, session);
|
||||
const trace = [];
|
||||
|
||||
const turns = typeof questionOrMessages === 'string'
|
||||
? [{ role: 'user', content: questionOrMessages }]
|
||||
: questionOrMessages.filter(m => m.role !== 'system');
|
||||
const messages = [{ role: 'system', content: SYSTEM_PROMPT }, ...turns];
|
||||
// ask(turns, profiles) → { text, spec }. turns is the conversation so far,
|
||||
// [{ role, content, attachments }], newest last. spec is the last spec
|
||||
// validate_spec accepted in this turn, or null.
|
||||
export async function ask(turns, profiles) {
|
||||
const current = currentSpec(turns);
|
||||
const messages = [{ role: 'system', content: await systemPrompt(current, profiles) },
|
||||
...withCurrentSpec(turns.map(withSpecHint), current)];
|
||||
let spec = null;
|
||||
|
||||
for (let round = 0; round < MAX_ROUNDS; round++) {
|
||||
const reply = await chat(messages);
|
||||
messages.push(reply.message);
|
||||
const calls = reply.message.tool_calls || [];
|
||||
if (!calls.length) {
|
||||
return { text: reply.message.content || '', trace };
|
||||
}
|
||||
const { message } = await chat(newestSpecOnly(messages), TOOLS);
|
||||
messages.push(message);
|
||||
const calls = message.tool_calls || [];
|
||||
if (!calls.length) return { text: message.content || '', spec };
|
||||
if (calls.some(c => c.function.name === 'validate_spec')) message.leftOut = { tool_calls: calls.map(leaveOutSpec) };
|
||||
|
||||
for (const call of calls) {
|
||||
const args = parseArgs(call.function.arguments);
|
||||
trace.push({ name: call.function.name, args });
|
||||
const fn = handlers.get(call.function.name);
|
||||
let resultText;
|
||||
if (!fn) {
|
||||
resultText = `unknown tool "${call.function.name}"`;
|
||||
} else {
|
||||
try {
|
||||
const out = await fn(args);
|
||||
resultText = out.content?.[0]?.text ?? JSON.stringify(out);
|
||||
} catch (e) {
|
||||
resultText = `tool failed: ${String(e.message || e).split('\n')[0]}`;
|
||||
}
|
||||
let result;
|
||||
try {
|
||||
result = await runTool(call.function.name, args, profiles);
|
||||
} catch (e) {
|
||||
result = `tool failed: ${String(e.message || e).split('\n')[0]}`;
|
||||
}
|
||||
messages.push({ role: 'tool', tool_call_id: call.id, content: resultText });
|
||||
trace[trace.length - 1].result = resultText;
|
||||
// validate_spec answers "ok" (plus notes) when the spec checks out;
|
||||
// that spec becomes the bijlage of the answer.
|
||||
// One answer carries one spec: say so when this one replaces another.
|
||||
if (call.function.name === 'validate_spec' && /^ok(\n|$)/.test(result)) {
|
||||
if (spec && spec !== args.spec) {
|
||||
result += '\nThis spec replaces the one you validated before. Your answer carries only this one.';
|
||||
}
|
||||
spec = args.spec;
|
||||
}
|
||||
const reply = { role: 'tool', tool_call_id: call.id, content: result };
|
||||
if (call.function.name === 'read_spec' && result.includes('ReportSpec')) reply.leftOut = { content: LEFT_OUT };
|
||||
messages.push(reply);
|
||||
}
|
||||
}
|
||||
return { text: '(loop did not finish)', trace };
|
||||
return { text: 'Sorry, dit rapport is te moeilijk. Contacteer de developers.', spec };
|
||||
}
|
||||
|
||||
async function chat(messages) {
|
||||
// The history carries the spec as a bijlage, never as text. Paste one hint
|
||||
// line per spec under the message text, so the model knows a spec is there
|
||||
// and can open it with read_spec when the user asks for it. A spec the model
|
||||
// printed in its answer anyway is cut from the text.
|
||||
function withSpecHint(turn) {
|
||||
const content = turn.role === 'assistant' ? withoutSpecText(turn.content) : turn.content;
|
||||
const specs = (turn.attachments || []).filter(a => a.mediaType === SPEC_MEDIA_TYPE);
|
||||
if (!specs.length) return { ...turn, content };
|
||||
const hints = specs.map(a => `Bijlage "${a.name}", a spec file.`);
|
||||
return { ...turn, content: `${content}\n\n${hints.join('\n\n')}` };
|
||||
}
|
||||
|
||||
function withoutSpecText(text) {
|
||||
return text.replace(/```[\s\S]*?```/g, block => (block.includes('ReportSpec') ? '(spec left out)' : block));
|
||||
}
|
||||
|
||||
// The current spec: the last spec bijlage in the conversation, or null.
|
||||
function currentSpec(turns) {
|
||||
const name = turns.flatMap(t => t.attachments || [])
|
||||
.filter(a => a.mediaType === SPEC_MEDIA_TYPE).at(-1)?.name;
|
||||
const turtle = name && readSpecFile(name);
|
||||
return turtle ? { name, turtle } : null;
|
||||
}
|
||||
|
||||
// The system prompt also carries what the model would otherwise fetch at the
|
||||
// start of every turn: the profile list, and the menu of the current spec's
|
||||
// profile. Without a menu the model calls describe_profile itself.
|
||||
async function systemPrompt(current, profiles) {
|
||||
const parts = [REFINE_PROMPT, `The profiles:\n${listProfiles(profiles)}`];
|
||||
const uri = current && profileOf(current.turtle);
|
||||
if (profiles.has(uri)) {
|
||||
try {
|
||||
parts.push(`The menu of the current spec's profile <${uri}>:\n\n${await describeProfile(profiles.get(uri))}`);
|
||||
} catch (e) {
|
||||
console.error('[llm] could not describe the current profile:', e.message);
|
||||
}
|
||||
}
|
||||
return parts.join('\n\n');
|
||||
}
|
||||
|
||||
function profileOf(turtle) {
|
||||
try {
|
||||
return parseSpec(turtle).profileUri;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// The current spec goes in full under the user's new message: the one spec
|
||||
// the history holds as text.
|
||||
function withCurrentSpec(messages, current) {
|
||||
if (!current) return messages;
|
||||
const { name, turtle } = current;
|
||||
const last = messages.at(-1);
|
||||
return [...messages.slice(0, -1), {
|
||||
...last,
|
||||
content: `${last.content}\n\nThe current spec, bijlage "${name}":\n\n${turtle}`,
|
||||
leftOut: { content: `${last.content}\n\nThe current spec, bijlage "${name}": ${LEFT_OUT}` },
|
||||
}];
|
||||
}
|
||||
|
||||
// Only the newest spec goes to the model in full. A message that holds an
|
||||
// older one goes in its leftOut form; read_spec opens that spec again.
|
||||
function newestSpecOnly(messages) {
|
||||
const newest = messages.findLastIndex(m => m.leftOut);
|
||||
return messages.map((m, i) => (m.leftOut && i !== newest ? { ...m, ...m.leftOut } : m));
|
||||
}
|
||||
|
||||
function leaveOutSpec(call) {
|
||||
if (call.function.name !== 'validate_spec') return call;
|
||||
return { ...call, function: { ...call.function, arguments: JSON.stringify({ spec: LEFT_OUT }) } };
|
||||
}
|
||||
|
||||
// chat(messages, tools) → { message }. The one place that talks to the
|
||||
// provider. Without tools it is a plain completion (the mode check).
|
||||
export async function chat(messages, tools = null) {
|
||||
// Send only the fields every provider accepts back.
|
||||
const wire = messages.map(m => {
|
||||
const out = { role: m.role, content: m.content ?? null };
|
||||
@@ -90,15 +223,16 @@ async function chat(messages) {
|
||||
if (m.role === 'tool' && m.tool_call_id) out.tool_call_id = m.tool_call_id;
|
||||
return out;
|
||||
});
|
||||
const body = {
|
||||
model: LLM_MODEL,
|
||||
messages: wire,
|
||||
tools: toolDefs(),
|
||||
tool_choice: 'auto',
|
||||
};
|
||||
const body = { model: LLM_MODEL, messages: wire };
|
||||
if (tools) {
|
||||
body.tools = tools;
|
||||
body.tool_choice = 'auto';
|
||||
}
|
||||
if (LOG_LLM) {
|
||||
console.log(`[llm] POST ${LLM_BASE_URL}/chat/completions request:\n${
|
||||
JSON.stringify(body, null, 2)}`);
|
||||
} else {
|
||||
console.log('[llm] sending instructions to llm');
|
||||
}
|
||||
const res = await fetch(`${LLM_BASE_URL}/chat/completions`, {
|
||||
method: 'POST',
|
||||
@@ -123,16 +257,3 @@ function parseArgs(s) {
|
||||
if (!s) return {};
|
||||
try { return JSON.parse(s); } catch { return {}; }
|
||||
}
|
||||
|
||||
// The tool definitions sent to the model.
|
||||
import { toolSchemas } from './schemas.js';
|
||||
function toolDefs() {
|
||||
return toolSchemas().map(t => ({
|
||||
type: 'function',
|
||||
function: {
|
||||
name: t.name,
|
||||
description: t.description,
|
||||
parameters: t.inputSchema,
|
||||
},
|
||||
}));
|
||||
}
|
||||
56
src/llm/mode.js
Normal file
56
src/llm/mode.js
Normal file
@@ -0,0 +1,56 @@
|
||||
// The mode classifier. One LLM call per turn: a guestimate of whether the
|
||||
// user wants to keep refining the query or has given a clear command to
|
||||
// execute it now. Returns true (execute) / false (refine). Deterministic
|
||||
// code branches on the answer; refine is the safe default when the call
|
||||
// fails or the answer does not parse.
|
||||
|
||||
import { chat } from './loop.js';
|
||||
|
||||
const MODE_PROMPT = `You read a conversation between a user and a report assistant.
|
||||
The assistant proposes report specs and asks the user to confirm before
|
||||
running them.
|
||||
|
||||
Decide whether the user's LAST message is a clear instruction to run the
|
||||
report now (for example "ja", "voer uit", "doe het", "maak het rapport",
|
||||
"go", "execute", "run it") or is still refining, asking, correcting or
|
||||
answering a question.
|
||||
|
||||
Rules:
|
||||
- Only return true when the user clearly tells the assistant to execute the
|
||||
agreed report right now. A short "ja" or "ok" after a proposal counts.
|
||||
- A new or amended question, a clarification, a "no", a question about the
|
||||
proposal, or the absence of an earlier proposal means false.
|
||||
- When in doubt, return false.
|
||||
|
||||
Reply with exactly one token: true or false. No other words, no punctuation,
|
||||
no markdown.`;
|
||||
|
||||
// wantsExecution(turns) → true when the user's last message says "run it
|
||||
// now". turns: [{ role, content }], newest last; the last few are enough.
|
||||
export async function wantsExecution(turns) {
|
||||
let message;
|
||||
try {
|
||||
({ message } = await chat([{ role: 'system', content: MODE_PROMPT }, ...turns.slice(-6)]));
|
||||
} catch (e) {
|
||||
// The call failed (no LLM, provider down). Refine: safer than running a
|
||||
// report on a guess, and the refine loop will surface the real error.
|
||||
return false;
|
||||
}
|
||||
return stripMarkdown(firstText(message)).trim().toLowerCase().startsWith('true');
|
||||
}
|
||||
|
||||
// Reasoning models may leave `content` empty and put the visible text in a
|
||||
// separate field (reasoning_content / thinking / reasoning), or may not
|
||||
// answer at all when the thinking burned the token budget. Read the visible
|
||||
// fields in order and take the first non-empty one.
|
||||
function firstText(message) {
|
||||
for (const key of ['content', 'reasoning_content', 'reasoning', 'thinking']) {
|
||||
const v = message?.[key];
|
||||
if (typeof v === 'string' && v.trim()) return v;
|
||||
}
|
||||
return '';
|
||||
}
|
||||
|
||||
function stripMarkdown(s) {
|
||||
return String(s).replace(/[*_`>#]/g, ' ');
|
||||
}
|
||||
@@ -1,39 +0,0 @@
|
||||
// The JSON schemas of the four tools. The descriptions are what the LLM
|
||||
// reads to decide which tool to call.
|
||||
|
||||
export function toolSchemas() {
|
||||
return [
|
||||
{
|
||||
name: 'list_profiles',
|
||||
description: 'List the report profiles available, with their id and title.',
|
||||
inputSchema: { type: 'object', properties: {}, required: [] },
|
||||
},
|
||||
{
|
||||
name: 'describe_profile',
|
||||
description: 'Describe one profile: its entities, fields, relations and labels. Inline the values of short code lists.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { profile_id: { type: 'string', description: 'The profile URI returned by list_profiles' } },
|
||||
required: ['profile_id'],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'validate_spec',
|
||||
description: 'Validate a report spec (Turtle). Returns "ok" or one clear sentence saying what is wrong and what to write instead. No database is touched.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { spec: { type: 'string', description: 'The spec as Turtle text' } },
|
||||
required: ['spec'],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: 'run_report',
|
||||
description: 'Run a valid spec and return the report URI, the row count and the file URI. Runs as the caller, so it sees only what they may see. Waits as long as the run keeps answering; gives up after RUN_TIMEOUT seconds of silence. The report is named by the dct:title in the spec. Call this once per question, after validate_spec returns ok.',
|
||||
inputSchema: {
|
||||
type: 'object',
|
||||
properties: { spec: { type: 'string' } },
|
||||
required: ['spec'],
|
||||
},
|
||||
},
|
||||
];
|
||||
}
|
||||
187
src/llm/tools.js
187
src/llm/tools.js
@@ -1,113 +1,88 @@
|
||||
// The tools the LLM calls: list_profiles, describe_profile, validate_spec,
|
||||
// run_report. They return a report URI, a count, or an error we wrote —
|
||||
// never a cell value. The shape every tool returns is
|
||||
// { content: [{ type: 'text', text }] }, which the loop passes straight back
|
||||
// to the model.
|
||||
|
||||
import { checkSpec } from '../runner/check.js';
|
||||
// The tools the LLM calls, refinement only: describe_profile,
|
||||
// validate_spec, read_spec and lookup_values. Each returns
|
||||
// plain text for the model. No report is created, no spec is evaluated; the
|
||||
// database reads run under the service scope (public graph).
|
||||
import { parseSpec } from '../runner/spec.js';
|
||||
import { describeProfile, codeListValues } from './describe.js';
|
||||
import { run, slug } from '../runner/run.js';
|
||||
import { checkSpec, whereNotes, profileError } from '../runner/check.js';
|
||||
import { describeProfile } from './describe.js';
|
||||
import { lookupValues } from './explore.js';
|
||||
import { readSpecFile } from '../chat.js';
|
||||
|
||||
const MAX_PATH_DEPTH = Number(process.env.MAX_PATH_DEPTH || 8);
|
||||
const RUN_TIMEOUT = Number(process.env.RUN_TIMEOUT || 60) * 1000;
|
||||
// The tool definitions sent to the model. The descriptions are what the LLM
|
||||
// reads to decide which tool to call.
|
||||
export const TOOLS = [
|
||||
tool('describe_profile',
|
||||
'Describe one profile: its entities, fields, relations and labels. Inline the values of short code lists.',
|
||||
{ profile_id: 'The profile URI from the profile list' }),
|
||||
tool('validate_spec',
|
||||
'Validate a report spec (Turtle). Returns "ok", followed by one line per rep:where saying where it applies, or one clear sentence saying what is wrong and what to write instead. No database is touched.',
|
||||
{ spec: 'The spec as Turtle text' }),
|
||||
tool('read_spec',
|
||||
'Return the Turtle of a spec bijlage, a ttl file attached to an earlier message in this conversation. The current spec already comes with the user\'s message; use this only when the user asks for an older spec. Read-only.',
|
||||
{ file_name: 'The file name of the spec bijlage, e.g. specificatie-<uuid>.ttl' }),
|
||||
tool('lookup_values',
|
||||
'Search a code list for a term so you can suggest concrete values for a filter (for sh:hasValue or sh:in). Returns up to 25 matches with their URIs and labels, a total, and an exact flag when a label equals the term. Reads candidate values only; it never runs the spec and never tells you how many subjects a spec would match. Read-only, no report is created.',
|
||||
{
|
||||
profile_id: 'The profile URI from the profile list',
|
||||
field: 'The field as "entityLabel.fieldLabel", e.g. "bestuurseenheid.naam"',
|
||||
term: 'The Dutch word or name to search for',
|
||||
}),
|
||||
];
|
||||
|
||||
// buildTools(profiles, session) → { handlers }
|
||||
// run_report waits while the run keeps answering: every query that comes
|
||||
// back (a seed page, a column batch) restarts the RUN_TIMEOUT clock, and
|
||||
// only that much silence gives up. The batch count is known upfront — the
|
||||
// subject list is complete after the seed — so a slow run with many small
|
||||
// batches finishes, however long it takes in total.
|
||||
export function buildTools(profiles, session) {
|
||||
const profileList = () => [...profiles.values()].map(p => ({ id: p.uri, title: p.title }));
|
||||
|
||||
const findProfile = (id) => profiles.get(id);
|
||||
|
||||
async function describe_profile({ profile_id }) {
|
||||
const p = findProfile(profile_id);
|
||||
if (!p) return text(`no profile <${profile_id}>. Available: ${profileList().map(x => x.title).join(', ')}.`);
|
||||
const inlined = await codeListValues(p, profiles);
|
||||
return text(describeProfile(p, inlined));
|
||||
}
|
||||
|
||||
async function validate_spec({ spec }) {
|
||||
let parsed;
|
||||
try { parsed = parseSpec(spec); }
|
||||
catch (e) { return text(`the spec did not parse: ${e.message}`); }
|
||||
const p = findProfile(parsed.profileUri);
|
||||
if (!p) return text(`no profile <${parsed.profileUri}>. Available: ${profileList().map(x => x.title).join(', ')}.`);
|
||||
const errors = checkSpec(parsed, p, MAX_PATH_DEPTH, profiles);
|
||||
return text(errors.length ? errors[0] : 'ok');
|
||||
}
|
||||
|
||||
async function list_profiles() {
|
||||
return text(profileList().map(x => `${x.title} <${x.id}>`).join('\n'));
|
||||
}
|
||||
|
||||
async function run_report({ spec }) {
|
||||
let parsed;
|
||||
try { parsed = parseSpec(spec); }
|
||||
catch (e) { return text(`the spec did not parse: ${e.message}`); }
|
||||
const p = findProfile(parsed.profileUri);
|
||||
if (!p) return text(`no profile <${parsed.profileUri}>.`);
|
||||
const errors = checkSpec(parsed, p, MAX_PATH_DEPTH, profiles);
|
||||
if (errors.length) return text(`the spec is not valid: ${errors[0]}`);
|
||||
try {
|
||||
// The spec's dct:title names the report.
|
||||
const name = parsed.title || 'report';
|
||||
await session.onReportStart?.({ title: name, fileName: `${slug(name)}.csv` });
|
||||
const waiter = newRunWaiter(RUN_TIMEOUT);
|
||||
const running = run(session.query, session.update, parsed, p, name, { spec }, waiter.tick);
|
||||
// The end callback also fires after the wait gave up: the run is not
|
||||
// cancelled, the file arrives when it arrives. The .catch keeps a late
|
||||
// failure from crashing the process.
|
||||
running
|
||||
.then((r) => session.onReportEnd?.(null, r), (e) => session.onReportEnd?.(e))
|
||||
.catch((e) => console.error('[reports] end callback failed:', e));
|
||||
const result = await waiter.finish(running);
|
||||
return text(`done. report <${result.reportUri}>, ${result.rowCount} rows, file <${result.fileUri}>.`);
|
||||
} catch (e) {
|
||||
return text(`the report failed: ${String(e.message || e).split('\n')[0]}`);
|
||||
}
|
||||
}
|
||||
|
||||
const handlers = new Map([
|
||||
['list_profiles', list_profiles],
|
||||
['describe_profile', describe_profile],
|
||||
['validate_spec', validate_spec],
|
||||
['run_report', run_report],
|
||||
]);
|
||||
|
||||
return { handlers };
|
||||
}
|
||||
|
||||
function text(s) { return { content: [{ type: 'text', text: s }] }; }
|
||||
|
||||
// Waits for a report run while it makes progress. tick restarts the silence
|
||||
// clock; idleMs without a single tick rejects. The run is never cancelled:
|
||||
// finish's handlers only pass its own settlement on, so a run that answers
|
||||
// after the wait gave up still delivers through the end callback.
|
||||
function newRunWaiter(idleMs) {
|
||||
let deferred;
|
||||
const promise = new Promise((resolve, reject) => { deferred = { resolve, reject }; });
|
||||
let settled = false;
|
||||
let timer = setTimeout(fail, idleMs);
|
||||
function fail() {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
deferred.reject(new Error(`no query answered for ${idleMs / 1000}s`));
|
||||
function tool(name, description, params) {
|
||||
const properties = {};
|
||||
for (const [key, text] of Object.entries(params)) {
|
||||
properties[key] = { type: 'string', description: text };
|
||||
}
|
||||
return {
|
||||
tick() {
|
||||
if (settled) return;
|
||||
clearTimeout(timer);
|
||||
timer = setTimeout(fail, idleMs);
|
||||
},
|
||||
finish(running) {
|
||||
running.then(
|
||||
(r) => { settled = true; clearTimeout(timer); deferred.resolve(r); },
|
||||
(e) => { settled = true; clearTimeout(timer); deferred.reject(e); });
|
||||
return promise;
|
||||
type: 'function',
|
||||
function: {
|
||||
name,
|
||||
description,
|
||||
parameters: { type: 'object', properties, required: Object.keys(params) },
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
export async function runTool(name, args, profiles) {
|
||||
switch (name) {
|
||||
case 'describe_profile': return describe(args, profiles);
|
||||
case 'validate_spec': return validateSpec(args, profiles);
|
||||
case 'read_spec': return readSpec(args);
|
||||
case 'lookup_values': return lookup(args, profiles);
|
||||
default: return `unknown tool "${name}"`;
|
||||
}
|
||||
}
|
||||
|
||||
// The profile list, in the system prompt: one line per profile.
|
||||
export function listProfiles(profiles) {
|
||||
return [...profiles.values()].map(p => `${p.title} <${p.uri}>`).join('\n');
|
||||
}
|
||||
|
||||
async function describe({ profile_id }, profiles) {
|
||||
return profileError(profiles, profile_id) || describeProfile(profiles.get(profile_id));
|
||||
}
|
||||
|
||||
function validateSpec({ spec }, profiles) {
|
||||
let parsed;
|
||||
try {
|
||||
parsed = parseSpec(spec);
|
||||
} catch (e) {
|
||||
return `the spec did not parse: ${e.message}`;
|
||||
}
|
||||
const error = checkSpec(parsed, profiles);
|
||||
if (error) return error;
|
||||
return ['ok', ...whereNotes(parsed, profiles.get(parsed.profileUri))].join('\n');
|
||||
}
|
||||
|
||||
// Only spec files this service made are readable.
|
||||
function readSpec({ file_name }) {
|
||||
return readSpecFile(file_name) || 'no such spec file.';
|
||||
}
|
||||
|
||||
async function lookup({ profile_id, field, term }, profiles) {
|
||||
const error = profileError(profiles, profile_id);
|
||||
if (error) return error;
|
||||
if (!term) return 'no term to search for.';
|
||||
return lookupValues(profiles.get(profile_id), field, term);
|
||||
}
|
||||
|
||||
@@ -1,48 +1,71 @@
|
||||
// What this service answers a chat turn with: the LLM loop with the
|
||||
// conversation so far, and a report run becomes an interim message with a
|
||||
// pending file.
|
||||
// What the assistant answers a chat turn with: { text, attachments }. When
|
||||
// the answer is written, every bijlage already exists.
|
||||
//
|
||||
// Refine: the LLM proposes a spec; the last spec that checked out goes on
|
||||
// the answer as a bijlage. The prompt never carries the Turtle.
|
||||
//
|
||||
// Execute: no LLM runs. The last spec bijlage in the conversation is the
|
||||
// agreed spec; it is read from the share and fed straight to the runner.
|
||||
// The answer is fixed text. When the run cannot happen or fails, the answer
|
||||
// is a fixed failure message: nothing is retried and no spec is rewritten.
|
||||
import { uuid } from 'mu';
|
||||
import { ask } from './llm/loop.js';
|
||||
import { parseSpec } from './runner/spec.js';
|
||||
import { checkSpec } from './runner/check.js';
|
||||
import { run, registerReport, slug } from './runner/run.js';
|
||||
import { storeSpecFile, readSpecFile, writeShareFile, registerFile,
|
||||
SPEC_MEDIA_TYPE, SPEC_TYPE, RESULT_TYPE } from './chat.js';
|
||||
|
||||
const TEXT = {
|
||||
making: 'Ik maak het rapport. Het verschijnt hier zodra het klaar is.',
|
||||
failed: 'Het rapport is mislukt. Stel de vraag opnieuw, of anders geformuleerd.',
|
||||
executed: 'Het rapport is uitgevoerd. De bijlagen staan erbij.',
|
||||
noSpec: 'Er is nog geen voorstel om uit te voeren. Beschrijf eerst welk rapport je wilt.',
|
||||
invalid: 'Het voorstel is niet meer geldig. Vraag een nieuw voorstel.',
|
||||
failed: 'Het rapport kon niet uitgevoerd worden. Probeer het later opnieuw.',
|
||||
noAnswer: 'Er kwam geen antwoord. Probeer het opnieuw.',
|
||||
};
|
||||
|
||||
// The chat mounts before the profiles are in; whenProfiles waits for the load.
|
||||
export function reportAssistant(whenProfiles) {
|
||||
return async function answer(turn) {
|
||||
// One pending file per turn: a second run_report overwrites it before the
|
||||
// first end callback fires.
|
||||
let pendingDocument = null;
|
||||
|
||||
const session = {
|
||||
query: turn.query,
|
||||
update: turn.update,
|
||||
async onReportStart({ fileName }) {
|
||||
const said = await turn.say(TEXT.making, [{ name: fileName, mediaType: 'text/csv' }]);
|
||||
pendingDocument = said.documents[0].uri;
|
||||
},
|
||||
async onReportEnd(error, result) {
|
||||
const documentUri = pendingDocument;
|
||||
pendingDocument = null;
|
||||
if (!documentUri) return;
|
||||
if (error) {
|
||||
await turn.drop(documentUri);
|
||||
if (turn.finished) await turn.say(TEXT.failed);
|
||||
return;
|
||||
}
|
||||
await turn.setUrl(documentUri, downloadHref(result.fileUri));
|
||||
},
|
||||
};
|
||||
|
||||
const profiles = await whenProfiles();
|
||||
const { text } = await ask([...turn.history, { role: 'user', content: turn.content }], profiles, session);
|
||||
return text;
|
||||
};
|
||||
// An empty answer (a reasoning model can end with no text) gets a fixed
|
||||
// text and no spec: only a spec the user saw explained may run.
|
||||
export async function refineSpec(turns, profiles) {
|
||||
const { text, spec } = await ask(turns, profiles);
|
||||
if (!text.trim()) return { text: TEXT.noAnswer };
|
||||
const attachments = spec ? [await storeSpecFile(spec)] : [];
|
||||
return { text: text.trim(), attachments };
|
||||
}
|
||||
|
||||
// The file service serves /files/<uuid>/download; the logical file URI ends
|
||||
// in that uuid.
|
||||
function downloadHref(fileUri) {
|
||||
return `/files/${fileUri.split('/').pop()}/download`;
|
||||
export async function executeSpec(history, conversationTitle, profiles) {
|
||||
const attachment = lastSpecAttachment(history);
|
||||
const content = attachment && readSpecFile(attachment.name);
|
||||
if (!content) return { text: TEXT.noSpec };
|
||||
|
||||
try {
|
||||
const spec = parseSpec(content);
|
||||
const error = checkSpec(spec, profiles);
|
||||
if (error) {
|
||||
console.error(`[reports] the agreed spec is not valid: ${error}`);
|
||||
return { text: TEXT.invalid };
|
||||
}
|
||||
|
||||
// The conversation's title (named by the first question) names the
|
||||
// file; the share name is unique, so a second run never overwrites the
|
||||
// first. The report itself keeps the dct:title of the spec.
|
||||
const shareName = `${uuid()}.csv`;
|
||||
writeShareFile(shareName, await run(spec, profiles.get(spec.profileUri)));
|
||||
const csvFile = await registerFile(shareName, 'text/csv', RESULT_TYPE, `${slug(conversationTitle)}.csv`);
|
||||
await registerReport(spec.title, csvFile);
|
||||
|
||||
// Same physical spec file, registered as a bijlage for this message.
|
||||
const specFile = await registerFile(attachment.name, SPEC_MEDIA_TYPE, SPEC_TYPE);
|
||||
|
||||
return { text: TEXT.executed, attachments: [csvFile, specFile] };
|
||||
} catch (e) {
|
||||
console.error('[reports] execution failed:', e);
|
||||
return { text: TEXT.failed };
|
||||
}
|
||||
}
|
||||
|
||||
function lastSpecAttachment(history) {
|
||||
const specs = history.flatMap(message => message.attachments)
|
||||
.filter(a => a.mediaType === SPEC_MEDIA_TYPE && a.name?.startsWith('specificatie-'));
|
||||
return specs.at(-1) || null;
|
||||
}
|
||||
|
||||
@@ -1,83 +1,132 @@
|
||||
// Values to rows. Subjects in stable order, dedup on term, collect per cell.
|
||||
// rep:self columns take the subject URI; a column with no values is an
|
||||
// empty cell, never a dropped row. rep:row expands one subject into several
|
||||
// lines; at most one column per spec may do that (checked in check.js).
|
||||
import { sharedSteps } from './spec.js';
|
||||
|
||||
// Values to rows, paired by URI on every step. Two columns that walk the
|
||||
// same steps (with the same rep:where on them) share the nodes on those
|
||||
// steps, the way a rep:where shares nodes with its column. Per subject the
|
||||
// paths of the columns make a tree of the nodes found; every row is one way
|
||||
// down that tree. Columns through the same node stay on one row; branches
|
||||
// under one node multiply. A column with no value is an empty cell, never a
|
||||
// dropped row. sh:groupConcat, sh:min and sh:max fold a column into one cell
|
||||
// per node it shares with the other columns. rep:self takes the subject
|
||||
// URI. Subjects in stable order; identical rows appear once.
|
||||
|
||||
const SH = 'http://www.w3.org/ns/shacl#';
|
||||
const REP = 'http://mu.semte.ch/vocabularies/reporting/';
|
||||
const XSD = 'http://www.w3.org/2001/XMLSchema#';
|
||||
const FOLDING = [SH + 'groupConcat', SH + 'min', SH + 'max'];
|
||||
|
||||
export const NUMERIC_DATATYPES = new Set([
|
||||
'integer', 'decimal', 'double', 'float', 'long', 'int', 'short', 'byte',
|
||||
'nonNegativeInteger', 'positiveInteger', 'nonPositiveInteger', 'negativeInteger',
|
||||
'unsignedLong', 'unsignedInt', 'unsignedShort', 'unsignedByte',
|
||||
].map(d => XSD + d));
|
||||
export const TEMPORAL_DATATYPES = new Set([XSD + 'date', XSD + 'dateTime']);
|
||||
|
||||
// assemble(subjects, values, spec) → rows, the header first. values:
|
||||
// subject → column index → chains, one term per step (see columns.js).
|
||||
export function assemble(subjects, values, spec) {
|
||||
const sorted = [...subjects].sort();
|
||||
const tree = columnTree(spec.columns);
|
||||
const rows = [spec.columns.map(c => c.label)];
|
||||
for (const subject of sorted) {
|
||||
const perColumn = values.get(subject) || new Map();
|
||||
const cells = spec.columns.map((col, i) => {
|
||||
if (!col.path.length) return [subject]; // rep:self
|
||||
const list = dedup(perColumn.get(i) || []);
|
||||
return collect(list, col);
|
||||
});
|
||||
const lines = Math.max(1, ...cells.map(c => c.length));
|
||||
for (let r = 0; r < lines; r++) {
|
||||
// every cell except the expanding one keeps its value on each line
|
||||
rows.push(cells.map(c => c[Math.min(r, c.length - 1)]));
|
||||
for (const subject of [...subjects].sort()) {
|
||||
const top = { term: { value: subject }, below: new Map() };
|
||||
for (const [index, chains] of values.get(subject) || []) {
|
||||
for (const chain of chains) addChain(top, tree.paths[index], chain);
|
||||
}
|
||||
const seen = new Set();
|
||||
for (const cells of rowsAt(tree.root, top, tree, spec.columns)) {
|
||||
const row = spec.columns.map((col, i) => cells.get(i) ?? '');
|
||||
const key = JSON.stringify(row);
|
||||
if (!seen.has(key)) {
|
||||
seen.add(key);
|
||||
rows.push(row);
|
||||
}
|
||||
}
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
function collect(terms, col) {
|
||||
switch (col.collect) {
|
||||
case SH + 'groupConcat':
|
||||
case null:
|
||||
case undefined:
|
||||
return [terms.map(t => t.value).join(col.separator ?? ',')];
|
||||
case SH + 'min':
|
||||
return [extreme(terms, col, -1)];
|
||||
case SH + 'max':
|
||||
return [extreme(terms, col, 1)];
|
||||
case REP + 'row':
|
||||
return terms.length ? terms.map(t => t.value) : [''];
|
||||
default:
|
||||
throw new Error(`column "${col.label}" has an unknown rep:collect <${col.collect}>. Use sh:groupConcat, rep:row, sh:min or sh:max.`);
|
||||
}
|
||||
// The steps of all columns as one tree of branches. Two columns share a
|
||||
// branch as long as they took the same steps with the same rep:where on
|
||||
// them. paths[i] lists the branches column i goes through; a branch knows
|
||||
// the columns that end on it and the columns that go through it.
|
||||
function columnTree(columns) {
|
||||
const newBranch = () => ({ next: new Map(), ends: [], columns: [] });
|
||||
const root = newBranch();
|
||||
const paths = columns.map((col, index) => {
|
||||
let branch = root;
|
||||
const branches = col.path.map((hop, k) => {
|
||||
const where = col.where.filter(w => sharedSteps(col.path, w.path) === k + 1);
|
||||
const key = `${hop.inverse ? '^' : ''}${hop.predicate} ${JSON.stringify(where)}`;
|
||||
if (!branch.next.has(key)) branch.next.set(key, newBranch());
|
||||
branch = branch.next.get(key);
|
||||
branch.columns.push(index);
|
||||
return branch;
|
||||
});
|
||||
branch.ends.push(index);
|
||||
return branches;
|
||||
});
|
||||
return { root, paths };
|
||||
}
|
||||
|
||||
// min/max over values that share one datatype. Numbers compare as numbers,
|
||||
// dates and times as instants. Mixed datatypes throw: check.js should have
|
||||
// caught it, and a silently wrong cell is worse than a failed job.
|
||||
// One path the store returned, subject to value, into the tree of nodes
|
||||
// found for that subject. Nodes are keyed by value: a node reached twice is
|
||||
// one node, a value found twice under one node is one value.
|
||||
function addChain(found, branches, chain) {
|
||||
branches.forEach((branch, k) => {
|
||||
if (!found.below.has(branch)) found.below.set(branch, new Map());
|
||||
const byValue = found.below.get(branch);
|
||||
if (!byValue.has(chain[k].value)) byValue.set(chain[k].value, { term: chain[k], below: new Map() });
|
||||
found = byValue.get(chain[k].value);
|
||||
});
|
||||
}
|
||||
|
||||
// The rows under one found node: its own cells, times the rows of each
|
||||
// branch below it. A branch where nothing was found adds empty cells.
|
||||
function rowsAt(branch, found, tree, columns) {
|
||||
let rows = [new Map(branch.ends.map(i => [i, found.term.value]))];
|
||||
for (const next of branch.next.values()) {
|
||||
const nodes = [...(found.below.get(next)?.values() || [])];
|
||||
const below = next.columns.every(i => FOLDING.includes(columns[i].collect))
|
||||
? [fold(next, nodes, tree, columns)]
|
||||
: nodes.length ? nodes.flatMap(node => rowsAt(next, node, tree, columns)) : [new Map()];
|
||||
rows = rows.flatMap(row => below.map(cells => new Map([...row, ...cells])));
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
// A branch where every column folds gives one set of cells: per column, all
|
||||
// its values in the branch, folded into one cell.
|
||||
function fold(branch, nodes, tree, columns) {
|
||||
const cells = new Map();
|
||||
for (const i of branch.columns) {
|
||||
const rest = tree.paths[i].slice(tree.paths[i].indexOf(branch) + 1);
|
||||
const terms = nodes.flatMap(node => termsBelow(node, rest));
|
||||
const unique = [...new Map(terms.map(t => [t.value, t])).values()];
|
||||
const col = columns[i];
|
||||
cells.set(i, col.collect === SH + 'groupConcat'
|
||||
? unique.map(t => t.value).join(col.separator ?? ',')
|
||||
: extreme(unique, col, col.collect === SH + 'min' ? -1 : 1));
|
||||
}
|
||||
return cells;
|
||||
}
|
||||
|
||||
function termsBelow(node, rest) {
|
||||
if (!rest.length) return [node.term];
|
||||
const byValue = node.below.get(rest[0]);
|
||||
return byValue ? [...byValue.values()].flatMap(next => termsBelow(next, rest.slice(1))) : [];
|
||||
}
|
||||
|
||||
// min/max. Numbers compare as numbers. Dates and times compare as text:
|
||||
// xsd:date and xsd:dateTime are ISO strings, so text order is date order.
|
||||
// Numbers mixed with other values throw: check.js should have caught it,
|
||||
// and a silently wrong cell is worse than a failed job. The terms are SPARQL
|
||||
// JSON bindings, so the datatype is a plain string.
|
||||
function extreme(terms, col, direction) {
|
||||
if (!terms.length) return '';
|
||||
const datatypes = [...new Set(terms.map(t => t.datatype?.value || ''))];
|
||||
if (datatypes.length > 1) {
|
||||
throw new Error(`column "${col.label}" asks for sh:${direction < 0 ? 'min' : 'max'} but its values carry ${datatypes.length} datatypes (${datatypes.join(', ')}).`);
|
||||
const numbers = terms.filter(t => NUMERIC_DATATYPES.has(t.datatype)).length;
|
||||
if (numbers && numbers < terms.length) {
|
||||
throw new Error(`column "${col.label}" asks for sh:${direction < 0 ? 'min' : 'max'} but mixes numbers with other values.`);
|
||||
}
|
||||
const dt = datatypes[0];
|
||||
const key = NUMERIC_DATATYPES.has(dt) ? t => Number(t.value)
|
||||
: TEMPORAL_DATATYPES.has(dt) ? t => new Date(t.value).getTime()
|
||||
: t => t.value;
|
||||
const key = numbers ? t => Number(t.value) : t => t.value;
|
||||
const cmp = (a, b) => a < b ? -1 : a > b ? 1 : 0;
|
||||
const best = terms.reduce((a, b) => (cmp(key(b), key(a)) * direction > 0 ? b : a));
|
||||
return best.value;
|
||||
}
|
||||
|
||||
// Two terms are the same when the URIs match, or lexical form, datatype
|
||||
// and language tag all match.
|
||||
export function dedup(terms) {
|
||||
const seen = new Set();
|
||||
const out = [];
|
||||
for (const t of terms) {
|
||||
const key = t.termType === 'NamedNode'
|
||||
? `u:${t.value}`
|
||||
: `l:${t.value}|${t.datatype?.value || ''}|${t.language || ''}`;
|
||||
if (!seen.has(key)) { seen.add(key); out.push(t); }
|
||||
}
|
||||
return out;
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import { fieldsOf } from './profile.js';
|
||||
import { startShape, fieldsOf } from './profile.js';
|
||||
import { NUMERIC_DATATYPES } from './assemble.js';
|
||||
import { sharedSteps } from './spec.js';
|
||||
|
||||
const SH = 'http://www.w3.org/ns/shacl#';
|
||||
const REP = 'http://mu.semte.ch/vocabularies/reporting/';
|
||||
@@ -8,23 +9,18 @@ const XSD_DATE = XSD + 'date';
|
||||
const XSD_DATETIME = XSD + 'dateTime';
|
||||
|
||||
// The validator. checkSpec runs the checks in order and returns the first
|
||||
// error as a one-element array (empty if ok). Every message names the thing
|
||||
// that is wrong in the profile's own words and says what to write instead.
|
||||
// No database call anywhere in this file.
|
||||
// error, or null when the spec is fine. Every message names the thing that
|
||||
// is wrong in the profile's own words and says what to write instead. No
|
||||
// database call anywhere in this file.
|
||||
|
||||
const MAX_PATH_DEPTH = Number(process.env.MAX_PATH_DEPTH || 8);
|
||||
const ANYOF_MAX_TERMS = 24;
|
||||
const ANYOF_MAX_TERM_LENGTH = 100;
|
||||
|
||||
// The spec names a profile we have. knownProfiles is the profile Map loaded
|
||||
// at boot; the check is skipped when it is omitted.
|
||||
function profileNamed(spec, knownProfiles) {
|
||||
if (!spec.profileUri) {
|
||||
return `no rep:profile named. Available: ${knownProfiles ? [...knownProfiles.values()].map(p => p.title).join(', ') : '(none)'} .`;
|
||||
}
|
||||
if (knownProfiles && !knownProfiles.has(spec.profileUri)) {
|
||||
return `no profile <${spec.profileUri}>. Available: ${knownProfiles ? [...knownProfiles.values()].map(p => p.title).join(', ') : '(none)'}.`;
|
||||
}
|
||||
return null;
|
||||
// null when the profile exists, else what to write instead.
|
||||
export function profileError(profiles, uri) {
|
||||
if (profiles.has(uri)) return null;
|
||||
return `no profile <${uri}>. Available: ${[...profiles.values()].map(p => p.title).join(', ')}.`;
|
||||
}
|
||||
|
||||
// The spec names an entity we have: by rep:entity, or by a targetClass only
|
||||
@@ -57,14 +53,20 @@ function shapeList(profile) {
|
||||
return profile.shapes.map(s => s.label || s.targetClass).join(', ');
|
||||
}
|
||||
|
||||
// The shape a spec starts from, after startEntity has approved it: rep:entity
|
||||
// wins, otherwise the one shape carrying the targetClass.
|
||||
function startShapeOf(spec, profile) {
|
||||
if (spec.entity) {
|
||||
return profile.shapes.find(s => s.uri === spec.entity)?.uri || null;
|
||||
}
|
||||
const matches = profile.shapes.filter(s => s.targetClass === spec.targetClass);
|
||||
return matches.length === 1 ? matches[0].uri : null;
|
||||
// Every condition in the spec: the filters, and the rep:where conditions on
|
||||
// filters and columns, with the words that name each one in a message. A
|
||||
// rep:where also carries its host, the filter or column it hangs on.
|
||||
function conditions(spec) {
|
||||
const out = spec.filters.map((f, i) => ({ what: `filter ${i + 1}`, cond: f }));
|
||||
spec.filters.forEach((f, i) => f.where.forEach((w, j) => out.push({
|
||||
what: `rep:where ${j + 1} on filter ${i + 1}`, cond: w,
|
||||
host: { what: `filter ${i + 1}`, path: f.path },
|
||||
})));
|
||||
spec.columns.forEach(c => c.where.forEach((w, j) => out.push({
|
||||
what: `rep:where ${j + 1} on column "${c.label}"`, cond: w,
|
||||
host: { what: `column "${c.label}"`, path: c.path },
|
||||
})));
|
||||
return out;
|
||||
}
|
||||
|
||||
// Columns exist
|
||||
@@ -83,26 +85,26 @@ function columnLabel(spec) {
|
||||
|
||||
// Labels are unique
|
||||
function columnLabelsUnique(spec) {
|
||||
const seen = new Map();
|
||||
const seen = new Set();
|
||||
for (const col of spec.columns) {
|
||||
if (seen.has(col.label)) {
|
||||
return `two columns are called "${col.label}". Labels must differ.`;
|
||||
}
|
||||
seen.set(col.label, true);
|
||||
seen.add(col.label);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Every path resolves, columns end on a value, depth capped
|
||||
function pathsResolve(spec, profile, maxPathDepth) {
|
||||
const startShapeUri = startShapeOf(spec, profile);
|
||||
function pathsResolve(spec, profile) {
|
||||
const startShapeUri = startShape(profile, spec).uri;
|
||||
for (const col of spec.columns) {
|
||||
const err = walk(profile, startShapeUri, col.path, maxPathDepth,
|
||||
`column "${col.label}"`, true);
|
||||
const err = walk(profile, startShapeUri, col.path,
|
||||
`column "${col.label}"`, true, col.nodeKind === SH + 'IRI');
|
||||
if (err) return err;
|
||||
}
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
const err = walk(profile, startShapeUri, filter.path, maxPathDepth,
|
||||
const err = walk(profile, startShapeUri, filter.path,
|
||||
`filter ${i + 1}`, false);
|
||||
if (err) return err;
|
||||
}
|
||||
@@ -111,22 +113,17 @@ function pathsResolve(spec, profile, maxPathDepth) {
|
||||
|
||||
// sh:min/sh:max only on numbers and dates
|
||||
function collectMinMaxTyped(spec, profile) {
|
||||
const startShapeUri = startShapeOf(spec, profile);
|
||||
const startShapeUri = startShape(profile, spec).uri;
|
||||
for (const col of spec.columns) {
|
||||
if (col.collect !== SH + 'min' && col.collect !== SH + 'max') continue;
|
||||
const name = col.collect === SH + 'min' ? 'sh:min' : 'sh:max';
|
||||
if (!col.path.length) return `${name} on a rep:self column does not apply. A subject URI is not a number or date.`;
|
||||
const last = col.path[col.path.length - 1];
|
||||
const shapeUri = shapeAfter(profile, startShapeUri, col.path);
|
||||
if (!shapeUri) continue; // pathsResolve already reported the real error
|
||||
const f = fieldsOf(profile, shapeUri)
|
||||
.find(x => x.path === last.predicate && !!x.inverse === !!last.inverse);
|
||||
if (!f) continue;
|
||||
const kind = f.datatype === XSD_DATETIME || f.datatype === XSD_DATE ? 'date'
|
||||
: NUMERIC_DATATYPES.has(f.datatype) ? 'number'
|
||||
: null;
|
||||
if (!kind) {
|
||||
return `"${f.name || last.predicate}" is text, so ${name} does not apply. Use rep:row, or leave it out to join the values.`;
|
||||
// no end field: pathsResolve already reported the real error
|
||||
const f = endFields(profile, startShapeUri, col.path)
|
||||
.find(x => x.datatype !== XSD_DATETIME && x.datatype !== XSD_DATE && !NUMERIC_DATATYPES.has(x.datatype));
|
||||
if (f) {
|
||||
return `"${f.name || last.predicate}" is text, so ${name} does not apply. Use rep:row (same as leaving rep:collect out: every value gets its own row).`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
@@ -147,20 +144,23 @@ function columnPure(spec) {
|
||||
for (const col of spec.columns) {
|
||||
if (Object.keys(col.constraints || {}).length) {
|
||||
const which = Object.keys(col.constraints)[0];
|
||||
return `a column carries ${which}. Constraints belong in sh:property.`;
|
||||
return `a column carries ${which}. Constraints belong in sh:property, or in rep:where for a condition on a step of this column.`;
|
||||
}
|
||||
if (col.nodeKind && col.nodeKind !== SH + 'IRI') {
|
||||
return `column "${col.label}" has sh:nodeKind <${col.nodeKind}>. A column takes only sh:nodeKind sh:IRI, to show the URI of the node its path ends on.`;
|
||||
}
|
||||
if (col.collect && ![SH + 'groupConcat', SH + 'min', SH + 'max', REP + 'row'].includes(col.collect)) {
|
||||
return `column "${col.label}" has an unknown rep:collect <${col.collect}>. Use sh:groupConcat (default), rep:row, sh:min or sh:max.`;
|
||||
return `column "${col.label}" has an unknown rep:collect <${col.collect}>. Use sh:groupConcat (joins the values), rep:row or nothing (each value gets its own row), sh:min or sh:max.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// No label inside a filter
|
||||
// No label inside a filter or rep:where
|
||||
function filterLabel(spec) {
|
||||
for (const filter of spec.filters) {
|
||||
if ((filter.constraints || {}).label) {
|
||||
return 'a filter carries rdfs:label. Move it to rep:columns to show it.';
|
||||
for (const { what, cond } of conditions(spec)) {
|
||||
if ((cond.constraints || {}).label) {
|
||||
return `${what} carries rdfs:label. Move it to rep:columns to show it.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
@@ -171,11 +171,11 @@ function filterLabel(spec) {
|
||||
function filterConstraint(spec) {
|
||||
const MEANINGFUL = ['minCount', 'maxCount', 'hasValue', 'in', 'minInclusive',
|
||||
'maxInclusive', 'minExclusive', 'maxExclusive', 'pattern', 'flags', 'anyOf'];
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
const c = filter.constraints || {};
|
||||
for (const { what, cond } of conditions(spec)) {
|
||||
const c = cond.constraints || {};
|
||||
if (!MEANINGFUL.some(k => c[k] !== undefined && c[k] !== null)) {
|
||||
const p = filter.path.map(h => h.predicate).join(' → ');
|
||||
return `filter ${i + 1} (path ${p}) has no condition. Add one: sh:hasValue for an exact value, sh:in for a list, rep:anyOf for words, or remove the filter.`;
|
||||
const p = cond.path.map(h => h.predicate).join(' → ');
|
||||
return `${what} (path ${p}) has no condition. Add one: sh:hasValue for an exact value, sh:in for a list, rep:anyOf for words, or remove the filter.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
@@ -183,12 +183,12 @@ function filterConstraint(spec) {
|
||||
|
||||
// sh:maxCount above 0 needs counting
|
||||
function maxCountZero(spec) {
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
const c = filter.constraints;
|
||||
if (c.maxCount !== null && c.maxCount !== undefined && c.maxCount > 0) {
|
||||
for (const { cond } of conditions(spec)) {
|
||||
const c = cond.constraints;
|
||||
if (c.maxCount > 0) {
|
||||
return `sh:maxCount ${c.maxCount} needs counting, which this service cannot do. Use sh:maxCount 0 for "has none", or drop it.`;
|
||||
}
|
||||
if (c.minCount !== null && c.minCount !== undefined && c.minCount > 2) {
|
||||
if (c.minCount > 2) {
|
||||
return `sh:minCount ${c.minCount} needs counting. Use sh:minCount 1 (at least one) or sh:minCount 2 (two distinct values); anything higher is beyond this service.`;
|
||||
}
|
||||
}
|
||||
@@ -197,8 +197,8 @@ function maxCountZero(spec) {
|
||||
|
||||
// rep:anyOf size limits
|
||||
function anyOfLimits(spec) {
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
const c = filter.constraints;
|
||||
for (const { cond } of conditions(spec)) {
|
||||
const c = cond.constraints;
|
||||
if (!c.anyOf) continue;
|
||||
if (c.anyOf.length > ANYOF_MAX_TERMS) {
|
||||
return `rep:anyOf has ${c.anyOf.length} terms. The limit is ${ANYOF_MAX_TERMS}. Narrow the words, or split the filter into two sh:property blocks.`;
|
||||
@@ -217,8 +217,8 @@ function anyOfLimits(spec) {
|
||||
|
||||
// sh:in must not mix URIs and literals
|
||||
function inHomogeneous(spec) {
|
||||
for (const filter of spec.filters) {
|
||||
const c = filter.constraints;
|
||||
for (const { cond } of conditions(spec)) {
|
||||
const c = cond.constraints;
|
||||
if (!c.in) continue;
|
||||
const uris = c.in.filter(t => t.type === 'uri').length;
|
||||
if (uris > 0 && uris < c.in.length) {
|
||||
@@ -231,8 +231,8 @@ function inHomogeneous(spec) {
|
||||
// Typed literals must parse
|
||||
function literalsParse(spec) {
|
||||
const constraintTerms = [];
|
||||
for (const filter of spec.filters) {
|
||||
const c = filter.constraints;
|
||||
for (const { cond } of conditions(spec)) {
|
||||
const c = cond.constraints;
|
||||
for (const key of ['hasValue', 'minInclusive', 'maxInclusive', 'minExclusive', 'maxExclusive']) {
|
||||
if (c[key]) constraintTerms.push(c[key]);
|
||||
}
|
||||
@@ -250,86 +250,177 @@ function literalsParse(spec) {
|
||||
return null;
|
||||
}
|
||||
|
||||
// rep:where: walks from the row through the profile, shares at least one
|
||||
// step with its filter or column, and holds no rep:where of its own.
|
||||
function whereConditions(spec, profile) {
|
||||
const startShapeUri = startShape(profile, spec).uri;
|
||||
for (const { what, cond, host } of conditions(spec)) {
|
||||
if (!host) continue;
|
||||
if (cond.where.length) {
|
||||
return `${what} holds a rep:where of its own. Put every rep:where directly on the filter or column.`;
|
||||
}
|
||||
const err = walk(profile, startShapeUri, cond.path, what, false);
|
||||
if (err) return err;
|
||||
const k = sharedSteps(host.path, cond.path);
|
||||
if (!k) {
|
||||
return `${what} shares no step with ${host.what}. Start its path with the same steps as ${host.what}; a condition on the row itself goes in sh:property.`;
|
||||
}
|
||||
const c = cond.constraints;
|
||||
if (k === cond.path.length && (c.maxCount === 0 || c.minCount === 2)) {
|
||||
return `${what} ends on a step of ${host.what}, so ${c.maxCount === 0 ? 'sh:maxCount 0' : 'sh:minCount 2'} has nothing to count. Add the step that must be missing or repeated.`;
|
||||
}
|
||||
}
|
||||
for (const [i, filter] of spec.filters.entries()) {
|
||||
if (filter.where.length && filter.constraints.minCount === 2) {
|
||||
return `filter ${i + 1} combines sh:minCount 2 with rep:where. Use one of the two.`;
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// A date or number condition compares values. On a link (an entity or a
|
||||
// code) it compares a URI and never matches, so its path must end on a field
|
||||
// with a datatype.
|
||||
function rangeOnValue(spec, profile) {
|
||||
const startShapeUri = startShape(profile, spec).uri;
|
||||
for (const { what, cond } of conditions(spec)) {
|
||||
const key = ['minInclusive', 'maxInclusive', 'minExclusive', 'maxExclusive']
|
||||
.find(k => cond.constraints[k]);
|
||||
if (!key || !cond.path.length) continue;
|
||||
// pathsResolve reports a missing step
|
||||
const f = endFields(profile, startShapeUri, cond.path).find(x => !x.datatype);
|
||||
if (!f) continue;
|
||||
const target = f.nodes.length
|
||||
? f.nodes.map(uri => shapeName(profile, uri)).join(' or ')
|
||||
: `the code "${f.name}"`;
|
||||
const steps = [...new Set(f.nodes.flatMap(uri => fieldsOf(profile, uri))
|
||||
.filter(x => x.datatype === XSD_DATE || x.datatype === XSD_DATETIME || NUMERIC_DATATYPES.has(x.datatype))
|
||||
.map(x => `${x.name} (${x.path})`))];
|
||||
const add = steps.length ? `Add a step: ${steps.join(' or ')}.` : 'End the path on a date or number field.';
|
||||
return `${what} ends on ${target}, not on a value, so sh:${key} never matches. ${add}`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
// Where each rep:where applies, in the profile's words, so the proposal can
|
||||
// say it. Profile only, no database.
|
||||
export function whereNotes(spec, profile) {
|
||||
const startShapeUri = startShape(profile, spec).uri;
|
||||
return conditions(spec).filter(x => x.host).map(({ what, cond, host }) => {
|
||||
const k = sharedSteps(host.path, cond.path);
|
||||
return `${what} applies at ${nodeName(profile, startShapeUri, host.path.slice(0, k))}, after step ${k} of ${host.what}.`;
|
||||
});
|
||||
}
|
||||
|
||||
// The entity a path lands on, or the value field when it ends on one. A
|
||||
// step that leads to several entities names them all.
|
||||
function nodeName(profile, startShapeUri, hops) {
|
||||
if (!hops.length) return shapeName(profile, startShapeUri);
|
||||
const ends = endFields(profile, startShapeUri, hops);
|
||||
if (!ends.length) return `step ${hops.length}`;
|
||||
const value = ends.find(f => !f.nodes.length);
|
||||
if (value) return `the value "${value.name}"`;
|
||||
return [...new Set(ends.flatMap(f => f.nodes))].map(uri => shapeName(profile, uri)).join(' or ');
|
||||
}
|
||||
|
||||
function shapeName(profile, shapeUri) {
|
||||
return `"${profile.shapes.find(s => s.uri === shapeUri)?.label || shapeUri}"`;
|
||||
}
|
||||
|
||||
function parsesAsDate(value, withTime) {
|
||||
const d = new Date(value);
|
||||
if (Number.isNaN(d.getTime())) return false;
|
||||
return withTime ? value.includes('T') : /^\d{4}-\d{2}-\d{2}$/.test(value);
|
||||
}
|
||||
|
||||
// In this order: startEntity comes first, the checks after it rely on a
|
||||
// start shape.
|
||||
const CHECKS = [
|
||||
['no entity or unknown class', startEntity],
|
||||
['no columns', columnsPresent],
|
||||
['column without rdfs:label', columnLabel],
|
||||
['two columns with one label', columnLabelsUnique],
|
||||
['column carries constraints', columnPure],
|
||||
['filter carries rdfs:label', filterLabel],
|
||||
['filter without a condition', filterConstraint],
|
||||
['sh:maxCount above 0', maxCountZero],
|
||||
['rep:anyOf too big', anyOfLimits],
|
||||
['sh:in mixes URIs and literals', inHomogeneous],
|
||||
['typed literal does not parse', literalsParse],
|
||||
['two rep:row columns', oneRowColumn],
|
||||
['path does not resolve', pathsResolve],
|
||||
['sh:min/sh:max on a non-number', collectMinMaxTyped],
|
||||
startEntity,
|
||||
columnsPresent,
|
||||
columnLabel,
|
||||
columnLabelsUnique,
|
||||
columnPure,
|
||||
filterLabel,
|
||||
filterConstraint,
|
||||
maxCountZero,
|
||||
anyOfLimits,
|
||||
inHomogeneous,
|
||||
literalsParse,
|
||||
oneRowColumn,
|
||||
pathsResolve,
|
||||
whereConditions,
|
||||
rangeOnValue,
|
||||
collectMinMaxTyped,
|
||||
];
|
||||
|
||||
export function checkSpec(spec, profile, maxPathDepth, knownProfiles = null) {
|
||||
const profileErr = profileNamed(spec, knownProfiles);
|
||||
if (profileErr) return [profileErr];
|
||||
for (const [name, check] of CHECKS) {
|
||||
const err = check(spec, profile, maxPathDepth);
|
||||
if (err) return [err];
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
// The shape a path lands in: follow sh:node through every hop but the
|
||||
// last. walk() has already validated the path, so a gap here means the walk
|
||||
// reported the real error and this can bail.
|
||||
function shapeAfter(profile, startShapeUri, hops) {
|
||||
let shapeUri = startShapeUri;
|
||||
for (let i = 0; i < hops.length - 1; i++) {
|
||||
const fields = fieldsOf(profile, shapeUri);
|
||||
const f = fields.find(x => x.path === hops[i].predicate && !!x.inverse === !!hops[i].inverse);
|
||||
if (!f || !f.node) return null;
|
||||
shapeUri = f.node;
|
||||
}
|
||||
return shapeUri;
|
||||
}
|
||||
|
||||
function walk(profile, startShapeUri, hops, maxPathDepth, what, isColumn) {
|
||||
if (hops.length > maxPathDepth) {
|
||||
return `${what} walks ${hops.length} hops. The limit is ${maxPathDepth}.`;
|
||||
}
|
||||
if (!hops.length) return null;
|
||||
let shapeUri = startShapeUri;
|
||||
for (let i = 0; i < hops.length; i++) {
|
||||
const hop = hops[i];
|
||||
const fields = fieldsOf(profile, shapeUri);
|
||||
const f = fields.find(x => x.path === hop.predicate && !!x.inverse === !!hop.inverse);
|
||||
if (!f) {
|
||||
const have = fields.map(x => (x.inverse ? `geen ${x.name} (${x.path}, omgekeerd)` : `${x.name} (${x.path})`)).join(', ');
|
||||
const shapeLabel = profile.shapes.find(s => s.uri === shapeUri)?.label || shapeUri;
|
||||
return `at "${shapeLabel}" there is no "${hop.predicate}". It has: ${have || '(nothing)'}.`;
|
||||
}
|
||||
if (i < hops.length - 1) {
|
||||
if (!f.node) {
|
||||
return `${what} hop ${i + 1} <${hop.predicate}> holds a value, not a link. You cannot walk past it.`;
|
||||
}
|
||||
shapeUri = f.node;
|
||||
} else if (isColumn) {
|
||||
if (!f.datatype && !f.class) {
|
||||
const valueFields = fieldsOf(profile, f.node)
|
||||
.filter(x => x.datatype || x.class)
|
||||
.map(x => `${x.name} (${x.path})`);
|
||||
return `column "${colLabel(what)}" ends on a link. Add a hop: ${valueFields.join(' or ')}.`;
|
||||
}
|
||||
}
|
||||
// checkSpec(spec, profiles) → the first error, or null.
|
||||
export function checkSpec(spec, profiles) {
|
||||
const error = profileError(profiles, spec.profileUri);
|
||||
if (error) return error;
|
||||
const profile = profiles.get(spec.profileUri);
|
||||
for (const check of CHECKS) {
|
||||
const err = check(spec, profile);
|
||||
if (err) return err;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function colLabel(what) {
|
||||
const m = what.match(/"([^"]+)"/);
|
||||
return m ? m[1] : what;
|
||||
// The fields a hop takes from any of the shapes a path has reached. A field
|
||||
// may link to several entities (sh:or of sh:node: an eenheid's political and
|
||||
// leidinggevende organen): the walk follows them all and the next steps
|
||||
// decide. The query follows predicates only, so this is what the data does.
|
||||
function hopFields(profile, shapeUris, hop) {
|
||||
return shapeUris.flatMap(uri => fieldsOf(profile, uri)
|
||||
.filter(f => f.path === hop.predicate && !!f.inverse === !!hop.inverse));
|
||||
}
|
||||
|
||||
// The fields a path can end on: follow sh:node through every hop but the
|
||||
// last. walk() has already validated the path, so none here means the walk
|
||||
// reported the real error and the caller can bail.
|
||||
function endFields(profile, startShapeUri, hops) {
|
||||
let shapeUris = [startShapeUri];
|
||||
for (const hop of hops.slice(0, -1)) {
|
||||
shapeUris = [...new Set(hopFields(profile, shapeUris, hop).flatMap(f => f.nodes))];
|
||||
}
|
||||
return hopFields(profile, shapeUris, hops[hops.length - 1]);
|
||||
}
|
||||
|
||||
function walk(profile, startShapeUri, hops, what, isColumn, wantsIri = false) {
|
||||
if (hops.length > MAX_PATH_DEPTH) {
|
||||
return `${what} walks ${hops.length} hops. The limit is ${MAX_PATH_DEPTH}.`;
|
||||
}
|
||||
if (!hops.length) return null;
|
||||
let shapeUris = [startShapeUri];
|
||||
for (let i = 0; i < hops.length; i++) {
|
||||
const hop = hops[i];
|
||||
const found = hopFields(profile, shapeUris, hop);
|
||||
if (!found.length) {
|
||||
const have = [...new Set(shapeUris.flatMap(uri => fieldsOf(profile, uri))
|
||||
.map(x => (x.inverse ? `${x.name} (${x.path}, inverse)` : `${x.name} (${x.path})`)))].join(', ');
|
||||
const shapeLabel = shapeUris.map(uri => shapeName(profile, uri)).join(' or ');
|
||||
return `at ${shapeLabel} there is no "${hop.predicate}". It has: ${have || '(nothing)'}.`;
|
||||
}
|
||||
if (i < hops.length - 1) {
|
||||
const links = found.filter(f => f.nodes.length);
|
||||
if (!links.length) {
|
||||
return `${what} hop ${i + 1} <${hop.predicate}> holds a value, not a link. You cannot walk past it.`;
|
||||
}
|
||||
shapeUris = [...new Set(links.flatMap(f => f.nodes))];
|
||||
} else if (isColumn) {
|
||||
// every field the path can end on must suit the column
|
||||
for (const f of found) {
|
||||
if (wantsIri && f.datatype) {
|
||||
return `${what} has sh:nodeKind sh:IRI but ends on the value "${f.name}", which is no URI. Drop sh:nodeKind.`;
|
||||
}
|
||||
if (!wantsIri && !f.datatype && !f.class) {
|
||||
const valueFields = [...new Set(f.nodes.flatMap(uri => fieldsOf(profile, uri))
|
||||
.filter(x => x.datatype || x.class)
|
||||
.map(x => `${x.name} (${x.path})`))];
|
||||
return `${what} ends on a link. Add a hop: ${valueFields.join(' or ')}. Only if the user asked for the URI of that node itself, add sh:nodeKind sh:IRI to the column instead.`;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
@@ -1,26 +1,25 @@
|
||||
import { sparqlEscapeUri } from '../db.js';
|
||||
import { query, sparqlEscapeUri } from 'mu';
|
||||
import { hopsPattern, wherePatterns } from './seed.js';
|
||||
|
||||
// Fetch column values for known subjects. Columns are grouped by their
|
||||
// shared non-leaf hops; two columns share a query only when every hop
|
||||
// before the leaf is identical, and the leaf is read the same way.
|
||||
// rep:self columns (empty path) are filled by assemble.js.
|
||||
// rep:self columns (empty path) are filled by assemble.js. Every query
|
||||
// returns the nodes on the path too, so assemble.js can pair columns by the
|
||||
// nodes they share.
|
||||
|
||||
const SUBJECT_CHUNK_SIZE = Math.max(1, Number(process.env.SUBJECT_CHUNK_SIZE) || 100);
|
||||
|
||||
export async function fetchColumns(sessionQuery, subjects, spec, values = new Map(), onProgress = null) {
|
||||
// fetchColumns(subjects, spec) → subject → column index → [chain], a chain
|
||||
// being the terms on the path, one per step, the value last.
|
||||
export async function fetchColumns(subjects, spec) {
|
||||
const values = new Map();
|
||||
const groups = groupColumns(spec.columns);
|
||||
// The subject list is complete and the chunk size is fixed, so the number
|
||||
// of batch queries is known before the first one flies. onProgress fires
|
||||
// after every answered batch with { done, total }.
|
||||
const total = groups.length * Math.ceil(subjects.length / SUBJECT_CHUNK_SIZE);
|
||||
let done = 0;
|
||||
for (const group of groups) {
|
||||
for (let i = 0; i < subjects.length; i += SUBJECT_CHUNK_SIZE) {
|
||||
const chunk = subjects.slice(i, i + SUBJECT_CHUNK_SIZE);
|
||||
const result = await sessionQuery(groupQuery(group, chunk));
|
||||
const result = await query(groupQuery(group, chunk));
|
||||
collect(result, group, values);
|
||||
done += 1;
|
||||
onProgress?.({ done, total });
|
||||
}
|
||||
}
|
||||
return values;
|
||||
@@ -28,14 +27,17 @@ export async function fetchColumns(sessionQuery, subjects, spec, values = new Ma
|
||||
|
||||
// Group columns by their shared prefix (every hop before the leaf) and the
|
||||
// direction of the leaf. A column with an empty path (rep:self) joins no group.
|
||||
// A column with rep:where only shares a group with the same conditions.
|
||||
export function groupColumns(columns) {
|
||||
const groups = new Map();
|
||||
columns.forEach((col, index) => {
|
||||
if (!col.path.length) return; // rep:self
|
||||
const leaf = col.path[col.path.length - 1];
|
||||
const key = col.path.map(h => `${h.inverse ? '^' : ''}${h.predicate}`).join(' > ');
|
||||
const where = col.where || [];
|
||||
const key = col.path.map(h => `${h.inverse ? '^' : ''}${h.predicate}`).join(' > ')
|
||||
+ (where.length ? ` | ${JSON.stringify(where)}` : '');
|
||||
if (!groups.has(key)) {
|
||||
groups.set(key, { hops: col.path, columns: [] });
|
||||
groups.set(key, { hops: col.path, where, columns: [] });
|
||||
}
|
||||
groups.get(key).columns.push({ index, leaf: leaf.predicate });
|
||||
});
|
||||
@@ -44,29 +46,25 @@ export function groupColumns(columns) {
|
||||
|
||||
export function groupQuery(group, chunk) {
|
||||
const subjects = chunk.map(sparqlEscapeUri).join(' ');
|
||||
const lines = [];
|
||||
let prev = '?s';
|
||||
// every hop except the last gets its own variable; the leaf is ?v
|
||||
group.hops.slice(0, -1).forEach((hop, i) => {
|
||||
const v = `?x${i}`;
|
||||
if (hop.inverse) lines.push(`${v} ${sparqlEscapeUri(hop.predicate)} ${prev} .`);
|
||||
else lines.push(`${prev} ${sparqlEscapeUri(hop.predicate)} ${v} .`);
|
||||
prev = v;
|
||||
});
|
||||
const leaf = group.hops[group.hops.length - 1];
|
||||
if (leaf.inverse) lines.push(`?v ${sparqlEscapeUri(leaf.predicate)} ${prev} .`);
|
||||
else lines.push(`${prev} ${sparqlEscapeUri(leaf.predicate)} ?v .`);
|
||||
return `SELECT ?s ?v WHERE {\n VALUES ?s { ${subjects} }\n ${lines.join('\n ')}\n}`;
|
||||
const lines = [hopsPattern(group.hops, 'x', '?s', '?v')];
|
||||
// rep:where hangs on the column's own nodes: ?x_<k> after k hops, ?v at
|
||||
// the end.
|
||||
const nodeVar = (k) => (k === group.hops.length ? '?v' : `?x_${k}`);
|
||||
lines.push(...wherePatterns(group.where, group.hops, nodeVar, 'c'));
|
||||
const vars = ['?s', ...group.hops.slice(1).map((hop, i) => `?x_${i + 1}`), '?v'];
|
||||
return `SELECT DISTINCT ${vars.join(' ')} WHERE {\n VALUES ?s { ${subjects} }\n ${lines.join('\n ')}\n}`;
|
||||
}
|
||||
|
||||
function collect(result, group, values) {
|
||||
for (const b of result.results.bindings) {
|
||||
const chain = [...group.hops.slice(1).map((hop, i) => b[`x_${i + 1}`]), b.v];
|
||||
let perColumn = values.get(b.s.value);
|
||||
if (!perColumn) { perColumn = new Map(); values.set(b.s.value, perColumn); }
|
||||
for (const col of group.columns) {
|
||||
let list = perColumn.get(col.index);
|
||||
if (!list) { list = []; perColumn.set(col.index, list); }
|
||||
list.push(b.v);
|
||||
list.push(chain);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,35 +0,0 @@
|
||||
import { mkdirSync, writeFileSync } from 'fs';
|
||||
|
||||
// Rows to a CSV file. CSV_SEPARATOR is ';' by default. A cell is quoted
|
||||
// only when it holds the separator, a quote or a newline; a quote inside a
|
||||
// cell is doubled, per RFC 4180.
|
||||
|
||||
const CSV_SEPARATOR = process.env.CSV_SEPARATOR || ';';
|
||||
const SHARE_DIR = process.env.SHARE_DIR || '/share';
|
||||
|
||||
export function rowsToCsv(rows) {
|
||||
return rows.map(row => row.map(escapeCell).join(CSV_SEPARATOR)).join('\n');
|
||||
}
|
||||
|
||||
function escapeCell(cell) {
|
||||
const s = String(cell ?? '');
|
||||
return s.includes(CSV_SEPARATOR) || s.includes('"') || /[\n\r]/.test(s)
|
||||
? `"${s.replace(/"/g, '""')}"`
|
||||
: s;
|
||||
}
|
||||
|
||||
export function writeCsv(name, csv) {
|
||||
mkdirSync(SHARE_DIR, { recursive: true });
|
||||
const file = `${SHARE_DIR}/${name}`;
|
||||
writeFileSync(file, csv, 'utf8');
|
||||
return file;
|
||||
}
|
||||
|
||||
// The spec of the run, next to its CSV (same base name, .ttl), so a report
|
||||
// can be debugged afterwards. The store keeps only the pointer.
|
||||
export function writeSpec(csvName, spec) {
|
||||
mkdirSync(SHARE_DIR, { recursive: true });
|
||||
const name = csvName.replace(/\.csv$/, '') + '.ttl';
|
||||
writeFileSync(`${SHARE_DIR}/${name}`, spec, 'utf8');
|
||||
return `share://${name}`;
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
import N3 from 'n3';
|
||||
import { readFileSync } from 'fs';
|
||||
import { readFileSync, readdirSync } from 'fs';
|
||||
import { parseFilter, readRdfList } from './spec.js';
|
||||
|
||||
// Parse a profile (a SHACL shapes graph) and answer questions about it.
|
||||
|
||||
@@ -8,13 +9,11 @@ const RDF = 'http://www.w3.org/1999/02/22-rdf-syntax-ns#';
|
||||
const OWL_ONTOLOGY = 'http://www.w3.org/2002/07/owl#Ontology';
|
||||
const REP = 'http://mu.semte.ch/vocabularies/reporting/';
|
||||
|
||||
export async function loadProfiles(dir) {
|
||||
const fs = await import('fs');
|
||||
const path = await import('path');
|
||||
const files = fs.readdirSync(dir).filter(f => f.endsWith('.ttl')).sort();
|
||||
export function loadProfiles(dir) {
|
||||
const files = readdirSync(dir).filter(f => f.endsWith('.ttl')).sort();
|
||||
const profiles = new Map();
|
||||
for (const file of files) {
|
||||
const profile = await loadProfile(path.join(dir, file));
|
||||
const profile = loadProfile(`${dir}/${file}`);
|
||||
profiles.set(profile.uri, profile);
|
||||
console.log(`[profile] "${profile.title}" (${file}): ${profile.shapes.length} entities, ` +
|
||||
`${profile.shapes.reduce((n, s) => n + s.fields.length, 0)} fields`);
|
||||
@@ -23,29 +22,19 @@ export async function loadProfiles(dir) {
|
||||
return profiles;
|
||||
}
|
||||
|
||||
export async function loadProfile(file) {
|
||||
const parser = new N3.Parser();
|
||||
const store = new N3.Store();
|
||||
await new Promise((resolve, reject) => {
|
||||
parser.parse(readFileSync(file, 'utf8'), (err, quad) => {
|
||||
if (err) return reject(new Error(`${file}: ${err.message}`));
|
||||
if (quad) store.addQuad(quad); else resolve();
|
||||
});
|
||||
});
|
||||
function loadProfile(file) {
|
||||
let store;
|
||||
try {
|
||||
store = new N3.Store(new N3.Parser().parse(readFileSync(file, 'utf8')));
|
||||
} catch (e) {
|
||||
throw new Error(`${file}: ${e.message}`);
|
||||
}
|
||||
|
||||
const ontology = store.getQuads(null, RDF + 'type', OWL_ONTOLOGY)[0];
|
||||
if (!ontology) throw new Error(`${file}: no owl:Ontology. A profile file must name itself.`);
|
||||
const uri = ontology.subject.value;
|
||||
const title = store.getQuads(ontology.subject, 'http://purl.org/dc/terms/title', null)[0]?.object.value || uri;
|
||||
|
||||
const prefixQuads = store.getQuads(null, SH + 'declare', null);
|
||||
const prefixes = {};
|
||||
for (const q of prefixQuads) {
|
||||
const p = store.getQuads(q.object, SH + 'prefix', null)[0]?.object.value;
|
||||
const ns = store.getQuads(q.object, SH + 'namespace', null)[0]?.object.value;
|
||||
if (p && ns) prefixes[p] = ns;
|
||||
}
|
||||
|
||||
const shapes = store.getQuads(null, RDF + 'type', SH + 'NodeShape').map(q => q.subject);
|
||||
if (!shapes.length) throw new Error(`${file}: no sh:NodeShape. A profile without entities is useless.`);
|
||||
|
||||
@@ -53,66 +42,72 @@ export async function loadProfile(file) {
|
||||
const targetClass = store.getQuads(shape, SH + 'targetClass', null)[0]?.object.value || null;
|
||||
const label = store.getQuads(shape, 'http://www.w3.org/2000/01/rdf-schema#label', null)[0]?.object.value || null;
|
||||
const fields = store.getQuads(shape, SH + 'property', null).map(q => parseField(store, q.object));
|
||||
const discriminator = parseDiscriminator(store, shape);
|
||||
return { uri: shape.value, targetClass, label, fields, discriminator };
|
||||
const discriminators = parseDiscriminators(store, shape);
|
||||
return { uri: shape.value, targetClass, label, fields, discriminators };
|
||||
});
|
||||
|
||||
return { uri, title, prefixes, shapes: shapeList, store };
|
||||
return { uri, title, shapes: shapeList };
|
||||
}
|
||||
|
||||
// A field's sh:path: one predicate, or [ sh:inversePath <predicate> ].
|
||||
function readPath(store, node) {
|
||||
const pathNode = store.getQuads(node, SH + 'path', null)[0]?.object;
|
||||
if (!pathNode) return { path: null, inverse: false };
|
||||
if (pathNode.termType !== 'BlankNode') return { path: pathNode.value, inverse: false };
|
||||
const inv = store.getQuads(pathNode, SH + 'inversePath', null)[0]?.object;
|
||||
return inv ? { path: inv.value, inverse: true } : { path: null, inverse: false };
|
||||
}
|
||||
|
||||
function parseField(store, p) {
|
||||
let path = null, inverse = false;
|
||||
const pathQuad = store.getQuads(p, SH + 'path', null)[0]?.object;
|
||||
if (pathQuad) {
|
||||
if (pathQuad.termType === 'BlankNode') {
|
||||
const inv = store.getQuads(pathQuad, SH + 'inversePath', null)[0]?.object;
|
||||
if (inv) { path = inv.value; inverse = true; }
|
||||
} else {
|
||||
path = pathQuad.value;
|
||||
}
|
||||
}
|
||||
const one = (pred) => store.getQuads(p, pred, null)[0]?.object || null;
|
||||
const one = (pred) => store.getQuads(p, pred, null)[0]?.object?.value || null;
|
||||
return {
|
||||
path, inverse,
|
||||
name: one(SH + 'name')?.value || null,
|
||||
description: one(SH + 'description')?.value || null,
|
||||
datatype: one(SH + 'datatype')?.value || null,
|
||||
class: one(SH + 'class')?.value || null,
|
||||
node: one(SH + 'node')?.value || null,
|
||||
nodeKind: one(SH + 'nodeKind')?.value || null,
|
||||
maxCount: one(SH + 'maxCount')?.value || null,
|
||||
...readPath(store, p),
|
||||
name: one(SH + 'name'),
|
||||
datatype: one(SH + 'datatype'),
|
||||
class: one(SH + 'class'),
|
||||
nodes: linkedShapes(store, p, one(SH + 'name')),
|
||||
};
|
||||
}
|
||||
|
||||
// rep:discriminator tells shapes that share a targetClass apart. sh:minCount 1
|
||||
// becomes a plain triple in the seed, sh:maxCount 0 a FILTER NOT EXISTS.
|
||||
function parseDiscriminator(store, shape) {
|
||||
const node = store.getQuads(shape, REP + 'discriminator', null)[0]?.object;
|
||||
if (!node) return null;
|
||||
let path = null, inverse = false;
|
||||
const pathQuad = store.getQuads(node, SH + 'path', null)[0]?.object;
|
||||
if (pathQuad) {
|
||||
if (pathQuad.termType === 'BlankNode') {
|
||||
const inv = store.getQuads(pathQuad, SH + 'inversePath', null)[0]?.object;
|
||||
if (inv) { path = inv.value; inverse = true; }
|
||||
} else {
|
||||
path = pathQuad.value;
|
||||
}
|
||||
}
|
||||
const one = (pred) => store.getQuads(node, pred, null)[0]?.object?.value || null;
|
||||
const minCount = one(SH + 'minCount');
|
||||
const maxCount = one(SH + 'maxCount');
|
||||
if (!path || (minCount === null && maxCount === null)) return null;
|
||||
return {
|
||||
path, inverse,
|
||||
minCount: minCount !== null ? Number(minCount) : null,
|
||||
maxCount: maxCount !== null ? Number(maxCount) : null,
|
||||
};
|
||||
// The entities a field links to: its sh:node, or the sh:node of each item of
|
||||
// its sh:or (SHACL for "each value is one of these").
|
||||
function linkedShapes(store, p, name) {
|
||||
const node = store.getQuads(p, SH + 'node', null)[0]?.object;
|
||||
if (node) return [node.value];
|
||||
const or = store.getQuads(p, SH + 'or', null)[0]?.object;
|
||||
if (!or) return [];
|
||||
return readRdfList(store, or).map(item => {
|
||||
const n = store.getQuads(item, SH + 'node', null)[0]?.object;
|
||||
if (!n) throw new Error(`field "${name}": every sh:or item needs a sh:node.`);
|
||||
return n.value;
|
||||
});
|
||||
}
|
||||
|
||||
export function entityForClass(profile, cls) {
|
||||
const shape = profile.shapes.find(s => s.targetClass === cls);
|
||||
return shape ? shape.uri : null;
|
||||
// rep:discriminator tells shapes that share a targetClass apart. A shape may
|
||||
// carry several; a subject meets them all.
|
||||
function parseDiscriminators(store, shape) {
|
||||
return store.getQuads(shape, REP + 'discriminator', null)
|
||||
.map(q => parseDiscriminator(store, q.object, shape.value));
|
||||
}
|
||||
|
||||
// One discriminator: a condition written like a spec filter (sh:path plus
|
||||
// sh:minCount, sh:maxCount, sh:in, sh:hasValue, ...), or sh:and / sh:or (a
|
||||
// list) or sh:not (one) over discriminators, as in SHACL.
|
||||
function parseDiscriminator(store, node, shapeUri) {
|
||||
const one = (pred) => store.getQuads(node, SH + pred, null)[0]?.object;
|
||||
const list = (pred) => {
|
||||
const items = readRdfList(store, one(pred)).map(x => parseDiscriminator(store, x, shapeUri));
|
||||
if (!items.length) throw new Error(`rep:discriminator on <${shapeUri}> has an empty sh:${pred}.`);
|
||||
return items;
|
||||
};
|
||||
if (one('and')) return { and: list('and') };
|
||||
if (one('or')) return { or: list('or') };
|
||||
if (one('not')) return { not: parseDiscriminator(store, one('not'), shapeUri) };
|
||||
const d = parseFilter(store, node);
|
||||
if (!d.path.length || !Object.keys(d.constraints).length) {
|
||||
throw new Error(`rep:discriminator on <${shapeUri}> needs a sh:path with a condition, or sh:and, sh:or or sh:not.`);
|
||||
}
|
||||
return d;
|
||||
}
|
||||
|
||||
// The shape a spec starts from. rep:entity wins; a targetClass only resolves
|
||||
@@ -131,11 +126,3 @@ export function shape(profile, shapeUri) {
|
||||
export function fieldsOf(profile, shapeUri) {
|
||||
return shape(profile, shapeUri)?.fields || [];
|
||||
}
|
||||
|
||||
export function field(profile, shapeUri, predicate, inverse) {
|
||||
return fieldsOf(profile, shapeUri).find(f => f.path === predicate && !!f.inverse === !!inverse) || null;
|
||||
}
|
||||
|
||||
export function prefixes(profile) {
|
||||
return profile.prefixes;
|
||||
}
|
||||
@@ -1,61 +0,0 @@
|
||||
import { randomUUID } from 'crypto';
|
||||
import { statSync } from 'fs';
|
||||
import { sessionUpdate, sparqlEscapeUri, sparqlEscapeString, sparqlEscapeInt,
|
||||
sparqlEscapeDateTime } from '../db.js';
|
||||
|
||||
const REPORT_CLASS = process.env.REPORT_CLASS || 'http://lblod.data.gift/vocabularies/reporting/Report';
|
||||
|
||||
// Register the CSV: a logical file (http://data.lblod.info/files/<uuid>) plus
|
||||
// a physical share:// file pointing back with nie:dataSource. The logical
|
||||
// URI is what you link to.
|
||||
export async function registerFile(sessionUpdateFn, fileName, filePath) {
|
||||
const size = statSync(filePath).size;
|
||||
const uuid = randomUUID();
|
||||
const uuid2 = randomUUID();
|
||||
const now = new Date();
|
||||
const logicalUri = `http://data.lblod.info/files/${uuid}`;
|
||||
const physicalUri = `share://${fileName}`;
|
||||
|
||||
await (sessionUpdateFn || sessionUpdate)(`
|
||||
INSERT DATA {
|
||||
${sparqlEscapeUri(logicalUri)} a ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject')} ;
|
||||
${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(uuid)} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName')} ${sparqlEscapeString(fileName)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/format')} ${sparqlEscapeString('text/csv')} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize')} ${sparqlEscapeInt(size)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/created')} ${sparqlEscapeDateTime(now)} .
|
||||
${sparqlEscapeUri(physicalUri)} a ${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject')} ;
|
||||
${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(uuid2)} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileName')} ${sparqlEscapeString(fileName)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/format')} ${sparqlEscapeString('text/csv')} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileSize')} ${sparqlEscapeInt(size)} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#fileCreated')} ${sparqlEscapeDateTime(now)} ;
|
||||
${sparqlEscapeUri('http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource')} ${sparqlEscapeUri(logicalUri)} .
|
||||
}`);
|
||||
|
||||
return logicalUri;
|
||||
}
|
||||
|
||||
// One report pointing at the logical file URI.
|
||||
export async function registerReport(sessionUpdateFn, title, fileUri, extra = {}) {
|
||||
const uuid = randomUUID();
|
||||
const now = new Date();
|
||||
const reportUri = `http://data.lblod.info/id/reports/${uuid}`;
|
||||
|
||||
const extraTriples = [];
|
||||
if (extra.question) extraTriples.push(`${sparqlEscapeUri(reportUri)} ${sparqlEscapeUri('http://mu.semte.ch/vocabularies/ext/question')} ${sparqlEscapeString(extra.question)} .`);
|
||||
if (extra.specFile) extraTriples.push(`${sparqlEscapeUri(reportUri)} ${sparqlEscapeUri('http://mu.semte.ch/vocabularies/ext/specFile')} ${sparqlEscapeString(extra.specFile)} .`);
|
||||
if (extra.creator) extraTriples.push(`${sparqlEscapeUri(reportUri)} ${sparqlEscapeUri('http://purl.org/dc/terms/creator')} ${sparqlEscapeUri(extra.creator)} .`);
|
||||
|
||||
await (sessionUpdateFn || sessionUpdate)(`
|
||||
INSERT DATA {
|
||||
${sparqlEscapeUri(reportUri)} a ${sparqlEscapeUri(REPORT_CLASS)} ;
|
||||
${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(uuid)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/title')} ${sparqlEscapeString(title)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/created')} ${sparqlEscapeDateTime(now)} ;
|
||||
${sparqlEscapeUri('http://www.w3.org/ns/prov#generated')} ${sparqlEscapeUri(fileUri)} .
|
||||
${extraTriples.join('\n ')}
|
||||
}`);
|
||||
|
||||
return reportUri;
|
||||
}
|
||||
@@ -1,42 +1,49 @@
|
||||
import { update, uuid, sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime } from 'mu';
|
||||
import { seed } from './seed.js';
|
||||
import { fetchColumns } from './columns.js';
|
||||
import { assemble } from './assemble.js';
|
||||
import { rowsToCsv, writeCsv, writeSpec } from './csv.js';
|
||||
import { registerFile, registerReport } from './report.js';
|
||||
import { startShape } from './profile.js';
|
||||
|
||||
// Ties the steps together: seed → columns → assemble → CSV → register.
|
||||
// Returns { reportUri, fileUri, rowCount }. onProgress fires on every
|
||||
// answered query with what the phase knows — { subjects } per seed page,
|
||||
// { done, total } per column batch — so a waiter can tell a live run from
|
||||
// a stalled one.
|
||||
const REPORT_CLASS = process.env.REPORT_CLASS || 'http://lblod.data.gift/vocabularies/reporting/Report';
|
||||
const CSV_SEPARATOR = process.env.CSV_SEPARATOR || ';';
|
||||
|
||||
export async function run(query, update, parsed, profile, title, extra = {}, onProgress = null) {
|
||||
const shape = startShape(profile, parsed);
|
||||
const subjects = await seed(query, parsed, shape,
|
||||
onProgress ? (n) => onProgress({ subjects: n }) : null);
|
||||
|
||||
const values = new Map();
|
||||
await fetchColumns(query, subjects, parsed, values, null, onProgress);
|
||||
|
||||
const rows = assemble(subjects, values, parsed);
|
||||
const fileName = `${slug(title)}.csv`;
|
||||
const csv = rowsToCsv(rows);
|
||||
const filePath = writeCsv(fileName, csv);
|
||||
const specFile = extra.spec && writeSpec(fileName, extra.spec);
|
||||
|
||||
const fileUri = await registerFile(update, fileName, filePath);
|
||||
const reportUri = await registerReport(update, title, fileUri,
|
||||
specFile ? { ...extra, specFile } : extra);
|
||||
|
||||
return { reportUri, fileUri, rowCount: rows.length - 1, filePath };
|
||||
// run(spec, profile) → the CSV text. seed → columns → assemble → CSV.
|
||||
export async function run(spec, profile) {
|
||||
const subjects = await seed(spec, startShape(profile, spec));
|
||||
const values = await fetchColumns(subjects, spec);
|
||||
const rows = assemble(subjects, values, spec);
|
||||
return rows.map(row => row.map(csvCell).join(CSV_SEPARATOR)).join('\n');
|
||||
}
|
||||
|
||||
// A cell is quoted only when it holds the separator, a quote or a newline;
|
||||
// a quote inside a cell is doubled, per RFC 4180.
|
||||
function csvCell(cell) {
|
||||
const s = String(cell ?? '');
|
||||
return s.includes(CSV_SEPARATOR) || s.includes('"') || /[\n\r]/.test(s)
|
||||
? `"${s.replace(/"/g, '""')}"`
|
||||
: s;
|
||||
}
|
||||
|
||||
// One report pointing at the logical file URI of its CSV.
|
||||
export async function registerReport(title, fileUri) {
|
||||
const id = uuid();
|
||||
const reportUri = `http://data.lblod.info/id/reports/${id}`;
|
||||
|
||||
await update(`
|
||||
INSERT DATA {
|
||||
${sparqlEscapeUri(reportUri)} a ${sparqlEscapeUri(REPORT_CLASS)} ;
|
||||
${sparqlEscapeUri('http://mu.semte.ch/vocabularies/core/uuid')} ${sparqlEscapeString(id)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/title')} ${sparqlEscapeString(title)} ;
|
||||
${sparqlEscapeUri('http://purl.org/dc/terms/created')} ${sparqlEscapeDateTime(new Date())} ;
|
||||
${sparqlEscapeUri('http://www.w3.org/ns/prov#generated')} ${sparqlEscapeUri(fileUri)} .
|
||||
}`);
|
||||
|
||||
return reportUri;
|
||||
}
|
||||
|
||||
// The file name must be known before the run starts, so the chat can show a
|
||||
// pending card named after the report.
|
||||
export function slug(title) {
|
||||
return (title || 'report').toLowerCase()
|
||||
.replace(/[^a-z0-9]+/g, '-')
|
||||
.replace(/^-+|-+$/g, '')
|
||||
.slice(0, 80) || 'report';
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,20 +1,17 @@
|
||||
import { sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime, sparqlEscapeDate } from '../db.js';
|
||||
import { query, sparqlEscapeUri, sparqlEscapeString, sparqlEscapeDateTime, sparqlEscapeDate } from 'mu';
|
||||
import { sharedSteps } from './spec.js';
|
||||
|
||||
// Subject selection. One paged query per page; stops at ROW_LIMIT.
|
||||
// onPage(n) fires after every answered page with the subject count so far,
|
||||
// so a caller waiting on this run can tell a live one from a stalled one.
|
||||
const ROW_LIMIT = Number(process.env.ROW_LIMIT || 200000);
|
||||
const PAGE_SIZE = Number(process.env.SEED_PAGE_SIZE || 5000);
|
||||
|
||||
export async function seed(sessionQuery, spec, shape = null, onPage = null) {
|
||||
export async function seed(spec, shape) {
|
||||
const subjects = [];
|
||||
let offset = 0;
|
||||
for (;;) {
|
||||
const query = seedPageQuery(spec, PAGE_SIZE, offset, shape);
|
||||
const result = await sessionQuery(query);
|
||||
const result = await query(seedPageQuery(spec, PAGE_SIZE, offset, shape));
|
||||
const rows = result.results.bindings.map(b => b.s.value);
|
||||
subjects.push(...rows);
|
||||
onPage?.(subjects.length);
|
||||
if (subjects.length > ROW_LIMIT) {
|
||||
throw new Error(`the question matches more than ${ROW_LIMIT} subjects. Add a filter, for example a date range or a bestuurseenheid.`);
|
||||
}
|
||||
@@ -28,10 +25,11 @@ export function seedPageQuery(spec, limit, offset, shape = null) {
|
||||
const parts = [];
|
||||
const targetClass = spec.targetClass || shape?.targetClass;
|
||||
parts.push(`?s a ${sparqlEscapeUri(targetClass)} .`);
|
||||
if (shape?.discriminator) parts.push(discriminatorPattern(shape.discriminator));
|
||||
(shape?.discriminators || []).forEach((d, di) => {
|
||||
parts.push(discriminatorPattern(d, `disc${di}`));
|
||||
});
|
||||
spec.filters.forEach((filter, fi) => {
|
||||
parts.push(hopsPattern(filter.path, `f${fi}`, '?s'));
|
||||
parts.push(constraintsPattern(filter.constraints, `f${fi}`));
|
||||
parts.push(filterPattern(filter, `f${fi}`));
|
||||
});
|
||||
return `SELECT DISTINCT ?s WHERE {
|
||||
${parts.join('\n ')}
|
||||
@@ -40,25 +38,32 @@ LIMIT ${limit} OFFSET ${offset}`;
|
||||
}
|
||||
|
||||
// A shape that shares its class with another says here what sets its
|
||||
// subjects apart: sh:minCount 1 is a plain triple, sh:maxCount 0 a NOT EXISTS.
|
||||
function discriminatorPattern(d) {
|
||||
const p = sparqlEscapeUri(d.path);
|
||||
const triple = d.inverse ? `?disc ${p} ?s .` : `?s ${p} ?disc .`;
|
||||
if (d.minCount !== null && d.minCount >= 1) return triple;
|
||||
if (d.maxCount === 0) return `FILTER NOT EXISTS { ${triple} }`;
|
||||
return null;
|
||||
// subjects apart. A plain discriminator is a filter; sh:and, sh:or and sh:not
|
||||
// become one FILTER over EXISTS tests, which see ?s from outside.
|
||||
function discriminatorPattern(d, varPrefix) {
|
||||
if (!d.and && !d.or && !d.not) return filterPattern(d, varPrefix);
|
||||
return `FILTER(${discriminatorTest(d, varPrefix)})`;
|
||||
}
|
||||
|
||||
function discriminatorTest(d, varPrefix) {
|
||||
if (d.and) return `(${d.and.map((x, i) => discriminatorTest(x, `${varPrefix}_${i}`)).join(' && ')})`;
|
||||
if (d.or) return `(${d.or.map((x, i) => discriminatorTest(x, `${varPrefix}_${i}`)).join(' || ')})`;
|
||||
if (d.not) return `!(${discriminatorTest(d.not, `${varPrefix}_n`)})`;
|
||||
const { maxCount, ...test } = d.constraints;
|
||||
if (maxCount === 0) return `NOT EXISTS { ${filterPattern({ ...d, constraints: test }, varPrefix)} }`;
|
||||
return `EXISTS { ${filterPattern(d, varPrefix)} }`;
|
||||
}
|
||||
|
||||
// Writes out every hop as its own triple pattern.
|
||||
// Forward: ?s <p1> ?f0_1 . ?f0_1 <p2> ?f0_v
|
||||
// Inverse: ?s ^<p1> → ?f0_1 <p1> ?s
|
||||
export function hopsPattern(hops, varPrefix, startVar) {
|
||||
export function hopsPattern(hops, varPrefix, startVar, lastVar = `?${varPrefix}_v`) {
|
||||
let prev = startVar;
|
||||
const patterns = [];
|
||||
for (let i = 0; i < hops.length; i++) {
|
||||
const hop = hops[i];
|
||||
const isLast = i === hops.length - 1;
|
||||
const v = isLast ? `?${varPrefix}_v` : `?${varPrefix}_${i + 1}`;
|
||||
const v = isLast ? lastVar : `?${varPrefix}_${i + 1}`;
|
||||
if (hop.inverse) {
|
||||
patterns.push(`${v} ${sparqlEscapeUri(hop.predicate)} ${prev} .`);
|
||||
} else {
|
||||
@@ -69,9 +74,40 @@ export function hopsPattern(hops, varPrefix, startVar) {
|
||||
return patterns.join('\n ');
|
||||
}
|
||||
|
||||
export function constraintsPattern(c, varPrefix) {
|
||||
const parts = [];
|
||||
const v = `?${varPrefix}_v`;
|
||||
// One filter: its condition, plus its rep:where conditions on the nodes it
|
||||
// walks through. With sh:maxCount 0 the whole of it becomes "none such";
|
||||
// with a value test too (sh:in, sh:hasValue, ...), "none with such a value".
|
||||
function filterPattern(filter, varPrefix) {
|
||||
const { maxCount, ...test } = filter.constraints;
|
||||
if (maxCount === 0) {
|
||||
return `FILTER NOT EXISTS {\n ${filterPattern({ ...filter, constraints: test }, varPrefix)}\n }`;
|
||||
}
|
||||
const nodeVar = (k) => (k === filter.path.length ? `?${varPrefix}_v` : `?${varPrefix}_${k}`);
|
||||
const wheres = wherePatterns(filter.where, filter.path, nodeVar, varPrefix);
|
||||
return [conditionPattern(filter.constraints, varPrefix, filter.path, '?s'), ...wheres].join('\n ');
|
||||
}
|
||||
|
||||
// The rep:where conditions of a filter or column. Each starts at the node
|
||||
// after the steps it shares with its host; nodeVar(k) names that node in the
|
||||
// host's own query.
|
||||
export function wherePatterns(where, hostHops, nodeVar, varPrefix) {
|
||||
return (where || []).map((w, wi) => {
|
||||
const k = sharedSteps(hostHops, w.path);
|
||||
return conditionPattern(w.constraints, `${varPrefix}_w${wi}`, w.path.slice(k), nodeVar(k));
|
||||
});
|
||||
}
|
||||
|
||||
// A condition: the hops from startVar and the test on where they end. A
|
||||
// filter starts at ?s; a rep:where starts at a node of its host and, with no
|
||||
// hops left, tests that node itself. sh:maxCount 0 means "none such", so its
|
||||
// hops and any value test only live inside the NOT EXISTS.
|
||||
export function conditionPattern(c, varPrefix, hops, startVar) {
|
||||
if (c.maxCount === 0) {
|
||||
const { maxCount, ...test } = c;
|
||||
return `FILTER NOT EXISTS { ${conditionPattern(test, varPrefix + '_nex', hops, startVar)} }`;
|
||||
}
|
||||
const v = hops.length ? `?${varPrefix}_v` : startVar;
|
||||
const parts = hops.length ? [hopsPattern(hops, varPrefix, startVar)] : [];
|
||||
|
||||
if (c.hasValue) parts.push(`FILTER(${v} = ${escapeTerm(c.hasValue)})`);
|
||||
if (c.in) parts.push(`FILTER(${v} IN (${c.in.map(escapeTerm).join(', ')}))`);
|
||||
@@ -89,33 +125,14 @@ export function constraintsPattern(c, varPrefix) {
|
||||
parts.push(`FILTER(REGEX(str(${v}), ${sparqlEscapeString(alternation)}${flags}))`);
|
||||
}
|
||||
if (c.minCount === 2) {
|
||||
// two distinct leaf values on the same path
|
||||
// two distinct leaf values: the same path again, to a second leaf
|
||||
const w = `?${varPrefix}_w`;
|
||||
parts.push(hopsPattern2(c, varPrefix, w));
|
||||
parts.push(hopsPattern(hops, `${varPrefix}_2`, startVar, w));
|
||||
parts.push(`FILTER(${v} != ${w})`);
|
||||
}
|
||||
if (c.maxCount === 0) {
|
||||
parts.push(`FILTER NOT EXISTS { ${hopsPattern(c.path, varPrefix + '_nex', '?s')} }`);
|
||||
}
|
||||
return parts.join('\n ');
|
||||
}
|
||||
|
||||
function hopsPattern2(filter, varPrefix, wVar) {
|
||||
// repeat the path to a second leaf variable
|
||||
const hops = filter.path.map(h => ({ ...h }));
|
||||
let prev = '?s';
|
||||
const patterns = [];
|
||||
for (let i = 0; i < hops.length; i++) {
|
||||
const hop = hops[i];
|
||||
const isLast = i === hops.length - 1;
|
||||
const v = isLast ? wVar : `?${varPrefix}_2_${i + 1}`;
|
||||
if (hop.inverse) patterns.push(`${v} ${sparqlEscapeUri(hop.predicate)} ${prev} .`);
|
||||
else patterns.push(`${prev} ${sparqlEscapeUri(hop.predicate)} ${v} .`);
|
||||
prev = v;
|
||||
}
|
||||
return patterns.join('\n ');
|
||||
}
|
||||
|
||||
function escapeTerm(t) {
|
||||
if (t.type === 'uri') return sparqlEscapeUri(t.value);
|
||||
if (t.datatype === 'http://www.w3.org/2001/XMLSchema#dateTime')
|
||||
|
||||
@@ -13,8 +13,13 @@ export function parseSpec(turtle) {
|
||||
const quads = parser.parse(turtle);
|
||||
const store = new N3.Store(quads);
|
||||
|
||||
const specNode = store.getQuads(null, RDF + 'type', REP + 'ReportSpec')[0]?.subject;
|
||||
if (!specNode) throw new Error('spec has no rep:ReportSpec');
|
||||
const specNodes = store.getQuads(null, RDF + 'type', REP + 'ReportSpec').map(q => q.subject);
|
||||
if (!specNodes.length) throw new Error('spec has no rep:ReportSpec');
|
||||
// One spec is one report. A second one would pass unchecked and never run.
|
||||
if (specNodes.length > 1) {
|
||||
throw new Error(`one spec per answer. This Turtle has ${specNodes.length} rep:ReportSpec. Keep one, and offer the other as the next report.`);
|
||||
}
|
||||
const specNode = specNodes[0];
|
||||
|
||||
const one = (s, pred) => store.getQuads(s, pred, null)[0]?.object || null;
|
||||
|
||||
@@ -23,35 +28,39 @@ export function parseSpec(turtle) {
|
||||
const profileUri = one(specNode, REP + 'profile')?.value || null;
|
||||
const title = one(specNode, 'http://purl.org/dc/terms/title')?.value || null;
|
||||
|
||||
const filters = store.getQuads(specNode, SH + 'property', null).map(q =>
|
||||
parseFilter(store, q.object, { withLabel: true }));
|
||||
const filters = store.getQuads(specNode, SH + 'property', null).map(q => parseFilter(store, q.object));
|
||||
|
||||
const columnsQuad = one(specNode, REP + 'columns');
|
||||
const columns = columnsQuad ? parseColumnList(store, columnsQuad) : [];
|
||||
const columns = columnsQuad ? readRdfList(store, columnsQuad).map(col => parseColumn(store, col)) : [];
|
||||
|
||||
return { uri: specNode.value, title, profileUri, targetClass, entity, filters, columns };
|
||||
}
|
||||
|
||||
function parseFilter(store, p, { withLabel = false } = {}) {
|
||||
export function parseFilter(store, p) {
|
||||
let path = [];
|
||||
const pathQuad = store.getQuads(p, SH + 'path', null)[0]?.object;
|
||||
if (pathQuad) path = parsePath(store, pathQuad);
|
||||
return {
|
||||
path,
|
||||
constraints: readConstraints(store, p, { withLabel }),
|
||||
constraints: readConstraints(store, p, { withLabel: true }),
|
||||
where: readWhere(store, p),
|
||||
};
|
||||
}
|
||||
|
||||
function parseColumnList(store, listNode) {
|
||||
const columns = [];
|
||||
let cur = listNode;
|
||||
while (cur && cur.value !== RDF + 'nil') {
|
||||
const item = store.getQuads(cur, RDF + 'first', null)[0]?.object;
|
||||
if (item) columns.push(parseColumn(store, item));
|
||||
cur = store.getQuads(cur, RDF + 'rest', null)[0]?.object;
|
||||
if (!cur) throw new Error('rep:columns list is broken (no rdf:rest terminator)');
|
||||
}
|
||||
return columns;
|
||||
// rep:where: conditions on the nodes a filter or column walks through. Each
|
||||
// is written like a filter, from the row; check.js refuses one nested in
|
||||
// another.
|
||||
function readWhere(store, node) {
|
||||
return store.getQuads(node, REP + 'where', null).map(q => parseFilter(store, q.object));
|
||||
}
|
||||
|
||||
// The number of steps two paths have in common from the row. A rep:where
|
||||
// shares those nodes with its filter or column.
|
||||
export function sharedSteps(a, b) {
|
||||
let k = 0;
|
||||
while (k < a.length && k < b.length
|
||||
&& a[k].predicate === b[k].predicate && !!a[k].inverse === !!b[k].inverse) k++;
|
||||
return k;
|
||||
}
|
||||
|
||||
function parseColumn(store, col) {
|
||||
@@ -69,7 +78,10 @@ function parseColumn(store, col) {
|
||||
label: store.getQuads(col, RDFS_LABEL, null)[0]?.object.value || null,
|
||||
collect: store.getQuads(col, REP + 'collect', null)[0]?.object.value || null,
|
||||
separator: store.getQuads(col, SH + 'separator', null)[0]?.object.value || null,
|
||||
// sh:nodeKind sh:IRI: the column shows the URI of the node it ends on
|
||||
nodeKind: store.getQuads(col, SH + 'nodeKind', null)[0]?.object.value || null,
|
||||
constraints: readConstraints(store, col), // kept so check.js can refuse it
|
||||
where: readWhere(store, col),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -86,21 +98,12 @@ function parsePath(store, pathQuad) {
|
||||
return [{ predicate: pathQuad.value, inverse: false }];
|
||||
}
|
||||
// RDF list of predicates
|
||||
const hops = [];
|
||||
let cur = pathQuad;
|
||||
while (cur && cur.value !== RDF + 'nil') {
|
||||
const item = store.getQuads(cur, RDF + 'first', null)[0]?.object;
|
||||
if (!item) throw new Error('sh:path list is broken');
|
||||
if (item.termType === 'BlankNode') {
|
||||
const inv = store.getQuads(item, SH + 'inversePath', null)[0]?.object;
|
||||
if (!inv) throw new Error('sh:path list items must be URIs or [ sh:inversePath <predicate> ]');
|
||||
hops.push({ predicate: inv.value, inverse: true });
|
||||
} else {
|
||||
hops.push({ predicate: item.value, inverse: false });
|
||||
}
|
||||
cur = store.getQuads(cur, RDF + 'rest', null)[0]?.object;
|
||||
}
|
||||
return hops;
|
||||
return readRdfList(store, pathQuad).map(item => {
|
||||
if (item.termType !== 'BlankNode') return { predicate: item.value, inverse: false };
|
||||
const inv = store.getQuads(item, SH + 'inversePath', null)[0]?.object;
|
||||
if (!inv) throw new Error('sh:path list items must be URIs or [ sh:inversePath <predicate> ]');
|
||||
return { predicate: inv.value, inverse: true };
|
||||
});
|
||||
}
|
||||
|
||||
function readConstraints(store, p, { withLabel = false } = {}) {
|
||||
@@ -136,7 +139,7 @@ function readConstraints(store, p, { withLabel = false } = {}) {
|
||||
return constraints;
|
||||
}
|
||||
|
||||
function readRdfList(store, node) {
|
||||
export function readRdfList(store, node) {
|
||||
const items = [];
|
||||
let cur = node;
|
||||
while (cur && cur.value !== RDF + 'nil') {
|
||||
|
||||
Reference in New Issue
Block a user