Remove change events from ckb's and update their status (#535)

* Remove change events from ckb's and update their status

* move status migration to triggering-indexing

* remove unneeded code

* update changelog

CHANGELOG.md

@@ -1,5 +1,66 @@
 # Changelog
 ## Unreleased
+### Backend
+- Removal of change events of 8 ckb's + update of their status [OP-3562]
+### Deploy Notes
+```
+drc restart migrations-triggering-indexing; drc logs -ft --tail=200 migrations-triggering-indexing
+drc restart migrations; drc logs -ft --tail=200 migrations
+```
+## v1.31.0 (2025-03-27)
+### Frontend
+- Bump to version [v1.30.0](https://github.com/lblod/frontend-organization-portal/blob/v1.30.0/CHANGELOG.md#v1300-2025-03-27)
+### Backend
+- Datafix: move status for municipalities and provinces to the `shared` graph such that it can be read by worship users [OP-3469]
+- Bump `triplestore` and `publication-triplestore` to `v1.3.0-rc.1`. [OP-2492] [OP-3547]
+- Add missing docker compose keys. [DL-6490]
+- Reorganize delta consumers config to harmonize with the ecosystem
+- Set up the dashboard app [OP-3103]
+- Datafix: correct date of change event of kerkfabriek st petrus [OP-3555]
+- Update OCMWv to VVMW [OP-3565] and back. It's technically a nil-operation. But you should run the migrations nevertheless.
+### Deploy notes
+```
+drc restart migrations-triggering-indexing
+drc restart migrations; drc logs -ft --tail=200 migrations
+drc pull frontend; drc up -d frontend
+drc up -d mandatarissen-consumer leidinggevenden-consumer worship-services-main-info-consumer worship-services-private-info-consumer
+```
+#### For upgrade databases
+[This README](https://github.com/Riadabd/upgrade-virtuoso) provides the necessary steps for upgrading the database. **NOTE**: This will involve shutting down the app for small period of time (around 30 minutes).
+#### For adding the missing keys
+```
+drc up -d search elasticsearch
+```
+
+#### Dashboard
+
+The dashboard requires some server specific configuration before it can be used.
+
+##### 1. Dashboard domains
+To configure the domain name that's used by the dashboard frontend, you can add it to the `VIRTUAL_HOST` and `LETSENCRYPT_HOST` environment variables of the identifier service.
+- DEV domain: dashboard.dev.app-organisation-portal.contacthub-dev.s.redhost.be
+- QA domain: dashboard.qa.app-organisation-portal.contacthub-dev.s.redhost.be
+- PROD domain: dashboard.app-organisation-portal.organisaties.s.redpencil.io
+
+##### 2. Dashboard users
+Before we can generate a user we need to add an application salt to the server. Add a `MU_APPLICATION_SALT` environment variable to the `dashboard-login` service in the docker-compose.override.yml file.
+
+Once this is done you can generate a user by running `mu script project-scripts generate-dashboard-login` and following the prompts. A new local migration will be generated which can be run by restarting the migrations service and which will insert the new user into the database.
+
+Store the login credentials as a comment in the docker-compose.override.yml file. 
+
+##### 3. (Re)start the needed services
+`drc up -d frontend-dashboard dashboard-login identifier; drc restart dispatcher resource db`
+
+## v1.30.5 (2025-03-10)
+### Backend
+ - Added `http://www.w3.org/ns/prov#Location` to delta stream 'public' [DL-6496]
+### Deploy notes
+```
+drc restart delta-producer-publication-graph-maintainer
+drc exec delta-producer-background-jobs-initiator curl -X POST http://localhost/public/healing-jobs
+
+```
 
 ## v1.30.6 (2025-03-25)
 ### Backend
@@ -71,7 +132,6 @@ drc restart migrations
 - Remove several OCMW bcsd faciliteitgemeenten [OP-3535]
 - Completely remove AGB Stekene's data. [OP-3409]
 - Remove reasoner service, replaced by new consumer logic
-
 ### Deploy Notes
 - Before pulling the changes
   + Shut down the reasoner and thank it for its good service: `drc down reasoner`

README.md

@@ -185,3 +185,26 @@ To make sure the app can share data, producers need to be set up. There is an in
        )
      }
 ```
+
+## Configuring the dashboard
+### Accessing the dashboard from your local machine
+
+Since we use dispatcher v2, which dispatches on hostname, we'll have to update `/etc/hosts`.
+Add an entry similar to the following. Ensure the first part of the domain starts with `dashboard`.:
+
+```
+127.0.0.1 dashboard.localhost
+```
+
+### Creating new users
+For now, we use specific logins for the dashboard users. Each environment has its own passwords.
+
+To add a user, make sure to have [mu-cli](https://github.com/mu-semtech/mu-cli) installed first.
+Then in `docker-compose.override.yml`
+```
+  dashboard-login:
+    environment:
+      MU_APPLICATION_SALT: 'a_random_string_with_sufficient_entropy_hence_not_this_one'
+```
+You can now generate a user by running `mu script project-scripts generate-dashboard-login` and following the steps.
+Restart `migrations` and it should insert the new user into the database.

config/authorization/config.ex

@@ -107,6 +107,19 @@ defmodule Acl.UserGroups.Config do
     }"
   end
 
+  defp can_access_dashboard() do
+    %AccessByQuery{
+      vars: [],
+      query: "PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
+        PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+        SELECT DISTINCT ?account WHERE {
+          <SESSION_ID> <http://mu.semte.ch/vocabularies/session/account> ?account.
+          ?account <http://mu.semte.ch/vocabularies/ext/sessionRole> ?session_role.
+          FILTER( ?session_role = \"dashboard-user\" )
+        }"
+      }
+  end
+
   def user_groups do
     [
       %GroupSpec{
@@ -206,6 +219,46 @@ defmodule Acl.UserGroups.Config do
           }
         ]
       },
+      # // dashboard users
+      %GroupSpec{
+        name: "dashboard-users",
+        useage: [:read],
+        access: can_access_dashboard(),
+        graphs: [
+          %GraphSpec{
+            graph: "http://mu.semte.ch/graphs/reports",
+            constraint: %ResourceConstraint{
+              resource_types: [
+                "http://lblod.data.gift/vocabularies/reporting/Report",
+                "http://open-services.net/ns/core#Error",
+                "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#DataContainer",
+                "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject"
+              ]
+            }
+          },
+          %GraphSpec{
+            graph: "http://mu.semte.ch/graphs/system/jobs",
+            constraint: %ResourceConstraint{
+              resource_types: [
+                "http://vocab.deri.ie/cogs#Job",
+                "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#DataContainer",
+                "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject"
+              ]
+            }
+          },
+          # OP stores jobs in multiple graphs, so we configure both until they are merged into one graph
+          %GraphSpec{
+            graph: "http://mu.semte.ch/graphs/jobs",
+            constraint: %ResourceConstraint{
+              resource_types: [
+                "http://vocab.deri.ie/cogs#Job",
+                "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#DataContainer",
+                "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject"
+              ]
+            }
+          },
+        ]
+      },
 
       # // CLEANUP
       #

config/db/virtuoso-production.ini

@@ -231,10 +231,12 @@ DefaultHost  = localhost:8890
 ;DefaultGraph           = http://localhost:8890/dataspace
 ;ImmutableGraphs        = http://localhost:8890/dataspace
 ResultSetMaxRows           = 1000000 ; 1 million
+MaxConstructTriples        = 1000000 ; More info on this addition here: https://github.com/redpencilio/docker-virtuoso/pull/29
 ;MaxQueryCostEstimationTime = 4000	; in seconds
 MaxQueryExecutionTime      = 1200	; in seconds
 DefaultQuery               = select distinct ?Concept where {[] a ?Concept} LIMIT 100
 DeferInferenceRulesInit    = 0	; controls inference rules loading
+MaxMemInUse                = 0 ; limits the amount of memory for construct dict (0=unlimited). More info on this addition here: https://github.com/redpencilio/docker-virtuoso/pull/29
 ;PingService            = http://rpc.pingthesemanticweb.com/
 
 [Plugins]

config/db/virtuoso.ini

@@ -231,10 +231,12 @@ DefaultHost  = localhost:8890
 ;DefaultGraph           = http://localhost:8890/dataspace
 ;ImmutableGraphs        = http://localhost:8890/dataspace
 ResultSetMaxRows           = 1000000 ; 1 million
+MaxConstructTriples        = 1000000 ; More info on this addition here: https://github.com/redpencilio/docker-virtuoso/pull/29
 ;MaxQueryCostEstimationTime = 4000	; in seconds
 MaxQueryExecutionTime      = 1200	; in seconds
 DefaultQuery               = select distinct ?Concept where {[] a ?Concept} LIMIT 100
 DeferInferenceRulesInit    = 0	; controls inference rules loading
+MaxMemInUse                = 0 ; limits the amount of memory for construct dict (0=unlimited). More info on this addition here: https://github.com/redpencilio/docker-virtuoso/pull/29
 ;PingService            = http://rpc.pingthesemanticweb.com/
 
 [Plugins]

config/dispatcher/dispatcher.ex

@@ -240,6 +240,47 @@ defmodule Dispatcher do
     Proxy.forward(conn, path, "http://resource/kbo-organizations/")
   end
 
+  # Dashboard
+  match "/jobs/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://cache/jobs/")
+  end
+
+  match "/tasks/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://cache/tasks/")
+  end
+
+  match "/data-containers/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://cache/data-containers/")
+  end
+
+  match "/job-errors/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://cache/job-errors/")
+  end
+
+  match "/reports/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://resource/reports/")
+  end
+
+  match "/log-entries/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://resource/log-entries/")
+  end
+
+  match "/log-levels/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://resource/log-levels/")
+  end
+
+  match "/status-codes/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://resource/status-codes/")
+  end
+
+  match "/log-sources/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://resource/log-sources/")
+  end
+
+  match "/remote-data-objects/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://resource/remote-data-objects/")
+  end
+
   ###############
   # LOGIN
   ###############
@@ -264,6 +305,10 @@ defmodule Dispatcher do
     Proxy.forward(conn, path, "http://mocklogin/sessions/")
   end
 
+  match "/sessions/*path", %{ reverse_host: ["dashboard" | _rest] } do
+    Proxy.forward conn, path, "http://dashboard-login/sessions/"
+  end
+
   match "/sessions/*path", %{accept: [:any], layer: :api} do
     Proxy.forward(conn, path, "http://login/sessions/")
   end
@@ -428,6 +473,20 @@ defmodule Dispatcher do
   # frontend layer
   ###############################################################
 
+  get "/assets/*path",  %{ reverse_host: ["dashboard" | _rest], layer: :api }  do
+    forward conn, path, "http://frontend-dashboard/assets/"
+  end
+
+  get "/@appuniversum/*path", %{ reverse_host: ["dashboard" | _rest], layer: :api} do
+    forward conn, path, "http://frontend-dashboard/@appuniversum/"
+  end
+
+  match "/*_path", %{ reverse_host: ["dashboard" | _rest], layer: :api } do
+    # *_path allows a path to be supplied, but will not yield
+    # an error that we don't use the path variable.
+    forward conn, [], "http://frontend-dashboard/index.html"
+  end
+
   match "/assets/*path", %{layer: :api} do
     Proxy.forward(conn, path, "http://frontend/assets/")
   end

config/migrations-triggering-indexing/2025/20250321162045-change-ocmwv-to-vvmw.sparql

@@ -0,0 +1,18 @@
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    ?s <http://www.w3.org/2004/02/skos/core#prefLabel> ?label
+  }
+}
+INSERT {
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    ?s <http://www.w3.org/2004/02/skos/core#prefLabel> "VVMW".
+  }
+}
+WHERE {
+  VALUES ?s {
+    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/cc4e2d67-603b-4784-9b61-e50bac1ec089>
+  }
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    ?s <http://www.w3.org/2004/02/skos/core#prefLabel> ?label.
+  }
+}

config/migrations-triggering-indexing/2025/20250321171109-change-vvmw-to-ocmwv.sparql

@@ -0,0 +1,18 @@
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    ?s <http://www.w3.org/2004/02/skos/core#prefLabel> ?label
+  }
+}
+INSERT {
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    ?s <http://www.w3.org/2004/02/skos/core#prefLabel> "OCMW vereniging".
+  }
+}
+WHERE {
+  VALUES ?s {
+    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/cc4e2d67-603b-4784-9b61-e50bac1ec089>
+  }
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    ?s <http://www.w3.org/2004/02/skos/core#prefLabel> ?label.
+  }
+}

config/migrations-triggering-indexing/2025/20250327135401-update-ckb-statuses.sparql

@@ -0,0 +1,42 @@
+PREFIX regorg: <http://www.w3.org/ns/regorg#>
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    ?s regorg:orgStatus ?activeStatus.
+  }
+} INSERT {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    ?s regorg:orgStatus ?notActiveStatus.
+  }
+} WHERE {
+  VALUES ?s {
+    <http://data.lblod.info/id/centraleBesturenVanDeEredienst/ed0914f40dfed83ca44af845a2bf4d81>
+    <http://data.lblod.info/id/centraleBesturenVanDeEredienst/9e0e8f060ca954da6263243642433f44>
+    <http://data.lblod.info/id/centraleBesturenVanDeEredienst/b93ae92a8eb4912125094d98b8b4841d>
+    <http://data.lblod.info/id/centraleBesturenVanDeEredienst/988321712204bd0104fb62e6f5dbe104>
+    <http://data.lblod.info/id/centraleBesturenVanDeEredienst/70e058024a3cd58f76005d645b892e01>
+    <http://data.lblod.info/id/centraleBesturenVanDeEredienst/cf4add06db226861767559753fb50206>
+    <http://data.lblod.info/id/centraleBesturenVanDeEredienst/5442c1374ed159347bc0c64518a9b1a9>
+  }
+  BIND (<http://lblod.data.gift/concepts/d02c4e12bf88d2fdf5123b07f29c9311> AS ?notActiveStatus)
+  BIND (<http://lblod.data.gift/concepts/63cc561de9188d64ba5840a42ae8f0d6> AS ?activeStatus)
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    ?s regorg:orgStatus ?notActiveStatus.
+  }
+} INSERT {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    ?s regorg:orgStatus ?activeStatus.
+  }
+} WHERE {
+  VALUES ?s {
+    <http://data.lblod.info/id/centraleBesturenVanDeEredienst/e12c4f9f7167cd0074a97a2f658d1d51>
+
+  }
+  BIND (<http://lblod.data.gift/concepts/d02c4e12bf88d2fdf5123b07f29c9311> AS ?notActiveStatus)
+  BIND (<http://lblod.data.gift/concepts/63cc561de9188d64ba5840a42ae8f0d6> AS ?activeStatus)
+}

config/migrations/2025/20250220171921-move-status-for-municipalities-and-provinces-to-shared-graph.sparql

@@ -0,0 +1,28 @@
+PREFIX regorg: <http://www.w3.org/ns/regorg#>
+PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>
+PREFIX org: <http://www.w3.org/ns/org#>
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?organisation regorg:orgStatus ?status.
+  }
+} INSERT {
+  GRAPH <http://mu.semte.ch/graphs/shared> {
+    ?organisation regorg:orgStatus ?status.
+  }
+} WHERE {
+  GRAPH <http://mu.semte.ch/graphs/shared> {
+    ?organisation a besluit:Bestuurseenheid;
+                  org:classification ?class.
+    FILTER (?class IN
+            (
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001>, # Municipality
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000000> # Province
+             )
+            )
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?organisation regorg:orgStatus ?status.
+  }
+}

config/migrations/2025/20250313103300-create-log-levels.sparql

@@ -0,0 +1,33 @@
+INSERT DATA {
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    <http://data.lblod.info/id/log-levels/0986a9b6-b543-4629-8ef8-08928ae42430> a <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#Level> ;
+      <http://mu.semte.ch/vocabularies/core/uuid> "0986a9b6-b543-4629-8ef8-08928ae42430" ;
+      <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#priority> 1 ;
+      <http://www.w3.org/2004/02/skos/core#prefLabel> "TRACE" .
+
+    <http://data.lblod.info/id/log-levels/fe6473d5-ac83-450c-83f8-3546357c7117> a <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#Level> ;
+      <http://mu.semte.ch/vocabularies/core/uuid> "fe6473d5-ac83-450c-83f8-3546357c7117" ;
+      <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#priority> 2 ;
+      <http://www.w3.org/2004/02/skos/core#prefLabel> "DEBUG" .
+
+    <http://data.lblod.info/id/log-levels/6f92baad-b4a1-42ce-8bdf-e28218b27ff9> a <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#Level> ;
+      <http://mu.semte.ch/vocabularies/core/uuid> "6f92baad-b4a1-42ce-8bdf-e28218b27ff9" ;
+      <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#priority> 3 ;
+      <http://www.w3.org/2004/02/skos/core#prefLabel> "INFO" .
+
+    <http://data.lblod.info/id/log-levels/62d2516b-e6b8-4866-aad1-e8347f2cdc70> a <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#Level> ;
+      <http://mu.semte.ch/vocabularies/core/uuid> "62d2516b-e6b8-4866-aad1-e8347f2cdc70" ;
+      <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#priority> 4 ;
+      <http://www.w3.org/2004/02/skos/core#prefLabel> "WARN" .
+
+    <http://data.lblod.info/id/log-levels/3af9ebe1-e6a8-495c-a392-16ced1f38ef1> a <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#Level> ;
+      <http://mu.semte.ch/vocabularies/core/uuid> "3af9ebe1-e6a8-495c-a392-16ced1f38ef1" ;
+      <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#priority> 5 ;
+      <http://www.w3.org/2004/02/skos/core#prefLabel> "ERROR" .
+
+    <http://data.lblod.info/id/log-levels/c679f915-1fdd-4b50-938d-6b565aa31278> a <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#Level> ;
+      <http://mu.semte.ch/vocabularies/core/uuid> "c679f915-1fdd-4b50-938d-6b565aa31278" ;
+      <http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#priority> 6 ;
+      <http://www.w3.org/2004/02/skos/core#prefLabel> "FATAL" .
+   }
+}

config/migrations/2025/20250313103900-create-log-source.sparql

@@ -0,0 +1,7 @@
+INSERT DATA {
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    <http://data.lblod.info/id/log-sources/c7806563-25a4-46c8-9be2-a0cdf0db1f98> a <http://mu.semte.ch/vocabularies/ext/LogSource> ;
+      <http://mu.semte.ch/vocabularies/core/uuid> "c7806563-25a4-46c8-9be2-a0cdf0db1f98" ;
+      <http://www.w3.org/2004/02/skos/core#prefLabel> "Acmidm login service" .
+   }
+}

config/migrations/2025/20250321102359-update-date-change-event-kerkfabriek-st-petrus.sparql

@@ -0,0 +1,15 @@
+PREFIX dct: <http://purl.org/dc/terms/>
+
+DELETE {
+  GRAPH ?g {
+    <http://data.lblod.info/id/veranderingsgebeurtenissen/67C816180F1E719C0DBB7B6C> dct:date ?oldDate .
+  }
+} INSERT {
+  GRAPH ?g {
+    <http://data.lblod.info/id/veranderingsgebeurtenissen/67C816180F1E719C0DBB7B6C> dct:date "2025-03-05"^^xsd:date .
+  }
+} WHERE {
+  GRAPH ?g {
+    <http://data.lblod.info/id/veranderingsgebeurtenissen/67C816180F1E719C0DBB7B6C> dct:date ?oldDate .
+  }
+}

config/migrations/2025/20250327135321-delete-change-events.sparql

@@ -0,0 +1,53 @@
+PREFIX core: <http://mu.semte.ch/vocabularies/core/>
+PREFIX org: <http://www.w3.org/ns/org#>
+PREFIX lblodorg: <http://lblod.data.gift/vocabularies/organisatie/>
+PREFIX euro: <http://data.europa.eu/m8g/>
+PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    ?event ?ep ?eo .
+
+    ?result ?rp ?ro .
+
+    ?decision ?de ?do .
+  }
+}
+WHERE {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    VALUES ?changeEventUuid {
+      "675AFDCA29299CC832892989" # CKB Beveren
+
+      "675AC8C529299CC83289296D" # CKB De Pinte
+
+      "675AC99229299CC83289296F" # CKB Merelbeke
+
+      "675AB94329299CC83289295D" # CKB Galmaarden
+      "675AD32529299CC83289297B" # CKB Galmaarden
+
+      "675AB98C29299CC83289295F" # CKB Gooik
+      "675AD47629299CC83289297D" # CKB Gooik
+
+      "675AD50229299CC83289297F" # CKB Herne
+      "675AB9D229299CC832892961" # CKB Herne
+
+      "675ACA5F29299CC832892971" # CKB Tessenderlo
+
+      "6774F73E29299CC8328929D3" # CKB Ham
+    }
+
+    ?event a org:ChangeEvent ;
+           core:uuid ?changeEventUuid ;
+           lblodorg:veranderingsgebeurtenisResultaat ?result ;
+           ?ep ?eo .
+
+    ?result a lblodorg:VeranderingsgebeurtenisResultaat ;
+            ?rp ?ro .
+
+    OPTIONAL {
+      ?event euro:hasFormalFramework ?decision .
+      ?decision a besluit:Besluit ;
+                ?de ?do .
+    }
+  }
+}

config/publication-triplestore/virtuoso-production.ini

@@ -231,10 +231,12 @@ DefaultHost  = localhost:8890
 ;DefaultGraph           = http://localhost:8890/dataspace
 ;ImmutableGraphs        = http://localhost:8890/dataspace
 ResultSetMaxRows           = 1000000 ; 1 million
+MaxConstructTriples        = 1000000 ; More info on this addition here: https://github.com/redpencilio/docker-virtuoso/pull/29
 ;MaxQueryCostEstimationTime = 4000	; in seconds
 MaxQueryExecutionTime      = 1200	; in seconds
 DefaultQuery               = select distinct ?Concept where {[] a ?Concept} LIMIT 100
 DeferInferenceRulesInit    = 0	; controls inference rules loading
+MaxMemInUse                = 0 ; limits the amount of memory for construct dict (0=unlimited). More info on this addition here: https://github.com/redpencilio/docker-virtuoso/pull/29
 ;PingService            = http://rpc.pingthesemanticweb.com/
 
 [Plugins]

config/publication-triplestore/virtuoso.ini

@@ -231,10 +231,12 @@ DefaultHost  = localhost:8890
 ;DefaultGraph           = http://localhost:8890/dataspace
 ;ImmutableGraphs        = http://localhost:8890/dataspace
 ResultSetMaxRows           = 1000000 ; 1 million
+MaxConstructTriples        = 1000000 ; More info on this addition here: https://github.com/redpencilio/docker-virtuoso/pull/29
 ;MaxQueryCostEstimationTime = 4000	; in seconds
 MaxQueryExecutionTime      = 1200	; in seconds
 DefaultQuery               = select distinct ?Concept where {[] a ?Concept} LIMIT 100
 DeferInferenceRulesInit    = 0	; controls inference rules loading
+MaxMemInUse                = 0 ; limits the amount of memory for construct dict (0=unlimited). More info on this addition here: https://github.com/redpencilio/docker-virtuoso/pull/29
 ;PingService            = http://rpc.pingthesemanticweb.com/
 
 [Plugins]

config/resources/domain.lisp

@@ -17,6 +17,10 @@
 (read-domain-file "domain.json")
 (read-domain-file "auth.json")
 (read-domain-file "change-events.json")
-(read-domain-file "files-domain.lisp")
+(read-domain-file "file.json")
 (read-domain-file "dcat.json")
 (read-domain-file "privacy-centric-service.json")
+(read-domain-file "job.json")
+(read-domain-file "report.json")
+(read-domain-file "log.json")
+(read-domain-file "harvest.json")

config/resources/file.json

@@ -0,0 +1,107 @@
+{
+    "version": "0.1",
+    "prefixes": {
+        "adms": "http://www.w3.org/ns/adms#",
+        "dct": "http://purl.org/dc/terms/",
+        "dbpedia": "http://dbpedia.org/ontology/",
+        "nfo": "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#",
+        "nie": "http://www.semanticdesktop.org/ontologies/2007/01/19/nie#",
+        "rpioHttp": "http://redpencil.data.gift/vocabularies/http/",
+        "skos": "http://www.w3.org/2004/02/skos/core#",
+        "task": "http://redpencil.data.gift/vocabularies/tasks/"
+    },
+    "resources": {
+        "files": {
+            "name": "file",
+            "class": "nfo:FileDataObject",
+            "attributes": {
+                "filename": {
+                    "type": "string",
+                    "predicate": "nfo:fileName"
+                },
+                "format": {
+                    "type": "string",
+                    "predicate": "dct:format"
+                },
+                "size": {
+                    "type": "number",
+                    "predicate": "nfo:fileSize"
+                },
+                "extension": {
+                    "type": "string",
+                    "predicate": "dbpedia:fileExtension"
+                },
+                "created": {
+                    "type": "datetime",
+                    "predicate": "nfo:fileCreated"
+                }
+            },
+            "relationships": {
+                "download": {
+                    "predicate": "nie:dataSource",
+                    "target": "file",
+                    "cardinality": "one",
+                    "inverse": true
+                },
+                "data-container": {
+                    "predicate": "task:hasFile",
+                    "target": "data-container",
+                    "cardinality": "one",
+                    "inverse": true
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://data.lblod.info/files/"
+        },
+        "remote-data-objects": {
+            "name": "remote-data-object",
+            "class": "nfo:RemoteDataObject",
+            "attributes": {
+                "source": {
+                    "type": "url",
+                    "predicate": "nie:url"
+                },
+                "created": {
+                    "type": "datetime",
+                    "predicate": "dct:created"
+                },
+                "modified": {
+                    "type": "datetime",
+                    "predicate": "dct:modified"
+                },
+                "request-header": {
+                    "type": "url",
+                    "predicate": "rpioHttp:requestHeader"
+                },
+                "status": {
+                    "type": "url",
+                    "predicate": "adms:status"
+                },
+                "comment": {
+                    "type": "string",
+                    "predicate": "skos:comment"
+                },
+                "creator": {
+                    "type": "url",
+                    "predicate": "dct:creator"
+                }
+            },
+            "relationships": {
+                "file": {
+                    "predicate": "nie:dataSource",
+                    "target": "file",
+                    "cardinality": "one",
+                    "inverse": true
+                },
+                "harvesting-collection": {
+                    "predicate": "dct:hasPart",
+                    "target": "harvesting-collection",
+                    "cardinality": "one",
+                    "inverse": true
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://data.lblod.info/id/remote-data-objects/"
+        }
+    }
+}

config/resources/files-domain.lisp

@@ -1,13 +0,0 @@
-(define-resource file ()
-  :class (s-prefix "nfo:FileDataObject")
-  :properties `((:filename :string ,(s-prefix "nfo:fileName"))
-                (:format :string ,(s-prefix "dct:format"))
-                (:size :number ,(s-prefix "nfo:fileSize"))
-                (:extension :string ,(s-prefix "dbpedia:fileExtension"))
-                (:created :datetime ,(s-prefix "nfo:fileCreated")))
-  :has-one `((file :via ,(s-prefix "nie:dataSource")
-                   :inverse t
-                   :as "download"))
-  :resource-base (s-url "http://data.lblod.info/files/")
-  :features `(no-pagination-defaults include-uri)
-  :on-path "files")

config/resources/harvest.json

@@ -0,0 +1,40 @@
+{
+    "version": "0.1",
+    "prefixes": {
+        "adms": "http://www.w3.org/ns/adms#",
+        "dct": "http://purl.org/dc/terms/",
+        "hrvst": "http://lblod.data.gift/vocabularies/harvesting/",
+        "task": "http://redpencil.data.gift/vocabularies/tasks/"
+    },
+    "resources": {
+        "harvesting-collections": {
+            "name": "harvesting-collection",
+            "class": "hrvst:HarvestingCollection",
+            "attributes": {
+                "status": {
+                    "type": "url",
+                    "predicate": "adms:status"
+                },
+                "creator": {
+                    "type": "url",
+                    "predicate": "dct:creator"
+                }
+            },
+            "relationships": {
+                "remote-data-objects": {
+                    "predicate": "dct:hasPart",
+                    "target": "remote-data-object",
+                    "cardinality": "many"
+                },
+                "data-container": {
+                    "predicate": "task:hasHarvestingCollection",
+                    "target": "data-container",
+                    "cardinality": "many",
+                    "inverse": true
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://data.lblod.info/id/harvesting-collection/"
+        }
+    }
+}

config/resources/job.json

@@ -0,0 +1,152 @@
+{
+    "version": "0.1",
+    "prefixes": {
+        "adms": "http://www.w3.org/ns/adms#",
+        "cogs": "http://vocab.deri.ie/cogs#",
+        "dct": "http://purl.org/dc/terms/",
+        "oslc": "http://open-services.net/ns/core#",
+        "nfo": "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#",
+        "task": "http://redpencil.data.gift/vocabularies/tasks/"
+    },
+    "resources": {
+        "jobs": {
+            "name": "job",
+            "class": "cogs:Job",
+            "attributes": {
+                "created": {
+                    "type": "datetime",
+                    "predicate": "dct:created"
+                },
+                "modified": {
+                    "type": "datetime",
+                    "predicate": "dct:modified"
+                },
+                "creator": {
+                    "type": "url",
+                    "predicate": "dct:creator"
+                },
+                "status": {
+                    "type": "url",
+                    "predicate": "adms:status"
+                },
+                "operation": {
+                    "type": "url",
+                    "predicate": "task:operation"
+                }
+            },
+            "relationships": {
+                "error": {
+                    "predicate": "task:error",
+                    "target": "job-error",
+                    "cardinality": "one"
+                },
+                "tasks": {
+                    "predicate": "dct:isPartOf",
+                    "target": "group",
+                    "cardinality": "many",
+                    "inverse": true
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://redpencil.data.gift/id/job/"
+        },
+        "tasks": {
+            "name": "task",
+            "class": "task:Task",
+            "attributes": {
+                "created": {
+                    "type": "datetime",
+                    "predicate": "dct:created"
+                },
+                "modified": {
+                    "type": "datetime",
+                    "predicate": "dct:modified"
+                },
+                "status": {
+                    "type": "url",
+                    "predicate": "adms:status"
+                },
+                "operation": {
+                    "type": "url",
+                    "predicate": "task:operation"
+                },
+                "index": {
+                    "type": "string",
+                    "predicate": "task:index"
+                }
+            },
+            "relationships": {
+                "error": {
+                    "predicate": "task:error",
+                    "target": "job-error",
+                    "cardinality": "one"
+                },
+                "job": {
+                    "predicate": "dct:isPartOf",
+                    "target": "job",
+                    "cardinality": "one"
+                },
+                "parent-tasks": {
+                    "predicate": "cogs:dependsOn",
+                    "target": "task",
+                    "cardinality": "many"
+                },
+                "results-containers": {
+                    "predicate": "task:resultsContainer",
+                    "target": "data-container",
+                    "cardinality": "many"
+                },
+                "input-containers": {
+                    "predicate": "task:inputContainer",
+                    "target": "data-container",
+                    "cardinality": "many"
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://redpencil.data.gift/id/task/"
+        },
+        "job-errors": {
+            "name": "job-error",
+            "class": "oslc:Error",
+            "attributes": {
+                "message": {
+                    "type": "string",
+                    "predicate": "oslc:message"
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://redpencil.data.gift/id/jobs/error/"
+        },
+        "data-containers": {
+            "name": "data-container",
+            "class": "nfo:DataContainer",
+            "attributes": {
+                "has-graph": {
+                    "type": "url",
+                    "predicate": "task:hasGraph"
+                }
+            },
+            "relationships": {
+                "harvesting-collections": {
+                    "predicate": "task:hasHarvestingCollection",
+                    "target": "harvesting-collection",
+                    "cardinality": "many"
+                },
+                "result-from-tasks": {
+                    "predicate": "task:resultsContainer",
+                    "target": "task",
+                    "cardinality": "many",
+                    "inverse": true
+                },
+                "input-from-tasks": {
+                    "predicate": "task:inputContainer",
+                    "target": "task",
+                    "cardinality": "many",
+                    "inverse":true
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://redpencil.data.gift/id/dataContainers/"
+        }
+    }
+}

config/resources/log.json

@@ -0,0 +1,92 @@
+{
+    "version": "0.1",
+    "prefixes": {
+        "dct": "http://purl.org/dc/terms/",
+        "ext": "http://mu.semte.ch/vocabularies/ext/",
+        "rlog": "http://persistence.uni-leipzig.org/nlp2rdf/ontologies/rlog#",
+        "skos": "http://www.w3.org/2004/02/skos/core#"
+    },
+    "resources": {
+        "log-entries": {
+            "name": "log-entry",
+            "class": "rlog:Entry",
+            "attributes": {
+                "class-name": {
+                    "type": "url",
+                    "predicate": "rlog:className"
+                },
+                "message": {
+                    "type": "string",
+                    "predicate": "rlog:message"
+                },
+                "specific-information": {
+                    "type": "string",
+                    "predicate": "ext:specificInformation"
+                },
+                "datetime": {
+                    "type": "datetime",
+                    "predicate": "rlog:date"
+                }
+            },
+            "relationships": {
+                "log-level": {
+                    "predicate": "rlog:level",
+                    "target": "log-level",
+                    "cardinality": "one"
+                },
+                "status-code": {
+                    "predicate": "rlog:hasCode",
+                    "target": "status-code",
+                    "cardinality": "one"
+                },
+                "log-source": {
+                    "predicate": "dct:source",
+                    "target": "log-source",
+                    "cardinality": "one"
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://data.lblod.info/id/log-entries/"
+        },
+        "log-levels": {
+            "name": "log-level",
+            "class": "rlog:Level",
+            "attributes": {
+                "priority": {
+                    "type": "number",
+                    "predicate": "rlog:priority"
+                },
+                "label": {
+                    "type": "string",
+                    "predicate": "skos:prefLabel"
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://data.lblod.info/id/log-levels/"
+        },
+        "status-codes": {
+            "name": "status-code",
+            "class": "rlog:StatusCode",
+            "attributes": {
+                "code": {
+                    "type": "number",
+                    "predicate": "rlog:codeId"
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://data.lblod.info/id/status-codes/"
+        },
+        "log-sources": {
+            "name": "log-source",
+            "class": "ext:LogSource",
+            "attributes": {
+                "label": {
+                    "type": "string",
+                    "predicate": "skos:prefLabel"
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://data.lblod.info/id/log-sources/"
+        }
+    }
+}

config/resources/report.json

@@ -0,0 +1,36 @@
+{
+    "version": "0.1",
+    "prefixes": {
+        "dct": "http://purl.org/dc/terms/",
+        "prov": "http://www.w3.org/ns/prov#"
+    },
+    "resources": {
+        "reports": {
+            "name": "report",
+            "class": "http://lblod.data.gift/vocabularies/reporting/Report",
+            "attributes": {
+                "created": {
+                    "type": "datetime",
+                    "predicate": "dct:created"
+                },
+                "title": {
+                    "type": "string",
+                    "predicate": "dct:title"
+                },
+                "description": {
+                    "type": "string",
+                    "predicate": "dct:description"
+                }
+            },
+            "relationships": {
+                "file": {
+                    "predicate": "prov:generated",
+                    "target": "file",
+                    "cardinality": "one"
+                }
+            },
+            "features": ["include-uri"],
+            "new-resource-base": "http://data.lblod.info/id/reports/"
+        }
+    }
+}

docker-compose.dev.yml

@@ -5,6 +5,8 @@ services:
     restart: "no"
     ports:
       - "80:80"
+    environment:
+      SESSION_COOKIE_SECURE: "off"
   dispatcher:
     restart: "no"
   triplestore:

docker-compose.yml

@@ -25,7 +25,7 @@ services:
     restart: always
     logging: *default-logging
   triplestore:
-    image: tenforce/virtuoso:1.3.2-virtuoso7.2.5.1
+    image: redpencil/virtuoso:1.3.0-rc.1
     environment:
       SPARQL_UPDATE: "true"
       DEFAULT_GRAPH: "http://mu.semte.ch/application"
@@ -147,6 +147,14 @@ services:
     logging: *default-logging
     links:
       - db:database
+  dashboard-login:
+    image: semtech/mu-login-service:3.0.0
+    links:
+      - db:database
+    labels:
+      - "logging=true"
+    restart: always
+    logging: *default-logging
   accountdetail:
     image: lblod/account-detail-service:0.1.0
     labels:
@@ -188,7 +196,7 @@ services:
     restart: always
     logging: *default-logging
   frontend:
-    image: lblod/frontend-organization-portal:1.29.1
+    image: lblod/frontend-organization-portal:1.30.0
     volumes:
       - ./config/frontend/add-x-frame-options-header.conf:/config/add-x-frame-options.conf
     environment:
@@ -202,6 +210,12 @@ services:
       - "logging=true"
     restart: always
     logging: *default-logging
+  frontend-dashboard:
+    image: lblod/frontend-dashboard:1.7.0
+    labels:
+      - "logging=true"
+    restart: always
+    logging: *default-logging
   uri-info:
     image: redpencil/mu-uri-info-service:0.2.1
     links:
@@ -310,7 +324,7 @@ services:
   # DELTA GENERAL
   ################################################################################
   publication-triplestore:
-    image: redpencil/virtuoso:1.2.0-rc.1
+    image: redpencil/virtuoso:1.3.0-rc.1
     environment:
       SPARQL_UPDATE: "true"
       DEFAULT_GRAPH: "http://mu.semte.ch/application"
@@ -388,7 +402,7 @@ services:
       SUDO_QUERY_RETRY_FOR_HTTP_STATUS_CODES: "404,500,503"
       SUDO_QUERY_RETRY: "true"
     volumes:
-      - ./config/loket-consumer/mandatarissen/mapping:/config/mapping
+      - ./config/delta-consumers/mandatarissen/mapping:/config/mapping
       - ./data/files/consumer-files/mandatarissen:/consumer-files/
     restart: always
     labels:
@@ -421,7 +435,7 @@ services:
       SUDO_QUERY_RETRY_FOR_HTTP_STATUS_CODES: "404,500,503"
       SUDO_QUERY_RETRY: "true"
     volumes:
-      - ./config/loket-consumer/leidinggevenden/mapping:/config/mapping
+      - ./config/delta-consumers/leidinggevenden/mapping:/config/mapping
       - ./data/files/consumer-files/leidinggevenden:/consumer-files/
     restart: always
     labels:
@@ -459,7 +473,7 @@ services:
       SUDO_QUERY_RETRY_FOR_HTTP_STATUS_CODES: "404,500,503"
       SUDO_QUERY_RETRY: "true"
     volumes:
-      - ./config/loket-consumer/worship-services-main-info/mapping:/config/mapping
+      - ./config/delta-consumers/worship-services-main-info/mapping:/config/mapping
       - ./data/files/consumer-files/worship-services-main-info:/consumer-files/
     restart: always
     labels:
@@ -494,7 +508,7 @@ services:
       SUDO_QUERY_RETRY_FOR_HTTP_STATUS_CODES: "404,500,503"
       SUDO_QUERY_RETRY: "true"
     volumes:
-      - ./config/loket-consumer/worship-services-private-info/mapping:/config/mapping
+      - ./config/delta-consumers/worship-services-private-info/mapping:/config/mapping
       - ./data/files/consumer-files/worship-services-private-info:/consumer-files/
     restart: always
     labels:
@@ -522,6 +536,7 @@ services:
     links:
       - db:database
     restart: always
+    logging: *default-logging
 
   ldes-backend:
     image: redpencil/fragmentation-producer:0.4.0
@@ -535,6 +550,7 @@ services:
       DATA_FOLDER: '/data'
       ENABLE_AUTH: 'false'
     restart: always
+    logging: *default-logging
     volumes:
       - './data/ldes-feed/:/data/'
 
@@ -548,6 +564,7 @@ services:
     restart: always
     volumes:
       - ./config/search:/config
+    logging: *default-logging
   elasticsearch:
     image: semtech/mu-search-elastic-backend:1.0.1
     volumes:
@@ -555,6 +572,7 @@ services:
     environment:
       - discovery.type=single-node
     restart: always
+    logging: *default-logging
 
   error-report-service:
     image: lblod/error-report-service:0.1.0

scripts/project/config.json

@@ -16,6 +16,21 @@
       "mounts": {
         "app": "/data/app/"
       }
+    },
+    {
+      "documentation": {
+        "command": "generate-dashboard-login",
+        "description": "generate a login account for the jobs dashboard",
+        "arguments": []
+      },
+      "environment": {
+        "image": "ruby",
+        "interactive": true,
+        "script": "generate-dashboard-login/run.sh"
+      },
+      "mounts": {
+        "app": "/project/"
+      }
     }
   ]
-}
+}

scripts/project/generate-dashboard-login/run.sh

@@ -0,0 +1,80 @@
+#!/usr/bin/env ruby
+require 'bundler/inline'
+require 'yaml'
+require 'securerandom'
+require 'date'
+
+$stdout.sync = true
+print "installing dependencies..."
+gemfile do
+  source 'https://rubygems.org'
+  gem 'bcrypt'
+end
+puts "done"
+
+salt = ""
+begin
+  File.open("/project/docker-compose.override.yml") do |file|
+    config = YAML.load(file)
+    salt = config.dig("services", "dashboard-login", "environment", "MU_APPLICATION_SALT")
+    throw "invalid salt" unless salt
+   end
+rescue
+  puts "Tried to locate login salt in docker-compose.override.yml, but couldn't find it."
+  puts "Please make sure a salt is configured:"
+  puts <<EOF
+  login:
+    restart: "no"
+    environment:
+      MU_APPLICATION_SALT: "#{SecureRandom.hex}"
+EOF
+end
+
+puts "this script will generate a new user account"
+print "username: "
+STDOUT.flush
+account_name = STDIN.gets.chomp
+
+print "password: "
+STDOUT.flush
+password = STDIN.gets.chomp
+
+account_salt = SecureRandom.hex
+person_uuid = SecureRandom.uuid
+account_uuid = SecureRandom.uuid
+domain = "data.gift"
+hashed_password = BCrypt::Password.create(password + salt + account_salt)
+now = DateTime.now.xmlschema
+query = %(
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX people: <http://#{domain}/people/>
+PREFIX accounts: <http://#{domain}/accounts/>
+PREFIX dcterms: <http://purl.org/dc/terms/>
+PREFIX mu:      <http://mu.semte.ch/vocabularies/core/>
+PREFIX ext:      <http://mu.semte.ch/vocabularies/ext/>
+PREFIX account: <http://mu.semte.ch/vocabularies/account/>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+INSERT DATA {
+   GRAPH <http://mu.semte.ch/application> {
+     people:#{person_uuid} a foaf:Person ;
+                   foaf:name "#{account_name}";
+                   foaf:account accounts:#{account_uuid};
+                   mu:uuid "#{person_uuid}";
+                   dcterms:created "#{now}"^^xsd:datetime;
+                   dcterms:modified "#{now}"^^xsd:datetime.
+     accounts:#{account_uuid} a foaf:OnlineAccount;
+                                                   foaf:accountName "#{account_name}";
+                                                   mu:uuid "#{account_uuid}";
+                                                   account:password """#{hashed_password}""";
+                                                   account:salt "#{account_salt}";
+                                                   account:status <http://mu.semte.ch/vocabularies/account/status/active>;
+                                                   dcterms:created "#{now}"^^xsd:datetime;
+                                                   ext:sessionRole "dashboard-user";
+                                                   dcterms:modified "#{now}"^^xsd:datetime.
+}}
+)
+
+timestamp = DateTime.now.strftime("%Y%m%d%H%M%L")
+filename = "#{timestamp}-create-user-#{account_name.gsub(" ", "-")}.sparql"
+File.write("/project/config/migrations/local/#{filename}", query)
+puts "generated migration ./config/migrations/local/#{filename} for #{account_name.inspect} : #{password.inspect}"