chore: update changelog

The governing bodies for an AGB have the same legislature as the corresponding
municipality. This adds new time specialised governing bodies for the
new (current) legislature for Raad van bestuur and Algemene vergadering.

.gitignore

@@ -20,6 +20,7 @@ docker-compose.override.yml
 
 /config/migrations/private-data/*
 /config/migrations/local/*
+/config/migrations-triggering-indexing/local/*
 
 /config/proxy/nginx.htpasswd
 

CHANGELOG.md

@@ -1,4 +1,262 @@
 # Changelog
+## Unreleased
+
+## v1.30.2 (2025-02-28)
+### Backend
+- Datafix: add new timed governing bodies for new legislature for AGBs [OP-3552]
+### Deploy notes
+- Restart the migration service
+- Start a healing job the public producer, or wait for it to kick in overnight
+#### Deploy commands
+```
+drc restart migrations; drc logs -ft --tail=200 migrations
+drc exec delta-producer-background-jobs-initiator curl -X POST http://localhost/public/healing-jobs
+```
+
+## v1.30.1 (2025-02-20)
+### Backend
+ - Bestuursorgaanclassificatiecode of VGC was in wrong graph, so it wouldn't export. [DL-6428]
+### Deploy notes
+```
+drc restart migrations
+```
+
+## v1.30.0 (2025-02-19)
+### Frontend
+- Bump to version [v1.29.0](https://github.com/lblod/frontend-organization-portal/releases/tag/v1.29.0)
+### Backend
+- Datafix: delete duplicate identifiers [OP-3157]
+- Feature: replace different relations between organisations by memberships [OP-3195 (epic), OP-2606, OP-3156, OP-3198, OP-3199, OP-3258, OP-3259, OP-3260, OP-3265, OP-3293, OP-3305]
+- Added `db-cleanup-service` to keep membership data backwards compatible [OP-3369]
+- Datafix: delete incorrect change event [OP-3363]
+- Feature: re-enable contact data editing [OP-3425]
+- Remove gemeente and provincie classifications from admin unit graph [OP-3393]
+- Remove several OCMW bcsd faciliteitgemeenten [OP-3535]
+- Completely remove AGB Stekene's data. [OP-3409]
+- Remove reasoner service, replaced by new consumer logic
+
+### Deploy Notes
+- Before pulling the changes
+  + Shut down the reasoner and thank it for its good service: `drc down reasoner`
+- Re-enable contact date editing:
+  + remove `EMBER_ENABLE_EDIT_CONTACT_DATA_FEATURE` flag in `docker-compose.override.yml`
+  + Stop and remove the `clb-contact-data-consumer` service
+  + Remove the `clb-contact-data-consumer` configuration in `docker-compose.override.yml`
+- Rebuild the `search` service before starting the re-indexing
+- To enable the membership functionality:
+  + Restart migration service
+  + Restart services for which the configuration was changed: `resource`,  `dispatcher` and `db`
+  + Start the `db-cleanup` service
+  + Pull and restart the frontend
+  + Perform a re-index using `reset-elastic.sh` script
+#### Deploy commands
+```
+drc down clb-contact-data-consumer
+drc up -d search
+drc restart migrations; drc logs -ft --tail=200 migrations
+drc restart migrations-triggering-indexing && drc logs -ft --tail=200 migrations-triggering-indexing
+drc restart resource dispatcher db
+drc up -d db-cleanup
+drc pull frontend; drc up -d frontend
+./scripts/reset-elastic.sh
+```
+
+## 1.29.2 (2025-02-17)
+### Data fix
+ - restore betrokken lokale besturen that have been accidentally flushed [OP-3546]
+
+## 1.29.1 (2025-02-14)
+### Backend
+ - re-init `worship-services-sensitive-consumer` [OP-3483]
+### Deploy notes
+:warning: Ensure first loket did a full re-sync again of this delta-stream.
+```
+drc stop worship-services-main-info-consumer worship-services-private-info-consumer
+drc retart migrations
+```
+Update `docker-compose.override.yml` to remove the config of the consumers and replace it by:
+```
+  worship-services-main-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_DISABLE_INITIAL_SYNC: "false"
+      DCR_DISABLE_DELTA_INGEST: "false"
+  worship-services-private-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_DISABLE_INITIAL_SYNC: "false"
+      DCR_DISABLE_DELTA_INGEST: "false"
+```
+Then
+```
+drc up -d db mandatarissen-consumer leidinggevenden-consumer worship-services-main-info-consumer worship-services-private-info-consumer
+# Wait until success of the previous step
+```
+Then, update `docker-compose.override.yml` to:
+```
+  worship-services-main-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "db"
+      DCR_REMAPPING_DATABASE: "db"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+  worship-services-private-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "database"
+      DCR_REMAPPING_DATABASE: "database"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+```
+```
+drc up -d
+sh scripts/reset-elastic.sh
+
+```
+
+## 1.29.0 (2025-02-07)
+### Backend
+#### Consumer
+- Upgraded `mandatarissen-consumer` [OP-3510]
+- Upgraded `leidinggevenden-consumer` [OP-3533], [OP-3511]
+- Upgraded `worship-services-sensitive-consumer` [OP-3337]
+
+### Deploy notes
+```
+drc down;
+```
+Update `docker-compose.override.yml` to remove the config of the consumers and replace it by:
+```
+  mandatarissen-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+  leidinggevenden-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+  worship-services-main-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_DISABLE_INITIAL_SYNC: "false"
+      DCR_DISABLE_DELTA_INGEST: "false"
+  worship-services-private-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_DISABLE_INITIAL_SYNC: "false"
+      DCR_DISABLE_DELTA_INGEST: "false"
+```
+Then:
+```
+drc up -d triplestore migrations
+drc up -d db mandatarissen-consumer leidinggevenden-consumer worship-services-main-info-consumer worship-services-private-info-consumer
+# Wait until success of the previous step
+```
+Then, update `docker-compose.override.yml` to:
+```
+  mandatarissen-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_LANDING_ZONE_DATABASE: "db"
+      DCR_REMAPPING_DATABASE: "db"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+  leidinggevenden-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # choose the correct endpoint
+      DCR_LANDING_ZONE_DATABASE: "db"
+      DCR_REMAPPING_DATABASE: "db"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+  worship-services-main-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "db"
+      DCR_REMAPPING_DATABASE: "db"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+  worship-services-private-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "database"
+      DCR_REMAPPING_DATABASE: "database"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+```
+```
+drc up -d
+sh scripts/reset-elastic.sh
+```
+
+## 1.28.5 (2025-01-23)
+### Frontend
+- Bump to [v1.28.3](https://github.com/lblod/frontend-organization-portal/releases/tag/v1.28.3)
+### Backend
+- Datafix: correct merger change events for worship organisations (OP-3534)
+- Avoid municipality filter to include partial matches (OP-3509)
+### Deploy notes
+- `drc restart migrations-triggering-indexing; drc logs -ft --tail=200 migrations-triggering-indexing`
+- `drc pull frontend; drc up -d frontend`
+- Perform a full re-index by executing `./scripts/reset-elastic.sh`
+
+## 1.28.4 (2025-01-16)
+### Backend
+#### Data
+- Performance fix on 1.28.3 [DL-6377]
+### Deploy commands
+```
+drc restart migrations
+drc exec delta-producer-background-jobs-initiator curl -X POST http://localhost/public/healing-jobs # or wait for the healing to kick in
+```
+## 1.28.3 (2025-01-16)
+### Backend
+#### Data
+- Datafix. Fusiegemeentes had different bestuursfuncties attached to OCMW and Gemeente. [DL-6377]
+  However, business-rule-wise, these bestuursfuncties are shared.
+### Deploy commands
+```
+drc restart migrations
+drc exec delta-producer-background-jobs-initiator curl -X POST http://localhost/public/healing-jobs # or wait for the healing to kick in
+```
+## 1.28.2 (2025-01-13)
+### Backend
+ - Added "Opdrachthoudende vereniging met private deelname" in delta public (DL-6368)
+### Deploy commands
+```
+drc restart migrations
+drc exec delta-producer-background-jobs-initiator curl -X POST http://localhost/public/healing-jobs # or wait for the healing to kick in
+```
 ## 1.28.1 (2025-01-11)
 ### Backend
 - Import 2025 NIS codes and update werkingsgebieden (OP-3342)

README.md

@@ -11,6 +11,145 @@ Backend for the organization portal application, based on the mu.semte.ch micros
 
 You can shut down using `docker-compose stop` and remove everything using `docker-compose rm`.
 
+#### Setup consumers
+
+The application depends on external data. You'll have to perform extra steps if you want this data to be available.
+These are multiple data sets, we'll try to be exhaustive on how to ingest these. Even though the procedures are similar.
+
+#### Setup mandatarissen sync
+Note: this is assuming you have never ran this before.
+```
+drc down;
+```
+Update `docker-compose.override.yml` to:
+```
+  mandatarissen-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
+      DCR_LANDING_ZONE_DATABASE: "triplestore"
+      DCR_REMAPPING_DATABASE: "triplestore"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+```
+Then:
+```
+drc up -d migrations
+drc up -d db mandatarissen-consumer
+# Wait until success of the previous step
+```
+Then, update `docker-compose.override.yml` to:
+```
+  mandatarissen-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
+      DCR_LANDING_ZONE_DATABASE: "db"
+      DCR_REMAPPING_DATABASE: "db"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+```
+Finally, put the stack back up and reindex elasticsearch
+```
+drc up -d
+sh scripts/reset-elastic.sh
+```
+
+#### Setup leidinggevenden sync (http://data.lblod.info/vocabularies/leidinggevenden/Functionaris)
+
+Note: this is assuming you have never ran this before.
+```
+drc down;
+```
+Update `docker-compose.override.yml` to:
+```
+  leidinggevenden-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
+      DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+```
+Then:
+```
+drc up -d migrations
+drc up -d db leidinggevenden-consumer
+# Wait until success of the previous step
+```
+Then, update `docker-compose.override.yml` to:
+```
+  leidinggevenden-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # choose the correct endpoint
+      DCR_LANDING_ZONE_DATABASE: "db"
+      DCR_REMAPPING_DATABASE: "db"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+```
+Finally, put the stack back up and reindex elasticsearch
+```
+drc up -d
+sh scripts/reset-elastic.sh
+```
+
+#### Setup worship mandatees sync
+
+Note: this is assuming you have never ran this before.
+```
+drc down;
+```
+Update `docker-compose.override.yml` to:
+```
+  worship-services-main-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "triplestore"
+      DCR_REMAPPING_DATABASE: "triplestore"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+  worship-services-private-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "triplestore"
+      DCR_REMAPPING_DATABASE: "triplestore"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+```
+Then:
+```
+drc up -d migrations
+drc up -d db worship-services-main-info-consumer worship-services-private-info-consumer
+# Wait until success of the previous step
+```
+Then, update `docker-compose.override.yml` to:
+```
+worship-services-main-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "db"
+      DCR_REMAPPING_DATABASE: "db"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+  worship-services-private-info-consumer:
+    environment:
+      DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
+      DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
+      DCR_SECRET_KEY: "key-of-the-producer"
+      DCR_LANDING_ZONE_DATABASE: "db"
+      DCR_REMAPPING_DATABASE: "db"
+      DCR_DISABLE_DELTA_INGEST: "false"
+      DCR_DISABLE_INITIAL_SYNC: "false"
+```
+Finally, put the stack back up and reindex elasticsearch
+```
+drc up -d
+sh scripts/reset-elastic.sh
+```
 
 ### Setting up the delta-producers related services
 To make sure the app can share data, producers need to be set up. There is an intial sync, that is potentially very expensive, and must be started manually

config/authorization/config.ex

@@ -79,6 +79,8 @@ defmodule Acl.UserGroups.Config do
     "http://www.w3.org/ns/org#ChangeEvent",
     "http://data.lblod.info/vocabularies/contacthub/AgentInPositie",
     "http://mu.semte.ch/vocabularies/ext/KboOrganisatie",
+    "http://www.w3.org/ns/org#Membership",
+    "http://www.w3.org/2006/time#ProperInterval"
   ]
 
   @shared_protected_type [

config/clb-consumer/contact-data/config.js

@@ -1,30 +0,0 @@
-const BATCH_SIZE = parseInt(process.env.BATCH_SIZE) || 1000;
-const MU_CALL_SCOPE_ID_INITIAL_SYNC = process.env.MU_CALL_SCOPE_ID_INITIAL_SYNC || 'http://redpencil.data.gift/id/concept/muScope/deltas/consumer/initialSync';
-const BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES = process.env.BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES == 'true' ? true : false;
-const DIRECT_DATABASE_ENDPOINT = process.env.DIRECT_DATABASE_ENDPOINT || 'http://virtuoso:8890/sparql';
-const MAX_DB_RETRY_ATTEMPTS = parseInt(process.env.MAX_DB_RETRY_ATTEMPTS || 5);
-const MAX_REASONING_RETRY_ATTEMPTS = parseInt(process.env.MAX_REASONING_RETRY_ATTEMPTS || 5);
-const SLEEP_BETWEEN_BATCHES = parseInt(process.env.SLEEP_BETWEEN_BATCHES || 100);
-const SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION || 10000);
-const SLEEP_TIME_AFTER_FAILED_DB_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_DB_OPERATION || 60000);
-
-// Graphs
-const LANDING_ZONE_GRAPH = process.env.DCR_LANDING_ZONE_GRAPH || `http://mu.semte.ch/graphs/landing-zone/clb-contact-data`;
-
-// ENDPOINTS
-const LANDING_ZONE_DATABASE = process.env.DCR_LANDING_ZONE_DATABASE || 'database';
-const LANDING_ZONE_DATABASE_ENDPOINT = process.env.DCR_LANDING_ZONE_DATABASE_ENDPOINT || `http://${LANDING_ZONE_DATABASE}:8890/sparql`;
-
-module.exports = {
-  BATCH_SIZE,
-  MU_CALL_SCOPE_ID_INITIAL_SYNC,
-  BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
-  DIRECT_DATABASE_ENDPOINT,
-  MAX_DB_RETRY_ATTEMPTS,
-  MAX_REASONING_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
-  LANDING_ZONE_GRAPH,
-  LANDING_ZONE_DATABASE_ENDPOINT
-};

config/clb-consumer/contact-data/delta-context-config.js

@@ -1,268 +0,0 @@
-
-const PREFIXES = `
-PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>
-  PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
-  PREFIX lblodgeneriek: <https://data.lblod.info/vocabularies/generiek/>
-  PREFIX org: <http://www.w3.org/ns/org#>
-  PREFIX code: <http://lblod.data.gift/vocabularies/organisatie/>
-  PREFIX adms: <http://www.w3.org/ns/adms#>
-  PREFIX generiek: <https://data.vlaanderen.be/ns/generiek#>
-  PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
-  PREFIX organisatie: <https://data.vlaanderen.be/ns/organisatie#>
-  PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
-  PREFIX euvoc: <http://publications.europa.eu/ontology/euvoc#>
-  PREFIX prov: <http://www.w3.org/ns/prov#>
-  PREFIX schema: <http://schema.org/>
-  PREFIX locn: <http://www.w3.org/ns/locn#>
-  PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>`
-
-const contextConfig = {
-  addTypes: {
-    scope: 'all', // 'inserts, 'deletes', 'all' or none. To add rdf:type to subjects of inserts, deletes or both
-    exhausitive: false, // true or false: find all types for a subject, even if one is already present in delta
-  },
-  contextQueries: [
-    {
-      trigger: { // subjectType or predicateValue
-        subjectType: "org:Site"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ${subject} ext:goesInGraph ?graphOne.
-          ${subject} ext:goesInGraph ?graphTwo.
-          ${subject} ext:goesInGraph ?graphThree.
-          ${subject} ext:goesInGraph ?graphFour.
-        } WHERE {
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-              ?adminUnitOne org:hasPrimarySite ${subject}.
-            }
-            BIND(<http://mu.semte.ch/graphs/administrative-unit> AS ?graphOne)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-              ?adminUnitTwo org:hasSite ${subject}.
-            }
-            BIND(<http://mu.semte.ch/graphs/administrative-unit> AS ?graphTwo)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/worship-service> {
-              ?adminUnitThree org:hasPrimarySite ${subject}.
-            }
-            BIND(<http://mu.semte.ch/graphs/worship-service> AS ?graphThree)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/worship-service> {
-              ?adminUnitFour org:hasSite ${subject}.
-            }
-            BIND(<http://mu.semte.ch/graphs/worship-service> AS ?graphFour)
-          }
-        }`
-    },
-    {
-      trigger: { // subjectType or predicateValue
-        subjectType: "schema:ContactPoint"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ${subject} ext:goesInGraph ?graphOne.
-          ${subject} ext:goesInGraph ?graphTwo.
-        } WHERE {
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-              ?siteOne org:siteAddress ${subject}.
-            }
-            BIND(<http://mu.semte.ch/graphs/administrative-unit> AS ?graphOne)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/worship-service> {
-              ?siteTwo org:siteAddress ${subject}.
-            }
-            BIND(<http://mu.semte.ch/graphs/worship-service> AS ?graphTwo)
-          }
-        }`
-    },
-    {
-      trigger: { // subjectType or predicateValue
-        subjectType: "locn:Address"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ${subject} ext:goesInGraph ?graphOne.
-          ${subject} ext:goesInGraph ?graphTwo.
-        } WHERE {
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-              ?siteOne organisatie:bestaatUit ${subject}.
-            }
-            BIND(<http://mu.semte.ch/graphs/administrative-unit> AS ?graphOne)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/worship-service> {
-              ?siteTwo organisatie:bestaatUit ${subject}.
-            }
-            BIND(<http://mu.semte.ch/graphs/worship-service> AS ?graphTwo)
-          }
-        }`
-    },
-    //Predicate triggers to solve out of order deltas
-    {
-      trigger: { // subjectType or predicateValue
-        predicateValue: "org:hasSite"
-      },
-      //We assume that only administrative units are in the shared graph of OP due to the need of sharing the prefLabel with worships services
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ${subject} ext:goesInGraph ?graphOne.
-          ${subject} ext:goesInGraph ?graphTwo.
-          ?site ?pSite ?oSite.
-          ?contact ?pContact ?oContact.
-          ?address ?pAddress ?oAddress.
-          ?addressContact ?pAddressContact ?oAddressContact.
-        } WHERE {
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-              ${subject} a ?type.
-            }
-            BIND(<http://mu.semte.ch/graphs/administrative-unit> AS ?graphOne)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/shared> {
-              ${subject} a ?type.
-            }
-            BIND(<http://mu.semte.ch/graphs/administrative-unit> AS ?graphOne)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/worship-service> {
-              ${subject} a ?type.
-            }
-            BIND(<http://mu.semte.ch/graphs/worship-service> AS ?graphTwo)
-          }
-          ${subject} org:hasSite ?site.
-          ?site ?pSite ?oSite.
-          {
-            ?site org:siteAddress ?contact.
-            ?contact ?pContact ?oContact.
-            ?contact locn:address ?addressContact.
-            ?addressContact ?pAddressContact ?oAddressContact.
-          }
-          UNION
-          {
-            ?site org:siteAddress ?contact.
-            ?contact ?pContact ?oContact.
-          }
-          UNION
-          {
-            ?site organisatie:bestaatUit ?address.
-            ?address ?pAddress ?oAddress.
-          }        
-        }`
-    },
-    {
-      trigger: { // subjectType or predicateValue
-        predicateValue: "org:hasPrimarySite"
-      },
-      //We assume that only administrative units are in the shared graph of OP due to the need of sharing the prefLabel with worships services
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ${subject} ext:goesInGraph ?graphOne.
-          ${subject} ext:goesInGraph ?graphTwo.
-          ?site ?pSite ?oSite.
-          ?contact ?pContact ?oContact.
-          ?address ?pAddress ?oAddress.
-          ?addressContact ?pAddressContact ?oAddressContact.
-        } WHERE {
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-              ${subject} a ?type.
-            }
-            BIND(<http://mu.semte.ch/graphs/administrative-unit> AS ?graphOne)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/shared> {
-              ${subject} a ?type.
-            }
-            BIND(<http://mu.semte.ch/graphs/administrative-unit> AS ?graphOne)
-          }
-          OPTIONAL {
-            GRAPH <http://mu.semte.ch/graphs/worship-service> {
-              ${subject} a ?type.
-            }
-            BIND(<http://mu.semte.ch/graphs/worship-service> AS ?graphTwo)
-          }
-          ${subject} org:hasPrimarySite ?site.
-          ?site ?pSite ?oSite.
-          {
-            ?site org:siteAddress ?contact.
-            ?contact ?pContact ?oContact.
-            ?contact locn:address ?addressContact.
-            ?addressContact ?pAddressContact ?oAddressContact.
-          }
-          UNION
-          {
-            ?site org:siteAddress ?contact.
-            ?contact ?pContact ?oContact.
-          }
-          UNION
-          {
-            ?site organisatie:bestaatUit ?address.
-            ?address ?pAddress ?oAddress.
-          }        
-        }`
-    },
-    {
-      trigger: { // subjectType or predicateValue
-        predicateValue: "org:siteAddress"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ?contact ?pContact ?oContact.
-          ?addressContact ?pAddressContact ?oAddressContact.
-        } WHERE {
-          ${subject} org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-          OPTIONAL {
-            ?contact locn:address ?addressContact.
-            ?addressContact ?pAddressContact ?oAddressContact.
-          }
-        }`
-    },
-    {
-      trigger: { // subjectType or predicateValue
-        predicateValue: "locn:address"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ?addressContact ?pAddressContact ?oAddressContact.
-        } WHERE {
-          ${subject} locn:address ?addressContact.
-          ?addressContact ?pAddressContact ?oAddressContact.
-        }`
-    },
-    {
-      trigger: { // subjectType or predicateValue
-        predicateValue: "organisatie:bestaatUit"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ?address ?pAddress ?oAddress.
-        } WHERE {
-          ${subject} organisatie:bestaatUit ?address.
-          ?address ?pAddress ?oAddress.
-        }`
-    }
-  ]
-}
-
-module.exports = {
-  contextConfig,
-  PREFIXES
-};

config/clb-consumer/contact-data/delta-sync-dispatching.js

@@ -1,104 +0,0 @@
-const {
-  deleteFromPublicGraph,
-  insertIntoPublicGraph,
-  deleteFromSpecificGraphs,
-  insertIntoSpecificGraphs
-} = require('./util');
-
-const {
-  LANDING_ZONE_GRAPH
-} = require('./config')
-
-const publicTypes = [
-  '<http://lblod.data.gift/vocabularies/organisatie/TypeVestiging>'
-]
-
-/**
-* Dispatch the fetched information to a target graph.
-* Note: <share://file/data> will be ADDED to it's own graph.
-*   We take only care of adding them, not updating triples, this is a TODO
-* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
-* @param { termObjectChangeSets: { deletes, inserts } } data - The fetched changes sets, which objects of serialized Terms
-*          [ {
-*              graph: "<http://foo>",
-*              subject: "<http://bar>",
-*              predicate: "<http://baz>",
-*              object: "<http://boom>^^<http://datatype>"
-*            }
-*         ]
-* @return {void} Nothing
-*/
-async function dispatch(lib, data) {
-  let { termObjectChangeSets, termObjectChangeSetsWithContext } = data;
-
-  // Both arrays are the same length, so we can zip them together for easier processing
-  zippedChangeSets = termObjectChangeSets.map((o, i) => ({
-    original: o,
-    withContext: termObjectChangeSetsWithContext && termObjectChangeSetsWithContext[i]
-  }));
-
-  console.log(`Received ${zippedChangeSets.length} change sets`)
-
-  for (let { original, withContext } of zippedChangeSets) {
-    
-    const {public: insertsOnPublic, target: insertsOnGraphs} = separateBetweenTargetAndPublic(original.inserts, withContext.inserts)
-    const {public: deletesOnPublic, target: deletesOnGraphs} = separateBetweenTargetAndPublic(original.deletes, withContext.deletes)
-
-    await deleteFromPublicGraph(lib, deletesOnPublic);
-    await deleteFromSpecificGraphs(lib, deletesOnGraphs);
-    await insertIntoPublicGraph(lib, insertsOnPublic);
-    await insertIntoSpecificGraphs(lib, insertsOnGraphs);
-
-  }
-}
-
-function separateBetweenTargetAndPublic(changes, context) {
-  const changesOnPublic = new Set();
-  const changesOnGraphs = {};
-  for(let change of changes) {
-    const subject = change.subject;
-    const contextTriples = context.filter((context) => context.subject === subject);
-    const graphTriple = contextTriples.find(
-      (context) => context.predicate === '<http://mu.semte.ch/vocabularies/ext/goesInGraph>' 
-        && context.object !== `<${LANDING_ZONE_GRAPH}>`
-    )
-    const otherContextTriples = context.filter(
-      (context) => context.predicate !== '<http://mu.semte.ch/vocabularies/ext/goesInGraph>' 
-        && context.predicate !== '<http://mu.semte.ch/vocabularies/ext/contextDataGoesInGraph>' 
-        && !(context.subject === subject && context.predicate === '<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>')
-    );
-    if(graphTriple) {
-      const graph = graphTriple.object.slice(1,-1); // We have to slice it to remove the "<" and ">"
-      if(change.object !== '') {
-        if(!changesOnGraphs[graph]) {
-          changesOnGraphs[graph] = new Set();
-        }  
-        changesOnGraphs[graph].add(`${change.subject} ${change.predicate} ${change.object}.`)
-      }                    
-      for(let triple of otherContextTriples) {
-        if(triple.object === '') continue;
-        if(!changesOnGraphs[graph]) {
-          changesOnGraphs[graph] = new Set();
-        }  
-        changesOnGraphs[graph].add(`${triple.subject} ${triple.predicate} ${triple.object}.`)                                       
-      }
-    }
-    const typeTriple = contextTriples.find(
-      (context) => context.subject === subject 
-        && context.predicate === '<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>'
-    );
-    if(!typeTriple) continue;
-    const type = typeTriple.object;
-    if(publicTypes.includes(type)) {
-      changesOnPublic.add(`${change.subject} ${change.predicate} ${change.object}.`)
-      for(let triple of otherContextTriples) {
-        changesOnPublic.add(`${triple.subject} ${triple.predicate} ${triple.object}.`)
-      }
-    }
-  }
-  return {public: changesOnPublic, target: changesOnGraphs}
-}
-
-module.exports = {
-  dispatch
-};

config/clb-consumer/contact-data/initial-sync-dispatching.js

@@ -1,69 +0,0 @@
-const { batchedDbUpdate, moveDataAfterInitialSync } = require('./util');
-const {
-  BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
-  DIRECT_DATABASE_ENDPOINT,
-  MU_CALL_SCOPE_ID_INITIAL_SYNC,
-  BATCH_SIZE,
-  MAX_DB_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  LANDING_ZONE_GRAPH,
-} = require('./config');
-
-const endpoint = BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES ? DIRECT_DATABASE_ENDPOINT : process.env.MU_SPARQL_ENDPOINT;
-
-/**
-* Dispatch the fetched information to a target graph.
-* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
-* @param { termObjects } data - The fetched quad information, which objects of serialized Terms
-*          [ {
-*              graph: "<http://foo>",
-*              subject: "<http://bar>",
-*              predicate: "<http://baz>",
-*              object: "<http://boom>^^<http://datatype>"
-*            }
-*         ]
-* @return {void} Nothing
-*/
-async function dispatch(lib, data) {
-  const { mu, muAuthSudo } = lib;
-  const { termObjects } = data;
-
-  // Steps:
-  // 1. Write the triples to the landing zone graph - without any mapping or filtering
-  // 2. One-time reasoning run with the full landing zone graph and write the results to the target graph (on-finish)
-
-  if (BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES) {
-    console.warn(`Service configured to skip MU_AUTH!`);
-  }
-  console.log(`Using ${endpoint} to insert triples`);
-
-  if (termObjects.length) {
-    const originalInsertTriples = termObjects.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-
-    await batchedDbUpdate(
-      muAuthSudo.updateSudo,
-      LANDING_ZONE_GRAPH,
-      originalInsertTriples,
-      { 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
-      endpoint,
-      BATCH_SIZE,
-      MAX_DB_RETRY_ATTEMPTS,
-      SLEEP_BETWEEN_BATCHES,
-      SLEEP_TIME_AFTER_FAILED_DB_OPERATION
-    );
-  }
-}
-
-async function onFinishInitialIngest(lib,data) {
-  const { muAuthSudo } = lib;
-
-  console.log(`!! On-finish triggered !!`);
-
-  await moveDataAfterInitialSync(muAuthSudo.updateSudo, endpoint)
-}
-
-module.exports = {
-  dispatch,
-  onFinishInitialIngest
-};

config/clb-consumer/contact-data/util.js

@@ -1,337 +0,0 @@
-const {
-  MAX_DB_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  LANDING_ZONE_GRAPH,
-  BATCH_SIZE,
-} = require('./config');
-
-const prefixes = `
-PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>
-PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
-PREFIX lblodgeneriek: <https://data.lblod.info/vocabularies/generiek/>
-PREFIX org: <http://www.w3.org/ns/org#>
-PREFIX code: <http://lblod.data.gift/vocabularies/organisatie/>
-PREFIX adms: <http://www.w3.org/ns/adms#>
-PREFIX generiek: <https://data.vlaanderen.be/ns/generiek#>
-PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
-PREFIX organisatie: <https://data.vlaanderen.be/ns/organisatie#>
-PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
-PREFIX euvoc: <http://publications.europa.eu/ontology/euvoc#>
-PREFIX prov: <http://www.w3.org/ns/prov#>
-PREFIX schema: <http://schema.org/>
-PREFIX locn: <http://www.w3.org/ns/locn#>
-PREFIX foaf: <http://xmlns.com/foaf/0.1/>
-PREFIX ext:<http://mu.semte.ch/vocabularies/ext/>
-`
-
-async function batchedDbUpdate(
-  muUpdate,
-  graph,
-  triples,
-  extraHeaders,
-  endpoint,
-  batchSize,
-  maxAttempts,
-  sleepBetweenBatches = 1000,
-  sleepTimeOnFail = 1000,
-  operation = 'INSERT'
-) {
-  for (let i = 0; i < triples.length; i += batchSize) {
-    console.log(`Inserting triples in batch: ${i}-${i + batchSize}`);
-
-    const batch = triples.slice(i, i + batchSize).join('\n');
-
-    const insertCall = async () => {
-      await muUpdate(`
-${operation} DATA {
-GRAPH <${graph}> {
-${batch}
-}
-}
-`, extraHeaders, endpoint);
-    };
-
-    await operationWithRetry(insertCall, 0, maxAttempts, sleepTimeOnFail);
-
-    console.log(`Sleeping before next query execution: ${sleepBetweenBatches}`);
-    await new Promise(r => setTimeout(r, sleepBetweenBatches));
-  }
-}
-
-async function operationWithRetry(callback,
-  attempt,
-  maxAttempts,
-  sleepTimeOnFail) {
-  try {
-    if (typeof callback === "function")
-      return await callback();
-    else // Catch error from promise - not how I would do it normally, but allows re use of existing code.
-      return await callback;
-  }
-  catch (e) {
-    console.log(`Operation failed for ${callback.toString()}, attempt: ${attempt} of ${maxAttempts}`);
-    console.log(`Error: ${e}`);
-    console.log(`Sleeping ${sleepTimeOnFail} ms`);
-
-    if (attempt >= maxAttempts) {
-      console.log(`Max attempts reached for ${callback.toString()}, giving up`);
-      throw e;
-    }
-
-    await new Promise(r => setTimeout(r, sleepTimeOnFail));
-    return operationWithRetry(callback, ++attempt, maxAttempts, sleepTimeOnFail);
-  }
-}
-
-
-async function insertIntoPublicGraph(lib, statements) {
-  console.log(`Inserting ${statements.length} statements into public graph`);
-
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    'http://mu.semte.ch/graphs/public',
-    Array.from(statements),
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'INSERT');
-}
-
-async function insertIntoSpecificGraphs(lib, statementsWithGraphs) {
-
-  for( let graph in statementsWithGraphs) {
-    console.log(`Inserting ${statementsWithGraphs[graph].length} statements into ${graph} graph`);
-    await batchedDbUpdate(
-      lib.muAuthSudo.updateSudo,
-      graph,
-      Array.from(statementsWithGraphs[graph]),
-      {},
-      process.env.MU_SPARQL_ENDPOINT,
-      BATCH_SIZE,
-      MAX_DB_RETRY_ATTEMPTS,
-      SLEEP_BETWEEN_BATCHES,
-      SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-      'INSERT');
-  }
-  
-}
-
-async function deleteFromPublicGraph(lib, statements) {
-  console.log(`Deleting ${statements.length} statements from public graph`);
-
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    'http://mu.semte.ch/graphs/public',
-    Array.from(statements),
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'DELETE');
-}
-
-async function deleteFromSpecificGraphs(lib, statementsWithGraphs) {
-  
-
-  for( let graph in statementsWithGraphs) {
-    console.log(`Deleting ${statementsWithGraphs[graph].length} statements from ${graph} graph`);
-    await batchedDbUpdate(
-      lib.muAuthSudo.updateSudo,
-      graph,
-      Array.from(statementsWithGraphs[graph]),
-      {},
-      process.env.MU_SPARQL_ENDPOINT,
-      BATCH_SIZE,
-      MAX_DB_RETRY_ATTEMPTS,
-      SLEEP_BETWEEN_BATCHES,
-      SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-      'DELETE');
-  }
-  
-}
-
-
-async function moveDataAfterInitialSync(muUpdate, endpoint) {
-  // Move primary sites admin units
-  await muUpdate(`
-    ${prefixes}
-    INSERT {
-      GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-        ?adminUnit org:hasPrimarySite ?site.
-        ?site ?pSite ?oSite.
-        ?contact ?pContact ?oContact.
-        ?address ?pAddress ?oAddress.
-        ?addressContact ?pAddressContact ?oAddressContact.
-      }
-    } WHERE {
-      GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-        ?adminUnit a ?type.
-      }
-      GRAPH <${LANDING_ZONE_GRAPH}> {
-        ?adminUnit org:hasPrimarySite ?site.
-        ?site ?pSite ?oSite.
-        {
-          ?site org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-          ?contact locn:address ?addressContact.
-          ?addressContact ?pAddressContact ?oAddressContact.
-        }
-        UNION
-        {
-          ?site org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-        }
-        UNION
-        {
-          ?site organisatie:bestaatUit ?address.
-          ?address ?pAddress ?oAddress.
-        }
-      }        
-    }
-  `, undefined, endpoint)
-
-  // Move sites administrative units
-  await muUpdate(`
-    ${prefixes}
-    INSERT {
-      GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-        ?adminUnit org:hasSite ?site.
-        ?site ?pSite ?oSite.
-        ?contact ?pContact ?oContact.
-        ?address ?pAddress ?oAddress.
-        ?addressContact ?pAddressContact ?oAddressContact.
-      }
-    } WHERE {
-      GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
-        ?adminUnit a ?type.
-      }
-      GRAPH <${LANDING_ZONE_GRAPH}> {
-        ?adminUnit org:hasSite ?site.
-        ?site ?pSite ?oSite.
-        {
-          ?site org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-          ?contact locn:address ?addressContact.
-          ?addressContact ?pAddressContact ?oAddressContact.
-        }
-        UNION
-        {
-          ?site org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-        }
-        UNION
-        {
-          ?site organisatie:bestaatUit ?address.
-          ?address ?pAddress ?oAddress.
-        }     
-      }   
-    }
-  `, undefined, endpoint)
-
-  // Move primary sites worships
-  await muUpdate(`
-    ${prefixes}
-    INSERT {
-      GRAPH <http://mu.semte.ch/graphs/worship-service> {
-        ?adminUnit org:hasPrimarySite ?site.
-        ?site ?pSite ?oSite.
-        ?contact ?pContact ?oContact.
-        ?address ?pAddress ?oAddress.
-        ?addressContact ?pAddressContact ?oAddressContact.
-      }
-    } WHERE {
-      GRAPH <http://mu.semte.ch/graphs/worship-service> {
-        ?adminUnit a ?type.
-      }
-      GRAPH <${LANDING_ZONE_GRAPH}> {
-        ?adminUnit org:hasPrimarySite ?site.
-        ?site ?pSite ?oSite.
-        {
-          ?site org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-          ?contact locn:address ?addressContact.
-          ?addressContact ?pAddressContact ?oAddressContact.
-        }
-        UNION
-        {
-          ?site org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-        }
-        UNION
-        {
-          ?site organisatie:bestaatUit ?address.
-          ?address ?pAddress ?oAddress.
-        }        
-      }
-    }
-  `, undefined, endpoint)
-
-  // Move sites worships
-  await muUpdate(`
-    ${prefixes}
-    INSERT {
-      GRAPH <http://mu.semte.ch/graphs/worship-service> {
-        ?adminUnit org:hasSite ?site.
-        ?site ?pSite ?oSite.
-        ?contact ?pContact ?oContact.
-        ?address ?pAddress ?oAddress.
-        ?addressContact ?pAddressContact ?oAddressContact.
-      }
-    } WHERE {
-      GRAPH <http://mu.semte.ch/graphs/worship-service> {
-        ?adminUnit a ?type.
-      }
-      GRAPH <${LANDING_ZONE_GRAPH}> {
-        ?adminUnit org:hasSite ?site.
-        ?site ?pSite ?oSite.
-        {
-          ?site org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-          ?contact locn:address ?addressContact.
-          ?addressContact ?pAddressContact ?oAddressContact.
-        }
-        UNION
-        {
-          ?site org:siteAddress ?contact.
-          ?contact ?pContact ?oContact.
-        }
-        UNION
-        {
-          ?site organisatie:bestaatUit ?address.
-          ?address ?pAddress ?oAddress.
-        }        
-      }
-    }
-  `, undefined, endpoint)
-
-  // Move sites worships
-  await muUpdate(`
-    ${prefixes}
-    INSERT {
-      GRAPH <http://mu.semte.ch/graphs/public> {
-        ?type a <http://lblod.data.gift/vocabularies/organisatie/TypeVestiging>.
-        ?type ?a ?b.
-      }
-    } WHERE {
-      GRAPH <${LANDING_ZONE_GRAPH}> {
-        ?type a <http://lblod.data.gift/vocabularies/organisatie/TypeVestiging>.
-        ?type ?a ?b.
-      }
-    }
-  `, undefined, endpoint)
-}
-
-module.exports = {
-  batchedDbUpdate,
-  moveDataAfterInitialSync,
-  insertIntoPublicGraph,
-  deleteFromPublicGraph,
-  insertIntoSpecificGraphs,
-  deleteFromSpecificGraphs,
-};

config/dev-migrations/20241003152155-remove-invalid-generic-memberships.sparql

@@ -0,0 +1,31 @@
+PREFIX org: <http://www.w3.org/ns/org#>
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?membership ?p ?o .
+  }
+} WHERE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?membership a org:Membership ;
+    org:member ?member ;
+    org:organization ?organization ;
+    org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> ; # has a relation with
+    ?p ?o .
+
+    ?organization org:classification ?organizationClass .
+    FILTER (?organizationClass IN
+            (
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/ea446861-2c51-45fa-afd3-4e4a37b71562> , # HVZ
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/80310756-ce0a-4a1b-9b8e-7c01b6cc7a2d> , # APB
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/d01bb1f6-2439-4e33-9c25-1fc295de2e71> , # Dienstverlenende vereniging
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/cd93f147-3ece-4308-acab-5c5ada3ec63d> , # Opdrachthoudende vereniging
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/4b8450cf-a326-4c66-9e63-b4ec10acc7f6> , # Opdrachthoudende vereniging met private deelname
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/b156b67f-c5f4-4584-9b30-4c090be02fdc> , # Projectvereniging
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/a3922c6d-425b-474f-9a02-ffb71a436bfc> , # Politiezone
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/34b5af85-dc9f-468f-9e03-ef89b174c267> , # Autonome verzorgingsinstelling
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/36a82ba0-7ff1-4697-a9dd-2e94df73b721> , # AGB
+             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/e8294b73-87c9-4fa2-9441-1937350763c9> # Welzijnsvereniging
+             )
+            )
+  }
+}

config/dispatcher/dispatcher.ex

@@ -1,364 +1,387 @@
 defmodule Dispatcher do
   use Matcher
 
-  define_accept_types [
+  define_accept_types(
     html: ["text/html", "application/xhtml+html"],
     json: ["application/json", "application/vnd.api+json"],
     upload: ["multipart/form-data"],
     sparql_json: ["application/sparql-results+json"],
-    any: [ "*/*" ],
+    any: ["*/*"]
-  ]
+  )
-
-  define_layers [ :api_services, :api, :frontend, :not_found ]
 
+  define_layers([:api_services, :api, :frontend, :not_found])
 
   ###############################################################
   # domain.json
   ###############################################################
 
-  match "/people/*path", %{ accept: [:json], layer: :api} do
+  match "/people/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/people/"
+    Proxy.forward(conn, path, "http://cache/people/")
   end
 
-  match "/agents-in-position/*path", %{ accept: [:json], layer: :api} do
+  match "/agents-in-position/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/agents-in-position/"
+    Proxy.forward(conn, path, "http://cache/agents-in-position/")
   end
 
-  match "/posts/*path", %{ accept: [:json], layer: :api} do
+  match "/posts/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/posts/"
+    Proxy.forward(conn, path, "http://cache/posts/")
   end
 
-  match "/mandatories/*path", %{ accept: [:json], layer: :api} do
+  match "/mandatories/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/mandatories/"
+    Proxy.forward(conn, path, "http://cache/mandatories/")
   end
 
-  match "/mandatory-status-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/mandatory-status-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/mandatory-status-codes/"
+    Proxy.forward(conn, path, "http://cache/mandatory-status-codes/")
   end
 
-  match "/worship-mandatories/*path", %{ accept: [:json], layer: :api} do
+  match "/worship-mandatories/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/worship-mandatories/"
+    Proxy.forward(conn, path, "http://cache/worship-mandatories/")
   end
 
-  match "/half-elections/*path", %{ accept: [:json], layer: :api} do
+  match "/half-elections/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/half-elections/"
+    Proxy.forward(conn, path, "http://cache/half-elections/")
   end
 
-  match "/contact-points/*path", %{ accept: [:json], layer: :api} do
+  match "/contact-points/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/contact-points/"
+    Proxy.forward(conn, path, "http://cache/contact-points/")
   end
 
-  match "/mandates/*path", %{ accept: [:json], layer: :api} do
+  match "/mandates/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/mandates/"
+    Proxy.forward(conn, path, "http://cache/mandates/")
   end
 
-  match "/board-positions/*path", %{ accept: [:json], layer: :api} do
+  match "/board-positions/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/board-positions/"
+    Proxy.forward(conn, path, "http://cache/board-positions/")
   end
 
-  match "/board-position-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/board-position-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/board-position-codes/"
+    Proxy.forward(conn, path, "http://cache/board-position-codes/")
   end
 
-  match "/functionaries/*path", %{ accept: [:json], layer: :api} do
+  match "/functionaries/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/functionaries/"
+    Proxy.forward(conn, path, "http://cache/functionaries/")
   end
 
-  match "/agent-status-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/agent-status-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/agent-status-codes/"
+    Proxy.forward(conn, path, "http://cache/agent-status-codes/")
   end
 
-  match "/roles/*path", %{ accept: [:json], layer: :api} do
+  match "/roles/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/roles/"
+    Proxy.forward(conn, path, "http://cache/roles/")
   end
 
-  match "/organizations/*path", %{ accept: [:json], layer: :api} do
+  match "/organizations/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/organizations/"
+    Proxy.forward(conn, path, "http://cache/organizations/")
   end
 
-  match "/registered-organizations/*path", %{ accept: [:json], layer: :api} do
+  match "/registered-organizations/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/registered-organizations/"
+    Proxy.forward(conn, path, "http://cache/registered-organizations/")
   end
 
-  match "/administrative-units/*path", %{ accept: [:json], layer: :api} do
+  match "/administrative-units/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/administrative-units/"
+    Proxy.forward(conn, path, "http://cache/administrative-units/")
   end
 
-  match "/administrative-unit-classification-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/administrative-unit-classification-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/administrative-unit-classification-codes/"
+    Proxy.forward(conn, path, "http://cache/administrative-unit-classification-codes/")
   end
 
-  match "/organization-classification-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/organization-classification-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/organization-classification-codes/"
+    Proxy.forward(conn, path, "http://cache/organization-classification-codes/")
   end
 
-  match "/registered-organization-classification-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/registered-organization-classification-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/registered-organization-classification-codes/"
+    Proxy.forward(conn, path, "http://cache/registered-organization-classification-codes/")
   end
 
-  match "/worship-administrative-units/*path", %{ accept: [:json], layer: :api} do
+  match "/worship-administrative-units/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/worship-administrative-units/"
+    Proxy.forward(conn, path, "http://cache/worship-administrative-units/")
   end
 
-  match "/worship-services/*path", %{ accept: [:json], layer: :api} do
+  match "/worship-services/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/worship-services/"
+    Proxy.forward(conn, path, "http://cache/worship-services/")
   end
 
-  match "/recognized-worship-types/*path", %{ accept: [:json], layer: :api} do
+  match "/recognized-worship-types/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/recognized-worship-types/"
+    Proxy.forward(conn, path, "http://cache/recognized-worship-types/")
   end
 
-  match "/central-worship-services/*path", %{ accept: [:json], layer: :api} do
+  match "/central-worship-services/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/central-worship-services/"
+    Proxy.forward(conn, path, "http://cache/central-worship-services/")
   end
 
-  match "/representative-bodies/*path", %{ accept: [:json], layer: :api} do
+  match "/representative-bodies/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/representative-bodies/"
+    Proxy.forward(conn, path, "http://cache/representative-bodies/")
   end
 
-  match "/governing-bodies/*path", %{ accept: [:json], layer: :api} do
+  match "/governing-bodies/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/governing-bodies/"
+    Proxy.forward(conn, path, "http://cache/governing-bodies/")
   end
 
-  match "/governing-body-classification-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/governing-body-classification-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/governing-body-classification-codes/"
+    Proxy.forward(conn, path, "http://cache/governing-body-classification-codes/")
   end
 
-  match "/local-involvements/*path", %{ accept: [:json], layer: :api} do
+  match "/local-involvements/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/local-involvements/"
+    Proxy.forward(conn, path, "http://cache/local-involvements/")
   end
 
-  match "/identifiers/*path", %{ accept: [:json], layer: :api} do
+  match "/identifiers/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/identifiers/"
+    Proxy.forward(conn, path, "http://cache/identifiers/")
   end
 
-  match "/structured-identifiers/*path", %{ accept: [:json], layer: :api} do
+  match "/structured-identifiers/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/structured-identifiers/"
+    Proxy.forward(conn, path, "http://cache/structured-identifiers/")
   end
 
-  match "/addresses/*path", %{ accept: [:json], layer: :api} do
+  match "/addresses/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/addresses/"
+    Proxy.forward(conn, path, "http://cache/addresses/")
   end
 
-  match "/sites/*path", %{ accept: [:json], layer: :api} do
+  match "/sites/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/sites/"
+    Proxy.forward(conn, path, "http://cache/sites/")
   end
 
-  match "/change-events/*path", %{ accept: [:json], layer: :api} do
+  match "/change-events/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/change-events/"
+    Proxy.forward(conn, path, "http://cache/change-events/")
   end
 
-  match "/change-event-types/*path", %{ accept: [:json], layer: :api} do
+  match "/change-event-types/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/change-event-types/"
+    Proxy.forward(conn, path, "http://cache/change-event-types/")
   end
 
-  match "/change-event-results/*path", %{ accept: [:json], layer: :api} do
+  match "/change-event-results/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/change-event-results/"
+    Proxy.forward(conn, path, "http://cache/change-event-results/")
   end
 
-  match "/organization-status-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/organization-status-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/organization-status-codes/"
+    Proxy.forward(conn, path, "http://cache/organization-status-codes/")
   end
 
-  match "/locations/*path", %{ accept: [:json], layer: :api} do
+  match "/locations/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/locations/"
+    Proxy.forward(conn, path, "http://cache/locations/")
   end
 
-  match "/involvement-types/*path", %{ accept: [:json], layer: :api} do
+  match "/involvement-types/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/involvement-types/"
+    Proxy.forward(conn, path, "http://cache/involvement-types/")
   end
 
-  match "/ministers/*path", %{ accept: [:json], layer: :api} do
+  match "/ministers/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/ministers/"
+    Proxy.forward(conn, path, "http://cache/ministers/")
   end
 
-  match "/minister-conditions/*path", %{ accept: [:json], layer: :api} do
+  match "/minister-conditions/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/minister-conditions/"
+    Proxy.forward(conn, path, "http://cache/minister-conditions/")
   end
 
-  match "/dates-of-birth/*path", %{ accept: [:json], layer: :api} do
+  match "/dates-of-birth/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/dates-of-birth/"
+    Proxy.forward(conn, path, "http://cache/dates-of-birth/")
   end
 
-  match "/nationalities/*path", %{ accept: [:json], layer: :api} do
+  match "/nationalities/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/nationalities/"
+    Proxy.forward(conn, path, "http://cache/nationalities/")
   end
 
-  match "/gender-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/gender-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/gender-codes/"
+    Proxy.forward(conn, path, "http://cache/gender-codes/")
-  end
-  match "/concepts/*path", %{ accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/concepts/"
   end
 
-  match "/minister-positions/*path", %{ accept: [:json], layer: :api} do
+  match "/concepts/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/minister-positions/"
+    Proxy.forward(conn, path, "http://cache/concepts/")
   end
 
-  match "/minister-position-functions/*path", %{ accept: [:json], layer: :api} do
+  match "/minister-positions/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/minister-position-functions/"
+    Proxy.forward(conn, path, "http://cache/minister-positions/")
   end
 
-  match "/financing-codes/*path", %{ accept: [:json], layer: :api} do
+  match "/minister-position-functions/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/financing-codes/"
+    Proxy.forward(conn, path, "http://cache/minister-position-functions/")
   end
 
-  match "/minister-condition-criterions/*path", %{ accept: [:json], layer: :api} do
+  match "/financing-codes/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/minister-condition-criterions/"
+    Proxy.forward(conn, path, "http://cache/financing-codes/")
   end
 
-  match "/document-types-criterions/*path", %{ accept: [:json], layer: :api} do
+  match "/minister-condition-criterions/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/document-types-criterions/"
+    Proxy.forward(conn, path, "http://cache/minister-condition-criterions/")
   end
 
-  match "/site-types/*path", %{ accept: [:json], layer: :api} do
+  match "/document-types-criterions/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/site-types/"
+    Proxy.forward(conn, path, "http://cache/document-types-criterions/")
   end
 
-  match "/request-reasons/*path", %{ accept: [:json], layer: :api} do
+  match "/site-types/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/request-reasons/"
+    Proxy.forward(conn, path, "http://cache/site-types/")
   end
 
-  match "/decisions/*path", %{ accept: [:json], layer: :api} do
+  match "/request-reasons/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/decisions/"
+    Proxy.forward(conn, path, "http://cache/request-reasons/")
   end
 
-  match "/decision-activities/*path", %{ accept: [:json], layer: :api} do
+  match "/decisions/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/decision-activities/"
+    Proxy.forward(conn, path, "http://cache/decisions/")
+  end
+
+  match "/decision-activities/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://cache/decision-activities/")
+  end
+
+  match "/memberships/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://cache/memberships/")
+  end
+
+  match "/membership-roles/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://cache/membership-roles/")
   end
 
   # There is currently no logic to invalidate the kboOrganization cache when wegwijse updates the content
-  match "/kbo-organizations/*path", %{ accept: [:any], layer: :api} do
+  match "/kbo-organizations/*path", %{accept: [:any], layer: :api} do
-    Proxy.forward conn, path, "http://resource/kbo-organizations/"
+    Proxy.forward(conn, path, "http://resource/kbo-organizations/")
   end
 
   ###############
   # LOGIN
   ###############
 
-  match "/accounts", %{ accept: [:json], layer: :api} do
+  match "/accounts", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, [], "http://resource/accounts/"
+    Proxy.forward(conn, [], "http://resource/accounts/")
-  end
-  match "/accounts/*path", %{ accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://accountdetail/accounts/"
-  end
-  match "/active-role/*path", %{ accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://roles/"
-  end
-  match "/groups/*path", %{ accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://resource/groups/"
   end
 
-  match "/mock/sessions/*path", %{ accept: [:any], layer: :api} do
+  match "/accounts/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://mocklogin/sessions/"
+    Proxy.forward(conn, path, "http://accountdetail/accounts/")
   end
 
-  match "/sessions/*path", %{ accept: [:any], layer: :api} do
+  match "/active-role/*path", %{accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://login/sessions/"
+    Proxy.forward(conn, path, "http://roles/")
+  end
+
+  match "/groups/*path", %{accept: [:json], layer: :api} do
+    Proxy.forward(conn, path, "http://resource/groups/")
+  end
+
+  match "/mock/sessions/*path", %{accept: [:any], layer: :api} do
+    Proxy.forward(conn, path, "http://mocklogin/sessions/")
+  end
+
+  match "/sessions/*path", %{accept: [:any], layer: :api} do
+    Proxy.forward(conn, path, "http://login/sessions/")
   end
 
   ###############################################################
   # SEARCH
   ###############################################################
 
-  match "/search/*path", %{  accept: %{ json: true }, layer: :api_services} do
+  match "/search/*path", %{accept: %{json: true}, layer: :api_services} do
-    Proxy.forward conn, path, "http://search/"
+    Proxy.forward(conn, path, "http://search/")
   end
 
   ###############
   # API SERVICES
   ###############
 
-  match "/v3/api-docs/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/v3/api-docs/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://kalliope-api/v3/api-docs/"
+    forward(conn, path, "http://kalliope-api/v3/api-docs/")
   end
 
-  match "/consolidated/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/consolidated/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://kalliope-api/consolidated/"
+    forward(conn, path, "http://kalliope-api/consolidated/")
   end
 
-  match "/adresses-register/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/adresses-register/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://adressenregister/"
+    forward(conn, path, "http://adressenregister/")
   end
 
-  match "/worship/person-information-updates/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/worship/person-information-updates/*path", %{
-    forward conn, path, "http://privacy-worship/person-information-updates"
+    layer: :api_services,
+    accept: %{json: true}
+  } do
+    forward(conn, path, "http://privacy-worship/person-information-updates")
   end
 
-  match "/worship/person-information-requests/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/worship/person-information-requests/*path", %{
-    forward conn, path, "http://privacy-worship/person-information-requests"
+    layer: :api_services,
+    accept: %{json: true}
+  } do
+    forward(conn, path, "http://privacy-worship/person-information-requests")
   end
 
-  match "/worship/person-information-ask/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/worship/person-information-ask/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://privacy-worship/person-information-ask/"
+    forward(conn, path, "http://privacy-worship/person-information-ask/")
   end
 
-  match "/worship/person-information-validate-ssn/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/worship/person-information-validate-ssn/*path", %{
-    forward conn, path, "http://privacy-worship/person-information-validate-ssn/"
+    layer: :api_services,
+    accept: %{json: true}
+  } do
+    forward(conn, path, "http://privacy-worship/person-information-validate-ssn/")
   end
 
-  match "/person-information-updates/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/person-information-updates/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://privacy-unit/person-information-updates"
+    forward(conn, path, "http://privacy-unit/person-information-updates")
   end
 
-  match "/person-information-requests/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/person-information-requests/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://privacy-unit/person-information-requests"
+    forward(conn, path, "http://privacy-unit/person-information-requests")
   end
 
-  match "/person-information-ask/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/person-information-ask/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://privacy-unit/person-information-ask/"
+    forward(conn, path, "http://privacy-unit/person-information-ask/")
   end
 
-  match "/person-information-validate-ssn/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/person-information-validate-ssn/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://privacy-unit/person-information-validate-ssn/"
+    forward(conn, path, "http://privacy-unit/person-information-validate-ssn/")
   end
 
-  get "/uri-info/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  get "/uri-info/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://uri-info/"
+    forward(conn, path, "http://uri-info/")
   end
 
-  post "/construct-organization-relationships/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  post "/construct-organization-relationships/*path", %{
-    forward conn, path, "http://construct-organization-relationships/create-relationships/"
+    layer: :api_services,
+    accept: %{json: true}
+  } do
+    forward(conn, path, "http://construct-organization-relationships/create-relationships/")
   end
 
-  post "/kbo-data-sync/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  post "/kbo-data-sync/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://kbo-data-sync/sync-kbo-data/"
+    forward(conn, path, "http://kbo-data-sync/sync-kbo-data/")
   end
 
-  get "/ldes/*path", %{ layer: :api_services } do
+  get "/ldes/*path", %{layer: :api_services} do
-    forward conn, path, "http://ldes-backend/"
+    forward(conn, path, "http://ldes-backend/")
   end
 
   #################################################################
   # ERROR REPORT
   #################################################################
 
-  match "/error-reports/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  match "/error-reports/*path", %{layer: :api_services, accept: %{json: true}} do
-    forward conn, path, "http://error-report-service/error-reports/"
+    forward(conn, path, "http://error-report-service/error-reports/")
   end
 
   #################################################################
   # FILES
   #################################################################
 
-  get "/files/:id/download", %{ accept: [:any], } do
+  get "/files/:id/download", %{accept: [:any]} do
-    Proxy.forward conn, [], "http://file/files/" <> id <> "/download"
+    Proxy.forward(conn, [], "http://file/files/" <> id <> "/download")
   end
 
-  get "/files/*path" , %{ layer: :api_services, accept: %{ json: true } } do
+  get "/files/*path", %{layer: :api_services, accept: %{json: true}} do
-    Proxy.forward conn, path, "http://resource/files/"
+    Proxy.forward(conn, path, "http://resource/files/")
   end
 
   #################################################################
   # DCAT
   #################################################################
 
-  get "/datasets/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  get "/datasets/*path", %{layer: :api_services, accept: %{json: true}} do
-    Proxy.forward conn, path, "http://resource/datasets/"
+    Proxy.forward(conn, path, "http://resource/datasets/")
   end
 
-  get "/distributions/*path", %{ layer: :api_services, accept: %{ json: true } } do
+  get "/distributions/*path", %{layer: :api_services, accept: %{json: true}} do
-    Proxy.forward conn, path, "http://resource/distributions/"
+    Proxy.forward(conn, path, "http://resource/distributions/")
   end
 
   #################################################################
@@ -398,44 +421,42 @@ defmodule Dispatcher do
   #################################################################
 
   get "/download-exports/*path" do
-    Proxy.forward conn, path, "http://download-exports/"
+    Proxy.forward(conn, path, "http://download-exports/")
   end
 
   ###############################################################
   # frontend layer
   ###############################################################
 
-  match "/assets/*path", %{ layer: :api } do
+  match "/assets/*path", %{layer: :api} do
-    Proxy.forward conn, path, "http://frontend/assets/"
+    Proxy.forward(conn, path, "http://frontend/assets/")
   end
 
-  match "/@appuniversum/*path", %{ layer: :api } do
+  match "/@appuniversum/*path", %{layer: :api} do
-    Proxy.forward conn, path, "http://frontend/@appuniversum/"
+    Proxy.forward(conn, path, "http://frontend/@appuniversum/")
   end
 
-  match "/*path", %{ accept: [:html], layer: :api } do
+  match "/*path", %{accept: [:html], layer: :api} do
-    Proxy.forward conn, [], "http://frontend/index.html"
+    Proxy.forward(conn, [], "http://frontend/index.html")
   end
 
-  match "/*_path", %{ layer: :frontend } do
+  match "/*_path", %{layer: :frontend} do
-    Proxy.forward conn, [], "http://frontend/index.html"
+    Proxy.forward(conn, [], "http://frontend/index.html")
   end
 
   ###############################################################
   # sparql endpoint
   ###############################################################
 
-  post "/sparql/*path", %{ layer: :api_services, accept: %{ sparql_json: true } } do
+  post "/sparql/*path", %{layer: :api_services, accept: %{sparql_json: true}} do
-    forward conn, path, "http://db:8890/sparql/"
+    forward(conn, path, "http://db:8890/sparql/")
   end
 
   ###############################################################
   # errors
   ###############################################################
 
-  match "/*_path", %{ accept: [:any], layer: :not_found} do
+  match "/*_path", %{accept: [:any], layer: :not_found} do
-    send_resp( conn, 404, "{\"error\": {\"code\": 404}")
+    send_resp(conn, 404, "{\"error\": {\"code\": 404}")
   end
-
-
 end

config/loket-consumer/leidinggevenden-dispatching/config.js

@@ -1,32 +0,0 @@
-const BATCH_SIZE = parseInt(process.env.BATCH_SIZE) || 100;
-const MU_CALL_SCOPE_ID_INITIAL_SYNC = process.env.MU_CALL_SCOPE_ID_INITIAL_SYNC || 'http://redpencil.data.gift/id/concept/muScope/deltas/consumer/initialSync';
-const BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES = process.env.BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES == 'true' ? true : false;
-const DIRECT_DATABASE_ENDPOINT = process.env.DIRECT_DATABASE_ENDPOINT || 'http://triplestore:8890/sparql';
-const MAX_DB_RETRY_ATTEMPTS = parseInt(process.env.MAX_DB_RETRY_ATTEMPTS || 5);
-const MAX_REASONING_RETRY_ATTEMPTS = parseInt(process.env.MAX_REASONING_RETRY_ATTEMPTS || 5);
-const SLEEP_BETWEEN_BATCHES = parseInt(process.env.SLEEP_BETWEEN_BATCHES || 1000);
-const SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION || 10000);
-const SLEEP_TIME_AFTER_FAILED_DB_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_DB_OPERATION || 60000);
-
-// Graphs
-const TARGET_GRAPH = process.env.TARGET_GRAPH || `http://mu.semte.ch/graphs/administrative-unit`;
-const LANDING_ZONE_GRAPH = process.env.DCR_LANDING_ZONE_GRAPH || `http://mu.semte.ch/graphs/landing-zone/leidinggevenden`;
-
-// ENDPOINTS
-const LANDING_ZONE_DATABASE = process.env.DCR_LANDING_ZONE_DATABASE || 'db';
-const LANDING_ZONE_DATABASE_ENDPOINT = process.env.DCR_LANDING_ZONE_DATABASE_ENDPOINT || `http://${LANDING_ZONE_DATABASE}:8890/sparql`;
-
-module.exports = {
-  BATCH_SIZE,
-  MU_CALL_SCOPE_ID_INITIAL_SYNC,
-  BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
-  DIRECT_DATABASE_ENDPOINT,
-  MAX_DB_RETRY_ATTEMPTS,
-  MAX_REASONING_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
-  LANDING_ZONE_GRAPH,
-  TARGET_GRAPH,
-  LANDING_ZONE_DATABASE_ENDPOINT
-};

config/loket-consumer/leidinggevenden-dispatching/delta-context-config.js

@@ -1,72 +0,0 @@
-
-const PREFIXES = `
-PREFIX org: <http://www.w3.org/ns/org#>
-PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
-PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
-PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>`
-
-const contextConfig = {
-  addTypes: {
-    scope: 'all', // 'inserts, 'deletes', 'all' or none. To add rdf:type to subjects of inserts, deletes or both
-    exhausitive: false, // true or false: find all types for a subject, even if one is already present in delta
-  },
-  contextQueries: [
-    // Case Leidnggevenden
-    // They can be triggered when being inserted or when their link to a bestuursfunctie is added
-    {
-      trigger: { // subjectType or predicateValue
-        subjectType: "lblodlg:Functionaris"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ?functionaris org:holds ?bestuursfunctie ;
-            ?p ?o .
-          ?orgaanInTime lblodlg:heeftBestuursfunctie ?bestuursfunctie ;
-            mandaat:isTijdspecialisatieVan ?orgaan .
-          ?orgaan besluit:bestuurt ?eenheid .
-          ?eenheid besluit:classificatie ?classification .
-        } WHERE {
-          GRAPH <http://mu.semte.ch/graphs/landing-zone/leidinggevenden> {
-            VALUES ?functionaris { ${subject} }
-            ?functionaris org:holds ?bestuursfunctie ;
-              ?p ?o .
-            ?orgaanInTime lblodlg:heeftBestuursfunctie ?bestuursfunctie ;
-              mandaat:isTijdspecialisatieVan ?orgaan .
-            ?orgaan besluit:bestuurt ?eenheid .
-            ?eenheid besluit:classificatie ?classification .
-          }
-        }`
-    },
-    {
-      trigger: {
-        predicateValue: "org:holds"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ?functionaris org:holds ?bestuursfunctie ;
-            ?p ?o .
-          ?orgaanInTime lblodlg:heeftBestuursfunctie ?bestuursfunctie ;
-            mandaat:isTijdspecialisatieVan ?orgaan .
-          ?orgaan besluit:bestuurt ?eenheid .
-          ?eenheid besluit:classificatie ?classification .
-        } WHERE {
-          GRAPH <http://mu.semte.ch/graphs/landing-zone/leidinggevenden> {
-            VALUES ?functionaris { ${subject} }
-            ?functionaris org:holds ?bestuursfunctie ;
-              ?p ?o .
-            ?orgaanInTime lblodlg:heeftBestuursfunctie ?bestuursfunctie ;
-              mandaat:isTijdspecialisatieVan ?orgaan .
-            ?orgaan besluit:bestuurt ?eenheid .
-            ?eenheid besluit:classificatie ?classification .
-          }
-        }`
-    }
-  ]
-}
-
-module.exports = {
-  contextConfig,
-  PREFIXES
-};

config/loket-consumer/leidinggevenden-dispatching/delta-sync-dispatching.js

@@ -1,85 +0,0 @@
-const {
-  transformStatements,
-  deleteFromTargetGraph,
-  insertIntoTargetGraph,
-} = require('./util');
-
-/**
-* Dispatch the fetched information to a target graph.
-* Note: <share://file/data> will be ADDED to it's own graph.
-*   We take only care of adding them, not updating triples, this is a TODO
-* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
-* @param { termObjectChangeSets: { deletes, inserts } } data - The fetched changes sets, which objects of serialized Terms
-*          [ {
-*              graph: "<http://foo>",
-*              subject: "<http://bar>",
-*              predicate: "<http://baz>",
-*              object: "<http://boom>^^<http://datatype>"
-*            }
-*         ]
-* @return {void} Nothing
-*/
-async function dispatch(lib, data) {
-  const { mu, fetch } = lib;
-  let { termObjectChangeSets, termObjectChangeSetsWithContext } = data;
-
-  // Both arrays are the same length, so we can zip them together for easier processing
-  zippedChangeSets = termObjectChangeSets.map((o, i) => ({
-    original: o,
-    withContext: termObjectChangeSetsWithContext[i]
-  }));
-
-  console.log(`Received ${zippedChangeSets.length} change sets`)
-
-  for (let { original, withContext } of zippedChangeSets) {
-    let originalDeletes, originalInserts, insertsWithContext, deletesWithContext;
-    try {
-      originalInserts = original.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-      originalDeletes = original.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-    } catch (e) {
-      console.log(`Error while processing change set: ${e}`);
-      console.log(`Original change set: ${JSON.stringify(original)}`);
-      throw e;
-    }
-
-    // Extra context needed for mapping from DL to OP model and filtering based on type.
-    try {
-      insertsWithContext = withContext.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-      deletesWithContext = withContext.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-    } catch (e) {
-      console.log(`Error while processing change set with context: ${e}`);
-      console.log(`Change set with context: ${JSON.stringify(withContext)}`);
-      throw e;
-    }
-
-    if (originalDeletes.length) {
-      // Map deletes from DL to OP model
-      const transformedDeletes = await transformStatements(fetch, deletesWithContext);
-
-      if (!transformedDeletes.length) {
-        console.log(`Warn: Delete statements mapped to empty result.`);
-        console.log(`Input: ${deletesWithContext}`);
-        console.log(`Output: ${transformedDeletes}`);
-      } else {
-        await deleteFromTargetGraph(lib, transformedDeletes);
-      }
-    }
-
-    if (originalInserts.length) {
-      // Map inserts from DL to OP model
-      const transformedInserts = await transformStatements(fetch, insertsWithContext);
-
-      if (!transformedInserts.length) {
-        console.log(`Warn: Insert statements mapped to empty result.`);
-        console.log(`Input: ${insertsWithContext}`);
-        console.log(`Output: ${transformedInserts}`);
-      } else {
-        await insertIntoTargetGraph(lib, transformedInserts);
-      }
-    }
-  }
-}
-
-module.exports = {
-  dispatch
-};

config/loket-consumer/leidinggevenden-dispatching/initial-sync-dispatching.js

@@ -1,85 +0,0 @@
-const { batchedDbUpdate, transformLandingZoneGraph } = require('./util');
-const {
-  BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
-  DIRECT_DATABASE_ENDPOINT,
-  MU_CALL_SCOPE_ID_INITIAL_SYNC,
-  BATCH_SIZE,
-  MAX_DB_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  TARGET_GRAPH,
-  LANDING_ZONE_GRAPH,
-  LANDING_ZONE_DATABASE_ENDPOINT
-} = require('./config');
-
-const endpoint = BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES ? DIRECT_DATABASE_ENDPOINT : process.env.MU_SPARQL_ENDPOINT;
-
-/**
-* Dispatch the fetched information to a target graph.
-* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
-* @param { termObjects } data - The fetched quad information, which objects of serialized Terms
-*          [ {
-*              graph: "<http://foo>",
-*              subject: "<http://bar>",
-*              predicate: "<http://baz>",
-*              object: "<http://boom>^^<http://datatype>"
-*            }
-*         ]
-* @return {void} Nothing
-*/
-async function dispatch(lib, data) {
-  const { mu, muAuthSudo } = lib;
-  const { termObjects } = data;
-
-  // Steps:
-  // 1. Write the triples to the landing zone graph - withouth any mapping or filtering
-  // 2. One-time reasoning run with the full landing zone graph and write the results to the target graph (on-finish)
-
-  if (BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES) {
-    console.warn(`Service configured to skip MU_AUTH!`);
-  }
-  console.log(`Using ${endpoint} to insert triples`);
-
-  if (termObjects.length) {
-    const originalInsertTriples = termObjects.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-
-    await batchedDbUpdate(
-      muAuthSudo.updateSudo,
-      LANDING_ZONE_GRAPH,
-      originalInsertTriples,
-      { 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
-      LANDING_ZONE_DATABASE_ENDPOINT,
-      BATCH_SIZE,
-      MAX_DB_RETRY_ATTEMPTS,
-      SLEEP_BETWEEN_BATCHES,
-      SLEEP_TIME_AFTER_FAILED_DB_OPERATION
-    );
-  }
-}
-
-async function onFinishInitialIngest(lib) {
-  const { muAuthSudo, fetch } = lib;
-
-  console.log(`!! On-finish triggered !!`);
-
-  const transformedInsertTriples = await transformLandingZoneGraph(fetch, endpoint);
-
-  console.log(`Transformed ${transformedInsertTriples.length} triples`);
-
-  await batchedDbUpdate(
-    muAuthSudo.updateSudo,
-    TARGET_GRAPH,
-    transformedInsertTriples,
-    { 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
-    endpoint,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION
-  );
-}
-
-module.exports = {
-  dispatch,
-  onFinishInitialIngest
-};

config/loket-consumer/leidinggevenden-dispatching/util.js

@@ -1,168 +0,0 @@
-const {
-  MAX_REASONING_RETRY_ATTEMPTS,
-  SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
-  MAX_DB_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  TARGET_GRAPH,
-  LANDING_ZONE_GRAPH,
-  BATCH_SIZE,
-} = require('./config');
-
-async function batchedDbUpdate(
-  muUpdate,
-  graph,
-  triples,
-  extraHeaders,
-  endpoint,
-  batchSize,
-  maxAttempts,
-  sleepBetweenBatches = 1000,
-  sleepTimeOnFail = 1000,
-  operation = 'INSERT'
-) {
-
-  for (let i = 0; i < triples.length; i += batchSize) {
-    console.log(`Inserting triples in batch: ${i}-${i + batchSize}`);
-
-    const batch = triples.slice(i, i + batchSize).join('\n');
-
-    const insertCall = async () => {
-      await muUpdate(`
-${operation} DATA {
-GRAPH <${graph}> {
-${batch}
-}
-}
-`, extraHeaders, endpoint);
-    };
-
-    await operationWithRetry(insertCall, 0, maxAttempts, sleepTimeOnFail);
-
-    console.log(`Sleeping before next query execution: ${sleepBetweenBatches}`);
-    await new Promise(r => setTimeout(r, sleepBetweenBatches));
-  }
-}
-
-async function operationWithRetry(callback,
-  attempt,
-  maxAttempts,
-  sleepTimeOnFail) {
-  try {
-    if (typeof callback === "function")
-      return await callback();
-    else // Catch error from promise - not how I would do it normally, but allows re use of existing code.
-      return await callback;
-  }
-  catch (e) {
-    console.log(`Operation failed for ${callback.toString()}, attempt: ${attempt} of ${maxAttempts}`);
-    console.log(`Error: ${e}`);
-    console.log(`Sleeping ${sleepTimeOnFail} ms`);
-
-    if (attempt >= maxAttempts) {
-      console.log(`Max attempts reached for ${callback.toString()}, giving up`);
-      throw e;
-    }
-
-    await new Promise(r => setTimeout(r, sleepTimeOnFail));
-    return operationWithRetry(callback, ++attempt, maxAttempts, sleepTimeOnFail);
-  }
-}
-
-/**
- * Splits an array into two parts, a part that passes and a part that fails a predicate function.
- * Credits: https://github.com/benjay10
- * @public
- * @function partition
- * @param {Array} arr - Array to be partitioned
- * @param {Function} fn - Function that accepts single argument: an element of the array, and should return a truthy or falsy value.
- * @returns {Object} Object that contains keys passes and fails, each representing an array with elemets that pass or fail the predicate respectively
- */
-function partition(arr, fn) {
-  let passes = [], fails = [];
-  arr.forEach((item) => (fn(item) ? passes : fails).push(item));
-  return { passes, fails };
-}
-
-/**
- * Send triples to reasoning service for conversion
- *
- */
-async function transformTriples(fetch, triples, mapping) {
-  return await operationWithRetry(mainConversion(fetch, triples, mapping), 0,
-    MAX_REASONING_RETRY_ATTEMPTS, SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION);
-}
-
-async function mainConversion(fetch, triples, mapping) {
-  let formdata = new URLSearchParams();
-  formdata.append("data", triples);
-
-  let requestOptions = {
-    method: 'POST',
-    body: formdata,
-    redirect: 'follow'
-  };
-
-  const response = await fetch(`http://reasoner/reason/dl2op/${mapping}`, requestOptions)
-  return await response.text();
-}
-
-async function transformStatements(fetch, triples, mapping = 'main') {
-  console.log(`Received ${JSON.stringify(triples)} to transform`);
-  const transformedTriples = await transformTriples(fetch, triples.join('\n'), mapping);
-  statements = transformedTriples ? transformedTriples.replace(/\n{2,}/g, '').split('\n') : [];
-  console.log(`CONVERSION: FROM ${triples.length} triples to ${statements.length}`);
-  return statements;
-}
-
-async function deleteFromTargetGraph(lib, statements) {
-  console.log(`Deleting ${statements.length} statements from target graph`);
-  console.log(`Statements:  ${JSON.stringify(statements)}`)
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    TARGET_GRAPH,
-    statements,
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'DELETE');
-}
-
-async function insertIntoTargetGraph(lib, statements) {
-  console.log(`Inserting ${statements.length} statements into target graph`);
-  console.log(`Statements:  ${JSON.stringify(statements)}`)
-
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    TARGET_GRAPH,
-    statements,
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'INSERT');
-}
-
-async function transformLandingZoneGraph(fetch, endpoint, mapping = 'main') {
-  console.log(`Transforming landing zone graph: ${LANDING_ZONE_GRAPH}`);
-
-  const response = await fetch(`http://reasoner/reason/dl2op/${mapping}?data=${encodeURIComponent(`${endpoint}?default-graph-uri=&query=CONSTRUCT+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D+WHERE+%7B%0D%0A+GRAPH+<${LANDING_ZONE_GRAPH}>+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D%0D%0A%7D&should-sponge=&format=text%2Fplain&timeout=0&run=+Run+Query`)}`);
-  const text = await response.text();
-  const statements = text.replace(/\n{2,}/g, '').split('\n');
-
-  return statements;
-}
-
-module.exports = {
-  batchedDbUpdate,
-  partition,
-  transformStatements,
-  transformLandingZoneGraph,
-  deleteFromTargetGraph,
-  insertIntoTargetGraph
-};

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/data-cleanup/phone.rq

@@ -0,0 +1,21 @@
+PREFIX schema: <http://schema.org/>
+
+CONSTRUCT {
+  ?s schema:telephone ?cleanedPhoneNumber .
+} WHERE {
+  ?s schema:telephone ?phoneNumber .
+
+  # Remove all non digits
+  BIND(REPLACE(?phoneNumber, """[^+\\d]""", "") AS ?scrapeResult)
+
+  # Replace leading 00 with +
+  BIND(REPLACE(?scrapeResult, """^00""", "+") AS ?replacedZeroZero)
+
+  BIND(REPLACE(?replacedZeroZero, """^0""", "+32") AS ?defaultBe)
+
+  BIND(REPLACE(?defaultBe, """\\+320""", "+32") AS ?extendedDefaultBe)
+
+  BIND(REPLACE(?extendedDefaultBe, """^320""", "+32") AS ?finalResult)
+
+  BIND(IF(REGEX(?finalResult, """^\\+((32\\d{8})|(324\\d{8})|((?!32)\\d+))$"""), CONCAT("tel:", ?finalResult), ?phoneNumber) AS ?cleanedPhoneNumber)
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/datetime-format/mandaat-einde.rq

@@ -0,0 +1,11 @@
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+
+CONSTRUCT {
+  ?s mandaat:einde ?endDateTime .
+} WHERE {
+  ?s mandaat:einde ?endDate .
+
+  BIND(SUBSTR(STR(?endDate), 1, 10) as ?endDateWithoutTimezone)
+  BIND(STRDT(CONCAT(?endDateWithoutTimezone, 'T02:00:00'), xsd:dateTime) as ?endDateTime)
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/datetime-format/mandaat-start.rq

@@ -0,0 +1,11 @@
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+
+CONSTRUCT {
+  ?s mandaat:start ?startDateTime .
+} WHERE {
+  ?s mandaat:start ?startDate .
+
+  BIND(SUBSTR(STR(?startDate), 1, 10) as ?startDateWithoutTimezone)
+  BIND(STRDT(CONCAT(?startDateWithoutTimezone, 'T02:00:00'), xsd:dateTime) as ?startDateTime)
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/inheritance/Agent.rq

@@ -0,0 +1,14 @@
+PREFIX dc_terms: <http://purl.org/dc/terms/>
+PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+CONSTRUCT {
+  ?s a dc_terms:Agent .
+} WHERE {
+  ?s a ?type .
+
+  VALUES ?type {
+    lblodlg:Functionaris
+    person:Person
+  }
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/inheritance/AgentInPositie.rq

@@ -0,0 +1,12 @@
+PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
+PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
+
+CONSTRUCT {
+  ?s a ch:AgentInPositie .
+} WHERE {
+  ?s a ?type .
+
+  VALUES ?type {
+    lblodlg:Functionaris
+  }
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/inverse/holds-to-held-by.rq

@@ -0,0 +1,8 @@
+PREFIX org: <http://www.w3.org/ns/org#>
+
+CONSTRUCT {
+  ?o org:heldBy ?s.
+}
+WHERE {
+  ?s org:holds ?o.
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/pass/address.sparql

@@ -0,0 +1,10 @@
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://www.w3.org/ns/locn#Address>;
+    ?p ?o.
+
+  FILTER NOT EXISTS {
+    ?s ?p "" .
+  }
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/pass/bestuursfunctie.sparql

@@ -0,0 +1,7 @@
+CONSTRUCT {
+  ?s <http://schema.org/contactPoint> ?contactpoint.
+}
+WHERE {
+  ?s a <http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie>.
+  ?s <http://schema.org/contactPoint> ?contactpoint.
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/pass/contactpoint.sparql

@@ -0,0 +1,11 @@
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://schema.org/ContactPoint>;
+    ?p ?o.
+ FILTER (?p NOT IN (<http://schema.org/telephone>))
+
+ FILTER NOT EXISTS {
+   ?s ?p "" .
+ }
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/pass/functionaris.sparql

@@ -0,0 +1,14 @@
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://data.lblod.info/vocabularies/leidinggevenden/Functionaris>;
+    ?p ?o.
+
+  FILTER (?p NOT IN (mandaat:start, mandaat:einde))
+
+  FILTER NOT EXISTS {
+    ?s ?p "" .
+  }
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/pass/person.sparql

@@ -0,0 +1,16 @@
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://www.w3.org/ns/person#Person>;
+    ?p ?o.
+
+  FILTER (?p IN (
+    <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>,
+    <http://xmlns.com/foaf/0.1/familyName>,
+    <http://mu.semte.ch/vocabularies/core/uuid>
+    ))
+
+  FILTER NOT EXISTS {
+   ?s ?p "" .
+  }
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/predicate/person-foaf-name.sparql

@@ -0,0 +1,21 @@
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
+
+# First name: persoon:gebruikteVoornaam => foaf:givenName
+# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
+
+# We do both in one query with an OPTIONAL because they need to happen simultaneously
+# not to end up in a loop
+
+# Part 1
+
+CONSTRUCT {
+  ?s persoon:gebruikteVoornaam ?name .
+} WHERE {
+  ?s a person:Person;
+    foaf:name ?name .
+}

config/loket-consumer/leidinggevenden/mapping/queries/dl2op/predicate/person-gebruikte-voornaam.sparql

@@ -0,0 +1,21 @@
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
+
+# First name: persoon:gebruikteVoornaam => foaf:givenName
+# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
+
+# We do both in one query with an OPTIONAL because they need to happen simultaneously
+# not to end up in a loop
+
+# Part 2
+
+CONSTRUCT {
+  ?s foaf:givenName ?gebruikteVoornaam .
+} WHERE {
+  ?s a person:Person ;
+    persoon_loket:gebruikteVoornaam ?gebruikteVoornaam .
+}

config/loket-consumer/mandatarissen-dispatching/config.js

@@ -1,33 +0,0 @@
-const BATCH_SIZE = parseInt(process.env.BATCH_SIZE) || 100;
-const MU_CALL_SCOPE_ID_INITIAL_SYNC = process.env.MU_CALL_SCOPE_ID_INITIAL_SYNC || 'http://redpencil.data.gift/id/concept/muScope/deltas/consumer/initialSync';
-const BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES = process.env.BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES == 'true' ? true : false;
-const DIRECT_DATABASE_ENDPOINT = process.env.DIRECT_DATABASE_ENDPOINT || 'http://triplestore:8890/sparql';
-const MAX_DB_RETRY_ATTEMPTS = parseInt(process.env.MAX_DB_RETRY_ATTEMPTS || 5);
-const MAX_REASONING_RETRY_ATTEMPTS = parseInt(process.env.MAX_REASONING_RETRY_ATTEMPTS || 5);
-const SLEEP_BETWEEN_BATCHES = parseInt(process.env.SLEEP_BETWEEN_BATCHES || 1000);
-const SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION || 10000);
-const SLEEP_TIME_AFTER_FAILED_DB_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_DB_OPERATION || 60000);
-
-// Graphs
-const TARGET_GRAPH = process.env.TARGET_GRAPH || `http://mu.semte.ch/graphs/administrative-unit`;
-const LANDING_ZONE_GRAPH = process.env.DCR_LANDING_ZONE_GRAPH || `http://mu.semte.ch/graphs/landing-zone/leidinggevenden`;
-
-// ENDPOINTS
-const LANDING_ZONE_DATABASE = process.env.DCR_LANDING_ZONE_DATABASE || 'db';
-const LANDING_ZONE_DATABASE_ENDPOINT = process.env.DCR_LANDING_ZONE_DATABASE_ENDPOINT || `http://${LANDING_ZONE_DATABASE}:8890/sparql`;
-
-module.exports = {
-  BATCH_SIZE,
-  MU_CALL_SCOPE_ID_INITIAL_SYNC,
-  BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
-  DIRECT_DATABASE_ENDPOINT,
-  MAX_DB_RETRY_ATTEMPTS,
-  MAX_REASONING_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
-  LANDING_ZONE_GRAPH,
-  LANDING_ZONE_DATABASE_ENDPOINT,
-  TARGET_GRAPH
-};
-

config/loket-consumer/mandatarissen-dispatching/delta-context-config.js

@@ -1,72 +0,0 @@
-
-const PREFIXES = `
-PREFIX org: <http://www.w3.org/ns/org#>
-PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
-PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
-PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>`
-
-const contextConfig = {
-  addTypes: {
-    scope: 'all', // 'inserts, 'deletes', 'all' or none. To add rdf:type to subjects of inserts, deletes or both
-    exhausitive: false, // true or false: find all types for a subject, even if one is already present in delta
-  },
-  contextQueries: [
-    // Case Mandatarissen
-    // They can be triggered when being inserted or when their link to a mandate is added
-    {
-      trigger: { // subjectType or predicateValue
-        subjectType: "mandaat:Mandataris"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ?mandataris org:holds ?mandate ;
-            ?p ?o .
-          ?orgaanInTime org:hasPost ?mandate ;
-            mandaat:isTijdspecialisatieVan ?orgaan .
-          ?orgaan besluit:bestuurt ?eenheid .
-          ?eenheid besluit:classificatie ?classification .
-        } WHERE {
-          GRAPH <http://mu.semte.ch/graphs/landing-zone/mandatarissen> {
-            VALUES ?mandataris { ${subject} }
-            ?mandataris org:holds ?mandate ;
-              ?p ?o .
-            ?orgaanInTime org:hasPost ?mandate ;
-              mandaat:isTijdspecialisatieVan ?orgaan .
-            ?orgaan besluit:bestuurt ?eenheid .
-            ?eenheid besluit:classificatie ?classification .
-          }
-        }`
-    },
-    {
-      trigger: {
-        predicateValue: "org:holds"
-      },
-      queryTemplate: (subject) => `
-        ${PREFIXES}
-        CONSTRUCT {
-          ?mandataris org:holds ?mandate ;
-            ?p ?o .
-          ?orgaanInTime org:hasPost ?mandate ;
-            mandaat:isTijdspecialisatieVan ?orgaan .
-          ?orgaan besluit:bestuurt ?eenheid .
-          ?eenheid besluit:classificatie ?classification .
-        } WHERE {
-          GRAPH <http://mu.semte.ch/graphs/landing-zone/mandatarissen> {
-            VALUES ?mandataris { ${subject} }
-            ?mandataris org:holds ?mandate ;
-              ?p ?o .
-            ?orgaanInTime org:hasPost ?mandate ;
-              mandaat:isTijdspecialisatieVan ?orgaan .
-            ?orgaan besluit:bestuurt ?eenheid .
-            ?eenheid besluit:classificatie ?classification .
-          }
-        }`
-    }
-  ]
-}
-
-module.exports = {
-  contextConfig,
-  PREFIXES
-};

config/loket-consumer/mandatarissen-dispatching/delta-sync-dispatching.js

@@ -1,85 +0,0 @@
-const {
-  transformStatements,
-  deleteFromTargetGraph,
-  insertIntoTargetGraph,
-} = require('./util');
-
-/**
-* Dispatch the fetched information to a target graph.
-* Note: <share://file/data> will be ADDED to it's own graph.
-*   We take only care of adding them, not updating triples, this is a TODO
-* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
-* @param { termObjectChangeSets: { deletes, inserts } } data - The fetched changes sets, which objects of serialized Terms
-*          [ {
-*              graph: "<http://foo>",
-*              subject: "<http://bar>",
-*              predicate: "<http://baz>",
-*              object: "<http://boom>^^<http://datatype>"
-*            }
-*         ]
-* @return {void} Nothing
-*/
-async function dispatch(lib, data) {
-  const { mu, fetch } = lib;
-  let { termObjectChangeSets, termObjectChangeSetsWithContext } = data;
-
-  // Both arrays are the same length, so we can zip them together for easier processing
-  zippedChangeSets = termObjectChangeSets.map((o, i) => ({
-    original: o,
-    withContext: termObjectChangeSetsWithContext[i]
-  }));
-
-  console.log(`Received ${zippedChangeSets.length} change sets`)
-
-  for (let { original, withContext } of zippedChangeSets) {
-    let originalDeletes, originalInserts, insertsWithContext, deletesWithContext;
-    try {
-      originalInserts = original.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-      originalDeletes = original.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-    } catch (e) {
-      console.log(`Error while processing change set: ${e}`);
-      console.log(`Original change set: ${JSON.stringify(original)}`);
-      throw e;
-    }
-
-    // Extra context needed for mapping from DL to OP model and filtering based on type.
-    try {
-      insertsWithContext = withContext.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-      deletesWithContext = withContext.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-    } catch (e) {
-      console.log(`Error while processing change set with context: ${e}`);
-      console.log(`Change set with context: ${JSON.stringify(withContext)}`);
-      throw e;
-    }
-
-    if (originalDeletes.length) {
-      // Map deletes from DL to OP model
-      const transformedDeletes = await transformStatements(fetch, deletesWithContext);
-
-      if (!transformedDeletes.length) {
-        console.log(`Warn: Delete statements mapped to empty result.`);
-        console.log(`Input: ${deletesWithContext}`);
-        console.log(`Output: ${transformedDeletes}`);
-      } else {
-        await deleteFromTargetGraph(lib, transformedDeletes);
-      }
-    }
-
-    if (originalInserts.length) {
-      // Map inserts from DL to OP model
-      const transformedInserts = await transformStatements(fetch, insertsWithContext);
-
-      if (!transformedInserts.length) {
-        console.log(`Warn: Insert statements mapped to empty result.`);
-        console.log(`Input: ${insertsWithContext}`);
-        console.log(`Output: ${transformedInserts}`);
-      } else {
-        await insertIntoTargetGraph(lib, transformedInserts);
-      }
-    }
-  }
-}
-
-module.exports = {
-  dispatch
-};

config/loket-consumer/mandatarissen-dispatching/initial-sync-dispatching.js

@@ -1,85 +0,0 @@
-const { batchedDbUpdate, transformLandingZoneGraph } = require('./util');
-const {
-  BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
-  DIRECT_DATABASE_ENDPOINT,
-  MU_CALL_SCOPE_ID_INITIAL_SYNC,
-  BATCH_SIZE,
-  MAX_DB_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  TARGET_GRAPH,
-  LANDING_ZONE_GRAPH,
-  LANDING_ZONE_DATABASE_ENDPOINT
-} = require('./config');
-
-const endpoint = BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES ? DIRECT_DATABASE_ENDPOINT : process.env.MU_SPARQL_ENDPOINT;
-
-/**
-* Dispatch the fetched information to a target graph.
-* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
-* @param { termObjects } data - The fetched quad information, which objects of serialized Terms
-*          [ {
-*              graph: "<http://foo>",
-*              subject: "<http://bar>",
-*              predicate: "<http://baz>",
-*              object: "<http://boom>^^<http://datatype>"
-*            }
-*         ]
-* @return {void} Nothing
-*/
-async function dispatch(lib, data) {
-  const { mu, muAuthSudo } = lib;
-  const { termObjects } = data;
-
-  // Steps:
-  // 1. Write the triples to the landing zone graph - withouth any mapping or filtering
-  // 2. One-time reasoning run with the full landing zone graph and write the results to the target graph (on-finish)
-
-  if (BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES) {
-    console.warn(`Service configured to skip MU_AUTH!`);
-  }
-  console.log(`Using ${endpoint} to insert triples`);
-
-  if (termObjects.length) {
-    const originalInsertTriples = termObjects.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-
-    await batchedDbUpdate(
-      muAuthSudo.updateSudo,
-      LANDING_ZONE_GRAPH,
-      originalInsertTriples,
-      { 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
-      LANDING_ZONE_DATABASE_ENDPOINT,
-      BATCH_SIZE,
-      MAX_DB_RETRY_ATTEMPTS,
-      SLEEP_BETWEEN_BATCHES,
-      SLEEP_TIME_AFTER_FAILED_DB_OPERATION
-    );
-  }
-}
-
-async function onFinishInitialIngest(lib) {
-  const { muAuthSudo, fetch } = lib;
-
-  console.log(`!! On-finish triggered !!`);
-
-  const transformedInsertTriples = await transformLandingZoneGraph(fetch, endpoint);
-
-  console.log(`Transformed ${transformedInsertTriples.length} triples`);
-
-  await batchedDbUpdate(
-    muAuthSudo.updateSudo,
-    TARGET_GRAPH,
-    transformedInsertTriples,
-    { 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
-    endpoint,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION
-  );
-}
-
-module.exports = {
-  dispatch,
-  onFinishInitialIngest
-};

config/loket-consumer/mandatarissen-dispatching/util.js

@@ -1,168 +0,0 @@
-const {
-  MAX_REASONING_RETRY_ATTEMPTS,
-  SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
-  MAX_DB_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  TARGET_GRAPH,
-  LANDING_ZONE_GRAPH,
-  BATCH_SIZE
-} = require('./config');
-
-async function batchedDbUpdate(
-  muUpdate,
-  graph,
-  triples,
-  extraHeaders,
-  endpoint,
-  batchSize,
-  maxAttempts,
-  sleepBetweenBatches = 1000,
-  sleepTimeOnFail = 1000,
-  operation = 'INSERT'
-) {
-
-  for (let i = 0; i < triples.length; i += batchSize) {
-    console.log(`Inserting triples in batch: ${i}-${i + batchSize}`);
-
-    const batch = triples.slice(i, i + batchSize).join('\n');
-
-    const insertCall = async () => {
-      await muUpdate(`
-${operation} DATA {
-GRAPH <${graph}> {
-${batch}
-}
-}
-`, extraHeaders, endpoint);
-    };
-
-    await operationWithRetry(insertCall, 0, maxAttempts, sleepTimeOnFail);
-
-    console.log(`Sleeping before next query execution: ${sleepBetweenBatches}`);
-    await new Promise(r => setTimeout(r, sleepBetweenBatches));
-  }
-}
-
-async function operationWithRetry(callback,
-  attempt,
-  maxAttempts,
-  sleepTimeOnFail) {
-  try {
-    if (typeof callback === "function")
-      return await callback();
-    else // Catch error from promise - not how I would do it normally, but allows re use of existing code.
-      return await callback;
-  }
-  catch (e) {
-    console.log(`Operation failed for ${callback.toString()}, attempt: ${attempt} of ${maxAttempts}`);
-    console.log(`Error: ${e}`);
-    console.log(`Sleeping ${sleepTimeOnFail} ms`);
-
-    if (attempt >= maxAttempts) {
-      console.log(`Max attempts reached for ${callback.toString()}, giving up`);
-      throw e;
-    }
-
-    await new Promise(r => setTimeout(r, sleepTimeOnFail));
-    return operationWithRetry(callback, ++attempt, maxAttempts, sleepTimeOnFail);
-  }
-}
-
-/**
- * Splits an array into two parts, a part that passes and a part that fails a predicate function.
- * Credits: https://github.com/benjay10
- * @public
- * @function partition
- * @param {Array} arr - Array to be partitioned
- * @param {Function} fn - Function that accepts single argument: an element of the array, and should return a truthy or falsy value.
- * @returns {Object} Object that contains keys passes and fails, each representing an array with elemets that pass or fail the predicate respectively
- */
-function partition(arr, fn) {
-  let passes = [], fails = [];
-  arr.forEach((item) => (fn(item) ? passes : fails).push(item));
-  return { passes, fails };
-}
-
-/**
- * Send triples to reasoning service for conversion
- *
- */
-async function transformTriples(fetch, triples, mapping) {
-  return await operationWithRetry(mainConversion(fetch, triples, mapping), 0,
-    MAX_REASONING_RETRY_ATTEMPTS, SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION);
-}
-
-async function mainConversion(fetch, triples, mapping) {
-  let formdata = new URLSearchParams();
-  formdata.append("data", triples);
-
-  let requestOptions = {
-    method: 'POST',
-    body: formdata,
-    redirect: 'follow'
-  };
-
-  const response = await fetch(`http://reasoner/reason/dl2op/${mapping}`, requestOptions)
-  return await response.text();
-}
-
-async function transformStatements(fetch, triples, mapping = 'main') {
-  console.log(`Received ${JSON.stringify(triples)} to transform`);
-  const transformedTriples = await transformTriples(fetch, triples.join('\n'), mapping);
-  statements = transformedTriples ? transformedTriples.replace(/\n{2,}/g, '').split('\n') : [];
-  console.log(`CONVERSION: FROM ${triples.length} triples to ${statements.length}`);
-  return statements;
-}
-
-async function deleteFromTargetGraph(lib, statements) {
-  console.log(`Deleting ${statements.length} statements from target graph`);
-  console.log(`Statements:  ${JSON.stringify(statements)}`)
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    TARGET_GRAPH,
-    statements,
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'DELETE');
-}
-
-async function insertIntoTargetGraph(lib, statements) {
-  console.log(`Inserting ${statements.length} statements into target graph`);
-  console.log(`Statements:  ${JSON.stringify(statements)}`)
-
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    TARGET_GRAPH,
-    statements,
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'INSERT');
-}
-
-async function transformLandingZoneGraph(fetch, endpoint, mapping = 'main') {
-  console.log(`Transforming landing zone graph: ${LANDING_ZONE_GRAPH}`);
-
-  const response = await fetch(`http://reasoner/reason/dl2op/${mapping}?data=${encodeURIComponent(`${endpoint}?default-graph-uri=&query=CONSTRUCT+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D+WHERE+%7B%0D%0A+GRAPH+<${LANDING_ZONE_GRAPH}>+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D%0D%0A%7D&should-sponge=&format=text%2Fplain&timeout=0&run=+Run+Query`)}`);
-  const text = await response.text();
-  const statements = text.replace(/\n{2,}/g, '').split('\n');
-
-  return statements;
-}
-
-module.exports = {
-  batchedDbUpdate,
-  partition,
-  transformStatements,
-  transformLandingZoneGraph,
-  deleteFromTargetGraph,
-  insertIntoTargetGraph
-};

config/loket-consumer/mandatarissen/mapping/queries/dl2op/datetime-format/mandaat-einde.rq

@@ -0,0 +1,11 @@
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+
+CONSTRUCT {
+  ?s mandaat:einde ?endDateTime .
+} WHERE {
+  ?s mandaat:einde ?endDate .
+
+  BIND(SUBSTR(STR(?endDate), 1, 10) as ?endDateWithoutTimezone)
+  BIND(STRDT(CONCAT(?endDateWithoutTimezone, 'T02:00:00'), xsd:dateTime) as ?endDateTime)
+}

config/loket-consumer/mandatarissen/mapping/queries/dl2op/datetime-format/mandaat-start.rq

@@ -0,0 +1,11 @@
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+
+CONSTRUCT {
+  ?s mandaat:start ?startDateTime .
+} WHERE {
+  ?s mandaat:start ?startDate .
+
+  BIND(SUBSTR(STR(?startDate), 1, 10) as ?startDateWithoutTimezone)
+  BIND(STRDT(CONCAT(?startDateWithoutTimezone, 'T02:00:00'), xsd:dateTime) as ?startDateTime)
+}

config/loket-consumer/mandatarissen/mapping/queries/dl2op/inheritance/Agent.rq

@@ -0,0 +1,14 @@
+PREFIX dc_terms: <http://purl.org/dc/terms/>
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+CONSTRUCT {
+  ?s a dc_terms:Agent .
+} WHERE {
+  ?s a ?type .
+
+  VALUES ?type {
+    mandaat:Mandataris
+    person:Person
+  }
+}

config/loket-consumer/mandatarissen/mapping/queries/dl2op/inheritance/AgentInPositie.rq

@@ -0,0 +1,11 @@
+PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+
+CONSTRUCT {
+  ?s a ch:AgentInPositie .
+} WHERE {
+  ?s a ?type .
+  VALUES ?type {
+    mandaat:Mandataris
+  }
+}

config/loket-consumer/mandatarissen/mapping/queries/dl2op/inverse/holds-to-held-by.rq

@@ -0,0 +1,8 @@
+PREFIX org: <http://www.w3.org/ns/org#>
+
+CONSTRUCT {
+  ?o org:heldBy ?s.
+}
+WHERE {
+  ?s org:holds ?o.
+}

config/loket-consumer/mandatarissen/mapping/queries/dl2op/pass/mandataris.sparql

@@ -0,0 +1,29 @@
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://data.vlaanderen.be/ns/mandaat#Mandataris>;
+    <http://www.w3.org/ns/org#holds> ?mandate;
+    ?p ?o.
+
+  ?orgaanInTime <http://www.w3.org/ns/org#hasPost> ?mandate ;
+    <http://data.vlaanderen.be/ns/mandaat#isTijdspecialisatieVan> ?orgaan .
+
+  ?orgaan <http://data.vlaanderen.be/ns/besluit#bestuurt> ?bestuur .
+  ?bestuur <http://data.vlaanderen.be/ns/besluit#classificatie> ?classification .
+
+
+  FILTER(?classification IN (
+    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000000>,
+    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001>,
+    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000002>,
+    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000003>
+  ))
+
+  FILTER (?p NOT IN (mandaat:start, mandaat:einde))
+
+  FILTER NOT EXISTS {
+    ?s ?p "" .
+  }
+}

config/loket-consumer/mandatarissen/mapping/queries/dl2op/pass/person.sparql

@@ -0,0 +1,16 @@
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://www.w3.org/ns/person#Person>;
+    ?p ?o.
+
+  FILTER (?p IN (
+    <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>,
+    <http://xmlns.com/foaf/0.1/familyName>,
+    <http://mu.semte.ch/vocabularies/core/uuid>
+    ))
+
+  FILTER NOT EXISTS {
+   ?s ?p "" .
+  }
+}

config/loket-consumer/mandatarissen/mapping/queries/dl2op/predicate/person-foaf-name.rq

@@ -0,0 +1,21 @@
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
+
+# First name: persoon:gebruikteVoornaam => foaf:givenName
+# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
+
+# We do both in one query with an OPTIONAL because they need to happen simultaneously
+# not to end up in a loop
+
+# Part 1
+
+CONSTRUCT {
+  ?s persoon:gebruikteVoornaam ?name .
+} WHERE {
+  ?s a person:Person;
+    foaf:name ?name .
+}

config/loket-consumer/mandatarissen/mapping/queries/dl2op/predicate/person-gebruikte-voornaam.rq

@@ -0,0 +1,22 @@
+
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
+
+# First name: persoon:gebruikteVoornaam => foaf:givenName
+# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
+
+# We do both in one query with an OPTIONAL because they need to happen simultaneously
+# not to end up in a loop
+
+# Part 2
+
+CONSTRUCT {
+  ?s foaf:givenName ?gebruikteVoornaam .
+} WHERE {
+  ?s a person:Person ;
+    persoon_loket:gebruikteVoornaam ?gebruikteVoornaam .
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/data-cleanup/phone.rq

@@ -0,0 +1,21 @@
+PREFIX schema: <http://schema.org/>
+
+CONSTRUCT {
+  ?s schema:telephone ?cleanedPhoneNumber .
+} WHERE {
+  ?s schema:telephone ?phoneNumber .
+
+  # Remove all non digits
+  BIND(REPLACE(?phoneNumber, """[^+\\d]""", "") AS ?scrapeResult)
+
+  # Replace leading 00 with +
+  BIND(REPLACE(?scrapeResult, """^00""", "+") AS ?replacedZeroZero)
+
+  BIND(REPLACE(?replacedZeroZero, """^0""", "+32") AS ?defaultBe)
+
+  BIND(REPLACE(?defaultBe, """\\+320""", "+32") AS ?extendedDefaultBe)
+
+  BIND(REPLACE(?extendedDefaultBe, """^320""", "+32") AS ?finalResult)
+
+  BIND(IF(REGEX(?finalResult, """^\\+((32\\d{8})|(324\\d{8})|((?!32)\\d+))$"""), CONCAT("tel:", ?finalResult), ?phoneNumber) AS ?cleanedPhoneNumber)
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/date-to-datetime/contacthub-eindedatum.rq

@@ -0,0 +1,10 @@
+PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+
+CONSTRUCT {
+  ?s ch:eindedatum ?endDateTime .
+} WHERE {
+  ?s ch:eindedatum ?endDate .
+
+  BIND(STRDT(CONCAT(STR(?endDate), 'T02:00:00'), xsd:dateTime) as ?endDateTime)
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/date-to-datetime/contacthub-startdatum.rq

@@ -0,0 +1,10 @@
+PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+
+CONSTRUCT {
+  ?s ch:startdatum ?startDateTime .
+} WHERE {
+  ?s ch:startdatum ?startDate .
+
+  BIND(STRDT(CONCAT(STR(?startDate), 'T02:00:00'), xsd:dateTime) as ?startDateTime)
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/date-to-datetime/ere-geplande-einddatum-aanstelling.rq

@@ -0,0 +1,10 @@
+PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
+PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
+
+CONSTRUCT {
+  ?s ere:geplandeEinddatumAanstelling ?dateTime .
+} WHERE {
+  ?s ere:geplandeEinddatumAanstelling ?date .
+
+  BIND(STRDT(CONCAT(STR(?date), 'T02:00:00'), xsd:dateTime) as ?dateTime)
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/inheritance/Agent.rq

@@ -0,0 +1,20 @@
+PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
+PREFIX dc_terms: <http://purl.org/dc/terms/>
+PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
+PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+CONSTRUCT {
+  ?s a dc_terms:Agent .
+} WHERE {
+  ?s a ?type .
+
+  VALUES ?type {
+    lblodlg:Functionaris
+    mandaat:Mandataris
+    ere:EredienstMandataris
+    ere:RolBedienaar
+    person:Person
+  }
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/inheritance/AgentInPositie.rq

@@ -0,0 +1,17 @@
+PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
+PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
+PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
+PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
+
+CONSTRUCT {
+  ?s a ch:AgentInPositie .
+} WHERE {
+  ?s a ?type .
+
+  VALUES ?type {
+    lblodlg:Functionaris
+    mandaat:Mandataris
+    ere:EredienstMandataris
+    ere:RolBedienaar
+  }
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/inverse/held-by-to-holds.rq

@@ -0,0 +1,8 @@
+PREFIX org: <http://www.w3.org/ns/org#>
+
+CONSTRUCT {
+  ?o org:holds ?s.
+}
+WHERE {
+  ?s org:heldBy ?o.
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/inverse/holds-to-held-by.rq

@@ -0,0 +1,8 @@
+PREFIX org: <http://www.w3.org/ns/org#>
+
+CONSTRUCT {
+  ?o org:heldBy ?s.
+}
+WHERE {
+  ?s org:holds ?o.
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/pass/address.sparql

@@ -0,0 +1,11 @@
+
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://www.w3.org/ns/locn#Address>;
+    ?p ?o.
+
+  FILTER NOT EXISTS {
+    ?s ?p "" .
+  }
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/pass/bestuursfunctie.sparql

@@ -0,0 +1,7 @@
+CONSTRUCT {
+  ?s <http://schema.org/contactPoint> ?contactpoint.
+}
+WHERE {
+  ?s a <http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie>.
+  ?s <http://schema.org/contactPoint> ?contactpoint.
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/pass/contactpoint.sparql

@@ -0,0 +1,12 @@
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://schema.org/ContactPoint>;
+    ?p ?o.
+
+ FILTER (?p NOT IN (<http://schema.org/telephone>))
+
+ FILTER NOT EXISTS {
+   ?s ?p "" .
+ }
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/pass/eredienst-mandataris.sparql

@@ -0,0 +1,15 @@
+PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
+PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
+
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://data.lblod.info/vocabularies/erediensten/EredienstMandataris>;
+    ?p ?o.
+
+ FILTER (?p NOT IN (ch:startdatum, ch:eindedatum, ere:geplandeEinddatumAanstelling))
+
+ FILTER NOT EXISTS {
+   ?s ?p "" .
+ }
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/pass/person.sparql

@@ -0,0 +1,16 @@
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://www.w3.org/ns/person#Person>;
+    ?p ?o.
+
+  FILTER (?p IN (
+    <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>,
+    <http://mu.semte.ch/vocabularies/core/uuid>,
+    <http://xmlns.com/foaf/0.1/familyName>
+  ))
+
+  FILTER NOT EXISTS {
+   ?s ?p "" .
+  }
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/pass/rol-bedienaar.sparql

@@ -0,0 +1,10 @@
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://data.lblod.info/vocabularies/erediensten/RolBedienaar>;
+    ?p ?o.
+
+ FILTER NOT EXISTS {
+   ?s ?p "" .
+ }
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/predicate/gebruikte-voornaam.rq

@@ -0,0 +1,14 @@
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
+
+# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
+
+CONSTRUCT {
+  ?s persoon:gebruikteVoornaam ?name .
+} WHERE {
+  ?s a person:Person ;
+    foaf:name ?name .
+}

config/loket-consumer/worship-services-main-info/mapping/queries/dl2op/predicate/given-name.rq

@@ -0,0 +1,14 @@
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+PREFIX person: <http://www.w3.org/ns/person#>
+
+# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
+
+# First name: persoon:gebruikteVoornaam => foaf:givenName
+
+CONSTRUCT {
+  ?s foaf:givenName ?gebruikteVoornaam .
+} WHERE {
+  ?s a person:Person ;
+    persoon_loket:gebruikteVoornaam ?gebruikteVoornaam .
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/pass/geboorte.sparql

@@ -0,0 +1,10 @@
+CONSTRUCT {
+  ?s ?p ?o.
+} WHERE {
+  ?s a <http://data.vlaanderen.be/ns/persoon#Geboorte>;
+    ?p ?o.
+
+ FILTER NOT EXISTS {
+   ?s ?p "" .
+ }
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/persoon-http-to-https/Geboorte.rq

@@ -0,0 +1,10 @@
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+
+# Map from http persoon to https persoon namespace
+
+CONSTRUCT {
+  ?s a persoon:Geboorte
+} WHERE {
+  ?s a persoon_loket:Geboorte .
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/persoon-http-to-https/datum.rq

@@ -0,0 +1,10 @@
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+
+# Map from http persoon to https persoon namespace
+
+CONSTRUCT {
+  ?s persoon:datum ?o .
+} WHERE {
+  ?s persoon_loket:datum ?o .
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/persoon-http-to-https/geslacht.rq

@@ -0,0 +1,10 @@
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+
+# Map from http persoon to https persoon namespace
+
+CONSTRUCT {
+  ?s persoon:geslacht ?o .
+} WHERE {
+  ?s persoon_loket:geslacht ?o .
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/persoon-http-to-https/heeftGeboorte.rq

@@ -0,0 +1,10 @@
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+
+# Map from http persoon to https persoon namespace
+
+CONSTRUCT {
+  ?s persoon:heeftGeboorte ?o .
+} WHERE {
+  ?s persoon_loket:heeftGeboorte ?o .
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/persoon-http-to-https/heeftNationaliteit.rq

@@ -0,0 +1,10 @@
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+
+# Map from http persoon to https persoon namespace
+
+CONSTRUCT {
+  ?s persoon:heeftNationaliteit ?o .
+} WHERE {
+  ?s persoon_loket:heeftNationaliteit ?o .
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/persoon-http-to-https/registratie.rq

@@ -0,0 +1,10 @@
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
+
+# Map from http persoon to https persoon namespace
+
+CONSTRUCT {
+  ?s persoon:registratie ?o .
+} WHERE {
+  ?s persoon_loket:registratie ?o .
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/rrn/identifier.rq

@@ -0,0 +1,19 @@
+PREFIX generiek: <https://data.vlaanderen.be/ns/generiek#>
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+
+CONSTRUCT {
+  ?identifier
+    generiek:gestructureerdeIdentificator ?structuredIdentifier ;
+    skos:notation "Rijksregisternummer" .
+
+  ?structuredIdentifier a generiek:GestructureerdeIdentificator ;
+    mu:uuid ?uuid ;
+    generiek:lokaleIdentificator ?notation .
+}
+WHERE {
+    ?identifier skos:notation ?notation.
+
+    BIND(MD5(CONCAT(STR(?identifier),"STRUCTURED_IDENTIFIER")) as ?uuid)
+    BIND(IRI(CONCAT("http://data.lblod.info/id/gestructureerdeIdentificatoren/", ?uuid)) AS ?structuredIdentifier)
+}

config/loket-consumer/worship-services-private-info/mapping/queries/dl2op/rrn/registratie.rq

@@ -0,0 +1,10 @@
+PREFIX adms: <http://www.w3.org/ns/adms#>
+PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
+
+# identifier property in the export is only used for RRN.
+CONSTRUCT {
+  ?person persoon:registratie ?rrn .
+}
+WHERE {
+  ?person adms:identifier ?rrn .
+}

config/loket-consumer/worship-services-sensitive-dispatching/config.js

@@ -1,40 +0,0 @@
-const BATCH_SIZE = parseInt(process.env.BATCH_SIZE) || 100;
-const MU_CALL_SCOPE_ID_INITIAL_SYNC = process.env.MU_CALL_SCOPE_ID_INITIAL_SYNC || 'http://redpencil.data.gift/id/concept/muScope/deltas/consumer/initialSync';
-const BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES = process.env.BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES == 'true' ? true : false;
-const DIRECT_DATABASE_ENDPOINT = process.env.DIRECT_DATABASE_ENDPOINT || 'http://triplestore:8890/sparql';
-const MAX_DB_RETRY_ATTEMPTS = parseInt(process.env.MAX_DB_RETRY_ATTEMPTS || 5);
-const MAX_REASONING_RETRY_ATTEMPTS = parseInt(process.env.MAX_REASONING_RETRY_ATTEMPTS || 5);
-const SLEEP_BETWEEN_BATCHES = parseInt(process.env.SLEEP_BETWEEN_BATCHES || 1000);
-const SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION || 10000);
-const SLEEP_TIME_AFTER_FAILED_DB_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_DB_OPERATION || 60000);
-
-// Graphs
-const TARGET_GRAPH = process.env.TARGET_GRAPH || `http://mu.semte.ch/graphs/worship-service`;
-const PRIVACY_SENSITIVE_GRAPH = process.env.PRIVACY_SENSITIVE_GRAPH || `http://mu.semte.ch/graphs/worship-privacy-centric-graph`;
-const LANDING_ZONE_GRAPH = process.env.DCR_LANDING_ZONE_GRAPH || `http://mu.semte.ch/graphs/landing-zone/leidinggevenden`;
-
-// Mappings
-const MAIN_INFO_MAPPING = "main-info";
-const PRIVATE_INFO_MAPPING = "private-info";
-
-// ENDPOINTS
-const LANDING_ZONE_DATABASE = process.env.DCR_LANDING_ZONE_DATABASE || 'db';
-const LANDING_ZONE_DATABASE_ENDPOINT = process.env.DCR_LANDING_ZONE_DATABASE_ENDPOINT || `http://${LANDING_ZONE_DATABASE}:8890/sparql`;
-
-module.exports = {
-  BATCH_SIZE,
-  MU_CALL_SCOPE_ID_INITIAL_SYNC,
-  BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
-  DIRECT_DATABASE_ENDPOINT,
-  MAX_DB_RETRY_ATTEMPTS,
-  MAX_REASONING_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
-  LANDING_ZONE_GRAPH,
-  LANDING_ZONE_DATABASE_ENDPOINT,
-  TARGET_GRAPH,
-  PRIVACY_SENSITIVE_GRAPH,
-  MAIN_INFO_MAPPING,
-  PRIVATE_INFO_MAPPING
-};

config/loket-consumer/worship-services-sensitive-dispatching/delta-context-config.js

@@ -1,11 +0,0 @@
-const contextConfig = {
-  addTypes: {
-    scope: 'all', // 'inserts, 'deletes', 'all' or none. To add rdf:type to subjects of inserts, deletes or both
-    exhausitive: false, // true or false: find all types for a subject, even if one is already present in delta
-  },
-  contextQueries: []
-}
-
-module.exports = {
-  contextConfig
-};

config/loket-consumer/worship-services-sensitive-dispatching/delta-sync-dispatching.js

@@ -1,114 +0,0 @@
-const {
-  transformStatements,
-  deleteFromTargetGraph,
-  insertIntoTargetGraph,
-  deleteFromPrivateGraph,
-  insertIntoPrivateGraph
-} = require('./util');
-
-const {
-  MAIN_INFO_MAPPING,
-  PRIVATE_INFO_MAPPING
-} = require('./config');
-
-/**
-* Dispatch the fetched information to a target graph.
-* Note: <share://file/data> will be ADDED to it's own graph.
-*   We take only care of adding them, not updating triples, this is a TODO
-* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
-* @param { termObjectChangeSets: { deletes, inserts } } data - The fetched changes sets, which objects of serialized Terms
-*          [ {
-*              graph: "<http://foo>",
-*              subject: "<http://bar>",
-*              predicate: "<http://baz>",
-*              object: "<http://boom>^^<http://datatype>"
-*            }
-*         ]
-* @return {void} Nothing
-*/
-async function dispatch(lib, data) {
-  const { mu, fetch } = lib;
-  let { termObjectChangeSets, termObjectChangeSetsWithContext } = data;
-
-  // Both arrays are the same length, so we can zip them together for easier processing
-  zippedChangeSets = termObjectChangeSets.map((o, i) => ({
-    original: o,
-    withContext: termObjectChangeSetsWithContext[i]
-  }));
-
-  console.log(`Received ${zippedChangeSets.length} change sets`)
-
-  for (let { original, withContext } of zippedChangeSets) {
-    let originalDeletes, originalInserts, insertsWithContext, deletesWithContext;
-    try {
-      originalInserts = original.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-      originalDeletes = original.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-    } catch (e) {
-      console.log(`Error while processing change set: ${e}`);
-      console.log(`Original change set: ${JSON.stringify(original)}`);
-      throw e;
-    }
-
-    // Extra context needed for mapping from DL to OP model and filtering based on type.
-    try {
-      insertsWithContext = withContext.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-      deletesWithContext = withContext.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-    } catch (e) {
-      console.log(`Error while processing change set with context: ${e}`);
-      console.log(`Change set with context: ${JSON.stringify(withContext)}`);
-      throw e;
-    }
-
-    if (originalDeletes.length) {
-      // Map main deletes from DL to OP model
-      const transformedMainDeletes = await transformStatements(fetch, deletesWithContext, MAIN_INFO_MAPPING);
-
-      if (!transformedMainDeletes.length) {
-        console.log(`Warn: Delete main statements mapped to empty result.`);
-        console.log(`Input: ${deletesWithContext}`);
-        console.log(`Output: ${transformedMainDeletes}`);
-      } else {
-        await deleteFromTargetGraph(lib, transformedMainDeletes);
-      }
-
-      // Map private deletes from DL to OP model
-      const transformedPrivateDeletes = await transformStatements(fetch, deletesWithContext, PRIVATE_INFO_MAPPING);
-
-      if (!transformedPrivateDeletes.length) {
-        console.log(`Warn: Delete private statements mapped to empty result.`);
-        console.log(`Input: ${deletesWithContext}`);
-        console.log(`Output: ${transformedPrivateDeletes}`);
-      } else {
-        await deleteFromPrivateGraph(lib, transformedPrivateDeletes);
-      }
-    }
-
-    if (originalInserts.length) {
-      // Map main inserts from DL to OP model
-      const transformedMainInserts = await transformStatements(fetch, insertsWithContext, MAIN_INFO_MAPPING);
-
-      if (!transformedMainInserts.length) {
-        console.log(`Warn: Insert statements mapped to empty result.`);
-        console.log(`Input: ${insertsWithContext}`);
-        console.log(`Output: ${transformedMainInserts}`);
-      } else {
-        await insertIntoTargetGraph(lib, transformedMainInserts);
-      }
-
-      // Map private inserts from DL to OP model
-      const transformedPrivateInserts = await transformStatements(fetch, insertsWithContext, PRIVATE_INFO_MAPPING);
-
-      if (!transformedPrivateInserts.length) {
-        console.log(`Warn: Insert statements mapped to empty result.`);
-        console.log(`Input: ${insertsWithContext}`);
-        console.log(`Output: ${transformedPrivateInserts}`);
-      } else {
-        await insertIntoPrivateGraph(lib, transformedPrivateInserts);
-      }
-    }
-  }
-}
-
-module.exports = {
-  dispatch
-};

config/loket-consumer/worship-services-sensitive-dispatching/initial-sync-dispatching.js

@@ -1,106 +0,0 @@
-const { batchedDbUpdate, transformLandingZoneGraph } = require('./util');
-const {
-  BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
-  DIRECT_DATABASE_ENDPOINT,
-  MU_CALL_SCOPE_ID_INITIAL_SYNC,
-  BATCH_SIZE,
-  MAX_DB_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  TARGET_GRAPH,
-  PRIVACY_SENSITIVE_GRAPH,
-  LANDING_ZONE_GRAPH,
-  LANDING_ZONE_DATABASE_ENDPOINT,
-  MAIN_INFO_MAPPING,
-  PRIVATE_INFO_MAPPING
-} = require('./config');
-
-const endpoint = BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES ? DIRECT_DATABASE_ENDPOINT : process.env.MU_SPARQL_ENDPOINT;
-
-/**
-* Dispatch the fetched information to a target graph.
-* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
-* @param { termObjects } data - The fetched quad information, which objects of serialized Terms
-*          [ {
-*              graph: "<http://foo>",
-*              subject: "<http://bar>",
-*              predicate: "<http://baz>",
-*              object: "<http://boom>^^<http://datatype>"
-*            }
-*         ]
-* @return {void} Nothing
-*/
-async function dispatch(lib, data) {
-  const { mu, muAuthSudo } = lib;
-  const { termObjects } = data;
-
-  // Steps:
-  // 1. Write the triples to the landing zone graph - withouth any mapping or filtering
-  // 2. One-time reasoning run with the full landing zone graph and write the results to the target graph (on-finish)
-
-  if (BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES) {
-    console.warn(`Service configured to skip MU_AUTH!`);
-  }
-  console.log(`Using ${endpoint} to insert triples`);
-
-  if (termObjects.length) {
-    const originalInsertTriples = termObjects.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
-
-    await batchedDbUpdate(
-      muAuthSudo.updateSudo,
-      LANDING_ZONE_GRAPH,
-      originalInsertTriples,
-      { 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
-      LANDING_ZONE_DATABASE_ENDPOINT,
-      BATCH_SIZE,
-      MAX_DB_RETRY_ATTEMPTS,
-      SLEEP_BETWEEN_BATCHES,
-      SLEEP_TIME_AFTER_FAILED_DB_OPERATION
-    );
-  }
-}
-
-async function onFinishInitialIngest(lib) {
-  const { muAuthSudo, fetch } = lib;
-
-  console.log(`!! On-finish triggered !!`);
-
-  const transformedMainInsertTriples = await transformLandingZoneGraph(fetch, endpoint, MAIN_INFO_MAPPING);
-  console.log(`Transformed ${transformedMainInsertTriples.length} main triples`);
-
-  if (transformedMainInsertTriples.length) {
-    await batchedDbUpdate(
-      muAuthSudo.updateSudo,
-      TARGET_GRAPH,
-      transformedMainInsertTriples,
-      { 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
-      endpoint,
-      BATCH_SIZE,
-      MAX_DB_RETRY_ATTEMPTS,
-      SLEEP_BETWEEN_BATCHES,
-      SLEEP_TIME_AFTER_FAILED_DB_OPERATION
-    );
-  }
-
-  const transformedPrivateInsertTriples = await transformLandingZoneGraph(fetch, endpoint, PRIVATE_INFO_MAPPING);
-  console.log(`Transformed ${transformedPrivateInsertTriples.length} private triples`);
-
-  if (transformedPrivateInsertTriples.length) {
-    await batchedDbUpdate(
-      muAuthSudo.updateSudo,
-      PRIVACY_SENSITIVE_GRAPH,
-      transformedPrivateInsertTriples,
-      { 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
-      endpoint,
-      BATCH_SIZE,
-      MAX_DB_RETRY_ATTEMPTS,
-      SLEEP_BETWEEN_BATCHES,
-      SLEEP_TIME_AFTER_FAILED_DB_OPERATION
-    );
-  }
-}
-
-module.exports = {
-  dispatch,
-  onFinishInitialIngest
-};

config/loket-consumer/worship-services-sensitive-dispatching/util.js

@@ -1,204 +0,0 @@
-const {
-  MAX_REASONING_RETRY_ATTEMPTS,
-  SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
-  MAX_DB_RETRY_ATTEMPTS,
-  SLEEP_BETWEEN_BATCHES,
-  SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-  TARGET_GRAPH,
-  PRIVACY_SENSITIVE_GRAPH,
-  LANDING_ZONE_GRAPH,
-  BATCH_SIZE
-} = require('./config');
-
-async function batchedDbUpdate(
-  muUpdate,
-  graph,
-  triples,
-  extraHeaders,
-  endpoint,
-  batchSize,
-  maxAttempts,
-  sleepBetweenBatches = 1000,
-  sleepTimeOnFail = 1000,
-  operation = 'INSERT'
-) {
-
-  for (let i = 0; i < triples.length; i += batchSize) {
-    console.log(`Inserting triples in batch: ${i}-${i + batchSize}`);
-
-    const batch = triples.slice(i, i + batchSize).join('\n');
-
-    const insertCall = async () => {
-      await muUpdate(`
-${operation} DATA {
-GRAPH <${graph}> {
-${batch}
-}
-}
-`, extraHeaders, endpoint);
-    };
-
-    await operationWithRetry(insertCall, 0, maxAttempts, sleepTimeOnFail);
-
-    console.log(`Sleeping before next query execution: ${sleepBetweenBatches}`);
-    await new Promise(r => setTimeout(r, sleepBetweenBatches));
-  }
-}
-
-async function operationWithRetry(callback,
-  attempt,
-  maxAttempts,
-  sleepTimeOnFail) {
-  try {
-    if (typeof callback === "function")
-      return await callback();
-    else // Catch error from promise - not how I would do it normally, but allows re use of existing code.
-      return await callback;
-  }
-  catch (e) {
-    console.log(`Operation failed for ${callback.toString()}, attempt: ${attempt} of ${maxAttempts}`);
-    console.log(`Error: ${e}`);
-    console.log(`Sleeping ${sleepTimeOnFail} ms`);
-
-    if (attempt >= maxAttempts) {
-      console.log(`Max attempts reached for ${callback.toString()}, giving up`);
-      throw e;
-    }
-
-    await new Promise(r => setTimeout(r, sleepTimeOnFail));
-    return operationWithRetry(callback, ++attempt, maxAttempts, sleepTimeOnFail);
-  }
-}
-
-/**
- * Splits an array into two parts, a part that passes and a part that fails a predicate function.
- * Credits: https://github.com/benjay10
- * @public
- * @function partition
- * @param {Array} arr - Array to be partitioned
- * @param {Function} fn - Function that accepts single argument: an element of the array, and should return a truthy or falsy value.
- * @returns {Object} Object that contains keys passes and fails, each representing an array with elemets that pass or fail the predicate respectively
- */
-function partition(arr, fn) {
-  let passes = [], fails = [];
-  arr.forEach((item) => (fn(item) ? passes : fails).push(item));
-  return { passes, fails };
-}
-
-/**
- * Send triples to reasoning service for conversion
- *
- */
-async function transformTriples(fetch, triples, mapping) {
-  return await operationWithRetry(mainConversion(fetch, triples, mapping), 0,
-    MAX_REASONING_RETRY_ATTEMPTS, SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION);
-}
-
-async function mainConversion(fetch, triples, mapping) {
-  let formdata = new URLSearchParams();
-  formdata.append("data", triples);
-
-  let requestOptions = {
-    method: 'POST',
-    body: formdata,
-    redirect: 'follow'
-  };
-
-  const response = await fetch(`http://reasoner/reason/dl2op/${mapping}`, requestOptions)
-  return await response.text();
-}
-
-async function transformStatements(fetch, triples, mapping = 'main') {
-  console.log(`Received ${JSON.stringify(triples)} to transform`);
-  const transformedTriples = await transformTriples(fetch, triples.join('\n'), mapping);
-  statements = transformedTriples ? transformedTriples.replace(/\n{2,}/g, '').split('\n') : [];
-  console.log(`CONVERSION: FROM ${triples.length} triples to ${statements.length}`);
-  return statements;
-}
-
-async function deleteFromTargetGraph(lib, statements) {
-  console.log(`Deleting ${statements.length} statements from target graph`);
-  console.log(`Statements:  ${JSON.stringify(statements)}`)
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    TARGET_GRAPH,
-    statements,
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'DELETE');
-}
-
-async function deleteFromPrivateGraph(lib, statements) {
-  console.log(`Deleting ${statements.length} statements from target graph`);
-  console.log(`Statements:  ${JSON.stringify(statements)}`)
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    PRIVACY_SENSITIVE_GRAPH,
-    statements,
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'DELETE');
-}
-
-async function insertIntoTargetGraph(lib, statements) {
-  console.log(`Inserting ${statements.length} statements into target graph`);
-  console.log(`Statements:  ${JSON.stringify(statements)}`)
-
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    TARGET_GRAPH,
-    statements,
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'INSERT');
-}
-
-async function insertIntoPrivateGraph(lib, statements) {
-  console.log(`Inserting ${statements.length} statements into target graph`);
-  console.log(`Statements:  ${JSON.stringify(statements)}`)
-
-  await batchedDbUpdate(
-    lib.muAuthSudo.updateSudo,
-    PRIVACY_SENSITIVE_GRAPH,
-    statements,
-    {},
-    process.env.MU_SPARQL_ENDPOINT,
-    BATCH_SIZE,
-    MAX_DB_RETRY_ATTEMPTS,
-    SLEEP_BETWEEN_BATCHES,
-    SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
-    'INSERT');
-}
-
-async function transformLandingZoneGraph(fetch, endpoint, mapping = 'main') {
-  console.log(`Transforming landing zone graph: ${LANDING_ZONE_GRAPH}`);
-
-  const response = await fetch(`http://reasoner/reason/dl2op/${mapping}?data=${encodeURIComponent(`${endpoint}?default-graph-uri=&query=CONSTRUCT+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D+WHERE+%7B%0D%0A+GRAPH+<${LANDING_ZONE_GRAPH}>+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D%0D%0A%7D&should-sponge=&format=text%2Fplain&timeout=0&run=+Run+Query`)}`);
-  const text = await response.text();
-  const statements = text.replace(/\n{2,}/g, '').split('\n');
-
-  return statements;
-}
-
-module.exports = {
-  batchedDbUpdate,
-  partition,
-  transformStatements,
-  transformLandingZoneGraph,
-  deleteFromTargetGraph,
-  insertIntoTargetGraph,
-  deleteFromPrivateGraph,
-  insertIntoPrivateGraph
-};

config/migrations-triggering-indexing/2024/20241212174128-archive-agb-stekene.sparql

@@ -0,0 +1,15 @@
+DELETE {
+  GRAPH ?g {
+    <http://data.lblod.info/id/bestuurseenheden/8d6609b4-1a20-46b8-97ed-150c6d53acd6> <http://www.w3.org/ns/regorg#orgStatus> ?orgStatus .
+  }
+}
+INSERT {
+  GRAPH ?g {
+    <http://data.lblod.info/id/bestuurseenheden/8d6609b4-1a20-46b8-97ed-150c6d53acd6> <http://www.w3.org/ns/regorg#orgStatus> <http://lblod.data.gift/concepts/d02c4e12bf88d2fdf5123b07f29c9311> .
+  }
+}
+WHERE {
+  GRAPH ?g {
+    <http://data.lblod.info/id/bestuurseenheden/8d6609b4-1a20-46b8-97ed-150c6d53acd6> <http://www.w3.org/ns/regorg#orgStatus> ?orgStatus .
+  }
+}

config/migrations-triggering-indexing/2025/20250109200145-archive-agb-stekene.sparql

@@ -0,0 +1,196 @@
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?bestuursorganen ?bestuursorganenP ?bestuursorganenO .
+  }
+}
+WHERE {
+  VALUES ?bestuursorganen {
+    <http://data.lblod.info/id/bestuursorganen/24d8e602a8ddbe0adf3dea8c40277706>
+    <http://data.lblod.info/id/bestuursorganen/dad1789d9b9f8f6cb3c81cc2ca88154f>
+    <http://data.lblod.info/id/bestuursorganen/b90d16f2c8043bc20b88da232c0389fc>
+    <http://data.lblod.info/id/bestuursorganen/2f27a688e1728c1906a4bbd8670d7810>
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?bestuursorganen ?bestuursorganenP ?bestuursorganenO .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?bestuursorganenInTijd ?bestuursorganenInTijdP ?bestuursorganenInTijdO .
+  }
+}
+WHERE {
+  VALUES ?bestuursorganenInTijd {
+    <http://data.lblod.info/id/bestuursorganen/c5bb7bca7363109b94b58b1aef6ed59c>
+    <http://data.lblod.info/id/bestuursorganen/8d43a29c6ed4a8af0cd43d4aba2c7014>
+    <http://data.lblod.info/id/bestuursorganen/f54979f47464bae1cc001788aedb8b86>
+    <http://data.lblod.info/id/bestuursorganen/c9a7d9130e1dd8f26ddb8c96fdd94c64>
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?bestuursorganenInTijd ?bestuursorganenInTijdP ?bestuursorganenInTijdO .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?identifier ?identifierP ?identifierO .
+  }
+}
+WHERE {
+  VALUES ?identifier {
+    <http://data.lblod.info/id/identificatoren/99554be7-29d2-403a-a7d5-bda842b4bd86>
+    <http://data.lblod.info/id/identificatoren/19847e1b-a2bf-48a1-872e-baf8d4efaa12>
+    <http://data.lblod.info/id/identificatoren/b2123d27-b6bc-4e70-b594-27d35c696713>
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?identifier ?identifierP ?identifierO .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?structuredIdentifier ?structuredIdentifierP ?structuredIdentifierO .
+  }
+}
+WHERE {
+  VALUES ?structuredIdentifier {
+    <http://data.lblod.info/id/gestructureerdeIdentificatoren/920f98a2-1785-4de1-81ed-5859c224dcb8>
+    <http://data.lblod.info/id/gestructureerdeIdentificatoren/5e1082d8-7751-4bed-a53f-457b347f7f28>
+    <http://data.lblod.info/id/gestructureerdeIdentificatoren/2e978403-d3ef-40ab-a9f9-3a4ae6a35609>
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?structuredIdentifier ?structuredIdentifierP ?structuredIdentifierO .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?site ?siteP ?siteO .
+  }
+}
+WHERE {
+  VALUES ?site {
+    <http://data.lblod.info/id/vestigingen/daa79855-1de7-4479-8a2e-a8d89db55ee3>
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?site ?siteP ?siteO .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?address ?addressP ?addressO .
+  }
+}
+WHERE {
+  VALUES ?address {
+    <http://data.lblod.info/id/adressen/df102cb4-8abb-4588-a199-1ce22c211812>
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?address ?addressP ?addressO .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?contactPoint ?contactPointP ?contactPointO .
+  }
+}
+WHERE {
+  VALUES ?contactPoint {
+    <http://data.lblod.info/id/contact-punten/042e8816-2e9d-45a1-89f6-dcf805882af7>
+    <http://data.lblod.info/id/contact-punten/364d8e1a-cb18-4970-aeaf-5f6f307370ad>
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?contactPoint ?contactPointP ?contactPointO .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?parentOrganization <http://www.w3.org/ns/org#hasSubOrganization> ?organization .
+  }
+}
+WHERE {
+  BIND(<http://data.lblod.info/id/bestuurseenheden/8d6609b4-1a20-46b8-97ed-150c6d53acd6> AS ?organization)
+  BIND(<http://data.lblod.info/id/bestuurseenheden/6025a5d1ca2262a784f002edd8ce9ca9073ae3d5ebc6b6b5531f05a29e9250af> AS ?parentOrganization)
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?parentOrganization <http://www.w3.org/ns/org#hasSubOrganization> ?organization .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?parentOrganization <http://mu.semte.ch/vocabularies/ext/isFounderOfOrganization> ?organization .
+  }
+}
+WHERE {
+  BIND(<http://data.lblod.info/id/bestuurseenheden/8d6609b4-1a20-46b8-97ed-150c6d53acd6> AS ?organization)
+  BIND(<http://data.lblod.info/id/bestuurseenheden/6025a5d1ca2262a784f002edd8ce9ca9073ae3d5ebc6b6b5531f05a29e9250af> AS ?parentOrganization)
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?parentOrganization <http://mu.semte.ch/vocabularies/ext/isFounderOfOrganization> ?organization .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?membership ?p ?o.
+  }
+}
+WHERE {
+  BIND(<http://data.lblod.info/id/bestuurseenheden/8d6609b4-1a20-46b8-97ed-150c6d53acd6> AS ?organization)
+
+  VALUES ?membershipP {
+    <http://www.w3.org/ns/org#organization>
+    <http://www.w3.org/ns/org#member>
+  }
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?membership a <http://www.w3.org/ns/org#Membership> ;
+      ?membershipP ?organization ;
+      ?p ?o .
+  }
+}
+
+;
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?organization ?organizationP ?organizationO .
+  }
+}
+WHERE {
+  BIND(<http://data.lblod.info/id/bestuurseenheden/8d6609b4-1a20-46b8-97ed-150c6d53acd6> AS ?organization)
+
+  GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+    ?organization ?organizationP ?organizationO .
+  }
+}

config/migrations-triggering-indexing/20250114093158-fix-merger-change-event-sint_martinus-st_amandus.sparql

@@ -0,0 +1,23 @@
+PREFIX regorg: <http://www.w3.org/ns/regorg#>
+PREFIX org: <http://www.w3.org/ns/org#>
+
+DELETE {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    ?stAmandus regorg:orgStatus ?oldStatus.
+  }
+} INSERT {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    ?stAmandus regorg:orgStatus ?notActiveStatus.
+
+    ?mergerChangeEvent org:originalOrganization ?stAmandus.
+  }
+} WHERE {
+  GRAPH <http://mu.semte.ch/graphs/worship-service> {
+    ?stAmandus regorg:orgStatus ?oldStatus.
+  }
+  VALUES ?stAmandus {
+    <http://data.lblod.info/id/besturenVanDeEredienst/b18a5d3943087d509d76dcceeba102c3>
+  }
+  BIND (<http://data.lblod.info/id/veranderingsgebeurtenissen/677E445E29299CC832892A0F> AS ?mergerChangeEvent)
+  BIND (<http://lblod.data.gift/concepts/d02c4e12bf88d2fdf5123b07f29c9311> AS ?notActiveStatus)
+}

config/migrations-triggering-indexing/local/README.md

@@ -0,0 +1,3 @@
+# Motivation
+
+This folder is designed to contain migration files specific to a specific deployed instance of the app. These files will be ignored by git. The typical use case for this folder includes migrations containing sensitive data or those related to a re-sync of consumers. Migrations that should be applied to all instances of the app should be stored outside of this folder, but within the 'migrations-triggering-indexing' folder.

config/migrations/2024/20240704112300-related-organizations-as-memberships/20240704112500-codelist-lidmaatschap-rollen.graph

@@ -0,0 +1 @@
+http://mu.semte.ch/graphs/public

config/migrations/2024/20240704112300-related-organizations-as-memberships/20240704112500-codelist-lidmaatschap-rollen.ttl

@@ -0,0 +1,31 @@
+@prefix org: <http://www.w3.org/ns/org#> .
+@prefix owl: <http://www.w3.org/2002/07/owl#> .
+@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
+@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
+@prefix schema: <http://schema.org/> .
+@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
+@prefix mu: <http://mu.semte.ch/vocabularies/core/> .
+
+<https://data.vlaanderen.be/id/conceptscheme/lidmaatschap-rollen> a skos:ConceptScheme;
+  mu:uuid "4755a6afc9044ff289d872092e202472";
+  skos:prefLabel "lidmaatschap rollen".
+
+<http://data.lblod.info/id/rollen/93c48754610c45e6bd9a894d2720a53d> a org:Role;
+  mu:uuid "93c48754610c45e6bd9a894d2720a53d";
+  skos:topConceptOf <https://data.vlaanderen.be/id/conceptscheme/lidmaatschap-rollen>;
+  skos:prefLabel "lid".
+
+<http://data.lblod.info/id/rollen/73d5e1cf250d42fab15926771f07505a> a org:Role;
+  mu:uuid "73d5e1cf250d42fab15926771f07505a";
+  skos:prefLabel "stichtend lid";
+  skos:broader <http://data.lblod.info/id/rollen/93c48754610c45e6bd9a894d2720a53d>.
+
+<http://data.lblod.info/id/rollen/2152eb830b1143bfb97a7dd9596d6c63> a org:Role;
+  mu:uuid "2152eb830b1143bfb97a7dd9596d6c63";
+  skos:prefLabel "participant";
+  skos:broader <http://data.lblod.info/id/rollen/93c48754610c45e6bd9a894d2720a53d>.
+
+<http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> a org:Role;
+  mu:uuid "4ec7d5c39bdc4e84b4174379b9e22ad8";
+  skos:prefLabel "gerelateerd";
+  skos:broader <http://data.lblod.info/id/rollen/93c48754610c45e6bd9a894d2720a53d>.

config/migrations/2024/20240704112300-related-organizations-as-memberships/20240704112600-founding-members.sparql

@@ -0,0 +1,35 @@
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+PREFIX org: <http://www.w3.org/ns/org#>
+PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
+
+DELETE {
+  GRAPH ?g {
+    ?foundingOrg ext:isFounderOfOrganization ?foundedOrg .
+  }
+}
+INSERT {
+  GRAPH ?g {
+    ?foundingOrg org:memberOf ?foundedOrg .
+
+    ?membership a org:Membership ;
+      mu:uuid ?uuid ;
+      org:organization ?foundedOrg ;
+      org:member ?foundingOrg ;
+      org:role <http://data.lblod.info/id/rollen/73d5e1cf250d42fab15926771f07505a> . # stichtend lid
+  }
+}
+WHERE {
+    VALUES ?g {
+      <http://mu.semte.ch/graphs/worship-service>
+      <http://mu.semte.ch/graphs/administrative-unit>
+      <http://mu.semte.ch/graphs/shared>
+    }
+
+    GRAPH ?g {
+      ?foundingOrg ext:isFounderOfOrganization ?foundedOrg .
+    }
+
+  BIND(SHA1(CONCAT(STR(?foundingOrg), STR(?foundedOrg), "founded by")) as ?uuid)
+  BIND(IRI(CONCAT("http://data.lblod.info/id/lidmaatschap/", ?uuid)) AS ?membership)
+}

config/migrations/2024/20240704112300-related-organizations-as-memberships/20240704112700-participants.sparql

@@ -0,0 +1,32 @@
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+PREFIX org: <http://www.w3.org/ns/org#>
+PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+
+DELETE {
+  GRAPH ?g {
+    ?memberOrganization org:memberOf ?organization .
+  }
+}
+INSERT {
+  GRAPH ?g {
+    ?membership a org:Membership ;
+      mu:uuid ?uuid ;
+      org:organization ?organization ;
+      org:member ?memberOrganization ;
+      org:role <http://data.lblod.info/id/rollen/2152eb830b1143bfb97a7dd9596d6c63> . # participant
+  }
+}
+WHERE {
+    VALUES ?g {
+      <http://mu.semte.ch/graphs/worship-service>
+      <http://mu.semte.ch/graphs/administrative-unit>
+      <http://mu.semte.ch/graphs/shared>
+    }
+
+    GRAPH ?g {
+      ?memberOrganization org:memberOf ?organization .
+    }
+
+  BIND(SHA1(CONCAT(STR(?memberOrganization), STR(?organization), "member of")) as ?uuid)
+  BIND(IRI(CONCAT("http://data.lblod.info/id/lidmaatschap/", ?uuid)) AS ?membership)
+}

config/migrations/2024/20240704112300-related-organizations-as-memberships/20240704112800-related-members.sparql

@@ -0,0 +1,26 @@
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+PREFIX org: <http://www.w3.org/ns/org#>
+PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+
+INSERT {
+  GRAPH ?g {
+    ?membership a org:Membership ;
+      mu:uuid ?uuid ;
+      org:organization ?organization ;
+      org:member ?subOrganization ;
+      org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> . # gerelateerd
+  }
+} WHERE {
+    VALUES ?g {
+      <http://mu.semte.ch/graphs/worship-service>
+      <http://mu.semte.ch/graphs/administrative-unit>
+      <http://mu.semte.ch/graphs/shared>
+    }
+
+    GRAPH ?g {
+      ?organization org:hasSubOrganization ?subOrganization.
+    }
+
+  BIND(SHA1(CONCAT(STR(?organization), STR(?subOrganization), "sub organization")) as ?uuid)
+  BIND(IRI(CONCAT("http://data.lblod.info/id/lidmaatschap/", ?uuid)) AS ?membership)
+}

config/migrations/2024/20240704112300-related-organizations-as-memberships/20240716151700-associated-with.sparql

@@ -0,0 +1,27 @@
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+PREFIX org: <http://www.w3.org/ns/org#>
+PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
+
+INSERT {
+  GRAPH ?g {
+    ?membership a org:Membership ;
+      mu:uuid ?uuid ;
+      org:organization ?linkedOrg ;
+      org:member ?org ;
+      org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> . # gerelateerd
+  }
+} WHERE {
+    VALUES ?g {
+      <http://mu.semte.ch/graphs/worship-service>
+      <http://mu.semte.ch/graphs/administrative-unit>
+      <http://mu.semte.ch/graphs/shared>
+    }
+
+    GRAPH ?g {
+      ?org org:linkedTo ?linkedOrg .
+    }
+
+  BIND(SHA1(CONCAT(STR(?org), STR(?linkedOrg), "associated with")) as ?uuid)
+  BIND(IRI(CONCAT("http://data.lblod.info/id/lidmaatschap/", ?uuid)) AS ?membership)
+}

config/migrations/2024/20240905164200-remove-https-BestuurseenheidClassificatieCode.sparql

@@ -0,0 +1,13 @@
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+
+DELETE DATA {
+  GRAPH <http://mu.semte.ch/graphs/public> {
+    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/36a82ba0-7ff1-4697-a9dd-2e94df73b721>
+      skos:inScheme <https://data.vlaanderen.be/id/conceptscheme/BestuurseenheidClassificatieCode> ;
+      skos:topConceptOf <https://data.vlaanderen.be/id/conceptscheme/BestuurseenheidClassificatieCode> .
+    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/80310756-ce0a-4a1b-9b8e-7c01b6cc7a2d>
+      skos:inScheme <https://data.vlaanderen.be/id/conceptscheme/BestuurseenheidClassificatieCode> ;
+      skos:topConceptOf <https://data.vlaanderen.be/id/conceptscheme/BestuurseenheidClassificatieCode> .
+  }
+}
+

config/migrations/2024/20240930124819-membership-cleanup-jobs/20240930124819-membership-cleanup--insert-worship-linked-to.graph

@@ -0,0 +1 @@
+http://mu.semte.ch/graphs/public

config/migrations/2024/20240930124819-membership-cleanup-jobs/20240930124819-membership-cleanup--insert-worship-linked-to.ttl

@@ -0,0 +1,39 @@
+@prefix cleanup: <http://mu.semte.ch/vocabularies/ext/cleanup/> .
+@prefix mu: <http://mu.semte.ch/vocabularies/core/> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+
+# Insert triples of the form:
+# <Representative Body> <http://www.w3.org/ns/org#linkedTo> <Worship Service>
+# <Representative Body> <http://www.w3.org/ns/org#linkedTo> <Central Worship Service>
+# matching memberships for which no such triple exists.
+
+# Worship Service: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/66ec74fd-8cfc-4e16-99c6-350b35012e86>
+# Central Worship Service: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/f9cac08a-13c1-49da-9bcb-f650b0604054>
+
+<http://data.lblod.info/id/cleanup-job/f0f7a304-34e9-4986-bf27-8434bb6f8111> a cleanup:Job ;
+    mu:uuid "f0f7a304-34e9-4986-bf27-8434bb6f8111" ;
+    dcterms:title "Insert missing `org:linkedTo` triples for Representative Bodies and [Central] Worship Services." ;
+    cleanup:randomQuery """
+      PREFIX org: <http://www.w3.org/ns/org#>
+
+      INSERT {
+        GRAPH <http://mu.semte.ch/graphs/worship-service> {
+          ?member org:linkedTo ?organization .
+        }
+      } WHERE {
+      GRAPH <http://mu.semte.ch/graphs/worship-service> {
+        ?membership a org:Membership ;
+          org:organization ?organization ;
+          org:member ?member ;
+          org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> .
+        ?member org:classification <http://data.vlaanderen.be/id/concept/RepresentatiefOrgaanClassificatieCode/89a00b5a-024f-4630-a722-65a5e68967e5> .
+        ?organization org:classification ?orgClass .
+        FILTER ( ?orgClass IN (
+                             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/66ec74fd-8cfc-4e16-99c6-350b35012e86> ,
+                             <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/f9cac08a-13c1-49da-9bcb-f650b0604054>
+                             ))
+
+        FILTER NOT EXISTS { ?member org:linkedTo ?organization . }
+      }
+    }
+    """ .

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241001071404-membership-cleanup--insert-worship-has-sub-organization.graph

@@ -0,0 +1 @@
+http://mu.semte.ch/graphs/public

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241001071404-membership-cleanup--insert-worship-has-sub-organization.ttl

@@ -0,0 +1,35 @@
+@prefix cleanup: <http://mu.semte.ch/vocabularies/ext/cleanup/> .
+@prefix mu: <http://mu.semte.ch/vocabularies/core/> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+
+# Insert triples of the form:
+# <Central Worship Service> <http://www.w3.org/ns/org#hasSubOrganization> <Worship Service>
+# matching memberships for which no such triple exists.
+
+# Central Worship Service: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/f9cac08a-13c1-49da-9bcb-f650b0604054>
+# Worship Service: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/66ec74fd-8cfc-4e16-99c6-350b35012e86>
+
+<http://data.lblod.info/id/cleanup-job/1352f2c5-0207-435b-9737-26d29769cde6> a cleanup:Job ;
+    mu:uuid "1352f2c5-0207-435b-9737-26d29769cde6" ;
+    dcterms:title "Insert missing `org:hasSubOrganization` triples for Central Worship Services and Worship Services." ;
+    cleanup:randomQuery """
+      PREFIX org: <http://www.w3.org/ns/org#>
+
+      INSERT {
+        GRAPH <http://mu.semte.ch/graphs/worship-service> {
+          ?organization org:hasSubOrganization ?member .
+        }
+      } WHERE {
+      GRAPH <http://mu.semte.ch/graphs/worship-service> {
+        ?membership a org:Membership ;
+          org:organization ?organization ;
+          org:member ?member ;
+          org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> .
+
+        ?organization org:classification <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/f9cac08a-13c1-49da-9bcb-f650b0604054> .
+        ?member org:classification <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/66ec74fd-8cfc-4e16-99c6-350b35012e86> .
+
+        FILTER NOT EXISTS { ?organization org:hasSubOrganization ?member . }
+      }
+    }
+    """ .

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241001145338-membership-cleanup--insert-province-has-sub-organization.graph

@@ -0,0 +1 @@
+http://mu.semte.ch/graphs/public

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241001145338-membership-cleanup--insert-province-has-sub-organization.ttl

@@ -0,0 +1,52 @@
+@prefix cleanup: <http://mu.semte.ch/vocabularies/ext/cleanup/> .
+@prefix mu: <http://mu.semte.ch/vocabularies/core/> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+
+# Insert triples of the form
+# <Province> <http://www.w3.org/ns/org#hasSubOrganization> <subOrg>
+# matching memberships for which no such triple exists.
+
+# Province: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000000>
+#
+# Municipality: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001>
+# OCMW: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000002>
+# AGB: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/36a82ba0-7ff1-4697-a9dd-2e94df73b721>
+# APB: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/80310756-ce0a-4a1b-9b8e-7c01b6cc7a2d>
+# PEVA municipality: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/2ad46df5-5c79-4d67-84d5-604c1377231e>
+# PEVA province: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/088784b6-e188-48bf-b94f-94665f9e1f53>
+
+<http://data.lblod.info/id/cleanup-job/434b8b52-b695-4358-a927-ddaceb22e7e3> a cleanup:Job ;
+    mu:uuid "434b8b52-b695-4358-a927-ddaceb22e7e3" ;
+    dcterms:title "Insert missing `org:hasSubOrganization` triples for Provinces." ;
+    cleanup:randomQuery """
+      PREFIX org: <http://www.w3.org/ns/org#>
+
+      INSERT {
+        GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+          ?organization org:hasSubOrganization ?member .
+        }
+      } WHERE {
+        GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+          ?membership a org:Membership ;
+            org:organization ?organization ;
+            org:member ?member ;
+            org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> .
+
+          FILTER NOT EXISTS { ?organization org:hasSubOrganization ?member . }
+        }
+
+        ?organization org:classification <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000000> .
+
+        ?member org:classification ?memberClass .
+        FILTER ( ?memberClass
+                 IN (
+                    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001> ,
+                    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000002> ,
+                    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/36a82ba0-7ff1-4697-a9dd-2e94df73b721> ,
+                    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/80310756-ce0a-4a1b-9b8e-7c01b6cc7a2d> ,
+                    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/2ad46df5-5c79-4d67-84d5-604c1377231e> ,
+                    <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/088784b6-e188-48bf-b94f-94665f9e1f53>
+                    )
+               )
+    }
+    """ .

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241002092839-membership-cleanup--insert-municipality-has-sub-organization.graph

@@ -0,0 +1 @@
+http://mu.semte.ch/graphs/public

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241002092839-membership-cleanup--insert-municipality-has-sub-organization.ttl

@@ -0,0 +1,57 @@
+@prefix cleanup: <http://mu.semte.ch/vocabularies/ext/cleanup/> .
+@prefix mu: <http://mu.semte.ch/vocabularies/core/> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+
+# Insert triples of the form
+# <Municipality> <http://www.w3.org/ns/org#hasSubOrganization> <subOrg>
+# matching memberships for which no such triple exists.
+
+# Municipality: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001>
+#
+# AGB: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/36a82ba0-7ff1-4697-a9dd-2e94df73b721>
+# Projectvereniging: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/b156b67f-c5f4-4584-9b30-4c090be02fdc>
+# Dientsverlenende vereniging <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/d01bb1f6-2439-4e33-9c25-1fc295de2e71>
+# Opdrachthoudende vereniging: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/cd93f147-3ece-4308-acab-5c5ada3ec63d>
+# Opdrachthoudende vereniging met private deelname: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/4b8450cf-a326-4c66-9e63-b4ec10acc7f6>
+# HVZ: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/ea446861-2c51-45fa-afd3-4e4a37b71562>
+# PZ: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/a3922c6d-425b-474f-9a02-ffb71a436bfc>
+# PEVA municipality: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/2ad46df5-5c79-4d67-84d5-604c1377231e>
+
+<http://data.lblod.info/id/cleanup-job/3b7f8739-aa56-4a6c-af56-eb9d80022379> a cleanup:Job ;
+    mu:uuid "3b7f8739-aa56-4a6c-af56-eb9d80022379" ;
+    dcterms:title "Insert missing `org:hasSubOrganization` triples for Municipalities." ;
+    cleanup:randomQuery """
+      PREFIX org: <http://www.w3.org/ns/org#>
+
+      INSERT {
+        GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+          ?organization org:hasSubOrganization ?member .
+        }
+      } WHERE {
+        GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+          ?membership a org:Membership ;
+            org:organization ?organization ;
+            org:member ?member ;
+            org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> .
+
+          FILTER NOT EXISTS { ?organization org:hasSubOrganization ?member . }
+
+          ?member org:classification ?memberClass .
+          FILTER ( ?memberClass
+                   IN (
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/36a82ba0-7ff1-4697-a9dd-2e94df73b721> ,
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/b156b67f-c5f4-4584-9b30-4c090be02fdc> ,
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/d01bb1f6-2439-4e33-9c25-1fc295de2e71> ,
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/cd93f147-3ece-4308-acab-5c5ada3ec63d> ,
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/4b8450cf-a326-4c66-9e63-b4ec10acc7f6> ,
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/ea446861-2c51-45fa-afd3-4e4a37b71562> ,
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/a3922c6d-425b-474f-9a02-ffb71a436bfc> ,
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/2ad46df5-5c79-4d67-84d5-604c1377231e>
+                    )
+               )
+        }
+        GRAPH <http://mu.semte.ch/graphs/shared> {
+          ?organization org:classification <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001> .
+        }
+    }
+    """ .

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241002092901-membership-cleanup--insert-municipality-linked-to.graph

@@ -0,0 +1 @@
+http://mu.semte.ch/graphs/public

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241002092901-membership-cleanup--insert-municipality-linked-to.ttl

@@ -0,0 +1,45 @@
+@prefix cleanup: <http://mu.semte.ch/vocabularies/ext/cleanup/> .
+@prefix mu: <http://mu.semte.ch/vocabularies/core/> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+
+# Insert triples of the form:
+# <Municipality> <http://www.w3.org/ns/org#linkedTo> <subOrg>
+# matching memberships for which no such triple exists.
+
+# Municipality: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001>
+#
+# OCMW: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000002>
+# APB: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/80310756-ce0a-4a1b-9b8e-7c01b6cc7a2d>
+
+<http://data.lblod.info/id/cleanup-job/a99bcd5c-9781-425d-8a34-fb11a6e3e262> a cleanup:Job ;
+    mu:uuid "a99bcd5c-9781-425d-8a34-fb11a6e3e262" ;
+    dcterms:title "Insert missing `org:linkedTo` triples for Municipalities." ;
+    cleanup:randomQuery """
+      PREFIX org: <http://www.w3.org/ns/org#>
+
+      INSERT {
+        GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+          ?organization org:linkedTo ?member .
+        }
+      } WHERE {
+        GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
+          ?membership a org:Membership ;
+            org:organization ?organization ;
+            org:member ?member ;
+            org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> .
+
+          FILTER NOT EXISTS { ?organization org:linkedTo ?member . }
+
+          ?member org:classification ?memberClass .
+          FILTER ( ?memberClass
+                   IN (
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000002> ,
+                      <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/80310756-ce0a-4a1b-9b8e-7c01b6cc7a2d>
+                    )
+               )
+        }
+        GRAPH <http://mu.semte.ch/graphs/shared> {
+          ?organization org:classification <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001> .
+        }
+      }
+    """ .

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241002151049-membership-cleanup--delete-worship-linked-to.graph

@@ -0,0 +1 @@
+http://mu.semte.ch/graphs/public

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241002151049-membership-cleanup--delete-worship-linked-to.ttl

@@ -0,0 +1,42 @@
+@prefix cleanup: <http://mu.semte.ch/vocabularies/ext/cleanup/> .
+@prefix mu: <http://mu.semte.ch/vocabularies/core/> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+
+# Delete triples of the form:
+# <Representative Body> <http://www.w3.org/ns/org#linkedTo> <Worship Service>
+# <Representative Body> <http://www.w3.org/ns/org#linkedTo> <Central Worship Service>
+# for which no matching membership exists.
+
+# Worship Service: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/66ec74fd-8cfc-4e16-99c6-350b35012e86>
+# Central Worship Service: <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/f9cac08a-13c1-49da-9bcb-f650b0604054>
+
+<http://data.lblod.info/id/cleanup-job/679eec55-0f69-4d45-84c1-9f73df966983> a cleanup:Job ;
+    mu:uuid "679eec55-0f69-4d45-84c1-9f73df966983" ;
+    dcterms:title "Delete leftover `org:linkedTo` triples for Representative Bodies and [Central] Worship Services." ;
+    cleanup:randomQuery """
+      PREFIX org: <http://www.w3.org/ns/org#>
+
+      DELETE {
+        GRAPH <http://mu.semte.ch/graphs/worship-service> {
+          ?s org:linkedTo ?o .
+        }
+      } WHERE {
+        GRAPH <http://mu.semte.ch/graphs/worship-service> {
+          ?s org:linkedTo ?o .
+          FILTER NOT EXISTS {
+            ?membership a org:Membership ;
+            org:organization ?o ;
+            org:member ?s ;
+            org:role <http://data.lblod.info/id/rollen/4ec7d5c39bdc4e84b4174379b9e22ad8> .
+          }
+
+          ?s org:classification                           <http://data.vlaanderen.be/id/concept/RepresentatiefOrgaanClassificatieCode/89a00b5a-024f-4630-a722-65a5e68967e5> .
+
+          ?o org:classification ?oClass .
+          FILTER ( ?oClass IN (
+                               <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/66ec74fd-8cfc-4e16-99c6-350b35012e86> ,
+                               <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/f9cac08a-13c1-49da-9bcb-f650b0604054>
+                               ))
+        }
+      }
+      """ .

config/migrations/2024/20240930124819-membership-cleanup-jobs/20241002162120-membership-cleanup--delete-worship-has-sub-organization.graph

@@ -0,0 +1 @@
+http://mu.semte.ch/graphs/public