Compare commits

...

74 Commits

Author SHA1 Message Date
Claire Lovisa
20f010bd42 Release v1.29.2 2025-02-17 16:47:32 +01:00
Claire Lovisa
7bb08363e2 Merge pull request #517 from lblod/hotfix/restore-betrokken-lokale-besturen
Restore betrokken lokale besturen
2025-02-17 16:45:29 +01:00
Claire Lovisa
b66278aec5 Restore betrokken lokale besturen 2025-02-17 16:15:17 +01:00
cecemel
b273a77923 Merge pull request #515 from lblod/hotfix/flush-eredienst-mandatarissen
Hotfix/flush eredienst mandatarissen
2025-02-14 16:45:59 +01:00
felix
f689f90e37 Update CHANGELOG.md 2025-02-14 14:31:17 +01:00
felix
20bc051103 Refine migrations 2025-02-13 17:37:52 +01:00
felix
8f57a160e0 update migration name 2025-02-13 17:37:41 +01:00
felix
6f731813bb Ensure different operation jobs for both consumers 2025-02-13 17:05:00 +01:00
felix
03635e4b12 bump consumers 2025-02-13 17:04:52 +01:00
felix
741a7cae2c added migrations to flush again 2025-02-13 17:04:23 +01:00
Claire Lovisa
9ea9d90751 Release v1.29.0 2025-02-07 20:12:22 +01:00
Claire Lovisa
d8114bec4e Release v1.29.0-1 2025-02-05 14:22:40 +01:00
Claire Lovisa
a352683c6f Merge pull request #512 from lblod/fix/mkissing-formatting-dates-consumers
Add dates mapping rules for mandatarissen and leidinggevenden consumers
2025-02-05 14:21:50 +01:00
Claire Lovisa
d601d7f34e Add dates mapping rules for consumers
The mandatarissen and leidinggevenden consumers were missing a date
formatting mapping that is making sure that the data we display to the
user is of the same date as the one shown in the original app. This
transcription was made by the reasoner and we forgot to include it in
the mapping configuration.

I also updated the timestamp of migrations that need to run again when
I'll re-deploy to QA and DEV. It will have no impact on the upcoming
PROD deploy.
2025-02-05 10:05:39 +01:00
Claire Lovisa
aab976bcc1 Release v1.29.0-0 2025-02-03 16:13:22 +01:00
Claire Lovisa
404a282a7d Merge pull request #507 from lblod/chore/bump-mandatarissen-consumer
Bump mandatarissen consumer to v0.1.0
2025-02-03 16:11:15 +01:00
Claire Lovisa
f79724132a Merge branch 'master' into chore/bump-mandatarissen-consumer 2025-02-03 16:11:05 +01:00
Claire Lovisa
0b8530db34 Merge pull request #506 from lblod/revert-505-revert-502-chore/update-leidinggevenden-consumer
Bump leidinggevenden consumer to v0.1.0
2025-02-03 16:04:35 +01:00
Claire Lovisa
17d48100d7 Merge branch 'master' into revert-505-revert-502-chore/update-leidinggevenden-consumer 2025-02-03 16:04:25 +01:00
Claire Lovisa
c6baf82dff Merge pull request #503 from lblod/feature/bump-worship-consumer
Bump worship consumer to v0.1.0
2025-02-03 15:58:59 +01:00
Claire Lovisa
cde5b49654 Merge branch 'master' into feature/bump-worship-consumer 2025-02-03 15:56:38 +01:00
mirdono
7316df1ab2 chore: added release date 2025-01-23 16:47:32 +01:00
Claire Lovisa
300e088b6c Bump mandatarissen consumer to v0.1.0 2025-01-23 16:24:02 +01:00
mirdono
43e7cb0dac chore: bump frontend 2025-01-22 20:26:33 +01:00
Claire Lovisa
e844c9f824 Add reasoner rules back, we'll clean everything up afterwards 2025-01-22 20:18:47 +01:00
mirdono
eb8790f40b chore: add changelog entry 2025-01-22 17:02:43 +01:00
mirdono
e10c825f75 fix: index municipality names as keywords instead of regular text
Index municipality names as keywords instead of as plain text. This prevents
them from being split at word separators. Thereby avoiding that searches for a
municipality name, as used by the frontend's municipality filter, return
organisations whose municipality is a partial match.

For example, previously the results when searching for "Puurs" also included the
organisations located in "Sint-Amands-Puurs". The latter was tokenized into
three terms, therefore also matching plain "Puurs". By treating the municipality
name as keyword this split non longer happens.
2025-01-22 17:02:43 +01:00
felix
0553cd7d66 Update consumer main 2025-01-22 16:47:04 +01:00
Claire Lovisa
55cb772e92 Update phone number mapping
The default behaviour should clean any number that can be cleaned BUT
let the other numbers flow without any modification. The previous
behaviour I'm fixing here was to NOT let them flow if they were not in a
"cleanable" format.
2025-01-22 16:47:04 +01:00
Claire Lovisa
83f247b051 Fixes on container names and env var 2025-01-22 16:47:04 +01:00
Claire Lovisa
950521f187 Bump consumer to v0.1.0 2025-01-22 16:47:04 +01:00
Claire Lovisa
61b88fe602 Adjust flushing migration to delete all mandates
and not only the ones coming from Loket. There is a risk that we delete
too much data here but if that would be the case, the data is
recuperable through the backups by looking at persons and worship
positions that were in OP but not in the worship services landing zone.
2025-01-22 16:47:04 +01:00
Claire Lovisa
dae66c97d3 Add sudo retry to worship consumers 2025-01-22 16:47:04 +01:00
Claire Lovisa
647e17900a Fixes on mapping rules 2025-01-22 16:47:04 +01:00
Claire Lovisa
72b4eaa68a Fix rules about names 2025-01-22 16:47:04 +01:00
Claire Lovisa
bbb0f7a373 Update Changelog and Readme
Inspired by @cecemel in
631ba827ee,
thanks!
2025-01-22 16:47:00 +01:00
Claire Lovisa
bf41ad4ad2 Make configuration more granular 2025-01-22 16:45:29 +01:00
Claire Lovisa
d3899762a9 Remove un-necessary remapping rules 2025-01-22 16:45:29 +01:00
Claire Lovisa
47c4dea630 Bump worship consumer to v0.27.0 into two consumers
The consumer needs to dispatch data to two different graphs: the worship
services graph and the privacy graph, based on triple type and/or
property. The latest version of the consumer doesn't allow it to be done
in one consumer instance, hence de split into two consumers.
2025-01-22 16:45:29 +01:00
Claire Lovisa
140626a936 Update changelog 2025-01-22 16:44:03 +01:00
Claire Lovisa
026ef435e1 Revert "Revert "Chore/update leidinggevenden consumer"" 2025-01-22 16:41:12 +01:00
Claire Lovisa
20d4613400 Merge pull request #505 from lblod/revert-502-chore/update-leidinggevenden-consumer
Revert "Chore/update leidinggevenden consumer"
2025-01-22 16:39:40 +01:00
Claire Lovisa
b358278067 Revert "Chore/update leidinggevenden consumer" 2025-01-22 16:31:44 +01:00
Claire Lovisa
f7a637f0f4 Merge pull request #502 from lblod/chore/update-leidinggevenden-consumer
Chore/update leidinggevenden consumer
2025-01-22 16:24:28 +01:00
Claire Lovisa
eb99585a1d Merge branch 'master' into chore/update-leidinggevenden-consumer 2025-01-22 16:23:19 +01:00
Claire Lovisa
841ac47db6 Merge branch 'chore/update-leidinggevenden-consumer' of github.com:lblod/app-organization-portal into chore/update-leidinggevenden-consumer 2025-01-22 16:12:06 +01:00
Claire Lovisa
3b812946e4 Reorganize migrations and split jobs removal 2025-01-22 16:11:59 +01:00
felix
9972451d0e fix typo 2025-01-22 16:09:25 +01:00
Claire Lovisa
0faa8100c8 Bump consumer to v0.1.0 2025-01-22 15:37:48 +01:00
Claire Lovisa
7834352617 Fix phone mapping 2025-01-22 15:37:39 +01:00
Claire Lovisa
261ba383b6 Update doc 2025-01-22 15:37:24 +01:00
mirdono
62769f855c chore: bump frontend 2025-01-21 12:07:29 +01:00
mirdono
5eca8a2e1d chore: bump frontend 2025-01-21 10:11:46 +01:00
mirdono
0e8885f4dd chore: update changelog 2025-01-21 10:04:23 +01:00
mirdono
3bc47a7fb0 fix: merger change event between St.-Amandus and Sint-Martinus
The worship service St.-Amandus was not included as `orginalOrganization` in the
merger change event somehow. As a result it was not listed as involved
organisation in the change event details. Furthermore, its status was not
correctly updated to "Not active".
2025-01-21 10:04:23 +01:00
felix
1698112a9d Merge branch 'master' into chore/update-leidinggevenden-consumer 2025-01-17 15:15:46 +01:00
felix
a001f6b9e4 Remove old dispatching rules 2025-01-17 15:12:52 +01:00
felix
dba1ada5e2 Updated mapping rules 2025-01-17 14:42:54 +01:00
felix
a6571eefa5 Current consumer is VERY picky on how you write the mapping-query
See:
https://github.com/lblod/delta-consumer/blob/v0.0.27/lib/delta-sparql-mapping.js#L267

make sure you write first the graph or "triple"-pattern and filter
after that. For now.
2025-01-17 12:09:07 +01:00
felix
347844e341 update the filter so empty strings are not ingested 2025-01-17 11:19:19 +01:00
felix
aaf87ee32f update consumer-config 2025-01-17 10:24:15 +01:00
felix
631ba827ee update doc 2025-01-17 10:24:04 +01:00
felix
19e3423202 updated migrations 2025-01-17 10:23:52 +01:00
felix
8750fddb2d added mapping rules 2025-01-17 10:23:41 +01:00
felix
8a7eb8ede0 adding migrations 2025-01-17 09:44:08 +01:00
cecemel
4ff40cb4e4 Merge pull request #500 from lblod/hotfix/peformance-fix-bestuursfuncties-fusiegemeentes
Hotfix/peformance fix bestuursfuncties fusiegemeentes
2025-01-16 14:37:28 +01:00
felix
d53bd27369 Release v1.28.4 2025-01-16 14:35:20 +01:00
felix
90fa373254 fix performance of migration 2025-01-16 14:34:08 +01:00
cecemel
d290187467 Merge pull request #499 from lblod/hotfix/betuursfuncties-fusiegemeentes
Hotfix/betuursfuncties fusiegemeentes
2025-01-16 12:12:39 +01:00
felix
dd08a03faa Update Changlog 2025-01-16 12:12:09 +01:00
felix
79bae4fd3c Updated changelog 2025-01-16 11:36:12 +01:00
cecemel
c570ab994e Merge pull request #498 from lblod/hotfix/add-opdrachthoudende-gemeente-private-deelname
Hotfix/add opdrachthoudende gemeente private deelname
2025-01-13 17:07:28 +01:00
cecemel
c9ba00fc23 Update CHANGELOG.md 2025-01-13 15:55:59 +01:00
felix
c993a681d9 Addendum changelog 2025-01-13 15:06:04 +01:00
126 changed files with 5594 additions and 1439 deletions

View File

@@ -1,20 +1,196 @@
# Changelog
## 1.29.2 (2025-02-17)
### Data fix
- restore bestrokken lokale besturen that have been accidentally flushed [OP-3546]
## 1.29.1 (2025-02-14)
### Backend
- re-init `worship-services-sensitive-consumer` [OP-3483]
### Deploy notes
:warning: Ensure first loket did a full re-sync again of this delta-stream.
```
drc stop worship-services-main-info-consumer worship-services-private-info-consumer
drc retart migrations
```
Update `docker-compose.override.yml` to remove the config of the consumers and replace it by:
```
worship-services-main-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_DISABLE_INITIAL_SYNC: "false"
DCR_DISABLE_DELTA_INGEST: "false"
worship-services-private-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_DISABLE_INITIAL_SYNC: "false"
DCR_DISABLE_DELTA_INGEST: "false"
```
Then
```
drc up -d db mandatarissen-consumer leidinggevenden-consumer worship-services-main-info-consumer worship-services-private-info-consumer
# Wait until success of the previous step
```
Then, update `docker-compose.override.yml` to:
```
worship-services-main-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "db"
DCR_REMAPPING_DATABASE: "db"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
worship-services-private-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "database"
DCR_REMAPPING_DATABASE: "database"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
```
```
drc up -d
sh scripts/reset-elastic.sh
```
## 1.29.0 (2025-02-07)
### Backend
#### Consumer
- Upgraded `mandatarissen-consumer` [OP-3510]
- Upgraded `leidinggevenden-consumer` [OP-3533], [OP-3511]
- Upgraded `worship-services-sensitive-consumer` [OP-3337]
### Deploy notes
```
drc down;
```
Update `docker-compose.override.yml` to remove the config of the consumers and replace it by:
```
mandatarissen-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
leidinggevenden-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
worship-services-main-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_DISABLE_INITIAL_SYNC: "false"
DCR_DISABLE_DELTA_INGEST: "false"
worship-services-private-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_DISABLE_INITIAL_SYNC: "false"
DCR_DISABLE_DELTA_INGEST: "false"
```
Then:
```
drc up -d triplestore migrations
drc up -d db mandatarissen-consumer leidinggevenden-consumer worship-services-main-info-consumer worship-services-private-info-consumer
# Wait until success of the previous step
```
Then, update `docker-compose.override.yml` to:
```
mandatarissen-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_LANDING_ZONE_DATABASE: "db"
DCR_REMAPPING_DATABASE: "db"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
leidinggevenden-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # choose the correct endpoint
DCR_LANDING_ZONE_DATABASE: "db"
DCR_REMAPPING_DATABASE: "db"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
worship-services-main-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "db"
DCR_REMAPPING_DATABASE: "db"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
worship-services-private-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lblod.info" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lblod.info/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "database"
DCR_REMAPPING_DATABASE: "database"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
```
```
drc up -d
sh scripts/reset-elastic.sh
```
## 1.28.5 (2025-01-23)
### Frontend
- Bump to [v1.28.3](https://github.com/lblod/frontend-organization-portal/releases/tag/v1.28.3)
### Backend
- Datafix: correct merger change events for worship organisations (OP-3534)
- Avoid municipality filter to include partial matches (OP-3509)
### Deploy notes
- `drc restart migrations-triggering-indexing; drc logs -ft --tail=200 migrations-triggering-indexing`
- `drc pull frontend; drc up -d frontend`
- Perform a full re-index by executing `./scripts/reset-elastic.sh`
## 1.28.4 (2025-01-16)
### Backend
#### Data
- Performance fix on 1.28.3 [DL-6377]
### Deploy commands
```
drc restart migrations
drc exec delta-producer-background-jobs-initiator curl -X POST http://localhost/public/healing-jobs # or wait for the healing to kick in
```
## 1.28.3 (2025-01-16)
### Backend
#### Data
- Datafix. Fusiegemeentes had different bestuursfuncties attached to OCMW and Gemeente.
- Datafix. Fusiegemeentes had different bestuursfuncties attached to OCMW and Gemeente. [DL-6377]
However, business-rule-wise, these bestuursfuncties are shared.
### Deploy commands
```
drc up -d migrations
drc restart migrations
drc exec delta-producer-background-jobs-initiator curl -X POST http://localhost/public/healing-jobs # or wait for the healing to kick in
```
## 1.28.2 (2025-01-13)
### Backend
- Added "Opdrachthoudende vereniging met private deelname" in delta public
- Added "Opdrachthoudende vereniging met private deelname" in delta public (DL-6368)
### Deploy commands
```
drc up -d migrations
drc restart migrations
drc exec delta-producer-background-jobs-initiator curl -X POST http://localhost/public/healing-jobs # or wait for the healing to kick in
```
## 1.28.1 (2025-01-11)

139
README.md
View File

@@ -11,6 +11,145 @@ Backend for the organization portal application, based on the mu.semte.ch micros
You can shut down using `docker-compose stop` and remove everything using `docker-compose rm`.
#### Setup consumers
The application depends on external data. You'll have to perform extra steps if you want this data to be available.
These are multiple data sets, we'll try to be exhaustive on how to ingest these. Even though the procedures are similar.
#### Setup mandatarissen sync
Note: this is assuming you have never ran this before.
```
drc down;
```
Update `docker-compose.override.yml` to:
```
mandatarissen-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
DCR_LANDING_ZONE_DATABASE: "triplestore"
DCR_REMAPPING_DATABASE: "triplestore"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
```
Then:
```
drc up -d migrations
drc up -d db mandatarissen-consumer
# Wait until success of the previous step
```
Then, update `docker-compose.override.yml` to:
```
mandatarissen-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
DCR_LANDING_ZONE_DATABASE: "db"
DCR_REMAPPING_DATABASE: "db"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
```
Finally, put the stack back up and reindex elasticsearch
```
drc up -d
sh scripts/reset-elastic.sh
```
#### Setup leidinggevenden sync (http://data.lblod.info/vocabularies/leidinggevenden/Functionaris)
Note: this is assuming you have never ran this before.
```
drc down;
```
Update `docker-compose.override.yml` to:
```
leidinggevenden-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
```
Then:
```
drc up -d migrations
drc up -d db leidinggevenden-consumer
# Wait until success of the previous step
```
Then, update `docker-compose.override.yml` to:
```
leidinggevenden-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # choose the correct endpoint
DCR_LANDING_ZONE_DATABASE: "db"
DCR_REMAPPING_DATABASE: "db"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
```
Finally, put the stack back up and reindex elasticsearch
```
drc up -d
sh scripts/reset-elastic.sh
```
#### Setup worship mandatees sync
Note: this is assuming you have never ran this before.
```
drc down;
```
Update `docker-compose.override.yml` to:
```
worship-services-main-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "triplestore"
DCR_REMAPPING_DATABASE: "triplestore"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
worship-services-private-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "triplestore"
DCR_REMAPPING_DATABASE: "triplestore"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
```
Then:
```
drc up -d migrations
drc up -d db worship-services-main-info-consumer worship-services-private-info-consumer
# Wait until success of the previous step
```
Then, update `docker-compose.override.yml` to:
```
worship-services-main-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "db"
DCR_REMAPPING_DATABASE: "db"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
worship-services-private-info-consumer:
environment:
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
DCR_SECRET_KEY: "key-of-the-producer"
DCR_LANDING_ZONE_DATABASE: "db"
DCR_REMAPPING_DATABASE: "db"
DCR_DISABLE_DELTA_INGEST: "false"
DCR_DISABLE_INITIAL_SYNC: "false"
```
Finally, put the stack back up and reindex elasticsearch
```
drc up -d
sh scripts/reset-elastic.sh
```
### Setting up the delta-producers related services
To make sure the app can share data, producers need to be set up. There is an intial sync, that is potentially very expensive, and must be started manually

View File

@@ -1,32 +0,0 @@
const BATCH_SIZE = parseInt(process.env.BATCH_SIZE) || 100;
const MU_CALL_SCOPE_ID_INITIAL_SYNC = process.env.MU_CALL_SCOPE_ID_INITIAL_SYNC || 'http://redpencil.data.gift/id/concept/muScope/deltas/consumer/initialSync';
const BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES = process.env.BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES == 'true' ? true : false;
const DIRECT_DATABASE_ENDPOINT = process.env.DIRECT_DATABASE_ENDPOINT || 'http://triplestore:8890/sparql';
const MAX_DB_RETRY_ATTEMPTS = parseInt(process.env.MAX_DB_RETRY_ATTEMPTS || 5);
const MAX_REASONING_RETRY_ATTEMPTS = parseInt(process.env.MAX_REASONING_RETRY_ATTEMPTS || 5);
const SLEEP_BETWEEN_BATCHES = parseInt(process.env.SLEEP_BETWEEN_BATCHES || 1000);
const SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION || 10000);
const SLEEP_TIME_AFTER_FAILED_DB_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_DB_OPERATION || 60000);
// Graphs
const TARGET_GRAPH = process.env.TARGET_GRAPH || `http://mu.semte.ch/graphs/administrative-unit`;
const LANDING_ZONE_GRAPH = process.env.DCR_LANDING_ZONE_GRAPH || `http://mu.semte.ch/graphs/landing-zone/leidinggevenden`;
// ENDPOINTS
const LANDING_ZONE_DATABASE = process.env.DCR_LANDING_ZONE_DATABASE || 'db';
const LANDING_ZONE_DATABASE_ENDPOINT = process.env.DCR_LANDING_ZONE_DATABASE_ENDPOINT || `http://${LANDING_ZONE_DATABASE}:8890/sparql`;
module.exports = {
BATCH_SIZE,
MU_CALL_SCOPE_ID_INITIAL_SYNC,
BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
DIRECT_DATABASE_ENDPOINT,
MAX_DB_RETRY_ATTEMPTS,
MAX_REASONING_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
LANDING_ZONE_GRAPH,
TARGET_GRAPH,
LANDING_ZONE_DATABASE_ENDPOINT
};

View File

@@ -1,72 +0,0 @@
const PREFIXES = `
PREFIX org: <http://www.w3.org/ns/org#>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>`
const contextConfig = {
addTypes: {
scope: 'all', // 'inserts, 'deletes', 'all' or none. To add rdf:type to subjects of inserts, deletes or both
exhausitive: false, // true or false: find all types for a subject, even if one is already present in delta
},
contextQueries: [
// Case Leidnggevenden
// They can be triggered when being inserted or when their link to a bestuursfunctie is added
{
trigger: { // subjectType or predicateValue
subjectType: "lblodlg:Functionaris"
},
queryTemplate: (subject) => `
${PREFIXES}
CONSTRUCT {
?functionaris org:holds ?bestuursfunctie ;
?p ?o .
?orgaanInTime lblodlg:heeftBestuursfunctie ?bestuursfunctie ;
mandaat:isTijdspecialisatieVan ?orgaan .
?orgaan besluit:bestuurt ?eenheid .
?eenheid besluit:classificatie ?classification .
} WHERE {
GRAPH <http://mu.semte.ch/graphs/landing-zone/leidinggevenden> {
VALUES ?functionaris { ${subject} }
?functionaris org:holds ?bestuursfunctie ;
?p ?o .
?orgaanInTime lblodlg:heeftBestuursfunctie ?bestuursfunctie ;
mandaat:isTijdspecialisatieVan ?orgaan .
?orgaan besluit:bestuurt ?eenheid .
?eenheid besluit:classificatie ?classification .
}
}`
},
{
trigger: {
predicateValue: "org:holds"
},
queryTemplate: (subject) => `
${PREFIXES}
CONSTRUCT {
?functionaris org:holds ?bestuursfunctie ;
?p ?o .
?orgaanInTime lblodlg:heeftBestuursfunctie ?bestuursfunctie ;
mandaat:isTijdspecialisatieVan ?orgaan .
?orgaan besluit:bestuurt ?eenheid .
?eenheid besluit:classificatie ?classification .
} WHERE {
GRAPH <http://mu.semte.ch/graphs/landing-zone/leidinggevenden> {
VALUES ?functionaris { ${subject} }
?functionaris org:holds ?bestuursfunctie ;
?p ?o .
?orgaanInTime lblodlg:heeftBestuursfunctie ?bestuursfunctie ;
mandaat:isTijdspecialisatieVan ?orgaan .
?orgaan besluit:bestuurt ?eenheid .
?eenheid besluit:classificatie ?classification .
}
}`
}
]
}
module.exports = {
contextConfig,
PREFIXES
};

View File

@@ -1,85 +0,0 @@
const {
transformStatements,
deleteFromTargetGraph,
insertIntoTargetGraph,
} = require('./util');
/**
* Dispatch the fetched information to a target graph.
* Note: <share://file/data> will be ADDED to it's own graph.
* We take only care of adding them, not updating triples, this is a TODO
* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
* @param { termObjectChangeSets: { deletes, inserts } } data - The fetched changes sets, which objects of serialized Terms
* [ {
* graph: "<http://foo>",
* subject: "<http://bar>",
* predicate: "<http://baz>",
* object: "<http://boom>^^<http://datatype>"
* }
* ]
* @return {void} Nothing
*/
async function dispatch(lib, data) {
const { mu, fetch } = lib;
let { termObjectChangeSets, termObjectChangeSetsWithContext } = data;
// Both arrays are the same length, so we can zip them together for easier processing
zippedChangeSets = termObjectChangeSets.map((o, i) => ({
original: o,
withContext: termObjectChangeSetsWithContext[i]
}));
console.log(`Received ${zippedChangeSets.length} change sets`)
for (let { original, withContext } of zippedChangeSets) {
let originalDeletes, originalInserts, insertsWithContext, deletesWithContext;
try {
originalInserts = original.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
originalDeletes = original.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
} catch (e) {
console.log(`Error while processing change set: ${e}`);
console.log(`Original change set: ${JSON.stringify(original)}`);
throw e;
}
// Extra context needed for mapping from DL to OP model and filtering based on type.
try {
insertsWithContext = withContext.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
deletesWithContext = withContext.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
} catch (e) {
console.log(`Error while processing change set with context: ${e}`);
console.log(`Change set with context: ${JSON.stringify(withContext)}`);
throw e;
}
if (originalDeletes.length) {
// Map deletes from DL to OP model
const transformedDeletes = await transformStatements(fetch, deletesWithContext);
if (!transformedDeletes.length) {
console.log(`Warn: Delete statements mapped to empty result.`);
console.log(`Input: ${deletesWithContext}`);
console.log(`Output: ${transformedDeletes}`);
} else {
await deleteFromTargetGraph(lib, transformedDeletes);
}
}
if (originalInserts.length) {
// Map inserts from DL to OP model
const transformedInserts = await transformStatements(fetch, insertsWithContext);
if (!transformedInserts.length) {
console.log(`Warn: Insert statements mapped to empty result.`);
console.log(`Input: ${insertsWithContext}`);
console.log(`Output: ${transformedInserts}`);
} else {
await insertIntoTargetGraph(lib, transformedInserts);
}
}
}
}
module.exports = {
dispatch
};

View File

@@ -1,85 +0,0 @@
const { batchedDbUpdate, transformLandingZoneGraph } = require('./util');
const {
BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
DIRECT_DATABASE_ENDPOINT,
MU_CALL_SCOPE_ID_INITIAL_SYNC,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
TARGET_GRAPH,
LANDING_ZONE_GRAPH,
LANDING_ZONE_DATABASE_ENDPOINT
} = require('./config');
const endpoint = BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES ? DIRECT_DATABASE_ENDPOINT : process.env.MU_SPARQL_ENDPOINT;
/**
* Dispatch the fetched information to a target graph.
* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
* @param { termObjects } data - The fetched quad information, which objects of serialized Terms
* [ {
* graph: "<http://foo>",
* subject: "<http://bar>",
* predicate: "<http://baz>",
* object: "<http://boom>^^<http://datatype>"
* }
* ]
* @return {void} Nothing
*/
async function dispatch(lib, data) {
const { mu, muAuthSudo } = lib;
const { termObjects } = data;
// Steps:
// 1. Write the triples to the landing zone graph - withouth any mapping or filtering
// 2. One-time reasoning run with the full landing zone graph and write the results to the target graph (on-finish)
if (BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES) {
console.warn(`Service configured to skip MU_AUTH!`);
}
console.log(`Using ${endpoint} to insert triples`);
if (termObjects.length) {
const originalInsertTriples = termObjects.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
await batchedDbUpdate(
muAuthSudo.updateSudo,
LANDING_ZONE_GRAPH,
originalInsertTriples,
{ 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
LANDING_ZONE_DATABASE_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION
);
}
}
async function onFinishInitialIngest(lib) {
const { muAuthSudo, fetch } = lib;
console.log(`!! On-finish triggered !!`);
const transformedInsertTriples = await transformLandingZoneGraph(fetch, endpoint);
console.log(`Transformed ${transformedInsertTriples.length} triples`);
await batchedDbUpdate(
muAuthSudo.updateSudo,
TARGET_GRAPH,
transformedInsertTriples,
{ 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
endpoint,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION
);
}
module.exports = {
dispatch,
onFinishInitialIngest
};

View File

@@ -1,168 +0,0 @@
const {
MAX_REASONING_RETRY_ATTEMPTS,
SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
TARGET_GRAPH,
LANDING_ZONE_GRAPH,
BATCH_SIZE,
} = require('./config');
async function batchedDbUpdate(
muUpdate,
graph,
triples,
extraHeaders,
endpoint,
batchSize,
maxAttempts,
sleepBetweenBatches = 1000,
sleepTimeOnFail = 1000,
operation = 'INSERT'
) {
for (let i = 0; i < triples.length; i += batchSize) {
console.log(`Inserting triples in batch: ${i}-${i + batchSize}`);
const batch = triples.slice(i, i + batchSize).join('\n');
const insertCall = async () => {
await muUpdate(`
${operation} DATA {
GRAPH <${graph}> {
${batch}
}
}
`, extraHeaders, endpoint);
};
await operationWithRetry(insertCall, 0, maxAttempts, sleepTimeOnFail);
console.log(`Sleeping before next query execution: ${sleepBetweenBatches}`);
await new Promise(r => setTimeout(r, sleepBetweenBatches));
}
}
async function operationWithRetry(callback,
attempt,
maxAttempts,
sleepTimeOnFail) {
try {
if (typeof callback === "function")
return await callback();
else // Catch error from promise - not how I would do it normally, but allows re use of existing code.
return await callback;
}
catch (e) {
console.log(`Operation failed for ${callback.toString()}, attempt: ${attempt} of ${maxAttempts}`);
console.log(`Error: ${e}`);
console.log(`Sleeping ${sleepTimeOnFail} ms`);
if (attempt >= maxAttempts) {
console.log(`Max attempts reached for ${callback.toString()}, giving up`);
throw e;
}
await new Promise(r => setTimeout(r, sleepTimeOnFail));
return operationWithRetry(callback, ++attempt, maxAttempts, sleepTimeOnFail);
}
}
/**
* Splits an array into two parts, a part that passes and a part that fails a predicate function.
* Credits: https://github.com/benjay10
* @public
* @function partition
* @param {Array} arr - Array to be partitioned
* @param {Function} fn - Function that accepts single argument: an element of the array, and should return a truthy or falsy value.
* @returns {Object} Object that contains keys passes and fails, each representing an array with elemets that pass or fail the predicate respectively
*/
function partition(arr, fn) {
let passes = [], fails = [];
arr.forEach((item) => (fn(item) ? passes : fails).push(item));
return { passes, fails };
}
/**
* Send triples to reasoning service for conversion
*
*/
async function transformTriples(fetch, triples, mapping) {
return await operationWithRetry(mainConversion(fetch, triples, mapping), 0,
MAX_REASONING_RETRY_ATTEMPTS, SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION);
}
async function mainConversion(fetch, triples, mapping) {
let formdata = new URLSearchParams();
formdata.append("data", triples);
let requestOptions = {
method: 'POST',
body: formdata,
redirect: 'follow'
};
const response = await fetch(`http://reasoner/reason/dl2op/${mapping}`, requestOptions)
return await response.text();
}
async function transformStatements(fetch, triples, mapping = 'main') {
console.log(`Received ${JSON.stringify(triples)} to transform`);
const transformedTriples = await transformTriples(fetch, triples.join('\n'), mapping);
statements = transformedTriples ? transformedTriples.replace(/\n{2,}/g, '').split('\n') : [];
console.log(`CONVERSION: FROM ${triples.length} triples to ${statements.length}`);
return statements;
}
async function deleteFromTargetGraph(lib, statements) {
console.log(`Deleting ${statements.length} statements from target graph`);
console.log(`Statements: ${JSON.stringify(statements)}`)
await batchedDbUpdate(
lib.muAuthSudo.updateSudo,
TARGET_GRAPH,
statements,
{},
process.env.MU_SPARQL_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
'DELETE');
}
async function insertIntoTargetGraph(lib, statements) {
console.log(`Inserting ${statements.length} statements into target graph`);
console.log(`Statements: ${JSON.stringify(statements)}`)
await batchedDbUpdate(
lib.muAuthSudo.updateSudo,
TARGET_GRAPH,
statements,
{},
process.env.MU_SPARQL_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
'INSERT');
}
async function transformLandingZoneGraph(fetch, endpoint, mapping = 'main') {
console.log(`Transforming landing zone graph: ${LANDING_ZONE_GRAPH}`);
const response = await fetch(`http://reasoner/reason/dl2op/${mapping}?data=${encodeURIComponent(`${endpoint}?default-graph-uri=&query=CONSTRUCT+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D+WHERE+%7B%0D%0A+GRAPH+<${LANDING_ZONE_GRAPH}>+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D%0D%0A%7D&should-sponge=&format=text%2Fplain&timeout=0&run=+Run+Query`)}`);
const text = await response.text();
const statements = text.replace(/\n{2,}/g, '').split('\n');
return statements;
}
module.exports = {
batchedDbUpdate,
partition,
transformStatements,
transformLandingZoneGraph,
deleteFromTargetGraph,
insertIntoTargetGraph
};

View File

@@ -0,0 +1,21 @@
PREFIX schema: <http://schema.org/>
CONSTRUCT {
?s schema:telephone ?cleanedPhoneNumber .
} WHERE {
?s schema:telephone ?phoneNumber .
# Remove all non digits
BIND(REPLACE(?phoneNumber, """[^+\\d]""", "") AS ?scrapeResult)
# Replace leading 00 with +
BIND(REPLACE(?scrapeResult, """^00""", "+") AS ?replacedZeroZero)
BIND(REPLACE(?replacedZeroZero, """^0""", "+32") AS ?defaultBe)
BIND(REPLACE(?defaultBe, """\\+320""", "+32") AS ?extendedDefaultBe)
BIND(REPLACE(?extendedDefaultBe, """^320""", "+32") AS ?finalResult)
BIND(IF(REGEX(?finalResult, """^\\+((32\\d{8})|(324\\d{8})|((?!32)\\d+))$"""), CONCAT("tel:", ?finalResult), ?phoneNumber) AS ?cleanedPhoneNumber)
}

View File

@@ -0,0 +1,11 @@
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
CONSTRUCT {
?s mandaat:einde ?endDateTime .
} WHERE {
?s mandaat:einde ?endDate .
BIND(SUBSTR(STR(?endDate), 1, 10) as ?endDateWithoutTimezone)
BIND(STRDT(CONCAT(?endDateWithoutTimezone, 'T02:00:00'), xsd:dateTime) as ?endDateTime)
}

View File

@@ -0,0 +1,11 @@
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
CONSTRUCT {
?s mandaat:start ?startDateTime .
} WHERE {
?s mandaat:start ?startDate .
BIND(SUBSTR(STR(?startDate), 1, 10) as ?startDateWithoutTimezone)
BIND(STRDT(CONCAT(?startDateWithoutTimezone, 'T02:00:00'), xsd:dateTime) as ?startDateTime)
}

View File

@@ -0,0 +1,14 @@
PREFIX dc_terms: <http://purl.org/dc/terms/>
PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
PREFIX person: <http://www.w3.org/ns/person#>
CONSTRUCT {
?s a dc_terms:Agent .
} WHERE {
?s a ?type .
VALUES ?type {
lblodlg:Functionaris
person:Person
}
}

View File

@@ -0,0 +1,12 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
CONSTRUCT {
?s a ch:AgentInPositie .
} WHERE {
?s a ?type .
VALUES ?type {
lblodlg:Functionaris
}
}

View File

@@ -0,0 +1,8 @@
PREFIX org: <http://www.w3.org/ns/org#>
CONSTRUCT {
?o org:heldBy ?s.
}
WHERE {
?s org:holds ?o.
}

View File

@@ -0,0 +1,10 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://www.w3.org/ns/locn#Address>;
?p ?o.
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,7 @@
CONSTRUCT {
?s <http://schema.org/contactPoint> ?contactpoint.
}
WHERE {
?s a <http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie>.
?s <http://schema.org/contactPoint> ?contactpoint.
}

View File

@@ -0,0 +1,11 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://schema.org/ContactPoint>;
?p ?o.
FILTER (?p NOT IN (<http://schema.org/telephone>))
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,14 @@
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://data.lblod.info/vocabularies/leidinggevenden/Functionaris>;
?p ?o.
FILTER (?p NOT IN (mandaat:start, mandaat:einde))
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,16 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://www.w3.org/ns/person#Person>;
?p ?o.
FILTER (?p IN (
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>,
<http://xmlns.com/foaf/0.1/familyName>,
<http://mu.semte.ch/vocabularies/core/uuid>
))
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,21 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
PREFIX person: <http://www.w3.org/ns/person#>
# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
# First name: persoon:gebruikteVoornaam => foaf:givenName
# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
# We do both in one query with an OPTIONAL because they need to happen simultaneously
# not to end up in a loop
# Part 1
CONSTRUCT {
?s persoon:gebruikteVoornaam ?name .
} WHERE {
?s a person:Person;
foaf:name ?name .
}

View File

@@ -0,0 +1,21 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
PREFIX person: <http://www.w3.org/ns/person#>
# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
# First name: persoon:gebruikteVoornaam => foaf:givenName
# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
# We do both in one query with an OPTIONAL because they need to happen simultaneously
# not to end up in a loop
# Part 2
CONSTRUCT {
?s foaf:givenName ?gebruikteVoornaam .
} WHERE {
?s a person:Person ;
persoon_loket:gebruikteVoornaam ?gebruikteVoornaam .
}

View File

@@ -1,33 +0,0 @@
const BATCH_SIZE = parseInt(process.env.BATCH_SIZE) || 100;
const MU_CALL_SCOPE_ID_INITIAL_SYNC = process.env.MU_CALL_SCOPE_ID_INITIAL_SYNC || 'http://redpencil.data.gift/id/concept/muScope/deltas/consumer/initialSync';
const BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES = process.env.BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES == 'true' ? true : false;
const DIRECT_DATABASE_ENDPOINT = process.env.DIRECT_DATABASE_ENDPOINT || 'http://triplestore:8890/sparql';
const MAX_DB_RETRY_ATTEMPTS = parseInt(process.env.MAX_DB_RETRY_ATTEMPTS || 5);
const MAX_REASONING_RETRY_ATTEMPTS = parseInt(process.env.MAX_REASONING_RETRY_ATTEMPTS || 5);
const SLEEP_BETWEEN_BATCHES = parseInt(process.env.SLEEP_BETWEEN_BATCHES || 1000);
const SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION || 10000);
const SLEEP_TIME_AFTER_FAILED_DB_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_DB_OPERATION || 60000);
// Graphs
const TARGET_GRAPH = process.env.TARGET_GRAPH || `http://mu.semte.ch/graphs/administrative-unit`;
const LANDING_ZONE_GRAPH = process.env.DCR_LANDING_ZONE_GRAPH || `http://mu.semte.ch/graphs/landing-zone/leidinggevenden`;
// ENDPOINTS
const LANDING_ZONE_DATABASE = process.env.DCR_LANDING_ZONE_DATABASE || 'db';
const LANDING_ZONE_DATABASE_ENDPOINT = process.env.DCR_LANDING_ZONE_DATABASE_ENDPOINT || `http://${LANDING_ZONE_DATABASE}:8890/sparql`;
module.exports = {
BATCH_SIZE,
MU_CALL_SCOPE_ID_INITIAL_SYNC,
BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
DIRECT_DATABASE_ENDPOINT,
MAX_DB_RETRY_ATTEMPTS,
MAX_REASONING_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
LANDING_ZONE_GRAPH,
LANDING_ZONE_DATABASE_ENDPOINT,
TARGET_GRAPH
};

View File

@@ -1,72 +0,0 @@
const PREFIXES = `
PREFIX org: <http://www.w3.org/ns/org#>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>`
const contextConfig = {
addTypes: {
scope: 'all', // 'inserts, 'deletes', 'all' or none. To add rdf:type to subjects of inserts, deletes or both
exhausitive: false, // true or false: find all types for a subject, even if one is already present in delta
},
contextQueries: [
// Case Mandatarissen
// They can be triggered when being inserted or when their link to a mandate is added
{
trigger: { // subjectType or predicateValue
subjectType: "mandaat:Mandataris"
},
queryTemplate: (subject) => `
${PREFIXES}
CONSTRUCT {
?mandataris org:holds ?mandate ;
?p ?o .
?orgaanInTime org:hasPost ?mandate ;
mandaat:isTijdspecialisatieVan ?orgaan .
?orgaan besluit:bestuurt ?eenheid .
?eenheid besluit:classificatie ?classification .
} WHERE {
GRAPH <http://mu.semte.ch/graphs/landing-zone/mandatarissen> {
VALUES ?mandataris { ${subject} }
?mandataris org:holds ?mandate ;
?p ?o .
?orgaanInTime org:hasPost ?mandate ;
mandaat:isTijdspecialisatieVan ?orgaan .
?orgaan besluit:bestuurt ?eenheid .
?eenheid besluit:classificatie ?classification .
}
}`
},
{
trigger: {
predicateValue: "org:holds"
},
queryTemplate: (subject) => `
${PREFIXES}
CONSTRUCT {
?mandataris org:holds ?mandate ;
?p ?o .
?orgaanInTime org:hasPost ?mandate ;
mandaat:isTijdspecialisatieVan ?orgaan .
?orgaan besluit:bestuurt ?eenheid .
?eenheid besluit:classificatie ?classification .
} WHERE {
GRAPH <http://mu.semte.ch/graphs/landing-zone/mandatarissen> {
VALUES ?mandataris { ${subject} }
?mandataris org:holds ?mandate ;
?p ?o .
?orgaanInTime org:hasPost ?mandate ;
mandaat:isTijdspecialisatieVan ?orgaan .
?orgaan besluit:bestuurt ?eenheid .
?eenheid besluit:classificatie ?classification .
}
}`
}
]
}
module.exports = {
contextConfig,
PREFIXES
};

View File

@@ -1,85 +0,0 @@
const {
transformStatements,
deleteFromTargetGraph,
insertIntoTargetGraph,
} = require('./util');
/**
* Dispatch the fetched information to a target graph.
* Note: <share://file/data> will be ADDED to it's own graph.
* We take only care of adding them, not updating triples, this is a TODO
* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
* @param { termObjectChangeSets: { deletes, inserts } } data - The fetched changes sets, which objects of serialized Terms
* [ {
* graph: "<http://foo>",
* subject: "<http://bar>",
* predicate: "<http://baz>",
* object: "<http://boom>^^<http://datatype>"
* }
* ]
* @return {void} Nothing
*/
async function dispatch(lib, data) {
const { mu, fetch } = lib;
let { termObjectChangeSets, termObjectChangeSetsWithContext } = data;
// Both arrays are the same length, so we can zip them together for easier processing
zippedChangeSets = termObjectChangeSets.map((o, i) => ({
original: o,
withContext: termObjectChangeSetsWithContext[i]
}));
console.log(`Received ${zippedChangeSets.length} change sets`)
for (let { original, withContext } of zippedChangeSets) {
let originalDeletes, originalInserts, insertsWithContext, deletesWithContext;
try {
originalInserts = original.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
originalDeletes = original.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
} catch (e) {
console.log(`Error while processing change set: ${e}`);
console.log(`Original change set: ${JSON.stringify(original)}`);
throw e;
}
// Extra context needed for mapping from DL to OP model and filtering based on type.
try {
insertsWithContext = withContext.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
deletesWithContext = withContext.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
} catch (e) {
console.log(`Error while processing change set with context: ${e}`);
console.log(`Change set with context: ${JSON.stringify(withContext)}`);
throw e;
}
if (originalDeletes.length) {
// Map deletes from DL to OP model
const transformedDeletes = await transformStatements(fetch, deletesWithContext);
if (!transformedDeletes.length) {
console.log(`Warn: Delete statements mapped to empty result.`);
console.log(`Input: ${deletesWithContext}`);
console.log(`Output: ${transformedDeletes}`);
} else {
await deleteFromTargetGraph(lib, transformedDeletes);
}
}
if (originalInserts.length) {
// Map inserts from DL to OP model
const transformedInserts = await transformStatements(fetch, insertsWithContext);
if (!transformedInserts.length) {
console.log(`Warn: Insert statements mapped to empty result.`);
console.log(`Input: ${insertsWithContext}`);
console.log(`Output: ${transformedInserts}`);
} else {
await insertIntoTargetGraph(lib, transformedInserts);
}
}
}
}
module.exports = {
dispatch
};

View File

@@ -1,85 +0,0 @@
const { batchedDbUpdate, transformLandingZoneGraph } = require('./util');
const {
BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
DIRECT_DATABASE_ENDPOINT,
MU_CALL_SCOPE_ID_INITIAL_SYNC,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
TARGET_GRAPH,
LANDING_ZONE_GRAPH,
LANDING_ZONE_DATABASE_ENDPOINT
} = require('./config');
const endpoint = BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES ? DIRECT_DATABASE_ENDPOINT : process.env.MU_SPARQL_ENDPOINT;
/**
* Dispatch the fetched information to a target graph.
* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
* @param { termObjects } data - The fetched quad information, which objects of serialized Terms
* [ {
* graph: "<http://foo>",
* subject: "<http://bar>",
* predicate: "<http://baz>",
* object: "<http://boom>^^<http://datatype>"
* }
* ]
* @return {void} Nothing
*/
async function dispatch(lib, data) {
const { mu, muAuthSudo } = lib;
const { termObjects } = data;
// Steps:
// 1. Write the triples to the landing zone graph - withouth any mapping or filtering
// 2. One-time reasoning run with the full landing zone graph and write the results to the target graph (on-finish)
if (BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES) {
console.warn(`Service configured to skip MU_AUTH!`);
}
console.log(`Using ${endpoint} to insert triples`);
if (termObjects.length) {
const originalInsertTriples = termObjects.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
await batchedDbUpdate(
muAuthSudo.updateSudo,
LANDING_ZONE_GRAPH,
originalInsertTriples,
{ 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
LANDING_ZONE_DATABASE_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION
);
}
}
async function onFinishInitialIngest(lib) {
const { muAuthSudo, fetch } = lib;
console.log(`!! On-finish triggered !!`);
const transformedInsertTriples = await transformLandingZoneGraph(fetch, endpoint);
console.log(`Transformed ${transformedInsertTriples.length} triples`);
await batchedDbUpdate(
muAuthSudo.updateSudo,
TARGET_GRAPH,
transformedInsertTriples,
{ 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
endpoint,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION
);
}
module.exports = {
dispatch,
onFinishInitialIngest
};

View File

@@ -1,168 +0,0 @@
const {
MAX_REASONING_RETRY_ATTEMPTS,
SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
TARGET_GRAPH,
LANDING_ZONE_GRAPH,
BATCH_SIZE
} = require('./config');
async function batchedDbUpdate(
muUpdate,
graph,
triples,
extraHeaders,
endpoint,
batchSize,
maxAttempts,
sleepBetweenBatches = 1000,
sleepTimeOnFail = 1000,
operation = 'INSERT'
) {
for (let i = 0; i < triples.length; i += batchSize) {
console.log(`Inserting triples in batch: ${i}-${i + batchSize}`);
const batch = triples.slice(i, i + batchSize).join('\n');
const insertCall = async () => {
await muUpdate(`
${operation} DATA {
GRAPH <${graph}> {
${batch}
}
}
`, extraHeaders, endpoint);
};
await operationWithRetry(insertCall, 0, maxAttempts, sleepTimeOnFail);
console.log(`Sleeping before next query execution: ${sleepBetweenBatches}`);
await new Promise(r => setTimeout(r, sleepBetweenBatches));
}
}
async function operationWithRetry(callback,
attempt,
maxAttempts,
sleepTimeOnFail) {
try {
if (typeof callback === "function")
return await callback();
else // Catch error from promise - not how I would do it normally, but allows re use of existing code.
return await callback;
}
catch (e) {
console.log(`Operation failed for ${callback.toString()}, attempt: ${attempt} of ${maxAttempts}`);
console.log(`Error: ${e}`);
console.log(`Sleeping ${sleepTimeOnFail} ms`);
if (attempt >= maxAttempts) {
console.log(`Max attempts reached for ${callback.toString()}, giving up`);
throw e;
}
await new Promise(r => setTimeout(r, sleepTimeOnFail));
return operationWithRetry(callback, ++attempt, maxAttempts, sleepTimeOnFail);
}
}
/**
* Splits an array into two parts, a part that passes and a part that fails a predicate function.
* Credits: https://github.com/benjay10
* @public
* @function partition
* @param {Array} arr - Array to be partitioned
* @param {Function} fn - Function that accepts single argument: an element of the array, and should return a truthy or falsy value.
* @returns {Object} Object that contains keys passes and fails, each representing an array with elemets that pass or fail the predicate respectively
*/
function partition(arr, fn) {
let passes = [], fails = [];
arr.forEach((item) => (fn(item) ? passes : fails).push(item));
return { passes, fails };
}
/**
* Send triples to reasoning service for conversion
*
*/
async function transformTriples(fetch, triples, mapping) {
return await operationWithRetry(mainConversion(fetch, triples, mapping), 0,
MAX_REASONING_RETRY_ATTEMPTS, SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION);
}
async function mainConversion(fetch, triples, mapping) {
let formdata = new URLSearchParams();
formdata.append("data", triples);
let requestOptions = {
method: 'POST',
body: formdata,
redirect: 'follow'
};
const response = await fetch(`http://reasoner/reason/dl2op/${mapping}`, requestOptions)
return await response.text();
}
async function transformStatements(fetch, triples, mapping = 'main') {
console.log(`Received ${JSON.stringify(triples)} to transform`);
const transformedTriples = await transformTriples(fetch, triples.join('\n'), mapping);
statements = transformedTriples ? transformedTriples.replace(/\n{2,}/g, '').split('\n') : [];
console.log(`CONVERSION: FROM ${triples.length} triples to ${statements.length}`);
return statements;
}
async function deleteFromTargetGraph(lib, statements) {
console.log(`Deleting ${statements.length} statements from target graph`);
console.log(`Statements: ${JSON.stringify(statements)}`)
await batchedDbUpdate(
lib.muAuthSudo.updateSudo,
TARGET_GRAPH,
statements,
{},
process.env.MU_SPARQL_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
'DELETE');
}
async function insertIntoTargetGraph(lib, statements) {
console.log(`Inserting ${statements.length} statements into target graph`);
console.log(`Statements: ${JSON.stringify(statements)}`)
await batchedDbUpdate(
lib.muAuthSudo.updateSudo,
TARGET_GRAPH,
statements,
{},
process.env.MU_SPARQL_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
'INSERT');
}
async function transformLandingZoneGraph(fetch, endpoint, mapping = 'main') {
console.log(`Transforming landing zone graph: ${LANDING_ZONE_GRAPH}`);
const response = await fetch(`http://reasoner/reason/dl2op/${mapping}?data=${encodeURIComponent(`${endpoint}?default-graph-uri=&query=CONSTRUCT+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D+WHERE+%7B%0D%0A+GRAPH+<${LANDING_ZONE_GRAPH}>+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D%0D%0A%7D&should-sponge=&format=text%2Fplain&timeout=0&run=+Run+Query`)}`);
const text = await response.text();
const statements = text.replace(/\n{2,}/g, '').split('\n');
return statements;
}
module.exports = {
batchedDbUpdate,
partition,
transformStatements,
transformLandingZoneGraph,
deleteFromTargetGraph,
insertIntoTargetGraph
};

View File

@@ -0,0 +1,11 @@
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
CONSTRUCT {
?s mandaat:einde ?endDateTime .
} WHERE {
?s mandaat:einde ?endDate .
BIND(SUBSTR(STR(?endDate), 1, 10) as ?endDateWithoutTimezone)
BIND(STRDT(CONCAT(?endDateWithoutTimezone, 'T02:00:00'), xsd:dateTime) as ?endDateTime)
}

View File

@@ -0,0 +1,11 @@
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
CONSTRUCT {
?s mandaat:start ?startDateTime .
} WHERE {
?s mandaat:start ?startDate .
BIND(SUBSTR(STR(?startDate), 1, 10) as ?startDateWithoutTimezone)
BIND(STRDT(CONCAT(?startDateWithoutTimezone, 'T02:00:00'), xsd:dateTime) as ?startDateTime)
}

View File

@@ -0,0 +1,14 @@
PREFIX dc_terms: <http://purl.org/dc/terms/>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX person: <http://www.w3.org/ns/person#>
CONSTRUCT {
?s a dc_terms:Agent .
} WHERE {
?s a ?type .
VALUES ?type {
mandaat:Mandataris
person:Person
}
}

View File

@@ -0,0 +1,11 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
CONSTRUCT {
?s a ch:AgentInPositie .
} WHERE {
?s a ?type .
VALUES ?type {
mandaat:Mandataris
}
}

View File

@@ -0,0 +1,8 @@
PREFIX org: <http://www.w3.org/ns/org#>
CONSTRUCT {
?o org:heldBy ?s.
}
WHERE {
?s org:holds ?o.
}

View File

@@ -0,0 +1,29 @@
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://data.vlaanderen.be/ns/mandaat#Mandataris>;
<http://www.w3.org/ns/org#holds> ?mandate;
?p ?o.
?orgaanInTime <http://www.w3.org/ns/org#hasPost> ?mandate ;
<http://data.vlaanderen.be/ns/mandaat#isTijdspecialisatieVan> ?orgaan .
?orgaan <http://data.vlaanderen.be/ns/besluit#bestuurt> ?bestuur .
?bestuur <http://data.vlaanderen.be/ns/besluit#classificatie> ?classification .
FILTER(?classification IN (
<http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000000>,
<http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001>,
<http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000002>,
<http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000003>
))
FILTER (?p NOT IN (mandaat:start, mandaat:einde))
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,16 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://www.w3.org/ns/person#Person>;
?p ?o.
FILTER (?p IN (
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>,
<http://xmlns.com/foaf/0.1/familyName>,
<http://mu.semte.ch/vocabularies/core/uuid>
))
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,21 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
PREFIX person: <http://www.w3.org/ns/person#>
# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
# First name: persoon:gebruikteVoornaam => foaf:givenName
# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
# We do both in one query with an OPTIONAL because they need to happen simultaneously
# not to end up in a loop
# Part 1
CONSTRUCT {
?s persoon:gebruikteVoornaam ?name .
} WHERE {
?s a person:Person;
foaf:name ?name .
}

View File

@@ -0,0 +1,22 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
PREFIX person: <http://www.w3.org/ns/person#>
# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
# First name: persoon:gebruikteVoornaam => foaf:givenName
# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
# We do both in one query with an OPTIONAL because they need to happen simultaneously
# not to end up in a loop
# Part 2
CONSTRUCT {
?s foaf:givenName ?gebruikteVoornaam .
} WHERE {
?s a person:Person ;
persoon_loket:gebruikteVoornaam ?gebruikteVoornaam .
}

View File

@@ -0,0 +1,21 @@
PREFIX schema: <http://schema.org/>
CONSTRUCT {
?s schema:telephone ?cleanedPhoneNumber .
} WHERE {
?s schema:telephone ?phoneNumber .
# Remove all non digits
BIND(REPLACE(?phoneNumber, """[^+\\d]""", "") AS ?scrapeResult)
# Replace leading 00 with +
BIND(REPLACE(?scrapeResult, """^00""", "+") AS ?replacedZeroZero)
BIND(REPLACE(?replacedZeroZero, """^0""", "+32") AS ?defaultBe)
BIND(REPLACE(?defaultBe, """\\+320""", "+32") AS ?extendedDefaultBe)
BIND(REPLACE(?extendedDefaultBe, """^320""", "+32") AS ?finalResult)
BIND(IF(REGEX(?finalResult, """^\\+((32\\d{8})|(324\\d{8})|((?!32)\\d+))$"""), CONCAT("tel:", ?finalResult), ?phoneNumber) AS ?cleanedPhoneNumber)
}

View File

@@ -0,0 +1,10 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
CONSTRUCT {
?s ch:eindedatum ?endDateTime .
} WHERE {
?s ch:eindedatum ?endDate .
BIND(STRDT(CONCAT(STR(?endDate), 'T02:00:00'), xsd:dateTime) as ?endDateTime)
}

View File

@@ -0,0 +1,10 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
CONSTRUCT {
?s ch:startdatum ?startDateTime .
} WHERE {
?s ch:startdatum ?startDate .
BIND(STRDT(CONCAT(STR(?startDate), 'T02:00:00'), xsd:dateTime) as ?startDateTime)
}

View File

@@ -0,0 +1,10 @@
PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
CONSTRUCT {
?s ere:geplandeEinddatumAanstelling ?dateTime .
} WHERE {
?s ere:geplandeEinddatumAanstelling ?date .
BIND(STRDT(CONCAT(STR(?date), 'T02:00:00'), xsd:dateTime) as ?dateTime)
}

View File

@@ -0,0 +1,20 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX dc_terms: <http://purl.org/dc/terms/>
PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX person: <http://www.w3.org/ns/person#>
CONSTRUCT {
?s a dc_terms:Agent .
} WHERE {
?s a ?type .
VALUES ?type {
lblodlg:Functionaris
mandaat:Mandataris
ere:EredienstMandataris
ere:RolBedienaar
person:Person
}
}

View File

@@ -0,0 +1,17 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
PREFIX lblodlg: <http://data.lblod.info/vocabularies/leidinggevenden/>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
CONSTRUCT {
?s a ch:AgentInPositie .
} WHERE {
?s a ?type .
VALUES ?type {
lblodlg:Functionaris
mandaat:Mandataris
ere:EredienstMandataris
ere:RolBedienaar
}
}

View File

@@ -0,0 +1,8 @@
PREFIX org: <http://www.w3.org/ns/org#>
CONSTRUCT {
?o org:holds ?s.
}
WHERE {
?s org:heldBy ?o.
}

View File

@@ -0,0 +1,8 @@
PREFIX org: <http://www.w3.org/ns/org#>
CONSTRUCT {
?o org:heldBy ?s.
}
WHERE {
?s org:holds ?o.
}

View File

@@ -0,0 +1,11 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://www.w3.org/ns/locn#Address>;
?p ?o.
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,7 @@
CONSTRUCT {
?s <http://schema.org/contactPoint> ?contactpoint.
}
WHERE {
?s a <http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie>.
?s <http://schema.org/contactPoint> ?contactpoint.
}

View File

@@ -0,0 +1,12 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://schema.org/ContactPoint>;
?p ?o.
FILTER (?p NOT IN (<http://schema.org/telephone>))
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,15 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX ere: <http://data.lblod.info/vocabularies/erediensten/>
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://data.lblod.info/vocabularies/erediensten/EredienstMandataris>;
?p ?o.
FILTER (?p NOT IN (ch:startdatum, ch:eindedatum, ere:geplandeEinddatumAanstelling))
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,16 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://www.w3.org/ns/person#Person>;
?p ?o.
FILTER (?p IN (
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>,
<http://mu.semte.ch/vocabularies/core/uuid>,
<http://xmlns.com/foaf/0.1/familyName>
))
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,10 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://data.lblod.info/vocabularies/erediensten/RolBedienaar>;
?p ?o.
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,14 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX person: <http://www.w3.org/ns/person#>
# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
# Alternatieve voornaam: foaf:name => persoon:gebruikteVoornaam
CONSTRUCT {
?s persoon:gebruikteVoornaam ?name .
} WHERE {
?s a person:Person ;
foaf:name ?name .
}

View File

@@ -0,0 +1,14 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
PREFIX person: <http://www.w3.org/ns/person#>
# DL and OP both use persoon:gebruikteVoornaam but with different semantics (!)
# First name: persoon:gebruikteVoornaam => foaf:givenName
CONSTRUCT {
?s foaf:givenName ?gebruikteVoornaam .
} WHERE {
?s a person:Person ;
persoon_loket:gebruikteVoornaam ?gebruikteVoornaam .
}

View File

@@ -0,0 +1,10 @@
CONSTRUCT {
?s ?p ?o.
} WHERE {
?s a <http://data.vlaanderen.be/ns/persoon#Geboorte>;
?p ?o.
FILTER NOT EXISTS {
?s ?p "" .
}
}

View File

@@ -0,0 +1,10 @@
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
# Map from http persoon to https persoon namespace
CONSTRUCT {
?s a persoon:Geboorte
} WHERE {
?s a persoon_loket:Geboorte .
}

View File

@@ -0,0 +1,10 @@
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
# Map from http persoon to https persoon namespace
CONSTRUCT {
?s persoon:datum ?o .
} WHERE {
?s persoon_loket:datum ?o .
}

View File

@@ -0,0 +1,10 @@
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
# Map from http persoon to https persoon namespace
CONSTRUCT {
?s persoon:geslacht ?o .
} WHERE {
?s persoon_loket:geslacht ?o .
}

View File

@@ -0,0 +1,10 @@
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
# Map from http persoon to https persoon namespace
CONSTRUCT {
?s persoon:heeftGeboorte ?o .
} WHERE {
?s persoon_loket:heeftGeboorte ?o .
}

View File

@@ -0,0 +1,10 @@
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
# Map from http persoon to https persoon namespace
CONSTRUCT {
?s persoon:heeftNationaliteit ?o .
} WHERE {
?s persoon_loket:heeftNationaliteit ?o .
}

View File

@@ -0,0 +1,10 @@
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
PREFIX persoon_loket: <http://data.vlaanderen.be/ns/persoon#>
# Map from http persoon to https persoon namespace
CONSTRUCT {
?s persoon:registratie ?o .
} WHERE {
?s persoon_loket:registratie ?o .
}

View File

@@ -0,0 +1,19 @@
PREFIX generiek: <https://data.vlaanderen.be/ns/generiek#>
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
CONSTRUCT {
?identifier
generiek:gestructureerdeIdentificator ?structuredIdentifier ;
skos:notation "Rijksregisternummer" .
?structuredIdentifier a generiek:GestructureerdeIdentificator ;
mu:uuid ?uuid ;
generiek:lokaleIdentificator ?notation .
}
WHERE {
?identifier skos:notation ?notation.
BIND(MD5(CONCAT(STR(?identifier),"STRUCTURED_IDENTIFIER")) as ?uuid)
BIND(IRI(CONCAT("http://data.lblod.info/id/gestructureerdeIdentificatoren/", ?uuid)) AS ?structuredIdentifier)
}

View File

@@ -0,0 +1,10 @@
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
# identifier property in the export is only used for RRN.
CONSTRUCT {
?person persoon:registratie ?rrn .
}
WHERE {
?person adms:identifier ?rrn .
}

View File

@@ -1,40 +0,0 @@
const BATCH_SIZE = parseInt(process.env.BATCH_SIZE) || 100;
const MU_CALL_SCOPE_ID_INITIAL_SYNC = process.env.MU_CALL_SCOPE_ID_INITIAL_SYNC || 'http://redpencil.data.gift/id/concept/muScope/deltas/consumer/initialSync';
const BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES = process.env.BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES == 'true' ? true : false;
const DIRECT_DATABASE_ENDPOINT = process.env.DIRECT_DATABASE_ENDPOINT || 'http://triplestore:8890/sparql';
const MAX_DB_RETRY_ATTEMPTS = parseInt(process.env.MAX_DB_RETRY_ATTEMPTS || 5);
const MAX_REASONING_RETRY_ATTEMPTS = parseInt(process.env.MAX_REASONING_RETRY_ATTEMPTS || 5);
const SLEEP_BETWEEN_BATCHES = parseInt(process.env.SLEEP_BETWEEN_BATCHES || 1000);
const SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION || 10000);
const SLEEP_TIME_AFTER_FAILED_DB_OPERATION = parseInt(process.env.SLEEP_TIME_AFTER_FAILED_DB_OPERATION || 60000);
// Graphs
const TARGET_GRAPH = process.env.TARGET_GRAPH || `http://mu.semte.ch/graphs/worship-service`;
const PRIVACY_SENSITIVE_GRAPH = process.env.PRIVACY_SENSITIVE_GRAPH || `http://mu.semte.ch/graphs/worship-privacy-centric-graph`;
const LANDING_ZONE_GRAPH = process.env.DCR_LANDING_ZONE_GRAPH || `http://mu.semte.ch/graphs/landing-zone/leidinggevenden`;
// Mappings
const MAIN_INFO_MAPPING = "main-info";
const PRIVATE_INFO_MAPPING = "private-info";
// ENDPOINTS
const LANDING_ZONE_DATABASE = process.env.DCR_LANDING_ZONE_DATABASE || 'db';
const LANDING_ZONE_DATABASE_ENDPOINT = process.env.DCR_LANDING_ZONE_DATABASE_ENDPOINT || `http://${LANDING_ZONE_DATABASE}:8890/sparql`;
module.exports = {
BATCH_SIZE,
MU_CALL_SCOPE_ID_INITIAL_SYNC,
BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
DIRECT_DATABASE_ENDPOINT,
MAX_DB_RETRY_ATTEMPTS,
MAX_REASONING_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
LANDING_ZONE_GRAPH,
LANDING_ZONE_DATABASE_ENDPOINT,
TARGET_GRAPH,
PRIVACY_SENSITIVE_GRAPH,
MAIN_INFO_MAPPING,
PRIVATE_INFO_MAPPING
};

View File

@@ -1,11 +0,0 @@
const contextConfig = {
addTypes: {
scope: 'all', // 'inserts, 'deletes', 'all' or none. To add rdf:type to subjects of inserts, deletes or both
exhausitive: false, // true or false: find all types for a subject, even if one is already present in delta
},
contextQueries: []
}
module.exports = {
contextConfig
};

View File

@@ -1,114 +0,0 @@
const {
transformStatements,
deleteFromTargetGraph,
insertIntoTargetGraph,
deleteFromPrivateGraph,
insertIntoPrivateGraph
} = require('./util');
const {
MAIN_INFO_MAPPING,
PRIVATE_INFO_MAPPING
} = require('./config');
/**
* Dispatch the fetched information to a target graph.
* Note: <share://file/data> will be ADDED to it's own graph.
* We take only care of adding them, not updating triples, this is a TODO
* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
* @param { termObjectChangeSets: { deletes, inserts } } data - The fetched changes sets, which objects of serialized Terms
* [ {
* graph: "<http://foo>",
* subject: "<http://bar>",
* predicate: "<http://baz>",
* object: "<http://boom>^^<http://datatype>"
* }
* ]
* @return {void} Nothing
*/
async function dispatch(lib, data) {
const { mu, fetch } = lib;
let { termObjectChangeSets, termObjectChangeSetsWithContext } = data;
// Both arrays are the same length, so we can zip them together for easier processing
zippedChangeSets = termObjectChangeSets.map((o, i) => ({
original: o,
withContext: termObjectChangeSetsWithContext[i]
}));
console.log(`Received ${zippedChangeSets.length} change sets`)
for (let { original, withContext } of zippedChangeSets) {
let originalDeletes, originalInserts, insertsWithContext, deletesWithContext;
try {
originalInserts = original.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
originalDeletes = original.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
} catch (e) {
console.log(`Error while processing change set: ${e}`);
console.log(`Original change set: ${JSON.stringify(original)}`);
throw e;
}
// Extra context needed for mapping from DL to OP model and filtering based on type.
try {
insertsWithContext = withContext.inserts.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
deletesWithContext = withContext.deletes.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
} catch (e) {
console.log(`Error while processing change set with context: ${e}`);
console.log(`Change set with context: ${JSON.stringify(withContext)}`);
throw e;
}
if (originalDeletes.length) {
// Map main deletes from DL to OP model
const transformedMainDeletes = await transformStatements(fetch, deletesWithContext, MAIN_INFO_MAPPING);
if (!transformedMainDeletes.length) {
console.log(`Warn: Delete main statements mapped to empty result.`);
console.log(`Input: ${deletesWithContext}`);
console.log(`Output: ${transformedMainDeletes}`);
} else {
await deleteFromTargetGraph(lib, transformedMainDeletes);
}
// Map private deletes from DL to OP model
const transformedPrivateDeletes = await transformStatements(fetch, deletesWithContext, PRIVATE_INFO_MAPPING);
if (!transformedPrivateDeletes.length) {
console.log(`Warn: Delete private statements mapped to empty result.`);
console.log(`Input: ${deletesWithContext}`);
console.log(`Output: ${transformedPrivateDeletes}`);
} else {
await deleteFromPrivateGraph(lib, transformedPrivateDeletes);
}
}
if (originalInserts.length) {
// Map main inserts from DL to OP model
const transformedMainInserts = await transformStatements(fetch, insertsWithContext, MAIN_INFO_MAPPING);
if (!transformedMainInserts.length) {
console.log(`Warn: Insert statements mapped to empty result.`);
console.log(`Input: ${insertsWithContext}`);
console.log(`Output: ${transformedMainInserts}`);
} else {
await insertIntoTargetGraph(lib, transformedMainInserts);
}
// Map private inserts from DL to OP model
const transformedPrivateInserts = await transformStatements(fetch, insertsWithContext, PRIVATE_INFO_MAPPING);
if (!transformedPrivateInserts.length) {
console.log(`Warn: Insert statements mapped to empty result.`);
console.log(`Input: ${insertsWithContext}`);
console.log(`Output: ${transformedPrivateInserts}`);
} else {
await insertIntoPrivateGraph(lib, transformedPrivateInserts);
}
}
}
}
module.exports = {
dispatch
};

View File

@@ -1,106 +0,0 @@
const { batchedDbUpdate, transformLandingZoneGraph } = require('./util');
const {
BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES,
DIRECT_DATABASE_ENDPOINT,
MU_CALL_SCOPE_ID_INITIAL_SYNC,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
TARGET_GRAPH,
PRIVACY_SENSITIVE_GRAPH,
LANDING_ZONE_GRAPH,
LANDING_ZONE_DATABASE_ENDPOINT,
MAIN_INFO_MAPPING,
PRIVATE_INFO_MAPPING
} = require('./config');
const endpoint = BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES ? DIRECT_DATABASE_ENDPOINT : process.env.MU_SPARQL_ENDPOINT;
/**
* Dispatch the fetched information to a target graph.
* @param { mu, muAuthSudo } lib - The provided libraries from the host service.
* @param { termObjects } data - The fetched quad information, which objects of serialized Terms
* [ {
* graph: "<http://foo>",
* subject: "<http://bar>",
* predicate: "<http://baz>",
* object: "<http://boom>^^<http://datatype>"
* }
* ]
* @return {void} Nothing
*/
async function dispatch(lib, data) {
const { mu, muAuthSudo } = lib;
const { termObjects } = data;
// Steps:
// 1. Write the triples to the landing zone graph - withouth any mapping or filtering
// 2. One-time reasoning run with the full landing zone graph and write the results to the target graph (on-finish)
if (BYPASS_MU_AUTH_FOR_EXPENSIVE_QUERIES) {
console.warn(`Service configured to skip MU_AUTH!`);
}
console.log(`Using ${endpoint} to insert triples`);
if (termObjects.length) {
const originalInsertTriples = termObjects.map(o => `${o.subject} ${o.predicate} ${o.object}.`);
await batchedDbUpdate(
muAuthSudo.updateSudo,
LANDING_ZONE_GRAPH,
originalInsertTriples,
{ 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
LANDING_ZONE_DATABASE_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION
);
}
}
async function onFinishInitialIngest(lib) {
const { muAuthSudo, fetch } = lib;
console.log(`!! On-finish triggered !!`);
const transformedMainInsertTriples = await transformLandingZoneGraph(fetch, endpoint, MAIN_INFO_MAPPING);
console.log(`Transformed ${transformedMainInsertTriples.length} main triples`);
if (transformedMainInsertTriples.length) {
await batchedDbUpdate(
muAuthSudo.updateSudo,
TARGET_GRAPH,
transformedMainInsertTriples,
{ 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
endpoint,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION
);
}
const transformedPrivateInsertTriples = await transformLandingZoneGraph(fetch, endpoint, PRIVATE_INFO_MAPPING);
console.log(`Transformed ${transformedPrivateInsertTriples.length} private triples`);
if (transformedPrivateInsertTriples.length) {
await batchedDbUpdate(
muAuthSudo.updateSudo,
PRIVACY_SENSITIVE_GRAPH,
transformedPrivateInsertTriples,
{ 'mu-call-scope-id': MU_CALL_SCOPE_ID_INITIAL_SYNC },
endpoint,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION
);
}
}
module.exports = {
dispatch,
onFinishInitialIngest
};

View File

@@ -1,204 +0,0 @@
const {
MAX_REASONING_RETRY_ATTEMPTS,
SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
TARGET_GRAPH,
PRIVACY_SENSITIVE_GRAPH,
LANDING_ZONE_GRAPH,
BATCH_SIZE
} = require('./config');
async function batchedDbUpdate(
muUpdate,
graph,
triples,
extraHeaders,
endpoint,
batchSize,
maxAttempts,
sleepBetweenBatches = 1000,
sleepTimeOnFail = 1000,
operation = 'INSERT'
) {
for (let i = 0; i < triples.length; i += batchSize) {
console.log(`Inserting triples in batch: ${i}-${i + batchSize}`);
const batch = triples.slice(i, i + batchSize).join('\n');
const insertCall = async () => {
await muUpdate(`
${operation} DATA {
GRAPH <${graph}> {
${batch}
}
}
`, extraHeaders, endpoint);
};
await operationWithRetry(insertCall, 0, maxAttempts, sleepTimeOnFail);
console.log(`Sleeping before next query execution: ${sleepBetweenBatches}`);
await new Promise(r => setTimeout(r, sleepBetweenBatches));
}
}
async function operationWithRetry(callback,
attempt,
maxAttempts,
sleepTimeOnFail) {
try {
if (typeof callback === "function")
return await callback();
else // Catch error from promise - not how I would do it normally, but allows re use of existing code.
return await callback;
}
catch (e) {
console.log(`Operation failed for ${callback.toString()}, attempt: ${attempt} of ${maxAttempts}`);
console.log(`Error: ${e}`);
console.log(`Sleeping ${sleepTimeOnFail} ms`);
if (attempt >= maxAttempts) {
console.log(`Max attempts reached for ${callback.toString()}, giving up`);
throw e;
}
await new Promise(r => setTimeout(r, sleepTimeOnFail));
return operationWithRetry(callback, ++attempt, maxAttempts, sleepTimeOnFail);
}
}
/**
* Splits an array into two parts, a part that passes and a part that fails a predicate function.
* Credits: https://github.com/benjay10
* @public
* @function partition
* @param {Array} arr - Array to be partitioned
* @param {Function} fn - Function that accepts single argument: an element of the array, and should return a truthy or falsy value.
* @returns {Object} Object that contains keys passes and fails, each representing an array with elemets that pass or fail the predicate respectively
*/
function partition(arr, fn) {
let passes = [], fails = [];
arr.forEach((item) => (fn(item) ? passes : fails).push(item));
return { passes, fails };
}
/**
* Send triples to reasoning service for conversion
*
*/
async function transformTriples(fetch, triples, mapping) {
return await operationWithRetry(mainConversion(fetch, triples, mapping), 0,
MAX_REASONING_RETRY_ATTEMPTS, SLEEP_TIME_AFTER_FAILED_REASONING_OPERATION);
}
async function mainConversion(fetch, triples, mapping) {
let formdata = new URLSearchParams();
formdata.append("data", triples);
let requestOptions = {
method: 'POST',
body: formdata,
redirect: 'follow'
};
const response = await fetch(`http://reasoner/reason/dl2op/${mapping}`, requestOptions)
return await response.text();
}
async function transformStatements(fetch, triples, mapping = 'main') {
console.log(`Received ${JSON.stringify(triples)} to transform`);
const transformedTriples = await transformTriples(fetch, triples.join('\n'), mapping);
statements = transformedTriples ? transformedTriples.replace(/\n{2,}/g, '').split('\n') : [];
console.log(`CONVERSION: FROM ${triples.length} triples to ${statements.length}`);
return statements;
}
async function deleteFromTargetGraph(lib, statements) {
console.log(`Deleting ${statements.length} statements from target graph`);
console.log(`Statements: ${JSON.stringify(statements)}`)
await batchedDbUpdate(
lib.muAuthSudo.updateSudo,
TARGET_GRAPH,
statements,
{},
process.env.MU_SPARQL_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
'DELETE');
}
async function deleteFromPrivateGraph(lib, statements) {
console.log(`Deleting ${statements.length} statements from target graph`);
console.log(`Statements: ${JSON.stringify(statements)}`)
await batchedDbUpdate(
lib.muAuthSudo.updateSudo,
PRIVACY_SENSITIVE_GRAPH,
statements,
{},
process.env.MU_SPARQL_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
'DELETE');
}
async function insertIntoTargetGraph(lib, statements) {
console.log(`Inserting ${statements.length} statements into target graph`);
console.log(`Statements: ${JSON.stringify(statements)}`)
await batchedDbUpdate(
lib.muAuthSudo.updateSudo,
TARGET_GRAPH,
statements,
{},
process.env.MU_SPARQL_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
'INSERT');
}
async function insertIntoPrivateGraph(lib, statements) {
console.log(`Inserting ${statements.length} statements into target graph`);
console.log(`Statements: ${JSON.stringify(statements)}`)
await batchedDbUpdate(
lib.muAuthSudo.updateSudo,
PRIVACY_SENSITIVE_GRAPH,
statements,
{},
process.env.MU_SPARQL_ENDPOINT,
BATCH_SIZE,
MAX_DB_RETRY_ATTEMPTS,
SLEEP_BETWEEN_BATCHES,
SLEEP_TIME_AFTER_FAILED_DB_OPERATION,
'INSERT');
}
async function transformLandingZoneGraph(fetch, endpoint, mapping = 'main') {
console.log(`Transforming landing zone graph: ${LANDING_ZONE_GRAPH}`);
const response = await fetch(`http://reasoner/reason/dl2op/${mapping}?data=${encodeURIComponent(`${endpoint}?default-graph-uri=&query=CONSTRUCT+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D+WHERE+%7B%0D%0A+GRAPH+<${LANDING_ZONE_GRAPH}>+%7B%0D%0A%3Fs+%3Fp+%3Fo%0D%0A%7D%0D%0A%7D&should-sponge=&format=text%2Fplain&timeout=0&run=+Run+Query`)}`);
const text = await response.text();
const statements = text.replace(/\n{2,}/g, '').split('\n');
return statements;
}
module.exports = {
batchedDbUpdate,
partition,
transformStatements,
transformLandingZoneGraph,
deleteFromTargetGraph,
insertIntoTargetGraph,
deleteFromPrivateGraph,
insertIntoPrivateGraph
};

View File

@@ -0,0 +1,23 @@
PREFIX regorg: <http://www.w3.org/ns/regorg#>
PREFIX org: <http://www.w3.org/ns/org#>
DELETE {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?stAmandus regorg:orgStatus ?oldStatus.
}
} INSERT {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?stAmandus regorg:orgStatus ?notActiveStatus.
?mergerChangeEvent org:originalOrganization ?stAmandus.
}
} WHERE {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?stAmandus regorg:orgStatus ?oldStatus.
}
VALUES ?stAmandus {
<http://data.lblod.info/id/besturenVanDeEredienst/b18a5d3943087d509d76dcceeba102c3>
}
BIND (<http://data.lblod.info/id/veranderingsgebeurtenissen/677E445E29299CC832892A0F> AS ?mergerChangeEvent)
BIND (<http://lblod.data.gift/concepts/d02c4e12bf88d2fdf5123b07f29c9311> AS ?notActiveStatus)
}

View File

@@ -0,0 +1,37 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?container ?containerP ?containerO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation;
dct:created ?created.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
task:resultsContainer ?container.
?container a nfo:DataContainer;
?containerP ?containerO.
}
}

View File

@@ -0,0 +1,34 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?task ?taskP ?taskO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
?taskP ?taskO.
}
}

View File

@@ -0,0 +1,32 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?job ?jobP ?jobO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation;
?jobP ?jobO.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
}
}

View File

@@ -0,0 +1,37 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?container ?containerP ?containerO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation;
dct:created ?created.
FILTER(?created < "2025-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
task:resultsContainer ?container.
?container a nfo:DataContainer;
?containerP ?containerO.
}
}

View File

@@ -0,0 +1,34 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?task ?taskP ?taskO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation.
FILTER(?created < "2025-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
?taskP ?taskO.
}
}

View File

@@ -0,0 +1,32 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?job ?jobP ?jobO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation;
?jobP ?jobO.
FILTER(?created < "2025-01-01T00:00:00"^^xsd:dateTime)
}
}

View File

@@ -0,0 +1,34 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?container ?containerP ?containerO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation.
?task dct:isPartOf ?job;
task:resultsContainer ?container.
?container a nfo:DataContainer;
?containerP ?containerO.
}
}

View File

@@ -0,0 +1,31 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?task ?taskP ?taskO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation.
?task dct:isPartOf ?job;
?taskP ?taskO.
}
}

View File

@@ -0,0 +1,29 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?job ?jobP ?jobO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/leidinggevenden>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/leidinggevenden>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation;
?jobP ?jobO.
}
}

View File

@@ -0,0 +1,25 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
DELETE {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?s a <http://www.w3.org/ns/person#Person>;
?p ?o.
}
} WHERE {
VALUES ?p {
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
<http://mu.semte.ch/vocabularies/core/uuid>
foaf:givenName
foaf:familyName
<http://data.vlaanderen.be/ns/persoon#gebruikteVoornaam>
}
?functionaris a <http://data.lblod.info/vocabularies/leidinggevenden/Functionaris>;
<http://data.vlaanderen.be/ns/mandaat#isBestuurlijkeAliasVan> ?s.
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?s a <http://www.w3.org/ns/person#Person>;
?p ?o.
}
}

View File

@@ -0,0 +1,25 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
DELETE {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?s a <http://data.lblod.info/vocabularies/leidinggevenden/Functionaris>;
?p ?o.
}
} WHERE {
VALUES ?p {
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
<http://mu.semte.ch/vocabularies/core/uuid>
<http://www.w3.org/ns/org#holds>
<http://data.vlaanderen.be/ns/mandaat#status>
<http://data.vlaanderen.be/ns/mandaat#start>
<http://data.vlaanderen.be/ns/mandaat#einde>
<http://data.vlaanderen.be/ns/mandaat#isBestuurlijkeAliasVan>
}
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?s a <http://data.lblod.info/vocabularies/leidinggevenden/Functionaris>;
?p ?o.
}
}

View File

@@ -0,0 +1,37 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
DELETE {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?s a <http://www.w3.org/ns/locn#Address>;
?p ?o.
}
} WHERE {
VALUES ?p {
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
<http://mu.semte.ch/vocabularies/core/uuid>
<https://data.vlaanderen.be/ns/adres#verwijstNaar>
<https://data.vlaanderen.be/ns/adres#Adresvoorstelling.huisnummer>
<http://www.w3.org/ns/locn#postCode>
<http://www.w3.org/ns/locn#thoroughfare>
<http://www.w3.org/ns/locn#fullAddress>
<https://data.vlaanderen.be/ns/adres#gemeentenaam>
<https://data.vlaanderen.be/ns/adres#land>
<http://www.w3.org/ns/locn#adminUnit1>
<http://www.w3.org/ns/locn#adminUnit2>
<http://www.w3.org/ns/locn#addressArea>
<https://data.vlaanderen.be/ns/adres#Adresvoorstelling.busnummer>
}
?bestuursfunctie a <http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie>;
<http://schema.org/contactPoint> ?contactpoint.
?contactpoint a <http://schema.org/ContactPoint>;
<http://www.w3.org/ns/locn#address> ?s.
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?s a <http://www.w3.org/ns/locn#Address>;
?p ?o.
}
}

View File

@@ -0,0 +1,28 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
DELETE {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?s a <http://schema.org/ContactPoint>;
?p ?o.
}
} WHERE {
VALUES ?p {
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
<http://mu.semte.ch/vocabularies/core/uuid>
<http://schema.org/telephone>
<http://schema.org/email>
<http://www.w3.org/ns/locn#address>
<http://schema.org/faxNumber>
<http://schema.org/url>
}
?bestuursfunctie a <http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie>;
<http://schema.org/contactPoint> ?s.
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?s a <http://schema.org/ContactPoint>;
?p ?o.
}
}

View File

@@ -0,0 +1,13 @@
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
DELETE {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?bestuursfunctie <http://schema.org/contactPoint> ?contactpoint.
}
} WHERE {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?bestuursfunctie a <http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie>;
<http://schema.org/contactPoint> ?contactpoint.
}
}

View File

@@ -0,0 +1 @@
CLEAR GRAPH <http://mu.semte.ch/graphs/landing-zone/leidinggevenden>

View File

@@ -0,0 +1,37 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?container ?containerP ?containerO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation;
dct:created ?created.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
task:resultsContainer ?container.
?container a nfo:DataContainer;
?containerP ?containerO.
}
}

View File

@@ -0,0 +1,34 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?task ?taskP ?taskO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
?taskP ?taskO.
}
}

View File

@@ -0,0 +1,32 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?job ?jobP ?jobO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation;
?jobP ?jobO.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
}
}

View File

@@ -0,0 +1,37 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?container ?containerP ?containerO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation;
dct:created ?created.
FILTER(?created < "2025-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
task:resultsContainer ?container.
?container a nfo:DataContainer;
?containerP ?containerO.
}
}

View File

@@ -0,0 +1,34 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?task ?taskP ?taskO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation.
FILTER(?created < "2025-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
?taskP ?taskO.
}
}

View File

@@ -0,0 +1,32 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?job ?jobP ?jobO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation;
?jobP ?jobO.
FILTER(?created < "2025-01-01T00:00:00"^^xsd:dateTime)
}
}

View File

@@ -0,0 +1,34 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?container ?containerP ?containerO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation.
?task dct:isPartOf ?job;
task:resultsContainer ?container.
?container a nfo:DataContainer;
?containerP ?containerO.
}
}

View File

@@ -0,0 +1,31 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?task ?taskP ?taskO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation.
?task dct:isPartOf ?job;
?taskP ?taskO.
}
}

View File

@@ -0,0 +1,29 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?job ?jobP ?jobO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitive>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/wsSensitiveDeltaFileSyncing>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation;
?jobP ?jobO.
}
}

View File

@@ -0,0 +1,33 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX org: <http://www.w3.org/ns/org#>
PREFIX person: <http://www.w3.org/ns/person#>
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
# Delete birthdates
DELETE {
GRAPH <http://mu.semte.ch/graphs/worship-privacy-centric-graph> {
?birthDate ?p ?o .
}
} WHERE {
VALUES ?g {
<http://mu.semte.ch/graphs/shared>
<http://mu.semte.ch/graphs/worship-service>
}
GRAPH ?g {
?person a person:Person .
}
FILTER EXISTS {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?agent a ch:AgentInPositie ;
mandaat:isBestuurlijkeAliasVan|org:heldBy ?person .
}
}
GRAPH <http://mu.semte.ch/graphs/worship-privacy-centric-graph> {
?person persoon:heeftGeboorte ?birthDate .
?birthDate ?p ?o .
}
}

View File

@@ -0,0 +1,39 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX generiek: <https://data.vlaanderen.be/ns/generiek#>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX org: <http://www.w3.org/ns/org#>
PREFIX person: <http://www.w3.org/ns/person#>
PREFIX persoon: <https://data.vlaanderen.be/ns/persoon#>
# Delete identifiers
DELETE {
GRAPH <http://mu.semte.ch/graphs/worship-privacy-centric-graph> {
?identifier ?pidentifier ?oidentifier .
?structuredIdentifier ?pstructuredIdentifier ?ostructuredIdentifier .
}
} WHERE {
VALUES ?g {
<http://mu.semte.ch/graphs/shared>
<http://mu.semte.ch/graphs/worship-service>
}
GRAPH ?g {
?person a person:Person .
}
FILTER EXISTS {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?agent a ch:AgentInPositie ;
mandaat:isBestuurlijkeAliasVan|org:heldBy ?person .
}
}
GRAPH <http://mu.semte.ch/graphs/worship-privacy-centric-graph> {
?person persoon:registratie ?identifier .
?identifier generiek:gestructureerdeIdentificator ?structuredIdentifier ;
?pidentifier ?oidentifier .
?structuredIdentifier ?pstructuredIdentifier ?ostructuredIdentifier .
}
}

View File

@@ -0,0 +1,36 @@
PREFIX ch: <http://data.lblod.info/vocabularies/contacthub/>
PREFIX mandaat: <http://data.vlaanderen.be/ns/mandaat#>
PREFIX org: <http://www.w3.org/ns/org#>
PREFIX person: <http://www.w3.org/ns/person#>
# Delete persons
DELETE {
GRAPH ?g {
?person ?p ?o .
}
GRAPH <http://mu.semte.ch/graphs/worship-privacy-centric-graph> {
?person ?pprivate ?oprivate .
}
} WHERE {
VALUES ?g {
<http://mu.semte.ch/graphs/shared>
<http://mu.semte.ch/graphs/worship-service>
}
GRAPH ?g {
?person a person:Person ;
?p ?o .
}
FILTER EXISTS {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?agent a ch:AgentInPositie ;
mandaat:isBestuurlijkeAliasVan|org:heldBy ?person .
}
}
GRAPH <http://mu.semte.ch/graphs/worship-privacy-centric-graph> {
?person ?pprivate ?oprivate .
}
}

View File

@@ -0,0 +1,22 @@
DELETE {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?contactPoint ?pcontactPoint ?ocontactPoint .
?address ?paddress ?oaddress .
}
} WHERE {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?agent a <http://data.lblod.info/vocabularies/contacthub/AgentInPositie> .
OPTIONAL {
?agent <http://schema.org/contactPoint> ?contactPoint .
?contactPoint ?pcontactPoint ?ocontactPoint .
OPTIONAL {
?contactPoint <http://www.w3.org/ns/locn#address> ?address .
?address ?paddress ?oaddress .
}
}
}
}

View File

@@ -0,0 +1,10 @@
DELETE {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?agent ?pagent ?oagent .
}
} WHERE {
GRAPH <http://mu.semte.ch/graphs/worship-service> {
?agent a <http://data.lblod.info/vocabularies/contacthub/AgentInPositie> ;
?pagent ?oagent .
}
}

View File

@@ -0,0 +1 @@
CLEAR GRAPH <http://mu.semte.ch/graphs/landing-zone/worship-services-sensitive>

View File

@@ -32,42 +32,4 @@ DELETE {
?gemeenteBestuursorgaanT <https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan> ?gemeenteBestuursorgaan;
<http://data.lblod.info/vocabularies/leidinggevenden/heeftBestuursfunctie> ?ocmwFunctie.
}
}
;
INSERT {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?ocmwBestuursorgaanT <http://data.lblod.info/vocabularies/leidinggevenden/heeftBestuursfunctie> ?gemeenteFunctie.
}
}
WHERE {
# OCMW
?ocmwBestuur a <http://data.vlaanderen.be/ns/besluit#Bestuurseenheid>;
<http://www.w3.org/ns/org#classification> <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000002>;
<http://www.w3.org/ns/org#linkedTo> ?gemeenteBestuur.
?ocmwBestuursorgaan a <http://data.vlaanderen.be/ns/besluit#Bestuursorgaan>;
<http://data.vlaanderen.be/ns/besluit#bestuurt> ?ocmwBestuur;
<http://www.w3.org/ns/org#classification> ?orgaanClassification.
?ocmwBestuursorgaanT <https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan> ?ocmwBestuursorgaan.
FILTER NOT EXISTS {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?ocmwBestuursorgaanT <http://data.lblod.info/vocabularies/leidinggevenden/heeftBestuursfunctie> ?ocmwFunctie.
}
}
?gemeenteBestuur a <http://data.vlaanderen.be/ns/besluit#Bestuurseenheid>;
# GEMEENTE
<http://www.w3.org/ns/org#classification> <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001>.
?gemeenteBestuursorgaan a <http://data.vlaanderen.be/ns/besluit#Bestuursorgaan>;
<http://data.vlaanderen.be/ns/besluit#bestuurt> ?gemeenteBestuur;
<http://www.w3.org/ns/org#classification> ?orgaanClassification.
?gemeenteBestuursorgaanT <https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan> ?gemeenteBestuursorgaan;
<http://data.lblod.info/vocabularies/leidinggevenden/heeftBestuursfunctie> ?gemeenteFunctie.
}

View File

@@ -0,0 +1,42 @@
INSERT {
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
?ocmwBestuursorgaanT <http://data.lblod.info/vocabularies/leidinggevenden/heeftBestuursfunctie> ?gemeenteFunctie.
}
}
WHERE {
VALUES ?orgaanClassification {
<http://data.vlaanderen.be/id/concept/BestuursorgaanClassificatieCode/62644b9c-4514-41dd-a660-4c35257f2b35>
<http://data.vlaanderen.be/id/concept/BestuursorgaanClassificatieCode/39854196-f214-4688-87a1-d6ad12baa2fa>
<http://data.vlaanderen.be/id/concept/BestuursorgaanClassificatieCode/11f0af9e-016c-4e0b-983a-d8bc73804abc>
<http://data.vlaanderen.be/id/concept/BestuursorgaanClassificatieCode/ed40469e-3b6f-4f38-99ba-18912ee352b0>
}
GRAPH <http://mu.semte.ch/graphs/administrative-unit> {
# OCMW
?ocmwBestuur a <http://data.vlaanderen.be/ns/besluit#Bestuurseenheid>;
<http://www.w3.org/ns/org#classification> <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000002>;
<http://www.w3.org/ns/org#linkedTo> ?gemeenteBestuur.
?ocmwBestuursorgaan a <http://data.vlaanderen.be/ns/besluit#Bestuursorgaan>;
<http://data.vlaanderen.be/ns/besluit#bestuurt> ?ocmwBestuur;
<http://www.w3.org/ns/org#classification> ?orgaanClassification.
?ocmwBestuursorgaanT <https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan> ?ocmwBestuursorgaan.
FILTER NOT EXISTS {
?ocmwBestuursorgaanT <http://data.lblod.info/vocabularies/leidinggevenden/heeftBestuursfunctie> ?ocmwFunctie.
}
?gemeenteBestuur a <http://data.vlaanderen.be/ns/besluit#Bestuurseenheid>;
# GEMEENTE
<http://www.w3.org/ns/org#classification> <http://data.vlaanderen.be/id/concept/BestuurseenheidClassificatieCode/5ab0e9b8a3b2ca7c5e000001>.
?gemeenteBestuursorgaan a <http://data.vlaanderen.be/ns/besluit#Bestuursorgaan>;
<http://data.vlaanderen.be/ns/besluit#bestuurt> ?gemeenteBestuur;
<http://www.w3.org/ns/org#classification> ?orgaanClassification.
?gemeenteBestuursorgaanT <https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan> ?gemeenteBestuursorgaan;
<http://data.lblod.info/vocabularies/leidinggevenden/heeftBestuursfunctie> ?gemeenteFunctie.
}
}

View File

@@ -0,0 +1,37 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?container ?containerP ?containerO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/mandatarissen>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/mandatarissen>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation;
dct:created ?created.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
task:resultsContainer ?container.
?container a nfo:DataContainer;
?containerP ?containerO.
}
}

View File

@@ -0,0 +1,34 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?task ?taskP ?taskO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/mandatarissen>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/mandatarissen>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
?task dct:isPartOf ?job;
?taskP ?taskO.
}
}

View File

@@ -0,0 +1,32 @@
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX dct: <http://purl.org/dc/terms/>
PREFIX prov: <http://www.w3.org/ns/prov#>
PREFIX nie: <http://www.semanticdesktop.org/ontologies/2007/01/19/nie#>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX oslc: <http://open-services.net/ns/core#>
PREFIX cogs: <http://vocab.deri.ie/cogs#>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX nfo: <http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#>
PREFIX dbpedia: <http://dbpedia.org/resource/>
DELETE {
GRAPH ?g {
?job ?jobP ?jobO.
}
} WHERE {
VALUES ?operation {
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/initialSync/mandatarissen>
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/consumer/deltaSync/mandatarissen>
}
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
dct:created ?created;
task:operation ?operation;
?jobP ?jobO.
FILTER(?created < "2024-01-01T00:00:00"^^xsd:dateTime)
}
}

Some files were not shown because too many files have changed in this diff Show More