felix.ruizdearcaute/app-organization-portal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Feature/delta producer administrative units (#67)

Browse Source 
* Setting up file-service

- pre-requisite to download dumps
- pre-requiiste to download the json-diff files
- security considerations: the content of the files created here, is
- assumed to be public data, hence we allow it to be in the public graph

* Bumping deltanotifier and mu-auth, which need

to be able to work with mu-scop-id's

* Delta-producer have multiple background-jobs with different tasks

Hence the introduction of the jobs-controller.
Note: in a more advanced configuration, you can easily extend the
tasks related to one job.

* A pre-requisite to serve dumps, is DCAT api support

The delta-producers make a periodic dump of what is supposed to be
published. They are published according to the DCAT data-model.

Security-considerations: the DCAT producer by the delta-producer flow,
are considered to be public data, hence added to the public graph.

* Introduction of the delta-producer-dump-file-publisher

Service responsible for creating dumps for the (potentially
many) data delta flows.
Here only for the administrative-units, but this can be extended in
the future, in the config.

* Introduction of delta-producer-background-jobs-initiator

This service is responsible for triggering the background jobs related
to the delta-productoin of the administrative-units.

It trigger currently three jobs:
- INITIAL_SYNC: this is the first job to fill the publication graph.
  In theory, this is only run once per deploy
- HEALING_JOB: This is a periodic job, which checks what information
is published vs. what information should have been
published (according to a configuration). If differences are found,
because delta-messages never arrived, they are compensated here/
- DUMP_JOB: A periodic job to create a dump of the publication
graph. This is especially useful for consumers, who don't have to
replay the whole delta-diffs, but just can start from the latest sync
point.

* Introduction of delta-producer-publication-graph-maintainer

This service is the core of the logic which determines what should be
published.
The configuration is provided here, and can be elaborated refined in
the config.
It also excutes the healing logic.

* Introduction of delta-producer-json-diff-file-publisher

This simple service, listens to the updates performed on the
publication graph, and serialized these to json-files which are
compatible with current consumers

* Missing dispatcher rules for DCAT

* Various small things:

- outline, whitespaces and comments

* Update README.md

* bump delta-producer-publication-graph-maintainer

* Add organizations (KAL) - Work in progress

* added organizations graph maintainer

* removed inverses, added all bestuur types

* refactored kalliope api

Co-authored-by: Boris De Vloed <boris.de.vloed@gmail.com>
This commit is contained in:
cecemel
2021-12-03 11:20:32 +01:00
committed by GitHub
parent 5b48852463
commit 566ec39d9a
13 changed files with 1115 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
README.md
View File

@@ -12,3 +12,37 @@ Backend for the Contact Hub application based on the mu.semte.ch microservices
You can shut down using `docker-compose stop` and remove everything using `docker-compose rm`.
### Setting up the delta-producers related services
To make sure the app can share data, producers need to be set up. There is an intial sync, that is potentially very expensive, and must be started manually
#### producers administrative-units
1. make sure the app is up and running, the migrations have run
2. in docker-compose.override.yml, make sure the following configuration is provided:
```
delta-producer-pub-graph-maintainer-administrative-units:
environment:
START_INITIAL_SYNC: 'true'
```
3. `drc up -d delta-producer-pub-graph-maintainer-administrative-units`
4. You can follow the status of the job, ideally through the dashboard, but this hasn't been setup yet. The following query should also give results:
```
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
PREFIX adms: <http://www.w3.org/ns/adms#>
PREFIX dct: <http://purl.org/dc/terms/>
SELECT DISTINCT ?job ?operation ?created ?modified ?status WHERE {
GRAPH ?g {
?job a <http://vocab.deri.ie/cogs#Job>;
task:operation ?operation;
adms:status ?status;
dct:created ?created;
dct:modified ?modified.
}
FILTER( ?operation IN (
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units>,
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units>
)
)
}
```

12
config/authorization/config.ex
View File

@@ -32,7 +32,7 @@ defmodule Acl.UserGroups.Config do
"http://schema.org/ContactPoint",
"http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie",
"http://data.lblod.info/vocabularies/erediensten/CentraalBestuurVanDeEredienst",
"http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan",
"http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan",
"http://www.w3.org/ns/org#Role",
"http://www.w3.org/ns/org#Post",
"http://data.lblod.info/vocabularies/contacthub/AgentInPositie",
@@ -107,17 +107,21 @@ defmodule Acl.UserGroups.Config do
%GroupSpec{
name: "public",
useage: [:read],
access: %AlwaysAccessible{},
access: %AlwaysAccessible{},
graphs: [ %GraphSpec{
graph: "http://mu.semte.ch/graphs/public", # mock login only
constraint: %ResourceConstraint{
resource_types: [
"http://data.vlaanderen.be/ns/besluit#Bestuurseenheid",
"http://xmlns.com/foaf/0.1/Person",
"http://xmlns.com/foaf/0.1/OnlineAccount"
"http://xmlns.com/foaf/0.1/OnlineAccount",
"http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject",
"http://www.w3.org/ns/dcat#Dataset",
"http://www.w3.org/ns/dcat#Distribution",
"http://www.w3.org/ns/dcat#Catalog"
]
} }]},
# // CLEANUP
#
%GraphCleanup{

65
config/delta-producer/administrative-units/export.json Normal file
View File

@@ -0,0 +1,65 @@
{
"export": [
{
"type": "http://lblod.data.gift/vocabularies/organisatie/BestuurseenheidClassificatieCode",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://mu.semte.ch/vocabularies/core/uuid",
"http://www.w3.org/2004/02/skos/core#prefLabel",
"http://www.w3.org/2004/02/skos/core#scopeNote"
]
},
{
"type": "http://data.vlaanderen.be/ns/besluit#Bestuurseenheid",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://mu.semte.ch/vocabularies/core/uuid",
"http://www.w3.org/2004/02/skos/core#prefLabel",
"http://www.w3.org/2004/02/skos/core#altLabel",
"http://data.vlaanderen.be/ns/besluit#werkingsgebied",
"http://www.w3.org/ns/org#classification"
]
},
{
"type": "http://data.vlaanderen.be/ns/besluit#Bestuursorgaan",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://mu.semte.ch/vocabularies/core/uuid",
"http://www.w3.org/2004/02/skos/core#prefLabel",
"http://data.vlaanderen.be/ns/besluit#bestuurt",
"http://www.w3.org/ns/org#classification",
"http://data.vlaanderen.be/ns/mandaat#bindingEinde",
"http://data.vlaanderen.be/ns/mandaat#bindingStart",
"https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/BestuursorgaanClassificatieCode",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://mu.semte.ch/vocabularies/core/uuid",
"http://www.w3.org/2004/02/skos/core#prefLabel",
"http://www.w3.org/2004/02/skos/core#scopeNote"
]
},
{
"type": "http://www.w3.org/ns/prov#Location",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://mu.semte.ch/vocabularies/core/uuid",
"http://www.w3.org/2000/01/rdf-schema#label",
"http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau"
]
}
]
}

22
config/delta-producer/dump-file-publisher/config.json Normal file
View File

@@ -0,0 +1,22 @@
{
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units" : {
"dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/AdministrativeUnitsCacheGraphDump",
"targetGraph": "http://redpencil.data.gift/id/deltas/producer/administrative-units",
"fileBaseName": "dump-administrative-units"
},
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units" : {
"dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/AdministrativeUnitsCacheGraphDump",
"targetGraph": "http://redpencil.data.gift/id/deltas/producer/administrative-units",
"fileBaseName": "dump-administrative-units"
},
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations" : {
"dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/OrganizationsCacheGraphDump",
"targetGraph": "http://redpencil.data.gift/id/deltas/producer/organizations",
"fileBaseName": "dump-administrative-units"
},
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations" : {
"dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/OrganizationsCacheGraphDump",
"targetGraph": "http://redpencil.data.gift/id/deltas/producer/organizations",
"fileBaseName": "dump-administrative-units"
}
}

419
config/delta-producer/organizations/export.json Normal file
View File

@@ -0,0 +1,419 @@
{
"export": [
{
"type": "http://www.w3.org/ns/person#Person",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://xmlns.com/foaf/0.1/givenName",
"http://xmlns.com/foaf/0.1/familyName",
"https://data.vlaanderen.be/ns/persoon#gebruikteVoornaam",
"http://data.vlaanderen.be/ns/mandaat#isAangesteldAls",
"http://www.w3.org/ns/org#heldBy"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/MandatarisStatusCode",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://data.lblod.info/vocabularies/erediensten/EredienstMandataris",
"strictTypeExport": "true",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://data.lblod.info/vocabularies/contacthub/startdatum",
"http://data.lblod.info/vocabularies/contacthub/eindedatum",
"http://www.w3.org/ns/org#holds",
"http://www.w3.org/ns/org#heldBy",
"http://schema.org/contactPoint",
"http://data.vlaanderen.be/ns/mandaat#start",
"http://data.vlaanderen.be/ns/mandaat#einde",
"http://data.vlaanderen.be/ns/mandaat#status",
"http://data.vlaanderen.be/ns/mandaat#isBestuurlijkeAliasVan",
"http://data.lblod.info/vocabularies/erediensten/geplandeEinddatumAanstelling",
"http://data.lblod.info/vocabularies/erediensten/redenVanStopzetting",
"http://data.lblod.info/vocabularies/erediensten/typeHelft"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/HelftVerkiezing",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://schema.org/ContactPoint",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://schema.org/email",
"http://schema.org/faxNumber",
"http://schema.org/telephone",
"http://www.w3.org/ns/locn#address"
]
},
{
"type": "http://data.vlaanderen.be/ns/mandaat#Mandaat",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/ns/org#role",
"http://www.w3.org/ns/org#postIn",
"http://www.w3.org/ns/org#role"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/BestuursfunctieCode",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://www.w3.org/ns/org#Role",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/BestuurseenheidClassificatieCode",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/TypeEredienst",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://data.lblod.info/vocabularies/erediensten/BestuurVanDeEredienst",
"strictTypeExport": "true",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel",
"http://www.w3.org/2004/02/skos/core#altLabel",
"http://www.w3.org/ns/adms#identifier",
"http://www.w3.org/ns/regorg#orgStatus",
"http://www.w3.org/ns/org#hasPrimarySite",
"http://www.w3.org/ns/org#hasSite",
"http://www.w3.org/ns/org#changedBy",
"http://www.w3.org/ns/org#resultedFrom",
"http://www.w3.org/ns/org#hasPost",
"http://www.w3.org/ns/org#linkedTo",
"http://www.w3.org/ns/org#hasSubOrganization",
"http://www.w3.org/ns/org#classification",
"http://data.lblod.info/vocabularies/erediensten/betrokkenBestuur",
"http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau",
"http://data.lblod.info/vocabularies/erediensten/denominatie",
"http://data.lblod.info/vocabularies/erediensten/grensoverschrijdend",
"http://data.lblod.info/vocabularies/erediensten/wordtBediendDoor"
]
},
{
"type": "http://data.lblod.info/vocabularies/erediensten/CentraalBestuurVanDeEredienst",
"strictTypeExport": "true",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel",
"http://www.w3.org/2004/02/skos/core#altLabel",
"http://www.w3.org/ns/adms#identifier",
"http://www.w3.org/ns/regorg#orgStatus",
"http://www.w3.org/ns/org#hasPrimarySite",
"http://www.w3.org/ns/org#hasSite",
"http://www.w3.org/ns/org#changedBy",
"http://www.w3.org/ns/org#resultedFrom",
"http://www.w3.org/ns/org#hasPost",
"http://www.w3.org/ns/org#linkedTo",
"http://www.w3.org/ns/org#hasSubOrganization",
"http://www.w3.org/ns/org#classification",
"http://data.lblod.info/vocabularies/erediensten/betrokkenBestuur",
"http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau"
]
},
{
"type": "http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan",
"strictTypeExport": "true",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel",
"http://www.w3.org/2004/02/skos/core#altLabel",
"http://www.w3.org/ns/adms#identifier",
"http://www.w3.org/ns/regorg#orgStatus",
"http://www.w3.org/ns/org#hasPrimarySite",
"http://www.w3.org/ns/org#hasSite",
"http://www.w3.org/ns/org#changedBy",
"http://www.w3.org/ns/org#resultedFrom",
"http://www.w3.org/ns/org#hasPost",
"http://www.w3.org/ns/org#linkedTo",
"http://www.w3.org/ns/org#hasSubOrganization",
"http://data.lblod.info/vocabularies/erediensten/typeEredienst"
]
},
{
"type": "http://data.vlaanderen.be/ns/besluit#Bestuursorgaan",
"strictTypeExport": "true",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://data.vlaanderen.be/ns/mandaat#bindingEinde",
"http://data.vlaanderen.be/ns/mandaat#bindingStart",
"http://data.vlaanderen.be/ns/besluit#bestuurt",
"http://www.w3.org/ns/org#classification",
"https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan",
"http://www.w3.org/ns/org#hasPost"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/BestuursorgaanClassificatieCode",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://data.lblod.info/vocabularies/erediensten/BetrokkenLokaleBesturen",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://data.lblod.info/vocabularies/erediensten/financieringspercentage",
"http://data.lblod.info/vocabularies/erediensten/typebetrokkenheid",
"http://www.w3.org/ns/org#organization"
]
},
{
"type": "http://www.w3.org/ns/adms#Identifier",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#notation",
"https://data.vlaanderen.be/ns/generiek#gestructureerdeIdentificator"
]
},
{
"type": "https://data.vlaanderen.be/ns/generiek#GestructureerdeIdentificator",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"https://data.vlaanderen.be/ns/generiek#lokaleIdentificator"
]
},
{
"type": "http://www.w3.org/ns/locn#Address",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"https://data.vlaanderen.be/ns/adres#Adresvoorstelling.huisnummer",
"https://data.vlaanderen.be/ns/adres#Adresvoorstelling.busnummer",
"http://www.w3.org/ns/locn#thoroughfare",
"http://www.w3.org/ns/locn#postCode",
"https://data.vlaanderen.be/ns/adres#gemeentenaam",
"http://www.w3.org/ns/locn#adminUnitL2",
"https://data.vlaanderen.be/ns/adres#land",
"http://www.w3.org/ns/locn#fullAddress",
"https://data.vlaanderen.be/ns/adres#verwijstNaar",
"http://purl.org/dc/terms/source"
]
},
{
"type": "http://www.w3.org/ns/org#Site",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/ns/org#siteAddress",
"https://data.vlaanderen.be/ns/organisatie#bestaatUit",
"http://data.lblod.info/vocabularies/erediensten/vestigingstype"
]
},
{
"type": "http://www.w3.org/ns/org#ChangeEvent",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://purl.org/dc/terms/date",
"http://purl.org/dc/terms/description",
"http://data.lblod.info/vocabularies/contacthub/typeWijziging",
"http://www.w3.org/ns/org#resultedFrom",
"http://www.w3.org/ns/org#changedBy"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/Veranderingsgebeurtenis",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://www.w3.org/2004/02/skos/core#Concept",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/OrganisatieStatusCode",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://www.w3.org/ns/prov#Location",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2000/01/rdf-schema#label",
"http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau",
"http://data.vlaanderen.be/ns/besluit#werkingsgebied"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/TypeBetrokkenheid",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://data.lblod.info/vocabularies/erediensten/RolBedienaar",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://data.lblod.info/vocabularies/contacthub/startdatum",
"http://data.lblod.info/vocabularies/contacthub/eindedatum",
"http://www.w3.org/ns/org#holds",
"http://www.w3.org/ns/org#heldBy",
"http://schema.org/contactPoint",
"http://data.lblod.info/vocabularies/erediensten/financiering",
"http://data.lblod.info/vocabularies/erediensten/voldoetAan"
]
},
{
"type": "http://data.lblod.info/vocabularies/erediensten/VoorwaardenBedienaar",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://data.lblod.info/vocabularies/erediensten/voldaan",
"http://data.lblod.info/vocabularies/erediensten/criterium",
"http://data.lblod.info/vocabularies/erediensten/criteriumbewijsstuktype"
]
},{
"type": "http://publications.europa.eu/ontology/euvoc#Country",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel",
"http://www.w3.org/2000/01/rdf-schema#label"
]
},{
"type": "http://data.lblod.info/vocabularies/erediensten/PositieBedienaar",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/ns/org#role",
"http://www.w3.org/ns/org#postIn",
"http://data.lblod.info/vocabularies/erediensten/functie",
"http://data.lblod.info/vocabularies/erediensten/behoort"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/EredienstBeroepen",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/BedienaarFinanceringCode",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/VoorwaardenBedienaarCriterium",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/BedienaarCriteriumBewijsstuk",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
},
{
"type": "http://lblod.data.gift/vocabularies/organisatie/TypeVestiging",
"graphsFilter": [
"http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
],
"properties": [
"http://www.w3.org/2004/02/skos/core#prefLabel"
]
}
]
}

115
config/delta/rules.js
View File

@@ -12,7 +12,118 @@ export default [
options: {
resourceFormat: "v0.0.1",
gracePeriod: 1000,
ignoreFromSelf: true
ignoreFromSelf: true,
optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
}
},
{
match: {
predicate: {
type: 'uri',
value: 'http://www.w3.org/ns/adms#status'
}
},
callback: {
method: 'POST',
url: 'http://jobs-controller/delta'
},
options: {
resourceFormat: 'v0.0.1',
gracePeriod: 1000,
ignoreFromSelf: true,
optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
}
]
},
{
match: {
},
callback: {
url: 'http://delta-producer-pub-graph-maintainer-administrative-units/delta',
method: 'POST'
},
options: {
resourceFormat: 'v0.0.1',
gracePeriod: 1000,
ignoreFromSelf: true,
optOutMuScopeIds: [
"http://redpencil.data.gift/id/concept/muScope/deltas/initialSync",
"http://redpencil.data.gift/id/concept/muScope/deltas/publicationGraphMaintenance"
]
}
},
{
match: {
graph: {
type: 'uri',
value: 'http://redpencil.data.gift/id/deltas/producer/administrative-units'
}
},
callback: {
url: 'http://delta-producer-json-diff-publisher-administrative-units/delta',
method: 'POST'
},
options: {
resourceFormat: 'v0.0.1',
gracePeriod: 1000,
ignoreFromSelf: true,
optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
}
},
{
match: {
},
callback: {
url: 'http://delta-producer-pub-graph-maintainer-organizations/delta',
method: 'POST'
},
options: {
resourceFormat: 'v0.0.1',
gracePeriod: 1000,
ignoreFromSelf: true,
optOutMuScopeIds: [
"http://redpencil.data.gift/id/concept/muScope/deltas/initialSync",
"http://redpencil.data.gift/id/concept/muScope/deltas/publicationGraphMaintenance"
]
}
},,
{
match: {
graph: {
type: 'uri',
value: 'http://redpencil.data.gift/id/deltas/producer/organizations'
}
},
callback: {
url: 'http://delta-producer-json-diff-publisher-organizations/delta',
method: 'POST'
},
options: {
resourceFormat: 'v0.0.1',
gracePeriod: 1000,
ignoreFromSelf: true,
optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
}
},
{
match: {
predicate: {
type: 'uri',
value: 'http://www.w3.org/ns/adms#status'
},
object: {
type: 'uri',
value: 'http://redpencil.data.gift/id/concept/JobStatus/scheduled'
}
},
callback: {
url: 'http://delta-producer-dump-file-publisher/delta',
method: 'POST'
},
options: {
resourceFormat: 'v0.0.1',
gracePeriod: 1000,
ignoreFromSelf: true,
optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
}
}
]

64
config/dispatcher/dispatcher.ex
View File

@@ -75,7 +75,7 @@ defmodule Dispatcher do
match "/worship-administrative-units/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/worship-administrative-units/"
end
match "/worship-services/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/worship-services/"
end
@@ -126,11 +126,11 @@ defmodule Dispatcher do
match "/change-event-types/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/change-event-types/"
end
end
match "/organization-status-codes/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/organization-status-codes/"
end
end
match "/locations/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/locations/"
@@ -142,11 +142,15 @@ defmodule Dispatcher do
match "/ministers/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/ministers/"
end
end
match "/minister-conditions/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/minister-conditions/"
end
end
match "/dates-of-birth/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/dates-of-birth/"
end
match "/nationalities/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/nationalities/"
@@ -177,7 +181,7 @@ defmodule Dispatcher do
match "/document-types-criterions/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/document-types-criterions/"
end
end
match "/site-types/*path", %{ accept: [:json], layer: :api} do
Proxy.forward conn, path, "http://cache/site-types/"
@@ -187,6 +191,7 @@ defmodule Dispatcher do
Proxy.forward conn, path, "http://cache/request-reasons/"
end
###############
# LOGIN
###############
@@ -208,7 +213,7 @@ defmodule Dispatcher do
match "/sessions/*path", %{ accept: [:any], layer: :api} do
Proxy.forward conn, path, "http://login/sessions/"
end
###############
# API SERVICES
@@ -243,6 +248,47 @@ defmodule Dispatcher do
end
#################################################################
# FILES
#################################################################
get "/files/:id/download", %{ accept: [:any], } do
Proxy.forward conn, [], "http://file/files/" <> id <> "/download"
end
get "/files/*path" , %{ layer: :api_services, accept: %{ json: true } } do
Proxy.forward conn, path, "http://cache/files/"
end
#################################################################
# DCAT
#################################################################
get "/datasets/*path", %{ layer: :api_services, accept: %{ json: true } } do
Proxy.forward conn, path, "http://cache/datasets/"
end
get "/distributions/*path", %{ layer: :api_services, accept: %{ json: true } } do
Proxy.forward conn, path, "http://cache/distributions/"
end
#################################################################
# DELTA: administrative-units
#################################################################
get "/sync/administrative-units/files/*path" do
Proxy.forward conn, path, "http://delta-producer-json-diff-file-publisher-administrative-units/files/"
end
#################################################################
# DELTA: organizations
#################################################################
get "/sync/organizations/files/*path" do
Proxy.forward conn, path, "http://delta-producer-json-diff-file-publisher-organizations/files/"
end
###############################################################
# frontend layer
###############################################################
@@ -266,11 +312,11 @@ defmodule Dispatcher do
match "/*_path", %{ layer: :frontend } do
Proxy.forward conn, [], "http://frontend/index.html"
end
###############################################################
# sparql endpoint
###############################################################
post "/sparql/*path", %{ layer: :api_services, accept: %{ sparql_json: true } } do
forward conn, path, "http://db:8890/sparql/"
end

66
config/jobs-controller/config.json Normal file
View File

@@ -0,0 +1,66 @@
{
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units": {
"tasksConfiguration": [
{
"currentOperation": null,
"nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing",
"nextIndex": "0"
},
{
"currentOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing",
"nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation",
"nextIndex": "1"
}
]
},
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units": {
"tasksConfiguration": [
{
"currentOperation": null,
"nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/healing/patchPublicationGraph",
"nextIndex": "0"
}
]
},
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units": {
"tasksConfiguration": [
{
"currentOperation": null,
"nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation",
"nextIndex": "0"
}
]
},
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations": {
"tasksConfiguration": [
{
"currentOperation": null,
"nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing",
"nextIndex": "0"
}
]
},
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations": {
"tasksConfiguration": [
{
"currentOperation": null,
"nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/healing/patchPublicationGraph",
"nextIndex": "0"
},
{
"currentOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing",
"nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation",
"nextIndex": "1"
}
]
},
"http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations": {
"tasksConfiguration": [
{
"currentOperation": null,
"nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation",
"nextIndex": "0"
}
]
}
}

157
config/resources/dcat.json Normal file
View File

@@ -0,0 +1,157 @@
{
"version": "0.1",
"prefixes": {
"dcat": "http://www.w3.org/ns/dcat#",
"dct": "http://purl.org/dc/terms/",
"prov": "http://www.w3.org/ns/prov#"
},
"resources": {
"catalogs": {
"class": "dcat:Catalog",
"attributes": {
"title": {
"type": "string",
"predicate": "dct:title"
},
"description": {
"type": "string",
"predicate": "dct:description"
},
"publisher": {
"type": "url",
"predicate": "dct:publisher"
},
"release-date": {
"type": "datetime",
"predicate": "dct:issued"
},
"modified": {
"type": "datetime",
"predicate": "dct:modified"
}
},
"relationships": {
"datasets": {
"target": "datasets",
"predicate": "dcat:dataset",
"cardinality": "many"
}
},
"features": [
"include-uri"
],
"new-resource-base": "http://data.lblod.info/id/catalog/"
},
"datasets": {
"class": "dcat:Dataset",
"attributes": {
"title": {
"type": "string",
"predicate": "dct:title"
},
"description": {
"type": "string",
"predicate": "dct:description"
},
"release-date": {
"type": "datetime",
"predicate": "dct:issued"
},
"modified": {
"type": "datetime",
"predicate": "dct:modified"
},
"type": {
"type": "url",
"predicate": "dct:type"
},
"subject": {
"type": "url",
"predicate": "dct:subject"
}
},
"relationships": {
"catalog": {
"target": "catalogs",
"predicate": "dcat:dataset",
"cardinality": "one",
"inverse": true
},
"distributions": {
"target": "distributions",
"predicate": "dcat:distribution",
"cardinality": "many"
},
"previous-version": {
"target": "datasets",
"predicate": "prov:wasRevisionOf",
"cardinality": "one"
},
"next-version": {
"target": "datasets",
"predicate": "prov:wasRevisionOf",
"cardinality": "one",
"inverse": true
}
},
"features": [
"include-uri"
],
"new-resource-base": "http://data.lblod.info/id/dataset/"
},
"distributions": {
"class": "dcat:Distribution",
"attributes": {
"title": {
"type": "string",
"predicate": "dct:title"
},
"description": {
"type": "string",
"predicate": "dct:description"
},
"release-date": {
"type": "datetime",
"predicate": "dct:issued"
},
"modified": {
"type": "datetime",
"predicate": "dct:modified"
},
"type": {
"type": "url",
"predicate": "dct:type"
},
"download-url": {
"type": "url",
"predicate": "dcat:downloadURL"
},
"format": {
"type": "string",
"predicate": "dct:format"
},
"byte-size": {
"type": "number",
"predicate": "dcat:byteSize"
}
},
"relationships": {
"dataset": {
"target": "datasets",
"predicate": "dcat:distribution",
"cardinality": "one",
"inverse": true
},
"subject": {
"target": "file",
"predicate": "dct:subject",
"cardinality": "one"
}
},
"features": [
"include-uri"
],
"new-resource-base": "http://data.lblod.info/id/distribution/"
}
}
}

2
config/resources/domain.json
View File

@@ -600,7 +600,7 @@
},
"relationships": {
"source": {
"predicate": "dct:source",
"predicate": "dc_terms:source",
"target": "concepts",
"cardinality": "one"
}

5
config/resources/domain.lisp
View File

@@ -1,4 +1,3 @@
(in-package :mu-cl-resources)
(defparameter *include-count-in-paginated-responses* t)
@@ -13,4 +12,6 @@
(read-domain-file "domain.json")
(read-domain-file "auth.json")
(read-domain-file "privacy-centric-service.json")
(read-domain-file "files-domain.lisp")
(read-domain-file "dcat.json")
(read-domain-file "privacy-centric-service.json")

13
config/resources/files-domain.lisp Normal file
View File

@@ -0,0 +1,13 @@
(define-resource file ()
:class (s-prefix "nfo:FileDataObject")
:properties `((:filename :string ,(s-prefix "nfo:fileName"))
(:format :string ,(s-prefix "dct:format"))
(:size :number ,(s-prefix "nfo:fileSize"))
(:extension :string ,(s-prefix "dbpedia:fileExtension"))
(:created :datetime ,(s-prefix "nfo:fileCreated")))
:has-one `((file :via ,(s-prefix "nie:dataSource")
:inverse t
:as "download"))
:resource-base (s-url "http://data.lblod.info/files/")
:features `(no-pagination-defaults include-uri)
:on-path "files")

162
docker-compose.yml
View File

@@ -36,7 +36,7 @@ services:
restart: always
logging: *default-logging
db:
image: semtech/mu-authorization:0.6.0-beta.7
image: cecemel/mu-authorization:0.6.0-beta.8
environment:
MU_SPARQL_ENDPOINT: "http://triplestore:8890/sparql"
LOG_OUTGOING_SPARQL_QUERIES: "true"
@@ -79,20 +79,23 @@ services:
restart: always
logging: *default-logging
kalliope-api:
image: lblod/jsonld-delta-service:0.3.2
image: lblod/jsonld-delta-service:0.4.0
volumes:
- ./config/kalliope:/config
- ./data/files:/share
links:
- db:database
environment:
SERVER_PORT: "80"
LOGGING_LEVEL: "INFO"
SPARQL_ENDPOINT: "http://db:8890/sparql"
LIMIT_SIZE: "10000"
labels:
- "logging=true"
restart: always
logging: *default-logging
deltanotifier:
image: semtech/mu-delta-notifier:0.1.0
image: cecemel/delta-notifier:0.2.0-beta.2
volumes:
- ./config/delta:/config
labels:
@@ -177,3 +180,156 @@ services:
- "logging=true"
restart: always
logging: *default-logging
file:
image: semtech/mu-file-service:3.2.0
volumes:
- ./data/files:/share
labels:
- "logging=true"
links:
- db:database
restart: always
logging: *default-logging
jobs-controller:
image: lblod/job-controller-service:0.7.0
links:
- db:database
volumes:
- ./config/jobs-controller/:/config/
################################################################################
# DELTA GENERAL
################################################################################
delta-producer-dump-file-publisher:
image: lblod/delta-producer-dump-file-publisher:0.4.0
volumes:
- ./config/delta-producer/dump-file-publisher:/config
- ./data/files:/share
environment:
FILES_GRAPH: 'http://mu.semte.ch/graphs/public' # Note: this will be extracted in a config per job one day
DCAT_DATASET_GRAPH: 'http://mu.semte.ch/graphs/public'
links:
- db:database
restart: always
logging: *default-logging
################################################################################
# DELTA ADMINISTRATIVE-UNITS: START
################################################################################
delta-producer-bg-jobs-initiator-administrative-units:
image: lblod/delta-producer-background-jobs-initiator:0.3.1
environment:
JOBS_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
DUMP_FILE_CREATION_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units'
INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units'
HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units'
CRON_PATTERN_DUMP_JOB: '0 0 4 * * 6' # weekly on saturday
CRON_PATTERN_HEALING_JOB: '0 0 4 * * *' # everyday at 4 AM
START_INITIAL_SYNC: 'false' # prefer to disable it now. We want to have feeling with how the DB acts.
labels:
- "logging=true"
links:
- db:database
restart: always
logging: *default-logging
delta-producer-pub-graph-maintainer-administrative-units:
image: lblod/delta-producer-publication-graph-maintainer:0.7.0
environment:
MAX_BODY_SIZE: "50mb"
JOBS_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
RELATIVE_FILE_PATH: "deltas/administrative-units"
PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/administrative-units'
HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units'
INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units'
QUEUE_POLL_INTERVAL: 3000
USE_VIRTUOSO_FOR_EXPENSIVE_SELECTS: "true" # some queries are just too expensive
SKIP_MU_AUTH_INITIAL_SYNC: 'true'
HEALING_PATCH_GRAPH_BATCH_SIZE: 500
volumes:
- ./config/delta-producer/administrative-units:/config
- ./data/files/:/share
links:
- db:database
- triplestore:virtuoso
restart: always
logging: *default-logging
delta-producer-json-diff-publisher-administrative-units:
image: lblod/delta-producer-json-diff-file-publisher:0.2.3
environment:
MAX_BODY_SIZE: "50mb"
ERROR_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
RELATIVE_FILE_PATH: "deltas/administrative-units"
PUBLISHER_URI: "http://data.lblod.info/services/delta-production-json-diff-file-manager/administrative-units"
PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/administrative-units'
DELTA_INTERVAL_MS: 10000
PRETTY_PRINT_DIFF_JSON: "true"
volumes:
- ./data/files:/share
labels:
- "logging=true"
links:
- db:database
restart: always
logging: *default-logging
################################################################################
# DELTA ADMINISTRATIVE-UNITS: END
################################################################################
################################################################################
# DELTA ORGANIZATIONS: START
################################################################################
delta-producer-bg-jobs-initiator-organizations:
image: lblod/delta-producer-background-jobs-initiator:0.3.1
environment:
JOBS_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
DUMP_FILE_CREATION_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations'
INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations'
HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations'
CRON_PATTERN_DUMP_JOB: '0 0 4 * * 6' # weekly on saturday
CRON_PATTERN_HEALING_JOB: '0 0 4 * * *' # everyday at 4 AM
START_INITIAL_SYNC: 'false' # prefer to disable it now. We want to have feeling with how the DB acts.
labels:
- "logging=true"
links:
- db:database
restart: always
logging: *default-logging
delta-producer-pub-graph-maintainer-organizations:
image: lblod/delta-producer-publication-graph-maintainer:0.7.0
environment:
MAX_BODY_SIZE: "50mb"
JOBS_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
RELATIVE_FILE_PATH: "deltas/organizations"
PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/organizations'
HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations'
INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations'
QUEUE_POLL_INTERVAL: 3000
USE_VIRTUOSO_FOR_EXPENSIVE_SELECTS: "true" # some queries are just too expensive
SKIP_MU_AUTH_INITIAL_SYNC: 'true'
HEALING_PATCH_GRAPH_BATCH_SIZE: 500
volumes:
- ./config/delta-producer/organizations:/config
- ./data/files/:/share
links:
- db:database
- triplestore:virtuoso
restart: always
logging: *default-logging
delta-producer-json-diff-publisher-organizations:
image: lblod/delta-producer-json-diff-file-publisher:0.2.3
environment:
MAX_BODY_SIZE: "50mb"
ERROR_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
RELATIVE_FILE_PATH: "deltas/organizations"
PUBLISHER_URI: "http://data.lblod.info/services/delta-production-json-diff-file-manager/organizations"
PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/organizations'
DELTA_INTERVAL_MS: 10000
PRETTY_PRINT_DIFF_JSON: "true"
volumes:
- ./data/files:/share
labels:
- "logging=true"
links:
- db:database
restart: always
logging: *default-logging
################################################################################
# DELTA ORGANIZATIONS: END
################################################################################