From 566ec39d9a832a3c40f677d71ae8fb03ea892fb1 Mon Sep 17 00:00:00 2001 From: cecemel Date: Fri, 3 Dec 2021 11:20:32 +0100 Subject: [PATCH] Feature/delta producer administrative units (#67) * Setting up file-service - pre-requisite to download dumps - pre-requiiste to download the json-diff files - security considerations: the content of the files created here, is - assumed to be public data, hence we allow it to be in the public graph * Bumping deltanotifier and mu-auth, which need to be able to work with mu-scop-id's * Delta-producer have multiple background-jobs with different tasks Hence the introduction of the jobs-controller. Note: in a more advanced configuration, you can easily extend the tasks related to one job. * A pre-requisite to serve dumps, is DCAT api support The delta-producers make a periodic dump of what is supposed to be published. They are published according to the DCAT data-model. Security-considerations: the DCAT producer by the delta-producer flow, are considered to be public data, hence added to the public graph. * Introduction of the delta-producer-dump-file-publisher Service responsible for creating dumps for the (potentially many) data delta flows. Here only for the administrative-units, but this can be extended in the future, in the config. * Introduction of delta-producer-background-jobs-initiator This service is responsible for triggering the background jobs related to the delta-productoin of the administrative-units. It trigger currently three jobs: - INITIAL_SYNC: this is the first job to fill the publication graph. In theory, this is only run once per deploy - HEALING_JOB: This is a periodic job, which checks what information is published vs. what information should have been published (according to a configuration). If differences are found, because delta-messages never arrived, they are compensated here/ - DUMP_JOB: A periodic job to create a dump of the publication graph. This is especially useful for consumers, who don't have to replay the whole delta-diffs, but just can start from the latest sync point. * Introduction of delta-producer-publication-graph-maintainer This service is the core of the logic which determines what should be published. The configuration is provided here, and can be elaborated refined in the config. It also excutes the healing logic. * Introduction of delta-producer-json-diff-file-publisher This simple service, listens to the updates performed on the publication graph, and serialized these to json-files which are compatible with current consumers * Missing dispatcher rules for DCAT * Various small things: - outline, whitespaces and comments * Update README.md * bump delta-producer-publication-graph-maintainer * Add organizations (KAL) - Work in progress * added organizations graph maintainer * removed inverses, added all bestuur types * refactored kalliope api Co-authored-by: Boris De Vloed --- README.md | 34 ++ config/authorization/config.ex | 12 +- .../administrative-units/export.json | 65 +++ .../dump-file-publisher/config.json | 22 + .../delta-producer/organizations/export.json | 419 ++++++++++++++++++ config/delta/rules.js | 115 ++++- config/dispatcher/dispatcher.ex | 64 ++- config/jobs-controller/config.json | 66 +++ config/resources/dcat.json | 157 +++++++ config/resources/domain.json | 2 +- config/resources/domain.lisp | 5 +- config/resources/files-domain.lisp | 13 + docker-compose.yml | 162 ++++++- 13 files changed, 1115 insertions(+), 21 deletions(-) create mode 100644 config/delta-producer/administrative-units/export.json create mode 100644 config/delta-producer/dump-file-publisher/config.json create mode 100644 config/delta-producer/organizations/export.json create mode 100644 config/jobs-controller/config.json create mode 100644 config/resources/dcat.json create mode 100644 config/resources/files-domain.lisp diff --git a/README.md b/README.md index 3afd554..d4741e2 100644 --- a/README.md +++ b/README.md @@ -12,3 +12,37 @@ Backend for the Contact Hub application based on the mu.semte.ch microservices You can shut down using `docker-compose stop` and remove everything using `docker-compose rm`. +### Setting up the delta-producers related services +To make sure the app can share data, producers need to be set up. There is an intial sync, that is potentially very expensive, and must be started manually + +#### producers administrative-units + +1. make sure the app is up and running, the migrations have run +2. in docker-compose.override.yml, make sure the following configuration is provided: +``` + delta-producer-pub-graph-maintainer-administrative-units: + environment: + START_INITIAL_SYNC: 'true' +``` +3. `drc up -d delta-producer-pub-graph-maintainer-administrative-units` +4. You can follow the status of the job, ideally through the dashboard, but this hasn't been setup yet. The following query should also give results: +``` + PREFIX task: + PREFIX adms: + PREFIX dct: + + SELECT DISTINCT ?job ?operation ?created ?modified ?status WHERE { + GRAPH ?g { + ?job a ; + task:operation ?operation; + adms:status ?status; + dct:created ?created; + dct:modified ?modified. + } + FILTER( ?operation IN ( + , + + ) + ) + } +``` diff --git a/config/authorization/config.ex b/config/authorization/config.ex index e4e380d..a94aaab 100644 --- a/config/authorization/config.ex +++ b/config/authorization/config.ex @@ -32,7 +32,7 @@ defmodule Acl.UserGroups.Config do "http://schema.org/ContactPoint", "http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie", "http://data.lblod.info/vocabularies/erediensten/CentraalBestuurVanDeEredienst", - "http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan", + "http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan", "http://www.w3.org/ns/org#Role", "http://www.w3.org/ns/org#Post", "http://data.lblod.info/vocabularies/contacthub/AgentInPositie", @@ -107,17 +107,21 @@ defmodule Acl.UserGroups.Config do %GroupSpec{ name: "public", useage: [:read], - access: %AlwaysAccessible{}, + access: %AlwaysAccessible{}, graphs: [ %GraphSpec{ graph: "http://mu.semte.ch/graphs/public", # mock login only constraint: %ResourceConstraint{ resource_types: [ "http://data.vlaanderen.be/ns/besluit#Bestuurseenheid", "http://xmlns.com/foaf/0.1/Person", - "http://xmlns.com/foaf/0.1/OnlineAccount" + "http://xmlns.com/foaf/0.1/OnlineAccount", + "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject", + "http://www.w3.org/ns/dcat#Dataset", + "http://www.w3.org/ns/dcat#Distribution", + "http://www.w3.org/ns/dcat#Catalog" ] } }]}, - + # // CLEANUP # %GraphCleanup{ diff --git a/config/delta-producer/administrative-units/export.json b/config/delta-producer/administrative-units/export.json new file mode 100644 index 0000000..f06ea4b --- /dev/null +++ b/config/delta-producer/administrative-units/export.json @@ -0,0 +1,65 @@ +{ + "export": [ + { + "type": "http://lblod.data.gift/vocabularies/organisatie/BestuurseenheidClassificatieCode", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://mu.semte.ch/vocabularies/core/uuid", + "http://www.w3.org/2004/02/skos/core#prefLabel", + "http://www.w3.org/2004/02/skos/core#scopeNote" + ] + }, + { + "type": "http://data.vlaanderen.be/ns/besluit#Bestuurseenheid", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://mu.semte.ch/vocabularies/core/uuid", + "http://www.w3.org/2004/02/skos/core#prefLabel", + "http://www.w3.org/2004/02/skos/core#altLabel", + "http://data.vlaanderen.be/ns/besluit#werkingsgebied", + "http://www.w3.org/ns/org#classification" + ] + }, + { + "type": "http://data.vlaanderen.be/ns/besluit#Bestuursorgaan", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://mu.semte.ch/vocabularies/core/uuid", + "http://www.w3.org/2004/02/skos/core#prefLabel", + "http://data.vlaanderen.be/ns/besluit#bestuurt", + "http://www.w3.org/ns/org#classification", + "http://data.vlaanderen.be/ns/mandaat#bindingEinde", + "http://data.vlaanderen.be/ns/mandaat#bindingStart", + "https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/BestuursorgaanClassificatieCode", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://mu.semte.ch/vocabularies/core/uuid", + "http://www.w3.org/2004/02/skos/core#prefLabel", + "http://www.w3.org/2004/02/skos/core#scopeNote" + ] + }, + { + "type": "http://www.w3.org/ns/prov#Location", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://mu.semte.ch/vocabularies/core/uuid", + "http://www.w3.org/2000/01/rdf-schema#label", + "http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau" + ] + } + ] +} diff --git a/config/delta-producer/dump-file-publisher/config.json b/config/delta-producer/dump-file-publisher/config.json new file mode 100644 index 0000000..8096db7 --- /dev/null +++ b/config/delta-producer/dump-file-publisher/config.json @@ -0,0 +1,22 @@ +{ + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units" : { + "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/AdministrativeUnitsCacheGraphDump", + "targetGraph": "http://redpencil.data.gift/id/deltas/producer/administrative-units", + "fileBaseName": "dump-administrative-units" + }, + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units" : { + "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/AdministrativeUnitsCacheGraphDump", + "targetGraph": "http://redpencil.data.gift/id/deltas/producer/administrative-units", + "fileBaseName": "dump-administrative-units" + }, + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations" : { + "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/OrganizationsCacheGraphDump", + "targetGraph": "http://redpencil.data.gift/id/deltas/producer/organizations", + "fileBaseName": "dump-administrative-units" + }, + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations" : { + "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/OrganizationsCacheGraphDump", + "targetGraph": "http://redpencil.data.gift/id/deltas/producer/organizations", + "fileBaseName": "dump-administrative-units" + } +} diff --git a/config/delta-producer/organizations/export.json b/config/delta-producer/organizations/export.json new file mode 100644 index 0000000..460beee --- /dev/null +++ b/config/delta-producer/organizations/export.json @@ -0,0 +1,419 @@ +{ + "export": [ + { + "type": "http://www.w3.org/ns/person#Person", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://xmlns.com/foaf/0.1/givenName", + "http://xmlns.com/foaf/0.1/familyName", + "https://data.vlaanderen.be/ns/persoon#gebruikteVoornaam", + "http://data.vlaanderen.be/ns/mandaat#isAangesteldAls", + "http://www.w3.org/ns/org#heldBy" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/MandatarisStatusCode", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://data.lblod.info/vocabularies/erediensten/EredienstMandataris", + "strictTypeExport": "true", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://data.lblod.info/vocabularies/contacthub/startdatum", + "http://data.lblod.info/vocabularies/contacthub/eindedatum", + "http://www.w3.org/ns/org#holds", + "http://www.w3.org/ns/org#heldBy", + "http://schema.org/contactPoint", + "http://data.vlaanderen.be/ns/mandaat#start", + "http://data.vlaanderen.be/ns/mandaat#einde", + "http://data.vlaanderen.be/ns/mandaat#status", + "http://data.vlaanderen.be/ns/mandaat#isBestuurlijkeAliasVan", + "http://data.lblod.info/vocabularies/erediensten/geplandeEinddatumAanstelling", + "http://data.lblod.info/vocabularies/erediensten/redenVanStopzetting", + "http://data.lblod.info/vocabularies/erediensten/typeHelft" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/HelftVerkiezing", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://schema.org/ContactPoint", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://schema.org/email", + "http://schema.org/faxNumber", + "http://schema.org/telephone", + "http://www.w3.org/ns/locn#address" + ] + }, + { + "type": "http://data.vlaanderen.be/ns/mandaat#Mandaat", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/ns/org#role", + "http://www.w3.org/ns/org#postIn", + "http://www.w3.org/ns/org#role" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/BestuursfunctieCode", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://www.w3.org/ns/org#Role", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/BestuurseenheidClassificatieCode", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/TypeEredienst", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://data.lblod.info/vocabularies/erediensten/BestuurVanDeEredienst", + "strictTypeExport": "true", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel", + "http://www.w3.org/2004/02/skos/core#altLabel", + "http://www.w3.org/ns/adms#identifier", + "http://www.w3.org/ns/regorg#orgStatus", + "http://www.w3.org/ns/org#hasPrimarySite", + "http://www.w3.org/ns/org#hasSite", + "http://www.w3.org/ns/org#changedBy", + "http://www.w3.org/ns/org#resultedFrom", + "http://www.w3.org/ns/org#hasPost", + "http://www.w3.org/ns/org#linkedTo", + "http://www.w3.org/ns/org#hasSubOrganization", + "http://www.w3.org/ns/org#classification", + "http://data.lblod.info/vocabularies/erediensten/betrokkenBestuur", + "http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau", + "http://data.lblod.info/vocabularies/erediensten/denominatie", + "http://data.lblod.info/vocabularies/erediensten/grensoverschrijdend", + "http://data.lblod.info/vocabularies/erediensten/wordtBediendDoor" + ] + }, + { + "type": "http://data.lblod.info/vocabularies/erediensten/CentraalBestuurVanDeEredienst", + "strictTypeExport": "true", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel", + "http://www.w3.org/2004/02/skos/core#altLabel", + "http://www.w3.org/ns/adms#identifier", + "http://www.w3.org/ns/regorg#orgStatus", + "http://www.w3.org/ns/org#hasPrimarySite", + "http://www.w3.org/ns/org#hasSite", + "http://www.w3.org/ns/org#changedBy", + "http://www.w3.org/ns/org#resultedFrom", + "http://www.w3.org/ns/org#hasPost", + "http://www.w3.org/ns/org#linkedTo", + "http://www.w3.org/ns/org#hasSubOrganization", + "http://www.w3.org/ns/org#classification", + "http://data.lblod.info/vocabularies/erediensten/betrokkenBestuur", + "http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau" + ] + }, + { + "type": "http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan", + "strictTypeExport": "true", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel", + "http://www.w3.org/2004/02/skos/core#altLabel", + "http://www.w3.org/ns/adms#identifier", + "http://www.w3.org/ns/regorg#orgStatus", + "http://www.w3.org/ns/org#hasPrimarySite", + "http://www.w3.org/ns/org#hasSite", + "http://www.w3.org/ns/org#changedBy", + "http://www.w3.org/ns/org#resultedFrom", + "http://www.w3.org/ns/org#hasPost", + "http://www.w3.org/ns/org#linkedTo", + "http://www.w3.org/ns/org#hasSubOrganization", + "http://data.lblod.info/vocabularies/erediensten/typeEredienst" + ] + }, + { + "type": "http://data.vlaanderen.be/ns/besluit#Bestuursorgaan", + "strictTypeExport": "true", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://data.vlaanderen.be/ns/mandaat#bindingEinde", + "http://data.vlaanderen.be/ns/mandaat#bindingStart", + "http://data.vlaanderen.be/ns/besluit#bestuurt", + "http://www.w3.org/ns/org#classification", + "https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan", + "http://www.w3.org/ns/org#hasPost" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/BestuursorgaanClassificatieCode", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://data.lblod.info/vocabularies/erediensten/BetrokkenLokaleBesturen", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://data.lblod.info/vocabularies/erediensten/financieringspercentage", + "http://data.lblod.info/vocabularies/erediensten/typebetrokkenheid", + "http://www.w3.org/ns/org#organization" + ] + }, + { + "type": "http://www.w3.org/ns/adms#Identifier", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#notation", + "https://data.vlaanderen.be/ns/generiek#gestructureerdeIdentificator" + ] + }, + { + "type": "https://data.vlaanderen.be/ns/generiek#GestructureerdeIdentificator", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "https://data.vlaanderen.be/ns/generiek#lokaleIdentificator" + ] + }, + { + "type": "http://www.w3.org/ns/locn#Address", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "https://data.vlaanderen.be/ns/adres#Adresvoorstelling.huisnummer", + "https://data.vlaanderen.be/ns/adres#Adresvoorstelling.busnummer", + "http://www.w3.org/ns/locn#thoroughfare", + "http://www.w3.org/ns/locn#postCode", + "https://data.vlaanderen.be/ns/adres#gemeentenaam", + "http://www.w3.org/ns/locn#adminUnitL2", + "https://data.vlaanderen.be/ns/adres#land", + "http://www.w3.org/ns/locn#fullAddress", + "https://data.vlaanderen.be/ns/adres#verwijstNaar", + "http://purl.org/dc/terms/source" + ] + }, + { + "type": "http://www.w3.org/ns/org#Site", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/ns/org#siteAddress", + "https://data.vlaanderen.be/ns/organisatie#bestaatUit", + "http://data.lblod.info/vocabularies/erediensten/vestigingstype" + ] + }, + { + "type": "http://www.w3.org/ns/org#ChangeEvent", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://purl.org/dc/terms/date", + "http://purl.org/dc/terms/description", + "http://data.lblod.info/vocabularies/contacthub/typeWijziging", + "http://www.w3.org/ns/org#resultedFrom", + "http://www.w3.org/ns/org#changedBy" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/Veranderingsgebeurtenis", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://www.w3.org/2004/02/skos/core#Concept", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/OrganisatieStatusCode", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://www.w3.org/ns/prov#Location", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2000/01/rdf-schema#label", + "http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau", + "http://data.vlaanderen.be/ns/besluit#werkingsgebied" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/TypeBetrokkenheid", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://data.lblod.info/vocabularies/erediensten/RolBedienaar", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://data.lblod.info/vocabularies/contacthub/startdatum", + "http://data.lblod.info/vocabularies/contacthub/eindedatum", + "http://www.w3.org/ns/org#holds", + "http://www.w3.org/ns/org#heldBy", + "http://schema.org/contactPoint", + "http://data.lblod.info/vocabularies/erediensten/financiering", + "http://data.lblod.info/vocabularies/erediensten/voldoetAan" + ] + }, + { + "type": "http://data.lblod.info/vocabularies/erediensten/VoorwaardenBedienaar", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://data.lblod.info/vocabularies/erediensten/voldaan", + "http://data.lblod.info/vocabularies/erediensten/criterium", + "http://data.lblod.info/vocabularies/erediensten/criteriumbewijsstuktype" + ] + },{ + "type": "http://publications.europa.eu/ontology/euvoc#Country", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel", + "http://www.w3.org/2000/01/rdf-schema#label" + ] + },{ + "type": "http://data.lblod.info/vocabularies/erediensten/PositieBedienaar", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/ns/org#role", + "http://www.w3.org/ns/org#postIn", + "http://data.lblod.info/vocabularies/erediensten/functie", + "http://data.lblod.info/vocabularies/erediensten/behoort" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/EredienstBeroepen", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/BedienaarFinanceringCode", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/VoorwaardenBedienaarCriterium", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/BedienaarCriteriumBewijsstuk", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + }, + { + "type": "http://lblod.data.gift/vocabularies/organisatie/TypeVestiging", + "graphsFilter": [ + "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + ], + "properties": [ + "http://www.w3.org/2004/02/skos/core#prefLabel" + ] + } + ] +} diff --git a/config/delta/rules.js b/config/delta/rules.js index b46db96..5476fc4 100644 --- a/config/delta/rules.js +++ b/config/delta/rules.js @@ -12,7 +12,118 @@ export default [ options: { resourceFormat: "v0.0.1", gracePeriod: 1000, - ignoreFromSelf: true + ignoreFromSelf: true, + optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ] } + }, + { + match: { + predicate: { + type: 'uri', + value: 'http://www.w3.org/ns/adms#status' + } + }, + callback: { + method: 'POST', + url: 'http://jobs-controller/delta' + }, + options: { + resourceFormat: 'v0.0.1', + gracePeriod: 1000, + ignoreFromSelf: true, + optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ] } - ] \ No newline at end of file + }, + { + match: { + }, + callback: { + url: 'http://delta-producer-pub-graph-maintainer-administrative-units/delta', + method: 'POST' + }, + options: { + resourceFormat: 'v0.0.1', + gracePeriod: 1000, + ignoreFromSelf: true, + optOutMuScopeIds: [ + "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync", + "http://redpencil.data.gift/id/concept/muScope/deltas/publicationGraphMaintenance" + ] + } + }, + { + match: { + graph: { + type: 'uri', + value: 'http://redpencil.data.gift/id/deltas/producer/administrative-units' + } + }, + callback: { + url: 'http://delta-producer-json-diff-publisher-administrative-units/delta', + method: 'POST' + }, + options: { + resourceFormat: 'v0.0.1', + gracePeriod: 1000, + ignoreFromSelf: true, + optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ] + } + }, + { + match: { + }, + callback: { + url: 'http://delta-producer-pub-graph-maintainer-organizations/delta', + method: 'POST' + }, + options: { + resourceFormat: 'v0.0.1', + gracePeriod: 1000, + ignoreFromSelf: true, + optOutMuScopeIds: [ + "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync", + "http://redpencil.data.gift/id/concept/muScope/deltas/publicationGraphMaintenance" + ] + } + },, + { + match: { + graph: { + type: 'uri', + value: 'http://redpencil.data.gift/id/deltas/producer/organizations' + } + }, + callback: { + url: 'http://delta-producer-json-diff-publisher-organizations/delta', + method: 'POST' + }, + options: { + resourceFormat: 'v0.0.1', + gracePeriod: 1000, + ignoreFromSelf: true, + optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ] + } + }, + { + match: { + predicate: { + type: 'uri', + value: 'http://www.w3.org/ns/adms#status' + }, + object: { + type: 'uri', + value: 'http://redpencil.data.gift/id/concept/JobStatus/scheduled' + } + }, + callback: { + url: 'http://delta-producer-dump-file-publisher/delta', + method: 'POST' + }, + options: { + resourceFormat: 'v0.0.1', + gracePeriod: 1000, + ignoreFromSelf: true, + optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ] + } + } +] diff --git a/config/dispatcher/dispatcher.ex b/config/dispatcher/dispatcher.ex index ea35dd2..cc7cce7 100644 --- a/config/dispatcher/dispatcher.ex +++ b/config/dispatcher/dispatcher.ex @@ -75,7 +75,7 @@ defmodule Dispatcher do match "/worship-administrative-units/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/worship-administrative-units/" end - + match "/worship-services/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/worship-services/" end @@ -126,11 +126,11 @@ defmodule Dispatcher do match "/change-event-types/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/change-event-types/" - end + end match "/organization-status-codes/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/organization-status-codes/" - end + end match "/locations/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/locations/" @@ -142,11 +142,15 @@ defmodule Dispatcher do match "/ministers/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/ministers/" - end + end match "/minister-conditions/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/minister-conditions/" - end + end + + match "/dates-of-birth/*path", %{ accept: [:json], layer: :api} do + Proxy.forward conn, path, "http://cache/dates-of-birth/" + end match "/nationalities/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/nationalities/" @@ -177,7 +181,7 @@ defmodule Dispatcher do match "/document-types-criterions/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/document-types-criterions/" - end + end match "/site-types/*path", %{ accept: [:json], layer: :api} do Proxy.forward conn, path, "http://cache/site-types/" @@ -187,6 +191,7 @@ defmodule Dispatcher do Proxy.forward conn, path, "http://cache/request-reasons/" end + ############### # LOGIN ############### @@ -208,7 +213,7 @@ defmodule Dispatcher do match "/sessions/*path", %{ accept: [:any], layer: :api} do Proxy.forward conn, path, "http://login/sessions/" end - + ############### # API SERVICES @@ -243,6 +248,47 @@ defmodule Dispatcher do end + ################################################################# + # FILES + ################################################################# + + get "/files/:id/download", %{ accept: [:any], } do + Proxy.forward conn, [], "http://file/files/" <> id <> "/download" + end + + get "/files/*path" , %{ layer: :api_services, accept: %{ json: true } } do + Proxy.forward conn, path, "http://cache/files/" + end + + ################################################################# + # DCAT + ################################################################# + + get "/datasets/*path", %{ layer: :api_services, accept: %{ json: true } } do + Proxy.forward conn, path, "http://cache/datasets/" + end + + get "/distributions/*path", %{ layer: :api_services, accept: %{ json: true } } do + Proxy.forward conn, path, "http://cache/distributions/" + end + + + ################################################################# + # DELTA: administrative-units + ################################################################# + + get "/sync/administrative-units/files/*path" do + Proxy.forward conn, path, "http://delta-producer-json-diff-file-publisher-administrative-units/files/" + end + + ################################################################# + # DELTA: organizations + ################################################################# + + get "/sync/organizations/files/*path" do + Proxy.forward conn, path, "http://delta-producer-json-diff-file-publisher-organizations/files/" + end + ############################################################### # frontend layer ############################################################### @@ -266,11 +312,11 @@ defmodule Dispatcher do match "/*_path", %{ layer: :frontend } do Proxy.forward conn, [], "http://frontend/index.html" end - + ############################################################### # sparql endpoint ############################################################### - + post "/sparql/*path", %{ layer: :api_services, accept: %{ sparql_json: true } } do forward conn, path, "http://db:8890/sparql/" end diff --git a/config/jobs-controller/config.json b/config/jobs-controller/config.json new file mode 100644 index 0000000..19432d7 --- /dev/null +++ b/config/jobs-controller/config.json @@ -0,0 +1,66 @@ +{ + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units": { + "tasksConfiguration": [ + { + "currentOperation": null, + "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing", + "nextIndex": "0" + }, + { + "currentOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing", + "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation", + "nextIndex": "1" + } + ] + }, + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units": { + "tasksConfiguration": [ + { + "currentOperation": null, + "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/healing/patchPublicationGraph", + "nextIndex": "0" + } + ] + }, + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units": { + "tasksConfiguration": [ + { + "currentOperation": null, + "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation", + "nextIndex": "0" + } + ] + }, + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations": { + "tasksConfiguration": [ + { + "currentOperation": null, + "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing", + "nextIndex": "0" + } + ] + }, + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations": { + "tasksConfiguration": [ + { + "currentOperation": null, + "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/healing/patchPublicationGraph", + "nextIndex": "0" + }, + { + "currentOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing", + "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation", + "nextIndex": "1" + } + ] + }, + "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations": { + "tasksConfiguration": [ + { + "currentOperation": null, + "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation", + "nextIndex": "0" + } + ] + } +} diff --git a/config/resources/dcat.json b/config/resources/dcat.json new file mode 100644 index 0000000..eba75ea --- /dev/null +++ b/config/resources/dcat.json @@ -0,0 +1,157 @@ +{ + "version": "0.1", + "prefixes": { + "dcat": "http://www.w3.org/ns/dcat#", + "dct": "http://purl.org/dc/terms/", + "prov": "http://www.w3.org/ns/prov#" + }, + "resources": { + "catalogs": { + "class": "dcat:Catalog", + "attributes": { + "title": { + "type": "string", + "predicate": "dct:title" + }, + "description": { + "type": "string", + "predicate": "dct:description" + }, + "publisher": { + "type": "url", + "predicate": "dct:publisher" + }, + "release-date": { + "type": "datetime", + "predicate": "dct:issued" + }, + "modified": { + "type": "datetime", + "predicate": "dct:modified" + } + }, + "relationships": { + "datasets": { + "target": "datasets", + "predicate": "dcat:dataset", + "cardinality": "many" + } + }, + "features": [ + "include-uri" + ], + "new-resource-base": "http://data.lblod.info/id/catalog/" + }, + "datasets": { + "class": "dcat:Dataset", + "attributes": { + "title": { + "type": "string", + "predicate": "dct:title" + }, + "description": { + "type": "string", + "predicate": "dct:description" + }, + "release-date": { + "type": "datetime", + "predicate": "dct:issued" + }, + "modified": { + "type": "datetime", + "predicate": "dct:modified" + }, + "type": { + "type": "url", + "predicate": "dct:type" + }, + "subject": { + "type": "url", + "predicate": "dct:subject" + } + }, + "relationships": { + "catalog": { + "target": "catalogs", + "predicate": "dcat:dataset", + "cardinality": "one", + "inverse": true + }, + "distributions": { + "target": "distributions", + "predicate": "dcat:distribution", + "cardinality": "many" + }, + "previous-version": { + "target": "datasets", + "predicate": "prov:wasRevisionOf", + "cardinality": "one" + }, + "next-version": { + "target": "datasets", + "predicate": "prov:wasRevisionOf", + "cardinality": "one", + "inverse": true + } + }, + "features": [ + "include-uri" + ], + "new-resource-base": "http://data.lblod.info/id/dataset/" + }, + "distributions": { + "class": "dcat:Distribution", + "attributes": { + "title": { + "type": "string", + "predicate": "dct:title" + }, + "description": { + "type": "string", + "predicate": "dct:description" + }, + "release-date": { + "type": "datetime", + "predicate": "dct:issued" + }, + "modified": { + "type": "datetime", + "predicate": "dct:modified" + }, + "type": { + "type": "url", + "predicate": "dct:type" + }, + "download-url": { + "type": "url", + "predicate": "dcat:downloadURL" + }, + "format": { + "type": "string", + "predicate": "dct:format" + }, + "byte-size": { + "type": "number", + "predicate": "dcat:byteSize" + } + }, + "relationships": { + "dataset": { + "target": "datasets", + "predicate": "dcat:distribution", + "cardinality": "one", + "inverse": true + }, + "subject": { + "target": "file", + "predicate": "dct:subject", + "cardinality": "one" + } + }, + "features": [ + "include-uri" + ], + "new-resource-base": "http://data.lblod.info/id/distribution/" + } + } +} diff --git a/config/resources/domain.json b/config/resources/domain.json index a6eb352..0786afe 100644 --- a/config/resources/domain.json +++ b/config/resources/domain.json @@ -600,7 +600,7 @@ }, "relationships": { "source": { - "predicate": "dct:source", + "predicate": "dc_terms:source", "target": "concepts", "cardinality": "one" } diff --git a/config/resources/domain.lisp b/config/resources/domain.lisp index ac4d7d6..964c273 100644 --- a/config/resources/domain.lisp +++ b/config/resources/domain.lisp @@ -1,4 +1,3 @@ - (in-package :mu-cl-resources) (defparameter *include-count-in-paginated-responses* t) @@ -13,4 +12,6 @@ (read-domain-file "domain.json") (read-domain-file "auth.json") -(read-domain-file "privacy-centric-service.json") \ No newline at end of file +(read-domain-file "files-domain.lisp") +(read-domain-file "dcat.json") +(read-domain-file "privacy-centric-service.json") diff --git a/config/resources/files-domain.lisp b/config/resources/files-domain.lisp new file mode 100644 index 0000000..b9dfdcf --- /dev/null +++ b/config/resources/files-domain.lisp @@ -0,0 +1,13 @@ +(define-resource file () + :class (s-prefix "nfo:FileDataObject") + :properties `((:filename :string ,(s-prefix "nfo:fileName")) + (:format :string ,(s-prefix "dct:format")) + (:size :number ,(s-prefix "nfo:fileSize")) + (:extension :string ,(s-prefix "dbpedia:fileExtension")) + (:created :datetime ,(s-prefix "nfo:fileCreated"))) + :has-one `((file :via ,(s-prefix "nie:dataSource") + :inverse t + :as "download")) + :resource-base (s-url "http://data.lblod.info/files/") + :features `(no-pagination-defaults include-uri) + :on-path "files") diff --git a/docker-compose.yml b/docker-compose.yml index 1a376bd..c528e2f 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -36,7 +36,7 @@ services: restart: always logging: *default-logging db: - image: semtech/mu-authorization:0.6.0-beta.7 + image: cecemel/mu-authorization:0.6.0-beta.8 environment: MU_SPARQL_ENDPOINT: "http://triplestore:8890/sparql" LOG_OUTGOING_SPARQL_QUERIES: "true" @@ -79,20 +79,23 @@ services: restart: always logging: *default-logging kalliope-api: - image: lblod/jsonld-delta-service:0.3.2 + image: lblod/jsonld-delta-service:0.4.0 volumes: - ./config/kalliope:/config - ./data/files:/share + links: + - db:database environment: SERVER_PORT: "80" LOGGING_LEVEL: "INFO" SPARQL_ENDPOINT: "http://db:8890/sparql" + LIMIT_SIZE: "10000" labels: - "logging=true" restart: always logging: *default-logging deltanotifier: - image: semtech/mu-delta-notifier:0.1.0 + image: cecemel/delta-notifier:0.2.0-beta.2 volumes: - ./config/delta:/config labels: @@ -177,3 +180,156 @@ services: - "logging=true" restart: always logging: *default-logging + file: + image: semtech/mu-file-service:3.2.0 + volumes: + - ./data/files:/share + labels: + - "logging=true" + links: + - db:database + restart: always + logging: *default-logging + jobs-controller: + image: lblod/job-controller-service:0.7.0 + links: + - db:database + volumes: + - ./config/jobs-controller/:/config/ +################################################################################ +# DELTA GENERAL +################################################################################ + delta-producer-dump-file-publisher: + image: lblod/delta-producer-dump-file-publisher:0.4.0 + volumes: + - ./config/delta-producer/dump-file-publisher:/config + - ./data/files:/share + environment: + FILES_GRAPH: 'http://mu.semte.ch/graphs/public' # Note: this will be extracted in a config per job one day + DCAT_DATASET_GRAPH: 'http://mu.semte.ch/graphs/public' + links: + - db:database + restart: always + logging: *default-logging +################################################################################ +# DELTA ADMINISTRATIVE-UNITS: START +################################################################################ + delta-producer-bg-jobs-initiator-administrative-units: + image: lblod/delta-producer-background-jobs-initiator:0.3.1 + environment: + JOBS_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + DUMP_FILE_CREATION_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units' + INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units' + HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units' + CRON_PATTERN_DUMP_JOB: '0 0 4 * * 6' # weekly on saturday + CRON_PATTERN_HEALING_JOB: '0 0 4 * * *' # everyday at 4 AM + START_INITIAL_SYNC: 'false' # prefer to disable it now. We want to have feeling with how the DB acts. + labels: + - "logging=true" + links: + - db:database + restart: always + logging: *default-logging + delta-producer-pub-graph-maintainer-administrative-units: + image: lblod/delta-producer-publication-graph-maintainer:0.7.0 + environment: + MAX_BODY_SIZE: "50mb" + JOBS_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + RELATIVE_FILE_PATH: "deltas/administrative-units" + PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/administrative-units' + HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units' + INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units' + QUEUE_POLL_INTERVAL: 3000 + USE_VIRTUOSO_FOR_EXPENSIVE_SELECTS: "true" # some queries are just too expensive + SKIP_MU_AUTH_INITIAL_SYNC: 'true' + HEALING_PATCH_GRAPH_BATCH_SIZE: 500 + volumes: + - ./config/delta-producer/administrative-units:/config + - ./data/files/:/share + links: + - db:database + - triplestore:virtuoso + restart: always + logging: *default-logging + delta-producer-json-diff-publisher-administrative-units: + image: lblod/delta-producer-json-diff-file-publisher:0.2.3 + environment: + MAX_BODY_SIZE: "50mb" + ERROR_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + RELATIVE_FILE_PATH: "deltas/administrative-units" + PUBLISHER_URI: "http://data.lblod.info/services/delta-production-json-diff-file-manager/administrative-units" + PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/administrative-units' + DELTA_INTERVAL_MS: 10000 + PRETTY_PRINT_DIFF_JSON: "true" + volumes: + - ./data/files:/share + labels: + - "logging=true" + links: + - db:database + restart: always + logging: *default-logging +################################################################################ +# DELTA ADMINISTRATIVE-UNITS: END +################################################################################ +################################################################################ +# DELTA ORGANIZATIONS: START +################################################################################ + delta-producer-bg-jobs-initiator-organizations: + image: lblod/delta-producer-background-jobs-initiator:0.3.1 + environment: + JOBS_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + DUMP_FILE_CREATION_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations' + INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations' + HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations' + CRON_PATTERN_DUMP_JOB: '0 0 4 * * 6' # weekly on saturday + CRON_PATTERN_HEALING_JOB: '0 0 4 * * *' # everyday at 4 AM + START_INITIAL_SYNC: 'false' # prefer to disable it now. We want to have feeling with how the DB acts. + labels: + - "logging=true" + links: + - db:database + restart: always + logging: *default-logging + delta-producer-pub-graph-maintainer-organizations: + image: lblod/delta-producer-publication-graph-maintainer:0.7.0 + environment: + MAX_BODY_SIZE: "50mb" + JOBS_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + RELATIVE_FILE_PATH: "deltas/organizations" + PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/organizations' + HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations' + INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations' + QUEUE_POLL_INTERVAL: 3000 + USE_VIRTUOSO_FOR_EXPENSIVE_SELECTS: "true" # some queries are just too expensive + SKIP_MU_AUTH_INITIAL_SYNC: 'true' + HEALING_PATCH_GRAPH_BATCH_SIZE: 500 + volumes: + - ./config/delta-producer/organizations:/config + - ./data/files/:/share + links: + - db:database + - triplestore:virtuoso + restart: always + logging: *default-logging + delta-producer-json-diff-publisher-organizations: + image: lblod/delta-producer-json-diff-file-publisher:0.2.3 + environment: + MAX_BODY_SIZE: "50mb" + ERROR_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin" + RELATIVE_FILE_PATH: "deltas/organizations" + PUBLISHER_URI: "http://data.lblod.info/services/delta-production-json-diff-file-manager/organizations" + PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/organizations' + DELTA_INTERVAL_MS: 10000 + PRETTY_PRINT_DIFF_JSON: "true" + volumes: + - ./data/files:/share + labels: + - "logging=true" + links: + - db:database + restart: always + logging: *default-logging +################################################################################ +# DELTA ORGANIZATIONS: END +################################################################################