Merge branch 'development'

corrected cron schedule
2021-12-16 18:04:04 +01:00 · 2021-12-16 18:03:01 +01:00 · 2021-12-16 17:33:20 +01:00 · 2021-12-16 17:16:00 +01:00 · 2021-12-16 17:14:51 +01:00 · 2021-12-16 17:13:41 +01:00
25 changed files with 1785 additions and 260 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -25,4 +25,5 @@ DS_STORE

 source.json
 out.json
-security.yml
+security.yml
+config/proxy-https/proxy_ssl.conf
--- a/README.md
+++ b/README.md
@@ -12,14 +12,37 @@ Backend for the Contact Hub application based on the  mu.semte.ch microservices
 You can shut down using `docker-compose stop` and remove everything using `docker-compose rm`.


-## Add username/password for Basic Auth
+### Setting up the delta-producers related services
+To make sure the app can share data, producers need to be set up. There is an intial sync, that is potentially very expensive, and must be started manually

-By default, there are two accounts you can use to log-in using basic auth:
+#### producers administrative-units

- root/root
- contacthub/contacthub
-
-To add new accounts, execute the following command, then add the result to `./config/proxy/nginx.htpasswd`
-
-`docker run --rm --entrypoint htpasswd registry:2.7.0 -bn username password`
+1. make sure the app is up and running, the migrations have run
+2. in docker-compose.override.yml, make sure the following configuration is provided:
+```
+  delta-producer-pub-graph-maintainer-administrative-units:
+    environment:
+      START_INITIAL_SYNC: 'true'
+```
+3. `drc up -d delta-producer-pub-graph-maintainer-administrative-units`
+4. You can follow the status of the job, ideally through the dashboard, but this hasn't been setup yet. The following query should also give results:
+```
+   PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
+   PREFIX adms: <http://www.w3.org/ns/adms#>
+   PREFIX dct: <http://purl.org/dc/terms/>

+    SELECT DISTINCT ?job ?operation ?created ?modified ?status WHERE {
+       GRAPH ?g {
+         ?job a <http://vocab.deri.ie/cogs#Job>;
+              task:operation ?operation;
+              adms:status ?status;
+              dct:created ?created;
+              dct:modified ?modified.
+       }
+       FILTER( ?operation IN (
+         <http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units>,
+         <http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units>
+        )
+       )
+     }
+```
--- a/config/authorization/config.ex
+++ b/config/authorization/config.ex
@@ -32,7 +32,7 @@ defmodule Acl.UserGroups.Config do
                        "http://schema.org/ContactPoint",
                        "http://data.lblod.info/vocabularies/leidinggevenden/Bestuursfunctie",
                        "http://data.lblod.info/vocabularies/erediensten/CentraalBestuurVanDeEredienst",
-                        "http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan",                          
+                        "http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan",
                        "http://www.w3.org/ns/org#Role",
                        "http://www.w3.org/ns/org#Post",
                        "http://data.lblod.info/vocabularies/contacthub/AgentInPositie",
@@ -59,7 +59,8 @@ defmodule Acl.UserGroups.Config do
 ]
  defp access_by_role( group_string ) do
    %AccessByQuery{
-      vars: ["session_group","session_role"],
+#      vars: ["session_group","session_role"], todo we might have to  create multiple graph depending on role
+       vars: [],
      query: sparql_query_for_access_role( group_string ) }
  end

@@ -76,20 +77,38 @@ defmodule Acl.UserGroups.Config do
  def user_groups do
    [
       %GroupSpec{
-        name: "acmidm",
-        useage: [:read, :write, :read_for_write],
-        access: %AccessByQuery{
-          query: "PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
-                  PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
-                  PREFIX session: <http://mu.semte.ch/vocabularies/session/>
-                  SELECT ?account WHERE {
-                    <SESSION_ID> session:account ?account.
-                  } LIMIT 1",
-          vars: []
-        },
+        name: "acmidm-lezer",
+        useage: [:read],
+        access: access_by_role( "ABBOrganisatiePortaalGebruiker-lezer" ),
        graphs: [
          %GraphSpec{
-            graph: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin",
+            graph: "http://mu.semte.ch/graphs/organisatieportaal",
+            constraint: %ResourceConstraint{
+              resource_types: @protected_resource_type
+            }
+          }
+        ]
+      },
+       %GroupSpec{
+        name: "acmidm-editeerder",
+        useage: [:read, :write, :read_for_write],
+        access: access_by_role( "ABBOrganisatiePortaalGebruiker-editeerder" ),
+        graphs: [
+          %GraphSpec{
+            graph: "http://mu.semte.ch/graphs/organisatieportaal",
+            constraint: %ResourceConstraint{
+              resource_types: @protected_resource_type
+            }
+          }
+        ]
+      },
+       %GroupSpec{
+        name: "acmidm-beheerder",
+        useage: [:read, :write, :read_for_write],
+        access: access_by_role( "ABBOrganisatiePortaalGebruiker-beheerder" ),
+        graphs: [
+          %GraphSpec{
+            graph: "http://mu.semte.ch/graphs/organisatieportaal",
            constraint: %ResourceConstraint{
              resource_types: @protected_resource_type
            }
@@ -97,27 +116,47 @@ defmodule Acl.UserGroups.Config do
        ]
      },
           %GroupSpec{
-        name: "ch-admin",
-        useage: [:read, :write, :read_for_write],
-        access: access_by_role( "ChAdmin" ),
+        name: "ch-lezer",
+        useage: [:read],
+        access: access_by_role( "ABBOrganisatiePortaalGebruiker-lezer" ),
        graphs: [ %GraphSpec{
-                    graph: "http://mu.semte.ch/graphs/contacthub/",
+                    graph: "http://mu.semte.ch/graphs/organisatieportaal",
+                    constraint: %ResourceConstraint{
+                      resource_types: @protected_resource_type } } ] },
+           %GroupSpec{
+        name: "ch-editeerder",
+        useage: [:read, :write, :read_for_write],
+        access: access_by_role( "ABBOrganisatiePortaalGebruiker-editeerder" ),
+        graphs: [ %GraphSpec{
+                    graph: "http://mu.semte.ch/graphs/organisatieportaal",
+                    constraint: %ResourceConstraint{
+                      resource_types: @protected_resource_type } } ] },
+           %GroupSpec{
+        name: "ch-beheerder",
+        useage: [:read, :write, :read_for_write],
+        access: access_by_role( "ABBOrganisatiePortaalGebruiker-beheerder" ),
+        graphs: [ %GraphSpec{
+                    graph: "http://mu.semte.ch/graphs/organisatieportaal",
                    constraint: %ResourceConstraint{
                      resource_types: @protected_resource_type } } ] },
      %GroupSpec{
        name: "public",
        useage: [:read],
-        access: %AlwaysAccessible{}, 
+        access: %AlwaysAccessible{},
        graphs: [ %GraphSpec{
                    graph: "http://mu.semte.ch/graphs/public", # mock login only
                    constraint: %ResourceConstraint{
                      resource_types: [
                        "http://data.vlaanderen.be/ns/besluit#Bestuurseenheid",
                        "http://xmlns.com/foaf/0.1/Person",
-                        "http://xmlns.com/foaf/0.1/OnlineAccount"
+                        "http://xmlns.com/foaf/0.1/OnlineAccount",
+                        "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject",
+                        "http://www.w3.org/ns/dcat#Dataset",
+                        "http://www.w3.org/ns/dcat#Distribution",
+                        "http://www.w3.org/ns/dcat#Catalog"
                      ]
                    } }]},
-  
+
      # // CLEANUP
      #
      %GraphCleanup{
--- a/config/db/virtuoso-production.ini
+++ b/config/db/virtuoso-production.ini
@@ -0,0 +1,253 @@
+;
+;  virtuoso.ini
+;
+;  Configuration file for the OpenLink Virtuoso VDBMS Server
+;
+;  To learn more about this product, or any other product in our
+;  portfolio, please check out our web site at:
+;
+;      http://virtuoso.openlinksw.com/
+;
+;  or contact us at:
+;
+;      general.information@openlinksw.com
+;
+;  If you have any technical questions, please contact our support
+;  staff at:
+;
+;      technical.support@openlinksw.com
+;
+;
+;  Database setup
+;
+[Database]
+DatabaseFile       = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.db
+ErrorLogFile       = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.log
+LockFile           = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.lck
+TransactionFile    = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.trx
+xa_persistent_file = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.pxa
+ErrorLogLevel      = 7
+FileExtend         = 200
+MaxCheckpointRemap = 17000 ;;niels 1/4 of the number of buffers (assuming db uses most of the available buffers, you can check with `status()` in isql-v
+Striping           = 0
+TempStorage        = TempDatabase
+Syslog             = 1	; write to syslog in case of failures where writing to log is not possible
+
+[TempDatabase]
+DatabaseFile       = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso-temp.db
+TransactionFile    = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso-temp.trx
+MaxCheckpointRemap = 2000
+Striping           = 0
+
+;
+;  Server parameters
+;
+[Parameters]
+ServerPort                 = 1111
+LiteMode                   = 0
+DisableUnixSocket          = 1
+DisableTcpSocket           = 0
+;SSLServerPort			= 2111
+;SSLCertificate			= cert.pem
+;SSLPrivateKey			= pk.pem
+;X509ClientVerify		= 0
+;X509ClientVerifyDepth		= 0
+;X509ClientVerifyCAFile		= ca.pem
+MaxClientConnections       = 10
+CheckpointInterval         = 60
+TraceOn                    = errors	; write traces
+O_DIRECT                   = 0
+CaseMode                   = 2
+MaxStaticCursorRows        = 5000
+CheckpointAuditTrail       = 0
+AllowOSCalls               = 0
+SchedulerInterval          = 10
+DirsAllowed                = ., /usr/local/virtuoso-opensource/share/virtuoso/vad
+ThreadCleanupInterval      = 10
+ThreadThreshold            = 10
+ResourcesCleanupInterval   = 10
+FreeTextBatchSize          = 100000
+SingleCPU                  = 0
+VADInstallDir              = /usr/local/virtuoso-opensource/share/virtuoso/vad/
+PrefixResultNames          = 0
+RdfFreeTextRulesSize       = 100
+IndexTreeMaps              = 256
+MaxMemPoolSize             = 200000000
+PrefixResultNames          = 0
+MacSpotlight               = 0
+MaxQueryMem                = 4G	; memory allocated to query processor ;;Niels said to 4g
+VectorSize                 = 1000	; initial parallel query vector (array of query operations) size
+MaxVectorSize              = 1000000	; query vector size threshold.
+AdjustVectorSize           = 0
+ThreadsPerQuery            = 4
+AsyncQueueMaxThreads       = 10
+TransactionAfterImageLimit = 150000000	; max 150MB transaction, total transient consumption is 450MB (roll forward + roll back)
+;;
+;; When running with large data sets, one should configure the Virtuoso
+;; process to use between 2/3 to 3/5 of free system memory and to stripe
+;; storage on all available disks.
+;;
+;; Uncomment next two lines if there is 2 GB system memory free
+;;NumberOfBuffers            = 170000
+;;MaxDirtyBuffers            = 130000
+
+;; Uncomment next two lines if there is 4 GB system memory free
+;;NumberOfBuffers          = 340000
+;;MaxDirtyBuffers          = 250000
+;; Uncomment next two lines if there is 8 GB system memory free
+NumberOfBuffers          = 680000
+MaxDirtyBuffers          = 500000
+;; Uncomment next two lines if there is 16 GB system memory free
+;NumberOfBuffers          = 1360000
+;MaxDirtyBuffers          = 1000000
+;; Uncomment next two lines if there is 32 GB system memory free
+;NumberOfBuffers          = 2720000
+;MaxDirtyBuffers          = 2000000
+;; Uncomment next two lines if there is 48 GB system memory free
+;NumberOfBuffers          = 4000000
+;MaxDirtyBuffers          = 3000000
+;; Uncomment next two lines if there is 64 GB system memory free
+;NumberOfBuffers          = 5450000
+;MaxDirtyBuffers          = 4000000
+;;
+;; Note the default settings will take very little memory
+;; but will not result in very good performance
+;;
+;NumberOfBuffers          = 10000
+;MaxDirtyBuffers          = 6000
+
+;; Introduced after bug with export service
+MaxSortedTopRows = 1000000 ; 1 million!
+
+[HTTPServer]
+ServerPort                  = 8890
+ServerRoot                  = /usr/local/virtuoso-opensource/var/lib/virtuoso/vsp
+MaxClientConnections        = 10
+DavRoot                     = DAV
+EnabledDavVSP               = 0
+HTTPProxyEnabled            = 0
+TempASPXDir                 = 0
+DefaultMailServer           = localhost:25
+ServerThreads               = 10
+MaxKeepAlives               = 10
+KeepAliveTimeout            = 10
+MaxCachedProxyConnections   = 10
+ProxyConnectionCacheTimeout = 15
+HTTPThreadSize              = 280000
+HttpPrintWarningsInOutput   = 0
+Charset                     = UTF-8
+;;HTTPLogFile             = http.log
+MaintenancePage             = atomic.html
+EnabledGzipContent          = 1
+
+[AutoRepair]
+BadParentLinks = 0
+
+[Client]
+SQL_PREFETCH_ROWS  = 100
+SQL_PREFETCH_BYTES = 16000
+SQL_QUERY_TIMEOUT  = 0
+SQL_TXN_TIMEOUT    = 0
+;SQL_NO_CHAR_C_ESCAPE		= 1
+;SQL_UTF8_EXECS			= 0
+;SQL_NO_SYSTEM_TABLES		= 0
+;SQL_BINARY_TIMESTAMP		= 1
+;SQL_ENCRYPTION_ON_PASSWORD	= -1
+
+[VDB]
+ArrayOptimization           = 0
+NumArrayParameters          = 10
+VDBDisconnectTimeout        = 1000
+KeepConnectionOnFixedThread = 0
+
+[Replication]
+ServerName   = db-D602566B774E
+ServerEnable = 1
+QueueMax     = 50000
+
+;
+;  Striping setup
+;
+;  These parameters have only effect when Striping is set to 1 in the
+;  [Database] section, in which case the DatabaseFile parameter is ignored.
+;
+;  With striping, the database is spawned across multiple segments
+;  where each segment can have multiple stripes.
+;
+;  Format of the lines below:
+;    Segment<number> = <size>, <stripe file name> [, <stripe file name> .. ]
+;
+;  <number> must be ordered from 1 up.
+;
+;  The <size> is the total size of the segment which is equally divided
+;  across all stripes forming  the segment. Its specification can be in
+;  gigabytes (g), megabytes (m), kilobytes (k) or in database blocks
+;  (b, the default)
+;
+;  Note that the segment size must be a multiple of the database page size
+;  which is currently 8k. Also, the segment size must be divisible by the
+;  number of stripe files forming  the segment.
+;
+;  The example below creates a 200 meg database striped on two segments
+;  with two stripes of 50 meg and one of 100 meg.
+;
+;  You can always add more segments to the configuration, but once
+;  added, do not change the setup.
+;
+[Striping]
+Segment1 = 100M, db-seg1-1.db, db-seg1-2.db
+Segment2 = 100M, db-seg2-1.db
+;...
+;[TempStriping]
+;Segment1			= 100M, db-seg1-1.db, db-seg1-2.db
+;Segment2			= 100M, db-seg2-1.db
+;...
+;[Ucms]
+;UcmPath			= <path>
+;Ucm1				= <file>
+;Ucm2				= <file>
+;...
+
+[Zero Config]
+ServerName = virtuoso (D602566B774E)
+;ServerDSN			= ZDSN
+;SSLServerName			=
+;SSLServerDSN			=
+
+[Mono]
+;MONO_TRACE			= Off
+;MONO_PATH			= <path_here>
+;MONO_ROOT			= <path_here>
+;MONO_CFG_DIR			= <path_here>
+;virtclr.dll			=
+
+[URIQA]
+DynamicLocal = 0
+DefaultHost  = localhost:8890
+
+[SPARQL]
+;ExternalQuerySource		= 1
+;ExternalXsltSource     = 1
+;DefaultGraph           = http://localhost:8890/dataspace
+;ImmutableGraphs        = http://localhost:8890/dataspace
+ResultSetMaxRows           = 1000000 ; 1 million
+;MaxQueryCostEstimationTime = 4000	; in seconds
+MaxQueryExecutionTime      = 1200	; in seconds
+DefaultQuery               = select distinct ?Concept where {[] a ?Concept} LIMIT 100
+DeferInferenceRulesInit    = 0	; controls inference rules loading
+;PingService            = http://rpc.pingthesemanticweb.com/
+
+[Plugins]
+LoadPath = /usr/local/virtuoso-opensource/lib/virtuoso/hosting
+;Load1				= plain, wikiv
+;Load2				= plain, mediawiki
+;Load3				= plain, creolewiki
+;Load4			= plain, im
+;Load5		= plain, wbxml2
+;Load6			= plain, hslookup
+;Load7			= attach, libphp5.so
+;Load8			= Hosting, hosting_php.so
+;Load9			= Hosting,hosting_perl.so
+;Load10		= Hosting,hosting_python.so
+;Load11		= Hosting,hosting_ruby.so
+;Load12				= msdtc,msdtc_sample
--- a/config/db/virtuoso.ini
+++ b/config/db/virtuoso.ini
@@ -17,85 +17,83 @@
 ;
 ;      technical.support@openlinksw.com
 ;
-
 ;
 ;  Database setup
 ;
 [Database]
-DatabaseFile			= /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.db
-ErrorLogFile			= /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.log
-LockFile			= /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.lck
-TransactionFile			= /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.trx
-xa_persistent_file		= /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.pxa
-ErrorLogLevel			= 7
-FileExtend			= 200
-MaxCheckpointRemap		= 2000
-Striping			= 0
-TempStorage			= TempDatabase
-
+DatabaseFile       = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.db
+ErrorLogFile       = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.log
+LockFile           = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.lck
+TransactionFile    = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.trx
+xa_persistent_file = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso.pxa
+ErrorLogLevel      = 7
+FileExtend         = 200
+MaxCheckpointRemap = 8500 ;;niels 1/4 of the number of buffers (assuming db uses most of the available buffers, you can check with `status()` in isql-v
+Striping           = 0
+TempStorage        = TempDatabase
+Syslog             = 1	; write to syslog in case of failures where writing to log is not possible

 [TempDatabase]
-DatabaseFile			= /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso-temp.db
-TransactionFile			= /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso-temp.trx
-MaxCheckpointRemap		= 2000
-Striping			= 0
-
+DatabaseFile       = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso-temp.db
+TransactionFile    = /usr/local/virtuoso-opensource/var/lib/virtuoso/db/virtuoso-temp.trx
+MaxCheckpointRemap = 2000
+Striping           = 0

 ;
 ;  Server parameters
 ;
 [Parameters]
-ServerPort			= 1111
-LiteMode			= 0
-DisableUnixSocket		= 1
-DisableTcpSocket		= 0
+ServerPort                 = 1111
+LiteMode                   = 0
+DisableUnixSocket          = 1
+DisableTcpSocket           = 0
 ;SSLServerPort			= 2111
 ;SSLCertificate			= cert.pem
 ;SSLPrivateKey			= pk.pem
 ;X509ClientVerify		= 0
 ;X509ClientVerifyDepth		= 0
 ;X509ClientVerifyCAFile		= ca.pem
-MaxClientConnections		= 10
-CheckpointInterval		= 60
-O_DIRECT			= 0
-TraceOn       = errors ; log errors
+MaxClientConnections       = 10
+CheckpointInterval         = 60
+TraceOn                    = errors	; write traces
+O_DIRECT                   = 0
+CaseMode                   = 2
+MaxStaticCursorRows        = 5000
+CheckpointAuditTrail       = 0
+AllowOSCalls               = 0
+SchedulerInterval          = 10
+DirsAllowed                = ., /usr/local/virtuoso-opensource/share/virtuoso/vad
+ThreadCleanupInterval      = 10
+ThreadThreshold            = 10
+ResourcesCleanupInterval   = 10
+FreeTextBatchSize          = 100000
+SingleCPU                  = 0
+VADInstallDir              = /usr/local/virtuoso-opensource/share/virtuoso/vad/
+PrefixResultNames          = 0
+RdfFreeTextRulesSize       = 100
+IndexTreeMaps              = 256
+MaxMemPoolSize             = 200000000
+PrefixResultNames          = 0
+MacSpotlight               = 0
+MaxQueryMem                = 2G
+VectorSize                 = 1000	; initial parallel query vector (array of query operations) size
+MaxVectorSize              = 1000000	; query vector size threshold.
+AdjustVectorSize           = 0
+ThreadsPerQuery            = 4
+AsyncQueueMaxThreads       = 10
 TransactionAfterImageLimit = 150000000	; max 150MB transaction, total transient consumption is 450MB (roll forward + roll back)
-CaseMode			= 2
-MaxStaticCursorRows		= 5000
-CheckpointAuditTrail		= 0
-AllowOSCalls			= 0
-SchedulerInterval		= 10
-DirsAllowed			= ., /usr/local/virtuoso-opensource/share/virtuoso/vad
-ThreadCleanupInterval		= 10
-ThreadThreshold			= 10
-ResourcesCleanupInterval	= 10
-FreeTextBatchSize		= 100000
-SingleCPU			= 0
-VADInstallDir			= /usr/local/virtuoso-opensource/share/virtuoso/vad/
-PrefixResultNames               = 0
-RdfFreeTextRulesSize		= 100
-IndexTreeMaps			= 256
-MaxMemPoolSize                  = 200000000
-PrefixResultNames               = 0
-MacSpotlight                    = 0
-IndexTreeMaps                   = 64
-MaxQueryMem 		 	= 2G		; memory allocated to query processor
-VectorSize 		 	= 1000		; initial parallel query vector (array of query operations) size
-MaxVectorSize 		 	= 1000000	; query vector size threshold.
-AdjustVectorSize 	 	= 1
-ThreadsPerQuery 	 	= 4
-AsyncQueueMaxThreads 	 	= 10
 ;;
 ;; When running with large data sets, one should configure the Virtuoso
 ;; process to use between 2/3 to 3/5 of free system memory and to stripe
 ;; storage on all available disks.
 ;;
 ;; Uncomment next two lines if there is 2 GB system memory free
-NumberOfBuffers          = 170000
-MaxDirtyBuffers          = 130000
+;;NumberOfBuffers            = 170000
+;;MaxDirtyBuffers            = 130000
+
 ;; Uncomment next two lines if there is 4 GB system memory free
-;NumberOfBuffers          = 340000
-; MaxDirtyBuffers          = 250000
+NumberOfBuffers          = 340000
+MaxDirtyBuffers          = 250000
 ;; Uncomment next two lines if there is 8 GB system memory free
 ;NumberOfBuffers          = 680000
 ;MaxDirtyBuffers          = 500000
@@ -111,38 +109,45 @@ MaxDirtyBuffers          = 130000
 ;; Uncomment next two lines if there is 64 GB system memory free
 ;NumberOfBuffers          = 5450000
 ;MaxDirtyBuffers          = 4000000
+;;
+;; Note the default settings will take very little memory
+;; but will not result in very good performance
+;;
+;NumberOfBuffers          = 10000
+;MaxDirtyBuffers          = 6000

+;; Introduced after bug with export service
+MaxSortedTopRows = 1000000 ; 1 million!

 [HTTPServer]
-ServerPort			= 8890
-ServerRoot			= /usr/local/virtuoso-opensource/var/lib/virtuoso/vsp
-MaxClientConnections		= 10
-DavRoot				= DAV
-EnabledDavVSP			= 0
-HTTPProxyEnabled		= 0
-TempASPXDir			= 0
-DefaultMailServer		= localhost:25
-ServerThreads			= 10
-MaxKeepAlives			= 10
-KeepAliveTimeout		= 10
-MaxCachedProxyConnections	= 10
-ProxyConnectionCacheTimeout	= 15
-HTTPThreadSize			= 280000
-HttpPrintWarningsInOutput	= 0
-Charset				= UTF-8
-;;HTTPLogFile		        = http.log
-MaintenancePage             	= atomic.html
-EnabledGzipContent          	= 1
-
+ServerPort                  = 8890
+ServerRoot                  = /usr/local/virtuoso-opensource/var/lib/virtuoso/vsp
+MaxClientConnections        = 10
+DavRoot                     = DAV
+EnabledDavVSP               = 0
+HTTPProxyEnabled            = 0
+TempASPXDir                 = 0
+DefaultMailServer           = localhost:25
+ServerThreads               = 10
+MaxKeepAlives               = 10
+KeepAliveTimeout            = 10
+MaxCachedProxyConnections   = 10
+ProxyConnectionCacheTimeout = 15
+HTTPThreadSize              = 280000
+HttpPrintWarningsInOutput   = 0
+Charset                     = UTF-8
+;;HTTPLogFile             = http.log
+MaintenancePage             = atomic.html
+EnabledGzipContent          = 1

 [AutoRepair]
-BadParentLinks			= 0
+BadParentLinks = 0

 [Client]
-SQL_PREFETCH_ROWS		= 100
-SQL_PREFETCH_BYTES		= 16000
-SQL_QUERY_TIMEOUT		= 0
-SQL_TXN_TIMEOUT			= 0
+SQL_PREFETCH_ROWS  = 100
+SQL_PREFETCH_BYTES = 16000
+SQL_QUERY_TIMEOUT  = 0
+SQL_TXN_TIMEOUT    = 0
 ;SQL_NO_CHAR_C_ESCAPE		= 1
 ;SQL_UTF8_EXECS			= 0
 ;SQL_NO_SYSTEM_TABLES		= 0
@@ -150,16 +155,15 @@ SQL_TXN_TIMEOUT			= 0
 ;SQL_ENCRYPTION_ON_PASSWORD	= -1

 [VDB]
-ArrayOptimization		= 0
-NumArrayParameters		= 10
-VDBDisconnectTimeout		= 1000
-KeepConnectionOnFixedThread	= 0
+ArrayOptimization           = 0
+NumArrayParameters          = 10
+VDBDisconnectTimeout        = 1000
+KeepConnectionOnFixedThread = 0

 [Replication]
-ServerName			= db-D602566B774E
-ServerEnable			= 1
-QueueMax			= 50000
-
+ServerName   = db-D602566B774E
+ServerEnable = 1
+QueueMax     = 50000

 ;
 ;  Striping setup
@@ -191,28 +195,24 @@ QueueMax			= 50000
 ;  added, do not change the setup.
 ;
 [Striping]
-Segment1			= 100M, db-seg1-1.db, db-seg1-2.db
-Segment2			= 100M, db-seg2-1.db
+Segment1 = 100M, db-seg1-1.db, db-seg1-2.db
+Segment2 = 100M, db-seg2-1.db
 ;...
-
 ;[TempStriping]
 ;Segment1			= 100M, db-seg1-1.db, db-seg1-2.db
 ;Segment2			= 100M, db-seg2-1.db
 ;...
-
 ;[Ucms]
 ;UcmPath			= <path>
 ;Ucm1				= <file>
 ;Ucm2				= <file>
 ;...

-
 [Zero Config]
-ServerName			= virtuoso (D602566B774E)
+ServerName = virtuoso (D602566B774E)
 ;ServerDSN			= ZDSN
-;SSLServerName			= 
-;SSLServerDSN			= 
-
+;SSLServerName			=
+;SSLServerDSN			=

 [Mono]
 ;MONO_TRACE			= Off
@@ -221,31 +221,28 @@ ServerName			= virtuoso (D602566B774E)
 ;MONO_CFG_DIR			= <path_here>
 ;virtclr.dll			=

-
 [URIQA]
-DynamicLocal			= 0
-DefaultHost			= localhost:8890
-
+DynamicLocal = 0
+DefaultHost  = localhost:8890

 [SPARQL]
 ;ExternalQuerySource		= 1
-;ExternalXsltSource 		= 1
-;DefaultGraph      		= http://localhost:8890/dataspace
-;ImmutableGraphs    		= http://localhost:8890/dataspace
-ResultSetMaxRows           	= 10000
-MaxQueryCostEstimationTime 	= 400	; in seconds
-MaxQueryExecutionTime      	= 60	; in seconds
-DefaultQuery               	= select distinct ?Concept where {[] a ?Concept} LIMIT 100
-DeferInferenceRulesInit    	= 0  ; controls inference rules loading
-;PingService       		= http://rpc.pingthesemanticweb.com/
-
+;ExternalXsltSource     = 1
+;DefaultGraph           = http://localhost:8890/dataspace
+;ImmutableGraphs        = http://localhost:8890/dataspace
+ResultSetMaxRows           = 1000000 ; 1 million
+;MaxQueryCostEstimationTime = 4000	; in seconds
+MaxQueryExecutionTime      = 1200	; in seconds
+DefaultQuery               = select distinct ?Concept where {[] a ?Concept} LIMIT 100
+DeferInferenceRulesInit    = 0	; controls inference rules loading
+;PingService            = http://rpc.pingthesemanticweb.com/

 [Plugins]
-LoadPath			= /usr/local/virtuoso-opensource/lib/virtuoso/hosting
+LoadPath = /usr/local/virtuoso-opensource/lib/virtuoso/hosting
 ;Load1				= plain, wikiv
 ;Load2				= plain, mediawiki
 ;Load3				= plain, creolewiki
-;Load4			= plain, im 
+;Load4			= plain, im
 ;Load5		= plain, wbxml2
 ;Load6			= plain, hslookup
 ;Load7			= attach, libphp5.so
@@ -253,4 +250,4 @@ LoadPath			= /usr/local/virtuoso-opensource/lib/virtuoso/hosting
 ;Load9			= Hosting,hosting_perl.so
 ;Load10		= Hosting,hosting_python.so
 ;Load11		= Hosting,hosting_ruby.so
-;Load12				= msdtc,msdtc_sample
+;Load12				= msdtc,msdtc_sample
--- a/config/delta-producer/administrative-units/export.json
+++ b/config/delta-producer/administrative-units/export.json
@@ -0,0 +1,65 @@
+{
+  "export": [
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/BestuurseenheidClassificatieCode",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://mu.semte.ch/vocabularies/core/uuid",
+        "http://www.w3.org/2004/02/skos/core#prefLabel",
+        "http://www.w3.org/2004/02/skos/core#scopeNote"
+      ]
+    },
+    {
+      "type": "http://data.vlaanderen.be/ns/besluit#Bestuurseenheid",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://mu.semte.ch/vocabularies/core/uuid",
+        "http://www.w3.org/2004/02/skos/core#prefLabel",
+        "http://www.w3.org/2004/02/skos/core#altLabel",
+        "http://data.vlaanderen.be/ns/besluit#werkingsgebied",
+        "http://www.w3.org/ns/org#classification"
+      ]
+    },
+    {
+      "type": "http://data.vlaanderen.be/ns/besluit#Bestuursorgaan",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://mu.semte.ch/vocabularies/core/uuid",
+        "http://www.w3.org/2004/02/skos/core#prefLabel",
+        "http://data.vlaanderen.be/ns/besluit#bestuurt",
+        "http://www.w3.org/ns/org#classification",
+        "http://data.vlaanderen.be/ns/mandaat#bindingEinde",
+        "http://data.vlaanderen.be/ns/mandaat#bindingStart",
+        "https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/BestuursorgaanClassificatieCode",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://mu.semte.ch/vocabularies/core/uuid",
+        "http://www.w3.org/2004/02/skos/core#prefLabel",
+        "http://www.w3.org/2004/02/skos/core#scopeNote"
+      ]
+    },
+    {
+      "type": "http://www.w3.org/ns/prov#Location",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://mu.semte.ch/vocabularies/core/uuid",
+        "http://www.w3.org/2000/01/rdf-schema#label",
+        "http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau"
+      ]
+    }
+  ]
+}
--- a/config/delta-producer/dump-file-publisher/config.json
+++ b/config/delta-producer/dump-file-publisher/config.json
@@ -0,0 +1,22 @@
+{
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units" : {
+    "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/AdministrativeUnitsCacheGraphDump",
+    "targetGraph": "http://redpencil.data.gift/id/deltas/producer/administrative-units",
+    "fileBaseName": "dump-administrative-units"
+  },
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units" : {
+    "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/AdministrativeUnitsCacheGraphDump",
+    "targetGraph": "http://redpencil.data.gift/id/deltas/producer/administrative-units",
+    "fileBaseName": "dump-administrative-units"
+  },
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations" : {
+    "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/OrganizationsCacheGraphDump",
+    "targetGraph": "http://redpencil.data.gift/id/deltas/producer/organizations",
+    "fileBaseName": "dump-organizations"
+  },
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations" : {
+    "dcatDataSetSubject": "http://data.lblod.info/datasets/delta-producer/dumps/OrganizationsCacheGraphDump",
+    "targetGraph": "http://redpencil.data.gift/id/deltas/producer/organizations",
+    "fileBaseName": "dump-organizations"
+  }
+}
--- a/config/delta-producer/organizations/export.json
+++ b/config/delta-producer/organizations/export.json
@@ -0,0 +1,421 @@
+{
+  "export": [
+    {
+      "type": "http://www.w3.org/ns/person#Person",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [ 
+        "http://xmlns.com/foaf/0.1/givenName",
+        "http://xmlns.com/foaf/0.1/familyName",
+        "https://data.vlaanderen.be/ns/persoon#gebruikteVoornaam",
+        "http://data.vlaanderen.be/ns/mandaat#isAangesteldAls",
+        "http://www.w3.org/ns/org#heldBy"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/MandatarisStatusCode",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://data.lblod.info/vocabularies/erediensten/EredienstMandataris",
+      "strictTypeExport": "true",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://data.lblod.info/vocabularies/contacthub/startdatum",
+        "http://data.lblod.info/vocabularies/contacthub/eindedatum",
+        "http://www.w3.org/ns/org#holds",
+        "http://www.w3.org/ns/org#heldBy",
+        "http://schema.org/contactPoint",
+        "http://data.vlaanderen.be/ns/mandaat#start",
+        "http://data.vlaanderen.be/ns/mandaat#einde",
+        "http://data.vlaanderen.be/ns/mandaat#status",
+        "http://data.vlaanderen.be/ns/mandaat#isBestuurlijkeAliasVan",
+        "http://data.lblod.info/vocabularies/erediensten/geplandeEinddatumAanstelling",
+        "http://data.lblod.info/vocabularies/erediensten/redenVanStopzetting",
+        "http://data.lblod.info/vocabularies/erediensten/typeHelft"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/HelftVerkiezing",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://schema.org/ContactPoint",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://schema.org/email",
+        "http://schema.org/faxNumber",
+        "http://schema.org/telephone",
+        "http://www.w3.org/ns/locn#address"
+      ]
+    },
+    {
+      "type": "http://data.vlaanderen.be/ns/mandaat#Mandaat",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/ns/org#role",
+        "http://www.w3.org/ns/org#postIn",
+        "http://www.w3.org/ns/org#role"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/BestuursfunctieCode",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://www.w3.org/ns/org#Role",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/BestuurseenheidClassificatieCode",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/TypeEredienst",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://data.lblod.info/vocabularies/erediensten/BestuurVanDeEredienst",
+      "strictTypeExport": "true",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel",
+        "http://www.w3.org/2004/02/skos/core#altLabel",
+        "http://www.w3.org/ns/adms#identifier",
+        "http://www.w3.org/ns/regorg#orgStatus",
+        "http://www.w3.org/ns/org#hasPrimarySite",
+        "http://www.w3.org/ns/org#hasSite",
+        "http://www.w3.org/ns/org#changedBy",
+        "http://www.w3.org/ns/org#resultedFrom",
+        "http://www.w3.org/ns/org#hasPost",
+        "http://www.w3.org/ns/org#linkedTo",
+        "http://www.w3.org/ns/org#hasSubOrganization",
+        "http://www.w3.org/ns/org#classification",
+        "http://data.lblod.info/vocabularies/erediensten/betrokkenBestuur",
+        "http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau",
+        "http://data.lblod.info/vocabularies/erediensten/denominatie",
+        "http://data.lblod.info/vocabularies/erediensten/grensoverschrijdend",
+        "http://data.lblod.info/vocabularies/erediensten/wordtBediendDoor",
+        "http://data.lblod.info/vocabularies/erediensten/typeEredienst"
+      ]
+    },
+    {
+      "type": "http://data.lblod.info/vocabularies/erediensten/CentraalBestuurVanDeEredienst",
+      "strictTypeExport": "true",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel",
+        "http://www.w3.org/2004/02/skos/core#altLabel",
+        "http://www.w3.org/ns/adms#identifier",
+        "http://www.w3.org/ns/regorg#orgStatus",
+        "http://www.w3.org/ns/org#hasPrimarySite",
+        "http://www.w3.org/ns/org#hasSite",
+        "http://www.w3.org/ns/org#changedBy",
+        "http://www.w3.org/ns/org#resultedFrom",
+        "http://www.w3.org/ns/org#hasPost",
+        "http://www.w3.org/ns/org#linkedTo",
+        "http://www.w3.org/ns/org#hasSubOrganization",
+        "http://www.w3.org/ns/org#classification",
+        "http://data.lblod.info/vocabularies/erediensten/betrokkenBestuur",
+        "http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau",
+        "http://data.lblod.info/vocabularies/erediensten/typeEredienst"
+      ]
+    },
+    {
+      "type": "http://data.lblod.info/vocabularies/erediensten/RepresentatiefOrgaan",
+      "strictTypeExport": "true",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel",
+        "http://www.w3.org/2004/02/skos/core#altLabel",
+        "http://www.w3.org/ns/adms#identifier",
+        "http://www.w3.org/ns/regorg#orgStatus",
+        "http://www.w3.org/ns/org#hasPrimarySite",
+        "http://www.w3.org/ns/org#hasSite",
+        "http://www.w3.org/ns/org#changedBy",
+        "http://www.w3.org/ns/org#resultedFrom",
+        "http://www.w3.org/ns/org#hasPost",
+        "http://www.w3.org/ns/org#linkedTo",
+        "http://www.w3.org/ns/org#hasSubOrganization",
+        "http://data.lblod.info/vocabularies/erediensten/typeEredienst"
+      ]
+    },
+    {
+      "type": "http://data.vlaanderen.be/ns/besluit#Bestuursorgaan",
+      "strictTypeExport": "true",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://data.vlaanderen.be/ns/mandaat#bindingEinde",
+        "http://data.vlaanderen.be/ns/mandaat#bindingStart",
+        "http://data.vlaanderen.be/ns/besluit#bestuurt",
+        "http://www.w3.org/ns/org#classification",
+        "https://data.vlaanderen.be/ns/generiek#isTijdspecialisatieVan",
+        "http://www.w3.org/ns/org#hasPost"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/BestuursorgaanClassificatieCode",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://data.lblod.info/vocabularies/erediensten/BetrokkenLokaleBesturen",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://data.lblod.info/vocabularies/erediensten/financieringspercentage",
+        "http://data.lblod.info/vocabularies/erediensten/typebetrokkenheid",
+        "http://www.w3.org/ns/org#organization"
+      ]
+    },
+    {
+      "type": "http://www.w3.org/ns/adms#Identifier",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#notation",
+        "https://data.vlaanderen.be/ns/generiek#gestructureerdeIdentificator"
+      ]
+    },
+    {
+      "type": "https://data.vlaanderen.be/ns/generiek#GestructureerdeIdentificator",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "https://data.vlaanderen.be/ns/generiek#lokaleIdentificator"
+      ]
+    },
+    {
+      "type": "http://www.w3.org/ns/locn#Address",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "https://data.vlaanderen.be/ns/adres#Adresvoorstelling.huisnummer",
+        "https://data.vlaanderen.be/ns/adres#Adresvoorstelling.busnummer",
+        "http://www.w3.org/ns/locn#thoroughfare",
+        "http://www.w3.org/ns/locn#postCode",
+        "https://data.vlaanderen.be/ns/adres#gemeentenaam",
+        "http://www.w3.org/ns/locn#adminUnitL2",
+        "https://data.vlaanderen.be/ns/adres#land",
+        "http://www.w3.org/ns/locn#fullAddress",
+        "https://data.vlaanderen.be/ns/adres#verwijstNaar",
+        "http://purl.org/dc/terms/source"
+      ]
+    },
+    {
+      "type": "http://www.w3.org/ns/org#Site",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/ns/org#siteAddress",
+        "https://data.vlaanderen.be/ns/organisatie#bestaatUit",
+        "http://data.lblod.info/vocabularies/erediensten/vestigingstype"
+      ]
+    },
+    {
+      "type": "http://www.w3.org/ns/org#ChangeEvent",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://purl.org/dc/terms/date",
+        "http://purl.org/dc/terms/description",
+        "http://data.lblod.info/vocabularies/contacthub/typeWijziging",
+        "http://www.w3.org/ns/org#resultedFrom",
+        "http://www.w3.org/ns/org#changedBy"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/Veranderingsgebeurtenis",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://www.w3.org/2004/02/skos/core#Concept",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/OrganisatieStatusCode",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://www.w3.org/ns/prov#Location",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2000/01/rdf-schema#label",
+        "http://mu.semte.ch/vocabularies/ext/werkingsgebiedNiveau",
+        "http://data.vlaanderen.be/ns/besluit#werkingsgebied"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/TypeBetrokkenheid",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://data.lblod.info/vocabularies/erediensten/RolBedienaar", 
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://data.lblod.info/vocabularies/contacthub/startdatum",
+        "http://data.lblod.info/vocabularies/contacthub/eindedatum",
+        "http://www.w3.org/ns/org#holds",
+        "http://www.w3.org/ns/org#heldBy",
+        "http://schema.org/contactPoint",
+        "http://data.lblod.info/vocabularies/erediensten/financiering",
+        "http://data.lblod.info/vocabularies/erediensten/voldoetAan"
+      ]
+    },
+    {
+      "type": "http://data.lblod.info/vocabularies/erediensten/VoorwaardenBedienaar",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://data.lblod.info/vocabularies/erediensten/voldaan",
+        "http://data.lblod.info/vocabularies/erediensten/criterium",
+        "http://data.lblod.info/vocabularies/erediensten/criteriumbewijsstuktype"
+      ]
+    },{
+      "type": "http://publications.europa.eu/ontology/euvoc#Country",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel",
+        "http://www.w3.org/2000/01/rdf-schema#label"
+      ]
+    },{
+      "type": "http://data.lblod.info/vocabularies/erediensten/PositieBedienaar",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/ns/org#role",
+        "http://www.w3.org/ns/org#postIn",
+        "http://data.lblod.info/vocabularies/erediensten/functie",
+        "http://data.lblod.info/vocabularies/erediensten/behoort"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/EredienstBeroepen",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/BedienaarFinanceringCode",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/VoorwaardenBedienaarCriterium",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/BedienaarCriteriumBewijsstuk",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    },
+    {
+      "type": "http://lblod.data.gift/vocabularies/organisatie/TypeVestiging",
+      "graphsFilter": [
+        "http://mu.semte.ch/graphs/organisatieportaal"
+      ],
+      "properties": [
+        "http://www.w3.org/2004/02/skos/core#prefLabel"
+      ]
+    }
+  ]
+}
--- a/config/delta/rules.js
+++ b/config/delta/rules.js
@@ -1,16 +1,112 @@
 export default [
-    {
-      match: {
-        subject: {
-        }
-      },
-      callback: {
-        url: "http://kalliope-api/delta", method: "POST"
-      },
-      options: {
-        resourceFormat: "v0.0.1",
-        gracePeriod: 1000,
-        ignoreFromSelf: true
+  {
+    match: {
+      predicate: {
+        type: 'uri',
+        value: 'http://www.w3.org/ns/adms#status'
      }
+    },
+    callback: {
+      method: 'POST',
+      url: 'http://jobs-controller/delta'
+    },
+    options: {
+      resourceFormat: 'v0.0.1',
+      gracePeriod: 1000,
+      ignoreFromSelf: true,
+      optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
    }
-  ]
+  },
+  {
+    match: {
+    },
+    callback: {
+      url: 'http://delta-producer-pub-graph-maintainer-administrative-units/delta',
+      method: 'POST'
+    },
+    options: {
+      resourceFormat: 'v0.0.1',
+      gracePeriod: 1000,
+      ignoreFromSelf: true,
+      optOutMuScopeIds: [
+                          "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync",
+                          "http://redpencil.data.gift/id/concept/muScope/deltas/publicationGraphMaintenance"
+                        ]
+    }
+  },
+  {
+    match: {
+      graph: {
+        type: 'uri',
+        value: 'http://redpencil.data.gift/id/deltas/producer/administrative-units'
+      }
+    },
+    callback: {
+      url: 'http://delta-producer-json-diff-publisher-administrative-units/delta',
+      method: 'POST'
+    },
+    options: {
+      resourceFormat: 'v0.0.1',
+      gracePeriod: 1000,
+      ignoreFromSelf: true,
+      optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
+    }
+  },
+  {
+    match: {
+    },
+    callback: {
+      url: 'http://delta-producer-pub-graph-maintainer-organizations/delta',
+      method: 'POST'
+    },
+    options: {
+      resourceFormat: 'v0.0.1',
+      gracePeriod: 1000,
+      ignoreFromSelf: true,
+      optOutMuScopeIds: [
+                          "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync",
+                          "http://redpencil.data.gift/id/concept/muScope/deltas/publicationGraphMaintenance"
+                        ]
+    }
+  },
+  {
+    match: {
+      graph: {
+        type: 'uri',
+        value: 'http://redpencil.data.gift/id/deltas/producer/organizations'
+      }
+    },
+    callback: {
+      url: 'http://delta-producer-json-diff-publisher-organizations/delta',
+      method: 'POST'
+    },
+    options: {
+      resourceFormat: 'v0.0.1',
+      gracePeriod: 1000,
+      ignoreFromSelf: true,
+      optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
+    }
+  },
+  {
+    match: {
+      predicate: {
+        type: 'uri',
+        value: 'http://www.w3.org/ns/adms#status'
+      },
+      object: {
+        type: 'uri',
+        value: 'http://redpencil.data.gift/id/concept/JobStatus/scheduled'
+      }
+    },
+    callback: {
+      url: 'http://delta-producer-dump-file-publisher/delta',
+      method: 'POST'
+    },
+    options: {
+      resourceFormat: 'v0.0.1',
+      gracePeriod: 1000,
+      ignoreFromSelf: true,
+      optOutMuScopeIds: [ "http://redpencil.data.gift/id/concept/muScope/deltas/initialSync" ]
+    }
+  }
+]
--- a/config/dispatcher/dispatcher.ex
+++ b/config/dispatcher/dispatcher.ex
@@ -5,6 +5,7 @@ defmodule Dispatcher do
    html: ["text/html", "application/xhtml+html"],
    json: ["application/json", "application/vnd.api+json"],
    upload: ["multipart/form-data"],
+    sparql_json: ["application/sparql-results+json"],
    any: [ "*/*" ],
  ]

@@ -74,7 +75,7 @@ defmodule Dispatcher do
  match "/worship-administrative-units/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/worship-administrative-units/"
  end
-  
+
  match "/worship-services/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/worship-services/"
  end
@@ -125,11 +126,11 @@ defmodule Dispatcher do

  match "/change-event-types/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/change-event-types/"
-  end 
+  end

  match "/organization-status-codes/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/organization-status-codes/"
-  end   
+  end

  match "/locations/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/locations/"
@@ -141,19 +142,15 @@ defmodule Dispatcher do

  match "/ministers/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/ministers/"
-  end 
+  end

  match "/minister-conditions/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/minister-conditions/"
-  end 
-
-  match "/associated-legal-structures/*path", %{ accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/associated-legal-structures/"
-  end 
+  end

  match "/dates-of-birth/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/dates-of-birth/"
-  end   
+  end

  match "/nationalities/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/nationalities/"
@@ -162,6 +159,9 @@ defmodule Dispatcher do
  match "/gender-codes/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/gender-codes/"
  end
+  match "/concepts/*path", %{ accept: [:json], layer: :api} do
+    Proxy.forward conn, path, "http://cache/concepts/"
+  end

  match "/minister-positions/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/minister-positions/"
@@ -179,18 +179,18 @@ defmodule Dispatcher do
    Proxy.forward conn, path, "http://cache/minister-condition-criterions/"
  end

-  match "/legal-form-types/*path", %{ accept: [:json], layer: :api} do
-    Proxy.forward conn, path, "http://cache/legal-form-types/"
-  end
-
  match "/document-types-criterions/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/document-types-criterions/"
-  end  
+  end

  match "/site-types/*path", %{ accept: [:json], layer: :api} do
    Proxy.forward conn, path, "http://cache/site-types/"
  end
  
+  match "/request-reasons/*path", %{ accept: [:json], layer: :api} do
+    Proxy.forward conn, path, "http://cache/request-reasons/"
+  end
+

  ###############
  # LOGIN
@@ -213,7 +213,7 @@ defmodule Dispatcher do
  match "/sessions/*path", %{ accept: [:any], layer: :api} do
    Proxy.forward conn, path, "http://login/sessions/"
  end
-  
+

  ###############
  # API SERVICES
@@ -248,6 +248,47 @@ defmodule Dispatcher do
  end


+  #################################################################
+  # FILES
+  #################################################################
+
+  get "/files/:id/download", %{ accept: [:any], } do
+    Proxy.forward conn, [], "http://file/files/" <> id <> "/download"
+  end
+
+  get "/files/*path" , %{ layer: :api_services, accept: %{ json: true } } do
+    Proxy.forward conn, path, "http://cache/files/"
+  end
+
+  #################################################################
+  # DCAT
+  #################################################################
+
+  get "/datasets/*path", %{ layer: :api_services, accept: %{ json: true } } do
+    Proxy.forward conn, path, "http://cache/datasets/"
+  end
+
+  get "/distributions/*path", %{ layer: :api_services, accept: %{ json: true } } do
+    Proxy.forward conn, path, "http://cache/distributions/"
+  end
+
+
+  #################################################################
+  #  DELTA: administrative-units
+  #################################################################
+
+  get "/sync/administrative-units/files/*path" do
+    Proxy.forward conn, path, "http://delta-producer-json-diff-file-publisher-administrative-units/files/"
+  end
+
+  #################################################################
+  #  DELTA: organizations
+  #################################################################
+
+  get "/sync/organizations/files/*path" do
+    Proxy.forward conn, path, "http://delta-producer-json-diff-file-publisher-organizations/files/"
+  end
+
  ###############################################################
  # frontend layer
  ###############################################################
@@ -271,7 +312,14 @@ defmodule Dispatcher do
  match "/*_path", %{ layer: :frontend } do
    Proxy.forward conn, [], "http://frontend/index.html"
  end
-  
+
+  ###############################################################
+  # sparql endpoint
+  ###############################################################
+
+  post "/sparql/*path", %{ layer: :api_services, accept: %{ sparql_json: true } } do
+    forward conn, path, "http://db:8890/sparql/"
+  end

  ###############################################################
  # errors
--- a/config/jobs-controller/config.json
+++ b/config/jobs-controller/config.json
@@ -0,0 +1,66 @@
+{
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units": {
+    "tasksConfiguration": [
+      {
+        "currentOperation": null,
+        "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing",
+        "nextIndex": "0"
+      },
+      {
+        "currentOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing",
+        "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation",
+        "nextIndex": "1"
+      }
+    ]
+  },
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units": {
+    "tasksConfiguration": [
+      {
+        "currentOperation": null,
+        "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/healing/patchPublicationGraph",
+        "nextIndex": "0"
+      }
+    ]
+  },
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units": {
+    "tasksConfiguration": [
+      {
+        "currentOperation": null,
+        "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation",
+        "nextIndex": "0"
+      }
+    ]
+  },
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations": {
+    "tasksConfiguration": [
+      {
+        "currentOperation": null,
+        "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing",
+        "nextIndex": "0"
+      },
+      {
+        "currentOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/initialPublicationGraphSyncing",
+        "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation",
+        "nextIndex": "1"
+      }
+    ]
+  },
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations": {
+    "tasksConfiguration": [
+      {
+        "currentOperation": null,
+        "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/healing/patchPublicationGraph",
+        "nextIndex": "0"
+      }
+    ]
+  },
+  "http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations": {
+    "tasksConfiguration": [
+      {
+        "currentOperation": null,
+        "nextOperation": "http://redpencil.data.gift/id/jobs/concept/TaskOperation/deltas/deltaDumpFileCreation",
+        "nextIndex": "0"
+      }
+    ]
+  }
+}
--- a/config/migrations/2021110315263111111111111111-source-crab.sparql
+++ b/config/migrations/2021110315263111111111111111-source-crab.sparql
@@ -0,0 +1,13 @@
+INSERT DATA {
+  GRAPH <http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin> {
+    <http://lblod.data.gift/concept-schemes/6f6f95d3-94e1-48a7-99ec-cdff57e6752d> a <http://www.w3.org/2004/02/skos/core#ConceptScheme>;    
+    <http://www.w3.org/2004/02/skos/core#prefLabel> "Data sources"@en, "Databronnen"@nl .
+    <http://lblod.data.gift/concepts/e59c97a9-4e95-4d65-9696-756de47fbc1f>  a <http://www.w3.org/2004/02/skos/core#Concept>;  
+    <http://www.w3.org/2004/02/skos/core#inScheme> <http://lblod.data.gift/concept-schemes/6f6f95d3-94e1-48a7-99ec-cdff57e6752d>;  
+    <http://www.w3.org/2004/02/skos/core#topConceptOf><http://lblod.data.gift/concept-schemes/6f6f95d3-94e1-48a7-99ec-cdff57e6752d>;  
+    <http://mu.semte.ch/vocabularies/core/uuid> "e59c97a9-4e95-4d65-9696-756de47fbc1f";  
+    <http://www.w3.org/2004/02/skos/core#prefLabel> "Centraal Referentieadressenbestand (CRAB)";  
+    <http://www.w3.org/2004/02/skos/core#altLabel> "CRAB".
+  }
+}
+
--- a/config/migrations/20211119151700-move-reason-codes.sparql
+++ b/config/migrations/20211119151700-move-reason-codes.sparql
@@ -0,0 +1,14 @@
+DELETE {
+  GRAPH  <http://mu.semte.ch/application> {
+    ?s ?p ?o
+  }
+} INSERT {
+  GRAPH  <http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin> {
+    ?s ?p ?o
+  }
+} WHERE {
+  GRAPH  <http://mu.semte.ch/application> {
+    ?s a <http://mu.semte.ch/vocabularies/ext/ReasonCode>;
+    ?p ?o.
+  }
+}
--- a/config/migrations/2022212411511111121112333222-private-data.sparql
+++ b/config/migrations/2022212411511111121112333222-private-data.sparql
@@ -0,0 +1,53 @@
+ PREFIX  ext:  <http://mu.semte.ch/vocabularies/ext/>
+ PREFIX  xsd:  <http://www.w3.org/2001/XMLSchema#>
+ PREFIX  mu:   <http://mu.semte.ch/vocabularies/core/>
+ PREFIX  generiek: <https://data.vlaanderen.be/ns/generiek#>
+ PREFIX  persoon: <https://data.vlaanderen.be/ns/persoon#>
+ PREFIX  person: <http://www.w3.org/ns/person#>
+ 
+ DELETE {
+   GRAPH <http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin> {
+     ?s persoon:heeftGeboorte ?heeftGeboorte .
+     ?s persoon:registratie ?registration .
+     ?registration generiek:gestructureerdeIdentificator ?identificator .
+     ?identificator ?x ?y .
+     ?registration ?t ?j .
+     ?s persoon:geslacht ?geslacht .
+     ?s persoon:heeftNationaliteit ?nationaliteit .
+   }
+ }
+ INSERT {
+   GRAPH <http://mu.semte.ch/graphs/privacy-centric-graph> {
+     ?s persoon:heeftGeboorte ?heeftGeboorte .
+     ?s persoon:registratie ?registration .
+     ?registration generiek:gestructureerdeIdentificator ?identificator .
+     ?identificator ?x ?y .
+     ?registration ?t ?j .
+     ?s persoon:geslacht ?geslacht .
+     ?s persoon:heeftNationaliteit ?nationaliteit .
+   }
+ }
+ WHERE
+   { GRAPH <http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin>
+       { 
+         ?s a person:Person.
+         OPTIONAL
+           { ?s        persoon:heeftGeboorte  ?heeftGeboorte .
+             ?heeftGeboorte
+                       ?p                    ?o
+           }
+         OPTIONAL
+           { ?s        persoon:registratie   ?registration .
+             ?registration
+                       generiek:gestructureerdeIdentificator  ?identificator .
+             ?identificator
+                       ?x                    ?y .
+             ?registration
+                       ?t                    ?j
+           }
+         OPTIONAL
+           { ?s  persoon:geslacht  ?geslacht }
+         OPTIONAL
+           { ?s  persoon:heeftNationaliteit  ?nationaliteit }
+       }
+   }
--- a/config/migrations/2022212411511111121112333224-bestuur-admidm.sparql
+++ b/config/migrations/2022212411511111121112333224-bestuur-admidm.sparql
@@ -0,0 +1,13 @@
+PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+PREFIX dcterms: <http://purl.org/dc/terms/>
+PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>
+
+INSERT DATA {
+  GRAPH <http://mu.semte.ch/graphs/contacthub/accounts> {
+    <http://data.lblod.info/id/bestuurseenheden/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> a besluit:Bestuurseenheid ;
+       mu:uuid "141d9d6b-54af-4d17-b313-8d1c30bc3f5b" ;
+       skos:prefLabel "Agentschap Binnenlands Bestuur" ;
+       dcterms:identifier "OVO001835" .
+  }
+}
--- a/config/migrations/2022212411511111121112333225-clear-mock-data.sparql
+++ b/config/migrations/2022212411511111121112333225-clear-mock-data.sparql
@@ -0,0 +1,3 @@
+
+CLEAR GRAPH  <http://mu.semte.ch/graphs/public> ;
+CLEAR GRAPH <http://mu.semte.ch/graphs/organizations/141d9d6b-54af-4d17-b313-8d1c30bc3f5b>;
--- a/config/migrations/2022212411511111121112333226-mock-user-roles.sparql
+++ b/config/migrations/2022212411511111121112333226-mock-user-roles.sparql
@@ -0,0 +1,83 @@
+PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+PREFIX dcterms: <http://purl.org/dc/terms/>
+PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX ext:<http://mu.semte.ch/vocabularies/ext/>
+ INSERT DATA {
+    GRAPH  <http://mu.semte.ch/graphs/public> {
+       <http://data.lblod.info/id/bestuurseenheden/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> a besluit:Bestuurseenheid ;
+          mu:uuid "141d9d6b-54af-4d17-b313-8d1c30bc3f5b" ;
+          skos:prefLabel "Agentschap Binnenlands Bestuur" ;
+          dcterms:identifier "OVO001835" .
+
+      <http://data.lblod.info/id/persoon/d39fe09e-d339-4e43-8680-dbbcfb3d3470> a foaf:Person ;
+          mu:uuid "d39fe09e-d339-4e43-8680-dbbcfb3d3470" ;
+          foaf:firstName "Josephine" ;
+          foaf:familyName "Lezer" ;
+          foaf:member <http://data.lblod.info/id/bestuurseenheden/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> ;
+          foaf:account <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0989> .
+      <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0989> a foaf:OnlineAccount ;
+          mu:uuid "3a91ff60-07c1-4136-ac5e-55cf401e0989" ;
+          foaf:accountServiceHomepage <https://github.com/lblod/mock-login-service>;
+          ext:sessionRole "ABBOrganisatiePortaalGebruiker-lezer".
+
+      <http://data.lblod.info/id/persoon/d39fe09e-d339-4e43-8680-dbbcfb3d3471> a foaf:Person ;
+          mu:uuid "d39fe09e-d339-4e43-8680-dbbcfb3d3471" ;
+          foaf:firstName "Josephine" ;
+          foaf:familyName "Editeerder" ;
+          foaf:member <http://data.lblod.info/id/bestuurseenheden/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> ;
+          foaf:account <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0957> .
+      <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0957> a foaf:OnlineAccount ;
+          mu:uuid "3a91ff60-07c1-4136-ac5e-55cf401e0957" ;
+          foaf:accountServiceHomepage <https://github.com/lblod/mock-login-service>;
+          ext:sessionRole "ABBOrganisatiePortaalGebruiker-editeerder".
+          
+      <http://data.lblod.info/id/persoon/d39fe09e-d339-4e43-8680-dbbcfb3d3472> a foaf:Person ;
+          mu:uuid "d39fe09e-d339-4e43-8680-dbbcfb3d3472" ;
+          foaf:firstName "Josephine" ;
+          foaf:familyName "Beheerder" ;
+          foaf:member <http://data.lblod.info/id/bestuurseenheden/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> ;
+          foaf:account <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0958> .
+      <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0958> a foaf:OnlineAccount ;
+          mu:uuid "3a91ff60-07c1-4136-ac5e-55cf401e0958" ;
+          foaf:accountServiceHomepage <https://github.com/lblod/mock-login-service>;
+          ext:sessionRole "ABBOrganisatiePortaalGebruiker-beheerder".
+    }
+
+    GRAPH <http://mu.semte.ch/graphs/organizations/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> {
+      
+      <http://data.lblod.info/id/persoon/d39fe09e-d339-4e43-8680-dbbcfb3d3470> a foaf:Person ;
+          mu:uuid "d39fe09e-d339-4e43-8680-dbbcfb3d3470" ;
+          foaf:firstName "Josephine" ;
+          foaf:familyName "Lezer" ;
+          foaf:member <http://data.lblod.info/id/bestuurseenheden/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> ;
+          foaf:account <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0989> .
+      <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0989> a foaf:OnlineAccount ;
+          mu:uuid "3a91ff60-07c1-4136-ac5e-55cf401e0989" ;
+          foaf:accountServiceHomepage <https://github.com/lblod/mock-login-service>;
+          ext:sessionRole "ABBOrganisatiePortaalGebruiker-lezer".
+
+      <http://data.lblod.info/id/persoon/d39fe09e-d339-4e43-8680-dbbcfb3d3471> a foaf:Person ;
+          mu:uuid "d39fe09e-d339-4e43-8680-dbbcfb3d3471" ;
+          foaf:firstName "Josephine" ;
+          foaf:familyName "Editeerder" ;
+          foaf:member <http://data.lblod.info/id/bestuurseenheden/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> ;
+          foaf:account <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0957> .
+      <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0957> a foaf:OnlineAccount ;
+          mu:uuid "3a91ff60-07c1-4136-ac5e-55cf401e0957" ;
+          foaf:accountServiceHomepage <https://github.com/lblod/mock-login-service>;
+          ext:sessionRole "ABBOrganisatiePortaalGebruiker-editeerder".
+          
+      <http://data.lblod.info/id/persoon/d39fe09e-d339-4e43-8680-dbbcfb3d3472> a foaf:Person ;
+          mu:uuid "d39fe09e-d339-4e43-8680-dbbcfb3d3472" ;
+          foaf:firstName "Josephine" ;
+          foaf:familyName "Beheerder" ;
+          foaf:member <http://data.lblod.info/id/bestuurseenheden/141d9d6b-54af-4d17-b313-8d1c30bc3f5b> ;
+          foaf:account <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0958> .
+      <http://data.lblod.info/id/account/3a91ff60-07c1-4136-ac5e-55cf401e0958> a foaf:OnlineAccount ;
+          mu:uuid "3a91ff60-07c1-4136-ac5e-55cf401e0958" ;
+          foaf:accountServiceHomepage <https://github.com/lblod/mock-login-service>;
+          ext:sessionRole "ABBOrganisatiePortaalGebruiker-beheerder".
+    }
+  } 
--- a/config/migrations/2022212411511111121112333227-move-graph.sparql
+++ b/config/migrations/2022212411511111121112333227-move-graph.sparql
@@ -0,0 +1,8 @@
+PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
+PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
+PREFIX dcterms: <http://purl.org/dc/terms/>
+PREFIX besluit: <http://data.vlaanderen.be/ns/besluit#>
+PREFIX foaf: <http://xmlns.com/foaf/0.1/>
+PREFIX ext:<http://mu.semte.ch/vocabularies/ext/>
+MOVE <http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin> TO 
+<http://mu.semte.ch/graphs/organisatieportaal>
--- a/config/resources/dcat.json
+++ b/config/resources/dcat.json
@@ -0,0 +1,157 @@
+{
+    "version": "0.1",
+    "prefixes": {
+        "dcat": "http://www.w3.org/ns/dcat#",
+        "dct": "http://purl.org/dc/terms/",
+        "prov": "http://www.w3.org/ns/prov#"
+    },
+    "resources": {
+        "catalogs": {
+            "class": "dcat:Catalog",
+            "attributes": {
+                "title": {
+                    "type": "string",
+                    "predicate": "dct:title"
+                },
+                "description": {
+                    "type": "string",
+                    "predicate": "dct:description"
+                },
+                "publisher": {
+                    "type": "url",
+                    "predicate": "dct:publisher"
+                },
+                "release-date": {
+                    "type": "datetime",
+                    "predicate": "dct:issued"
+                },
+                "modified": {
+                    "type": "datetime",
+                    "predicate": "dct:modified"
+                }
+            },
+            "relationships": {
+                "datasets": {
+                    "target": "datasets",
+                    "predicate": "dcat:dataset",
+                    "cardinality": "many"
+                }
+            },
+            "features": [
+                "include-uri"
+            ],
+            "new-resource-base": "http://data.lblod.info/id/catalog/"
+        },
+        "datasets": {
+            "class": "dcat:Dataset",
+            "attributes": {
+                "title": {
+                    "type": "string",
+                    "predicate": "dct:title"
+                },
+                "description": {
+                    "type": "string",
+                    "predicate": "dct:description"
+                },
+                "release-date": {
+                    "type": "datetime",
+                    "predicate": "dct:issued"
+                },
+                "modified": {
+                    "type": "datetime",
+                    "predicate": "dct:modified"
+                },
+                "type": {
+                    "type": "url",
+                    "predicate": "dct:type"
+                },
+                "subject": {
+                    "type": "url",
+                    "predicate": "dct:subject"
+                }
+            },
+            "relationships": {
+                "catalog": {
+                    "target": "catalogs",
+                    "predicate": "dcat:dataset",
+                    "cardinality": "one",
+                    "inverse": true
+                },
+                "distributions": {
+                    "target": "distributions",
+                    "predicate": "dcat:distribution",
+                    "cardinality": "many"
+                },
+                "previous-version": {
+                    "target": "datasets",
+                    "predicate": "prov:wasRevisionOf",
+                    "cardinality": "one"
+                },
+                "next-version": {
+                    "target": "datasets",
+                    "predicate": "prov:wasRevisionOf",
+                    "cardinality": "one",
+                    "inverse": true
+                }
+            },
+            "features": [
+                "include-uri"
+            ],
+            "new-resource-base": "http://data.lblod.info/id/dataset/"
+        },
+        "distributions": {
+            "class": "dcat:Distribution",
+            "attributes": {
+                "title": {
+                    "type": "string",
+                    "predicate": "dct:title"
+                },
+                "description": {
+                    "type": "string",
+                    "predicate": "dct:description"
+                },
+                "release-date": {
+                    "type": "datetime",
+                    "predicate": "dct:issued"
+                },
+                "modified": {
+                    "type": "datetime",
+                    "predicate": "dct:modified"
+                },
+                "type": {
+                    "type": "url",
+                    "predicate": "dct:type"
+                },
+                "download-url": {
+                    "type": "url",
+                    "predicate": "dcat:downloadURL"
+                },
+                "format": {
+                    "type": "string",
+                    "predicate": "dct:format"
+                },
+                "byte-size": {
+                    "type": "number",
+                    "predicate": "dcat:byteSize"
+                }
+            },
+            "relationships": {
+                "dataset": {
+                    "target": "datasets",
+                    "predicate": "dcat:distribution",
+                    "cardinality": "one",
+                    "inverse": true
+                },
+                "subject": {
+                    "target": "file",
+                    "predicate": "dct:subject",
+                    "cardinality": "one"
+                }
+            },
+            "features": [
+                "include-uri"
+            ],
+            "new-resource-base": "http://data.lblod.info/id/distribution/"
+        }
+    }
+}
--- a/config/resources/domain.json
+++ b/config/resources/domain.json
@@ -55,26 +55,6 @@
          "target": "agents-in-position",
          "cardinality": "many",
          "inverse": true
-        },
-        "registration": {
-          "predicate": "persoon:registratie",
-          "target": "identifiers",
-          "cardinality": "one"
-        },
-        "date-of-birth": {
-          "predicate": "persoon:heeftGeboorte",
-          "target": "dates-of-birth",
-          "cardinality": "one"
-        },
-        "nationalities": {
-          "predicate": "persoon:heeftNationaliteit",
-          "target": "nationalities",
-          "cardinality": "many"
-        },
-        "gender": {
-          "predicate": "persoon:geslacht",
-          "target": "gender-codes",
-          "cardinality": "one"
        }
      },
      "new-resource-base": "http://data.lblod.info/id/personen/"
@@ -429,11 +409,6 @@
          "cardinality": "many",
          "inverse": true
        },
-        "associated-structures": {
-          "predicate": "ere:heeftVerbondenEntiteit",
-          "target": "associated-legal-structures",
-          "cardinality": "many"
-        },
        "minister-positions": {
          "predicate": "ere:wordtBediendDoor",
          "target": "minister-positions",
@@ -616,6 +591,18 @@
        "full-address": {
          "type": "string",
          "predicate": "locn:fullAddress"
+        },
+        "address-register-uri": {
+          "type": "url",
+          "predicate": "adres:verwijstNaar"
+        }
+
+      },
+      "relationships": {
+        "source": {
+          "predicate": "dc_terms:source",
+          "target": "concepts",
+          "cardinality": "one"
        }
      },
      "new-resource-base": "http://data.lblod.info/id/adressen/"
@@ -684,6 +671,20 @@
      },
      "new-resource-base": "http://lblod.data.gift/concepts/"
    },
+    "concepts": {
+      "class": "skos:Concept",
+      "attributes": {
+        "label": {
+          "type": "string",
+          "predicate": "skos:prefLabel"
+        },
+        "alt-label": {
+          "type": "string",
+          "predicate": "skos:altLabel"
+        }
+      },
+      "new-resource-base": "http://lblod.data.gift/concepts/"
+    },
    "organization-status-codes": {
      "class": "code:OrganisatieStatusCode",
      "attributes": {
@@ -771,43 +772,6 @@
      },
      "new-resource-base": "http://data.lblod.info/id/voorwaardenBedienaar/"
    },
-    "associated-legal-structures": {
-      "class": "ere:VerbondenJuridischeStructuren",
-      "attributes": {
-        "name": {
-          "type": "string",
-          "predicate": "skos:prefLabel"
-        }
-      },
-      "relationships": {
-        "registration": {
-          "predicate": "regorg:registration",
-          "target": "identifiers",
-          "cardinality": "one"
-        },
-        "legal-type": {
-          "predicate": "organisatie:rechtsvorm",
-          "target": "legal-form-types",
-          "cardinality": "one"
-        },
-        "address": {
-          "predicate": "locn:address",
-          "target": "addresses",
-          "cardinality": "one"
-        }
-      },
-      "new-resource-base": "http://data.lblod.info/id/verbondenJuridischeStructuren/"
-    },
-    "dates-of-birth": {
-      "class": "persoon:Geboorte",
-      "attributes": {
-        "date": {
-          "type": "datetime",
-          "predicate": "persoon:datum"
-        } 
-      },
-      "new-resource-base": "http://data.lblod.info/id/geboortes/"
-    },
    "nationalities": {
      "class": "euvoc:Country",
      "attributes": {
@@ -891,16 +855,6 @@
      },
      "new-resource-base": "http://lblod.data.gift/concepts/"
    },
-    "legal-form-types": {
-      "class": "code:Rechtsvormtype",
-      "attributes": {
-        "label": {
-          "type": "string",
-          "predicate": "skos:prefLabel"
-        }
-      },
-      "new-resource-base": "http://lblod.data.gift/concepts/"
-    },
    "document-types-criterions": {
      "class": "code:BedienaarCriteriumBewijsstuk",
      "attributes": {
--- a/config/resources/domain.lisp
+++ b/config/resources/domain.lisp
@@ -1,4 +1,3 @@
-
 (in-package :mu-cl-resources)

 (defparameter *include-count-in-paginated-responses* t)
@@ -12,4 +11,7 @@
 (defparameter *supply-cache-headers-p* t)

 (read-domain-file "domain.json")
-(read-domain-file "auth.json")
+(read-domain-file "auth.json")
+(read-domain-file "files-domain.lisp")
+(read-domain-file "dcat.json")
+(read-domain-file "privacy-centric-service.json")
--- a/config/resources/files-domain.lisp
+++ b/config/resources/files-domain.lisp
@@ -0,0 +1,13 @@
+(define-resource file ()
+  :class (s-prefix "nfo:FileDataObject")
+  :properties `((:filename :string ,(s-prefix "nfo:fileName"))
+                (:format :string ,(s-prefix "dct:format"))
+                (:size :number ,(s-prefix "nfo:fileSize"))
+                (:extension :string ,(s-prefix "dbpedia:fileExtension"))
+                (:created :datetime ,(s-prefix "nfo:fileCreated")))
+  :has-one `((file :via ,(s-prefix "nie:dataSource")
+                   :inverse t
+                   :as "download"))
+  :resource-base (s-url "http://data.lblod.info/files/")
+  :features `(no-pagination-defaults include-uri)
+  :on-path "files")
--- a/config/resources/privacy-centric-service.json
+++ b/config/resources/privacy-centric-service.json
@@ -0,0 +1,20 @@
+{
+  "version": "0.1",
+  "prefixes": {
+    "skos": "http://www.w3.org/2004/02/skos/core#",
+    "ext": "http://mu.semte.ch/vocabularies/ext/"
+  },
+  "resources": {
+    "request-reasons": {
+      "name": "request-reason",
+      "class": "ext:ReasonCode",
+      "attributes": {
+        "label": {
+          "type": "string",
+          "predicate": "skos:prefLabel"
+        }
+      },
+      "new-resource-base": "http://data.lblod.info/id/information-request-reason/"
+    }
+  }
+}
--- a/docker-compose.dev.yml
+++ b/docker-compose.dev.yml
@@ -22,7 +22,7 @@ services:
  kalliope-api:
    environment:
      SPRING_SECURITY_CONFIG: "/config/security-dev.yml"
-      restart: "no"
+    restart: "no"
  deltanotifier:
    restart: "no"
  mocklogin:
@@ -33,7 +33,14 @@ services:
    restart: "no"
  login:
    restart: "no"
+  accountdetail:
+    restart: "no"
+  privacy:
+    restart: "no"
+  adressenregister:
+    restart: "no"
  frontend:
+    image: lblod/frontend-contact-hub:latest
    restart: "no"
  uri-info:
    restart: "no"
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -36,7 +36,7 @@ services:
    restart: always
    logging: *default-logging
  db:
-    image: semtech/mu-authorization:0.6.0-beta.7
+    image: cecemel/mu-authorization:0.6.0-beta.8
    environment:
      MU_SPARQL_ENDPOINT: "http://triplestore:8890/sparql"
      LOG_OUTGOING_SPARQL_QUERIES: "true"
@@ -79,20 +79,23 @@ services:
    restart: always
    logging: *default-logging
  kalliope-api:
-    image: lblod/jsonld-delta-service:0.3.2
+    image: lblod/jsonld-delta-service:0.5.4
    volumes:
      - ./config/kalliope:/config
      - ./data/files:/share
+    links:
+      - db:database
    environment:
      SERVER_PORT: "80"
      LOGGING_LEVEL: "INFO"
      SPARQL_ENDPOINT: "http://db:8890/sparql"
+      LIMIT_SIZE: "10000"
    labels:
      - "logging=true"
    restart: always
    logging: *default-logging
  deltanotifier:
-    image: semtech/mu-delta-notifier:0.1.0
+    image: cecemel/delta-notifier:0.2.0-beta.2
    volumes:
      - ./config/delta:/config
    labels:
@@ -108,17 +111,15 @@ services:
    restart: always
    logging: *default-logging
  login:
-    image: lblod/acmidm-login-service:0.9.3-organisatie_portaal
-    links:
-      - db:database
+    image: lblod/acmidm-login-service:0.9.2
    environment:
+      MU_APPLICATION_AUTH_USERID_CLAIM: "vo_id"
      MU_APPLICATION_GRAPH: "http://mu.semte.ch/graphs/contacthub/accounts"
+      MU_APPLICATION_AUTH_ROLE_CLAIM: "abb_organisatieportaal_rol_3d"
      MU_APPLICATION_AUTH_DISCOVERY_URL: "https://authenticatie-ti.vlaanderen.be/op"
      MU_APPLICATION_AUTH_CLIENT_ID: "677af572-2e2f-4e61-ad38-036723bb314c"
      MU_APPLICATION_AUTH_REDIRECT_URI: "https://organisaties.abb.lblod.info/authorization/callback"
-      MU_APPLICATION_AUTH_CLIENT_SECRET: ""
-    links:
-      - db:database
+      MU_APPLICATION_AUTH_CLIENT_SECRET: "secret"
    labels:
      - "logging=true"
    restart: always
@@ -136,14 +137,14 @@ services:
    restart: always
    logging: *default-logging
  privacy:
-    image: lblod/privacy-centric-service:0.0.1
+    image: lblod/privacy-centric-service:0.2.1
    labels:
      - "logging=true"
    environment:
      SERVER_PORT: "80"
      SPARQL_ENDPOINT: "http://db:8890/sparql"
      LOGGING_LEVEL: "INFO"
-      DEFAULT_GRAPH: "http://mu.semte.ch/graphs/contacthub/141d9d6b-54af-4d17-b313-8d1c30bc3f5b/ChAdmin"
+      DEFAULT_GRAPH: "http://mu.semte.ch/graphs/privacy-centric-graph"
      SESSION_GRAPH: "http://mu.semte.ch/graphs/sessions"
    labels:
      - "logging=true"
@@ -156,7 +157,7 @@ services:
    restart: always
    logging: *default-logging
  frontend:
-    image: lblod/frontend-contact-hub:1.1.0
+    image: lblod/frontend-contact-hub:1.4.0
    volumes:
      - ./config/frontend/add-x-frame-options-header.conf:/config/add-x-frame-options.conf
    environment:
@@ -177,3 +178,156 @@ services:
      - "logging=true"
    restart: always
    logging: *default-logging
+  file:
+    image: semtech/mu-file-service:3.2.0
+    volumes:
+      - ./data/files:/share
+    labels:
+      - "logging=true"
+    links:
+      - db:database
+    restart: always
+    logging: *default-logging
+  jobs-controller:
+    image: lblod/job-controller-service:0.7.0
+    links:
+      - db:database
+    volumes:
+      - ./config/jobs-controller/:/config/
+################################################################################
+# DELTA GENERAL
+################################################################################
+  delta-producer-dump-file-publisher:
+    image: lblod/delta-producer-dump-file-publisher:0.4.0
+    volumes:
+      - ./config/delta-producer/dump-file-publisher:/config
+      - ./data/files:/share
+    environment:
+      FILES_GRAPH: 'http://mu.semte.ch/graphs/public' # Note: this will be extracted in a config per job one day
+      DCAT_DATASET_GRAPH: 'http://mu.semte.ch/graphs/public'
+    links:
+      - db:database
+    restart: always
+    logging: *default-logging
+################################################################################
+# DELTA ADMINISTRATIVE-UNITS: START
+################################################################################
+  delta-producer-bg-jobs-initiator-administrative-units:
+    image: lblod/delta-producer-background-jobs-initiator:0.3.1
+    environment:
+      JOBS_GRAPH: "http://mu.semte.ch/graphs/organisatieportaal"
+      DUMP_FILE_CREATION_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/administrative-units'
+      INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units'
+      HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units'
+      CRON_PATTERN_DUMP_JOB: '0 0 4 * * 6' # weekly on saturday
+      CRON_PATTERN_HEALING_JOB: '0 0 4 * * *' # everyday at 4 AM
+      START_INITIAL_SYNC: 'false' # prefer to disable it now. We want to have feeling with how the DB acts.
+    labels:
+      - "logging=true"
+    links:
+      - db:database
+    restart: always
+    logging: *default-logging
+  delta-producer-pub-graph-maintainer-administrative-units:
+    image: lblod/delta-producer-publication-graph-maintainer:0.7.0
+    environment:
+      MAX_BODY_SIZE: "50mb"
+      JOBS_GRAPH: "http://mu.semte.ch/graphs/organisatieportaal"
+      RELATIVE_FILE_PATH: "deltas/administrative-units"
+      PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/administrative-units'
+      HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units'
+      INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units'
+      QUEUE_POLL_INTERVAL: 3000
+      USE_VIRTUOSO_FOR_EXPENSIVE_SELECTS: "true" # some queries are just too expensive
+      SKIP_MU_AUTH_INITIAL_SYNC: 'true'
+      HEALING_PATCH_GRAPH_BATCH_SIZE: 500
+    volumes:
+      - ./config/delta-producer/administrative-units:/config
+      - ./data/files/:/share
+    links:
+      - db:database
+      - triplestore:virtuoso
+    restart: always
+    logging: *default-logging
+  delta-producer-json-diff-publisher-administrative-units:
+    image: lblod/delta-producer-json-diff-file-publisher:0.2.3
+    environment:
+      MAX_BODY_SIZE: "50mb"
+      ERROR_GRAPH: "http://mu.semte.ch/graphs/organisatieportaal"
+      RELATIVE_FILE_PATH: "deltas/administrative-units"
+      PUBLISHER_URI: "http://data.lblod.info/services/delta-production-json-diff-file-manager/administrative-units"
+      PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/administrative-units'
+      DELTA_INTERVAL_MS: 10000
+      PRETTY_PRINT_DIFF_JSON: "true"
+    volumes:
+      - ./data/files:/share
+    labels:
+      - "logging=true"
+    links:
+      - db:database
+    restart: always
+    logging: *default-logging
+################################################################################
+# DELTA ADMINISTRATIVE-UNITS: END
+################################################################################
+################################################################################
+# DELTA ORGANIZATIONS: START
+################################################################################
+  delta-producer-bg-jobs-initiator-organizations:
+    image: lblod/delta-producer-background-jobs-initiator:0.3.1
+    environment:
+      JOBS_GRAPH: "http://mu.semte.ch/graphs/organisatieportaal"
+      DUMP_FILE_CREATION_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/deltaDumpFileCreation/organizations'
+      INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations'
+      HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations'
+      CRON_PATTERN_DUMP_JOB: '30 9 8-18/2 * * 1-5' # At second :30, at minute :09, every 2 hours between hours 08:00-18:00, on every Monday to Friday
+      CRON_PATTERN_HEALING_JOB: '0 0 4 * * *' # everyday at 4 AM
+      START_INITIAL_SYNC: 'false' # prefer to disable it now. We want to have feeling with how the DB acts.
+    labels:
+      - "logging=true"
+    links:
+      - db:database
+    restart: always
+    logging: *default-logging
+  delta-producer-pub-graph-maintainer-organizations:
+    image: lblod/delta-producer-publication-graph-maintainer:0.7.0
+    environment:
+      MAX_BODY_SIZE: "50mb"
+      JOBS_GRAPH: "http://mu.semte.ch/graphs/organisatieportaal"
+      RELATIVE_FILE_PATH: "deltas/organizations"
+      PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/organizations'
+      HEALING_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/organizations'
+      INITIAL_PUBLICATION_GRAPH_SYNC_JOB_OPERATION: 'http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/organizations'
+      QUEUE_POLL_INTERVAL: 3000
+      USE_VIRTUOSO_FOR_EXPENSIVE_SELECTS: "true" # some queries are just too expensive
+      SKIP_MU_AUTH_INITIAL_SYNC: 'true'
+      HEALING_PATCH_GRAPH_BATCH_SIZE: 500
+    volumes:
+      - ./config/delta-producer/organizations:/config
+      - ./data/files/:/share
+    links:
+      - db:database
+      - triplestore:virtuoso
+    restart: always
+    logging: *default-logging
+  delta-producer-json-diff-publisher-organizations:
+    image: lblod/delta-producer-json-diff-file-publisher:0.2.3
+    environment:
+      MAX_BODY_SIZE: "50mb"
+      ERROR_GRAPH: "http://mu.semte.ch/graphs/organisatieportaal"
+      RELATIVE_FILE_PATH: "deltas/organizations"
+      PUBLISHER_URI: "http://data.lblod.info/services/delta-production-json-diff-file-manager/organizations"
+      PUBLICATION_GRAPH: 'http://redpencil.data.gift/id/deltas/producer/organizations'
+      DELTA_INTERVAL_MS: 10000
+      PRETTY_PRINT_DIFF_JSON: "true"
+    volumes:
+      - ./data/files:/share
+    labels:
+      - "logging=true"
+    links:
+      - db:database
+    restart: always
+    logging: *default-logging
+################################################################################
+# DELTA ORGANIZATIONS: END
+################################################################################
Author	SHA1	Message	Date
Boris De Vloed	470b546d82	Merge branch 'development'	2021-12-16 18:04:04 +01:00
Boris De Vloed	f56d56d137	corrected cron schedule	2021-12-16 18:03:01 +01:00
Boris De Vloed	cfc4bb9c5d	Merge branch 'development'	2021-12-16 17:33:20 +01:00
Boris De Vloed	7d17f21ff1	Merge branch 'development'	2021-12-16 17:16:00 +01:00
Boris De Vloed	b41f35c90b	Merge pull request #84 from lblod/feature/bump-jsonld-api-2 bump jsonld api	2021-12-16 17:14:51 +01:00
Boris De Vloed	9b67988b76	bump jsonld api	2021-12-16 17:13:41 +01:00
Boris De Vloed	37bd122e60	Merge branch 'development'	2021-12-16 16:14:49 +01:00
Boris De Vloed	7438b38bae	bump jsonld api (#83 )	2021-12-16 16:11:41 +01:00
Boris De Vloed	78f1bd0233	Merge pull request #82 from lblod/chore/bump-log4j-spring bump privacy service & kalliope api	2021-12-15 11:41:30 +01:00
Nordine Bittich	e1f73b4179	bump privacy service & kalliope api	2021-12-15 10:33:59 +01:00
Nordine Bittich	421b5e20dd	Squashed commit of the following: commit `8b505d6fa8` Author: Nordine Bittich <nordine@bittich.be> Date: Tue Dec 14 17:32:38 2021 +0100 bump frontend	2021-12-14 17:34:12 +01:00
Nordine Bittich	8b505d6fa8	bump frontend	2021-12-14 17:32:38 +01:00
Boris De Vloed	59bdee66b1	removed unused delta rule (#81 )	2021-12-14 13:55:03 +01:00
Boris De Vloed	f556d22fa0	Merge pull request #80 from lblod/feature/bump-java-app bump java apps	2021-12-13 10:35:32 +01:00
Boris De Vloed	4fd77859ef	Merge branch 'development' into feature/bump-java-app	2021-12-13 10:34:48 +01:00
Nordine Bittich	ad05fff3ba	bump java app	2021-12-13 10:16:44 +01:00
Boris De Vloed	fed1fb8975	Merge pull request #79 from lblod/feature/use-ttl-dumps-for-api simplified cron config	2021-12-10 10:39:50 +01:00
Boris De Vloed	713ceb5f1a	simplified cron config	2021-12-10 10:38:22 +01:00
Boris De Vloed	7ff0480faf	Merge pull request #78 from lblod/feature/use-ttl-dumps-for-api Feature/use ttl dumps for api	2021-12-10 09:39:25 +01:00
Boris De Vloed	3930b87237	add config changes	2021-12-10 09:38:15 +01:00
Boris De Vloed	1008af8c08	Fix/type eredienst export (#77 ) * added typeEredienst * fix typo in graph name	2021-12-08 17:50:17 +01:00
Boris De Vloed	cbfe243b3e	fix typo in graph name	2021-12-08 16:53:39 +01:00
Boris De Vloed	834897c0c2	added typeEredienst	2021-12-08 15:24:46 +01:00
Boris De Vloed	f9b17ced4c	Merge pull request #76 from lblod/feature/acmidm-refactor acm idm roles	2021-12-06 11:05:49 +01:00
Nordine Bittich	a47c9c384d	order migrations	2021-12-06 10:54:17 +01:00
Nordine Bittich	9a74e1e1ba	bugfix	2021-12-03 14:21:23 +01:00
Nordine Bittich	4000c9a50f	changing graph	2021-12-03 11:32:03 +01:00
Nordine Bittich	9e63016553	change graph	2021-12-03 11:24:05 +01:00
Nordine Bittich	3541a5f66b	acm idm roles	2021-12-03 11:22:01 +01:00
cecemel	566ec39d9a	Feature/delta producer administrative units (#67 ) * Setting up file-service - pre-requisite to download dumps - pre-requiiste to download the json-diff files - security considerations: the content of the files created here, is - assumed to be public data, hence we allow it to be in the public graph * Bumping deltanotifier and mu-auth, which need to be able to work with mu-scop-id's * Delta-producer have multiple background-jobs with different tasks Hence the introduction of the jobs-controller. Note: in a more advanced configuration, you can easily extend the tasks related to one job. * A pre-requisite to serve dumps, is DCAT api support The delta-producers make a periodic dump of what is supposed to be published. They are published according to the DCAT data-model. Security-considerations: the DCAT producer by the delta-producer flow, are considered to be public data, hence added to the public graph. * Introduction of the delta-producer-dump-file-publisher Service responsible for creating dumps for the (potentially many) data delta flows. Here only for the administrative-units, but this can be extended in the future, in the config. * Introduction of delta-producer-background-jobs-initiator This service is responsible for triggering the background jobs related to the delta-productoin of the administrative-units. It trigger currently three jobs: - INITIAL_SYNC: this is the first job to fill the publication graph. In theory, this is only run once per deploy - HEALING_JOB: This is a periodic job, which checks what information is published vs. what information should have been published (according to a configuration). If differences are found, because delta-messages never arrived, they are compensated here/ - DUMP_JOB: A periodic job to create a dump of the publication graph. This is especially useful for consumers, who don't have to replay the whole delta-diffs, but just can start from the latest sync point. * Introduction of delta-producer-publication-graph-maintainer This service is the core of the logic which determines what should be published. The configuration is provided here, and can be elaborated refined in the config. It also excutes the healing logic. * Introduction of delta-producer-json-diff-file-publisher This simple service, listens to the updates performed on the publication graph, and serialized these to json-files which are compatible with current consumers * Missing dispatcher rules for DCAT * Various small things: - outline, whitespaces and comments * Update README.md * bump delta-producer-publication-graph-maintainer * Add organizations (KAL) - Work in progress * added organizations graph maintainer * removed inverses, added all bestuur types * refactored kalliope api Co-authored-by: Boris De Vloed <boris.de.vloed@gmail.com>	2021-12-03 11:20:32 +01:00
Nordine Bittich	5b48852463	Bump frontend (#74 )	2021-11-26 14:58:09 +01:00
Nordine Bittich	a44eab2e45	bump privacy service (#73 )	2021-11-25 15:26:11 +01:00
Nordine Bittich	819c434f3d	bump frontend (#72 )	2021-11-25 13:49:52 +01:00
Sam Van Campenhout	b5a7351189	Merge pull request #71 from lblod/chore/remove-privacy-centric-config Remove config related to the sensitive data feature	2021-11-23 16:38:55 +01:00
Nordine Bittich	3dc9b1a0b4	sparql endpoint for prod	2021-11-23 16:37:18 +01:00
Sam Van Campenhout	929dc59ee5	Remove config related to the sensitive data feature We no longer use mu-cl-resources to manage this data so we can remove it from the configuration.	2021-11-23 16:33:40 +01:00
Sam Van Campenhout	b6e1458d8a	Add a new `/request-reasons` endpoint (#70 ) This can be used to retrieve all the possible options.	2021-11-23 15:45:42 +01:00
Nordine Bittich	8c7ba7afec	migrate private data (#69 ) * migrate private data * bind subject before optional Co-authored-by: Boris De Vloed <boris.de.vloed@gmail.com>	2021-11-23 10:36:41 +01:00
Nordine Bittich	545622b972	bump frontend	2021-11-18 16:52:37 +01:00
Sam Van Campenhout	13e7337574	Merge pull request #68 from lblod/feature/only-allow-sparql-endpoint-to-acc-dev sparql endpoint only on dev/acc environment	2021-11-18 15:54:00 +01:00
Nordine Bittich	48810caf68	fix typo	2021-11-18 14:37:55 +01:00
Nordine Bittich	fe7c0d2c0f	only allow sparql endpoint to acc/dev	2021-11-18 14:32:21 +01:00
Boris De Vloed	2c26b0d25c	Merge pull request #66 from lblod/chore/tweak-database Added two tweaked database settings	2021-11-17 08:26:20 +01:00
felix	97cc023be5	Added two tweaked database settings Note: virtuoso-production.ini might currently be overkill. Could consider not using it.	2021-11-16 16:27:42 +01:00
Boris De Vloed	8d58d87a54	only send deltas for app graph (#64 )	2021-11-15 11:26:05 +01:00
Sam Van Campenhout	56c0583c62	Prevent services from restarting in development (#63 )	2021-11-12 09:47:00 +01:00
Sam Van Campenhout	ad919b1f0e	Merge pull request #60 from lblod/feature/crab-changes changes for crab	2021-11-10 16:38:52 +01:00
Nordine Bittich	abc5d9256c	fix dispatcher	2021-11-10 14:56:28 +01:00
Nordine Bittich	fb2fa6242f	fix dispatcher	2021-11-10 14:51:56 +01:00
Boris De Vloed	9e5d45adb9	Merge pull request #62 from lblod/feature/yasgui Feature/yasgui	2021-11-10 14:23:59 +01:00
Nordine Bittich	c716667801	change graph for privacy	2021-11-10 11:34:43 +01:00
Nordine Bittich	ed86245a2d	bugfix	2021-11-09 16:07:41 +01:00
Nordine Bittich	567740ae86	yasgui	2021-11-09 15:43:09 +01:00
Nordine Bittich	0bf3dca9bf	remove basic auth from readme (deprecated)	2021-11-08 18:53:23 +01:00
Nordine Bittich	45a5dff52a	fix file extension	2021-11-08 14:33:24 +01:00
Nordine Bittich	e6a167c402	ensure source crab migration will run	2021-11-05 10:40:38 +01:00
Nordine Bittich	670c1591c6	bugfix	2021-11-05 10:31:08 +01:00
Nordine Bittich	478affc609	bugfix	2021-11-05 10:29:14 +01:00
Sam Van Campenhout	65f9ca1558	Merge pull request #61 from lblod/chore/remove-connected-legal-structures Remove all code related to "verbonden juridische structuren"	2021-11-05 10:07:01 +01:00
Sam Van Campenhout	60ee996e36	Remove all code related to "verbonden juridische structuren" This functionality is no longer needed.	2021-11-03 12:59:18 +01:00
Nordine Bittich	52bc6e5d60	changes for crab	2021-11-03 10:48:12 +01:00