242 lines
8.4 KiB
Markdown
242 lines
8.4 KiB
Markdown
# Organization portal
|
|
|
|
Backend for the organization portal application, based on the mu.semte.ch microservices stack.
|
|
|
|
### Boot up the system in DEV environment
|
|
|
|
cd /path/to/app-organization-portal
|
|
docker compose -f docker-compose.yml -f docker-compose.dev.yml up
|
|
|
|
You can shut down using `docker compose stop` and remove everything using `docker compose rm`.
|
|
|
|
You need to wait until the migrations are ready.
|
|
|
|
After that, you need to trigger the ES index manually.
|
|
|
|
You can do it by:
|
|
```
|
|
# Assumes you are in root of app-organization-portal
|
|
/bin/bash scripts/reset-elastic.sh
|
|
```
|
|
And follow the steps. It takes a bit, from an install from scratch, about 10 mins. (Mileage may vary)
|
|
|
|
|
|
#### Tricks to make it a bit easier.
|
|
If you think typing the full command takes too long every time, run this in your shell:
|
|
|
|
```
|
|
# Assumes you are in root of app-organization-portal
|
|
echo "COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml:docker-compose.override.yml" > .env
|
|
touch docker-compose.override.yml
|
|
```
|
|
Now you can just run `docker compose up` and it will take the config files into account.
|
|
If you want to go a step further, and still think (like most of us) that `docker compose` is too long to type, you can alias it:
|
|
```
|
|
echo "alias drc='docker compose'" >> ~/.bashrc
|
|
echo "alias drcl='docker compose logs -ft --tail=100'" >> ~/.bashrc
|
|
```
|
|
Now everywhere `drc up -d` will work as shorthand for `docker compose up -d`.
|
|
|
|
#### Setup consumers
|
|
|
|
The application depends on external data. You'll have to perform extra steps if you want this data to be available.
|
|
These are multiple data sets, we'll try to be exhaustive on how to ingest these. Even though the procedures are similar.
|
|
|
|
#### Setup mandatarissen sync
|
|
Note: this is assuming you have never ran this before.
|
|
```
|
|
drc down;
|
|
```
|
|
Update `docker-compose.override.yml` to:
|
|
```
|
|
mandatarissen-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
|
|
DCR_LANDING_ZONE_DATABASE: "triplestore"
|
|
DCR_REMAPPING_DATABASE: "triplestore"
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
```
|
|
Then:
|
|
```
|
|
drc up -d migrations
|
|
drc up -d db mandatarissen-consumer
|
|
# Wait until success of the previous step
|
|
```
|
|
Then, update `docker-compose.override.yml` to:
|
|
```
|
|
mandatarissen-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
|
|
DCR_LANDING_ZONE_DATABASE: "db"
|
|
DCR_REMAPPING_DATABASE: "db"
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
```
|
|
Finally, put the stack back up and reindex elasticsearch
|
|
```
|
|
drc up -d
|
|
sh scripts/reset-elastic.sh
|
|
```
|
|
|
|
#### Setup leidinggevenden sync (http://data.lblod.info/vocabularies/leidinggevenden/Functionaris)
|
|
|
|
Note: this is assuming you have never ran this before.
|
|
```
|
|
drc down;
|
|
```
|
|
Update `docker-compose.override.yml` to:
|
|
```
|
|
leidinggevenden-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
|
|
DCR_LANDING_ZONE_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
|
|
DCR_REMAPPING_DATABASE: "triplestore" # for the initial sync, we go directly to virtuoso
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
```
|
|
Then:
|
|
```
|
|
drc up -d migrations
|
|
drc up -d db leidinggevenden-consumer
|
|
# Wait until success of the previous step
|
|
```
|
|
Then, update `docker-compose.override.yml` to:
|
|
```
|
|
leidinggevenden-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # choose the correct endpoint
|
|
DCR_LANDING_ZONE_DATABASE: "db"
|
|
DCR_REMAPPING_DATABASE: "db"
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
```
|
|
Finally, put the stack back up and reindex elasticsearch
|
|
```
|
|
drc up -d
|
|
sh scripts/reset-elastic.sh
|
|
```
|
|
|
|
#### Setup worship mandatees sync
|
|
|
|
Note: this is assuming you have never ran this before.
|
|
```
|
|
drc down;
|
|
```
|
|
Update `docker-compose.override.yml` to:
|
|
```
|
|
worship-services-main-info-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
|
|
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
|
|
DCR_SECRET_KEY: "key-of-the-producer"
|
|
DCR_LANDING_ZONE_DATABASE: "triplestore"
|
|
DCR_REMAPPING_DATABASE: "triplestore"
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
worship-services-private-info-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
|
|
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
|
|
DCR_SECRET_KEY: "key-of-the-producer"
|
|
DCR_LANDING_ZONE_DATABASE: "triplestore"
|
|
DCR_REMAPPING_DATABASE: "triplestore"
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
```
|
|
Then:
|
|
```
|
|
drc up -d migrations
|
|
drc up -d db worship-services-main-info-consumer worship-services-private-info-consumer
|
|
# Wait until success of the previous step
|
|
```
|
|
Then, update `docker-compose.override.yml` to:
|
|
```
|
|
worship-services-main-info-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
|
|
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
|
|
DCR_SECRET_KEY: "key-of-the-producer"
|
|
DCR_LANDING_ZONE_DATABASE: "db"
|
|
DCR_REMAPPING_DATABASE: "db"
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
worship-services-private-info-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://loket.lokaalbestuur.vlaanderen.be" # or another endpoint
|
|
DCR_SYNC_LOGIN_ENDPOINT: "https://loket.lokaalbestuur.vlaanderen.be/sync/worship-services-sensitive/login" # or another endpoint
|
|
DCR_SECRET_KEY: "key-of-the-producer"
|
|
DCR_LANDING_ZONE_DATABASE: "db"
|
|
DCR_REMAPPING_DATABASE: "db"
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
```
|
|
Finally, put the stack back up and reindex elasticsearch
|
|
```
|
|
drc up -d
|
|
sh scripts/reset-elastic.sh
|
|
```
|
|
|
|
#### Setup vendors sync
|
|
Note: this is assuming you have never ran this before.
|
|
```
|
|
drc down;
|
|
```
|
|
Update `docker-compose.override.yml` to:
|
|
```
|
|
vendor-management-consumer:
|
|
environment:
|
|
DCR_SYNC_BASE_URL: "https://dev.loket.lblod.info/" # or another endpoint
|
|
BATCH_SIZE: "500"
|
|
DCR_SECRET_KEY: "<key>>" # fill in key for selected endpoint
|
|
DCR_SYNC_LOGIN_ENDPOINT: "https://dev.loket.lblod.info/sync/vendor-management/login" # login for selected endpoint
|
|
DCR_DISABLE_DELTA_INGEST: "false"
|
|
DCR_DISABLE_INITIAL_SYNC: "false"
|
|
```
|
|
Then:
|
|
```
|
|
drc up -d migrations
|
|
drc up -d db vendor-management-consumer
|
|
# Wait until success of the previous step
|
|
|
|
Finally, put the stack back up and reindex elasticsearch:
|
|
```
|
|
drc up -d
|
|
sh scripts/reset-elastic.sh
|
|
```
|
|
|
|
### Setting up the delta-producers related services
|
|
To make sure the app can share data, producers need to be set up. There is an intial sync, that is potentially very expensive, and must be started manually
|
|
|
|
#### producers administrative-units
|
|
|
|
1. make sure the app is up and running, the migrations have run
|
|
2. in docker-compose.override.yml, make sure the following configuration is provided:
|
|
```
|
|
delta-producer-pub-graph-maintainer-administrative-units:
|
|
environment:
|
|
START_INITIAL_SYNC: 'true'
|
|
```
|
|
3. `drc up -d delta-producer-pub-graph-maintainer-administrative-units`
|
|
4. You can follow the status of the job, ideally through the dashboard, but this hasn't been setup yet. The following query should also give results:
|
|
```
|
|
PREFIX task: <http://redpencil.data.gift/vocabularies/tasks/>
|
|
PREFIX adms: <http://www.w3.org/ns/adms#>
|
|
PREFIX dct: <http://purl.org/dc/terms/>
|
|
|
|
SELECT DISTINCT ?job ?operation ?created ?modified ?status WHERE {
|
|
GRAPH ?g {
|
|
?job a <http://vocab.deri.ie/cogs#Job>;
|
|
task:operation ?operation;
|
|
adms:status ?status;
|
|
dct:created ?created;
|
|
dct:modified ?modified.
|
|
}
|
|
FILTER( ?operation IN (
|
|
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/initialPublicationGraphSyncing/administrative-units>,
|
|
<http://redpencil.data.gift/id/jobs/concept/JobOperation/deltas/healingOperation/administrative-units>
|
|
)
|
|
)
|
|
}
|
|
```
|