felix.ruizdearcaute/app-organization-portal
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Set up the jobs dashboard

Browse Source 
Based on the same setup in the worship decisions app: https://github.com/lblod/app-worship-decisions-database/pull/29
This commit is contained in:
Sam Van Campenhout
2025-03-12 10:51:52 +01:00
parent 2c729ddee3
commit b97b545add
11 changed files with 435 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
CHANGELOG.md
View File

@@ -7,6 +7,7 @@
- Bump `triplestore` and `publication-triplestore` to `v1.3.0-rc.1`. [OP-2492] [OP-3547]
- Add missing docker compose keys. [DL-6490]
- Reorganize delta consumers config to harmonize with the ecosystem
- Set up the dashboard app [OP-3103]
### Deploy notes
```
drc restart migrations; drc logs -ft --tail=200 migrations
@@ -20,6 +21,26 @@ drc up -d mandatarissen-consumer leidinggevenden-consumer worship-services-main-
drc up -d search elasticsearch
```
#### Dashboard
The dashboard requires some server specific configuration before it can be used.
##### 1. Dashboard domains
To configure the domain name that's used by the dashboard frontend, you can add it to the `VIRTUAL_HOST` and `LETSENCRYPT_HOST` environment variables of the identifier service.
- DEV domain: dev.dashboard.app-organisation-portal.contacthub-dev.s.redhost.be
- QA domain: qa.dashboard.app-organisation-portal.contacthub-dev.s.redhost.be
- PROD domain: dashboard.app-organisation-portal.organisaties.s.redpencil.io
##### 2. Dashboard users
Before we can generate a user we need to add an application salt to the server. Add a `MU_APPLICATION_SALT` environment variable to the `dashboard-login` service in the docker-compose.override.yml file.
Once this is done you can generate a user by running `mu script project-scripts generate-dashboard-login` and following the prompts. A new local migration will be generated which can be run by restarting the migrations service and which will insert the new user into the database.
Store the login credentials as a comment in the docker-compose.override.yml file.
##### 3. (Re)start the needed services
`drc up -d frontend-dashboard dashboard-login identifier; drc restart dispatcher resource db`
## v1.30.5 (2025-03-10)
### Backend
- Added `http://www.w3.org/ns/prov#Location` to delta stream 'public' [DL-6496]

23
README.md
View File

@@ -185,3 +185,26 @@ To make sure the app can share data, producers need to be set up. There is an in
)
}
```
## Configuring the dashboard
### Accessing the dashboard from your local machine
Since we use dispatcher v2, which dispatches on hostname, we'll have to update `/etc/hosts`.
Add an entry similar to the following. Ensure the first part of the domain starts with `dashboard`.:
```
127.0.0.1 dashboard.localhost
```
### Creating new users
For now, we use specific logins for the dashboard users. Each environment has its own passwords.
To add a user, make sure to have [mu-cli](https://github.com/mu-semtech/mu-cli) installed first.
Then in `docker-compose.override.yml`
```
dashboard-login:
environment:
MU_APPLICATION_SALT: 'a_random_string_with_sufficient_entropy_hence_not_this_one'
```
You can now generate a user by running `mu script project-scripts generate-dashboard-login` and following the steps.
Restart `migrations` and it should insert the new user into the database.

53
config/authorization/config.ex
View File

@@ -107,6 +107,19 @@ defmodule Acl.UserGroups.Config do
}"
end
defp can_access_dashboard() do
%AccessByQuery{
vars: [],
query: "PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
SELECT DISTINCT ?account WHERE {
<SESSION_ID> <http://mu.semte.ch/vocabularies/session/account> ?account.
?account <http://mu.semte.ch/vocabularies/ext/sessionRole> ?session_role.
FILTER( ?session_role = \"dashboard-user\" )
}"
}
end
def user_groups do
[
%GroupSpec{
@@ -206,6 +219,46 @@ defmodule Acl.UserGroups.Config do
}
]
},
# // dashboard users
%GroupSpec{
name: "dashboard-users",
useage: [:read],
access: can_access_dashboard(),
graphs: [
%GraphSpec{
graph: "http://mu.semte.ch/graphs/reports",
constraint: %ResourceConstraint{
resource_types: [
"http://lblod.data.gift/vocabularies/reporting/Report",
"http://open-services.net/ns/core#Error",
"http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#DataContainer",
"http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject"
]
}
},
%GraphSpec{
graph: "http://mu.semte.ch/graphs/system/jobs",
constraint: %ResourceConstraint{
resource_types: [
"http://vocab.deri.ie/cogs#Job",
"http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#DataContainer",
"http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject"
]
}
},
# OP stores jobs in multiple graphs, so we configure both until they are merged into one graph
%GraphSpec{
graph: "http://mu.semte.ch/graphs/jobs",
constraint: %ResourceConstraint{
resource_types: [
"http://vocab.deri.ie/cogs#Job",
"http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#DataContainer",
"http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#FileDataObject"
]
}
},
]
},
# // CLEANUP
#

39
config/dispatcher/dispatcher.ex
View File

@@ -240,6 +240,27 @@ defmodule Dispatcher do
Proxy.forward(conn, path, "http://resource/kbo-organizations/")
end
# Dashboard
match "/jobs/*path", %{accept: [:json], layer: :api} do
Proxy.forward(conn, path, "http://cache/jobs/")
end
match "/tasks/*path", %{accept: [:json], layer: :api} do
Proxy.forward(conn, path, "http://cache/tasks/")
end
match "/data-containers/*path", %{accept: [:json], layer: :api} do
Proxy.forward(conn, path, "http://cache/data-containers/")
end
match "/job-errors/*path", %{accept: [:json], layer: :api} do
Proxy.forward(conn, path, "http://cache/job-errors/")
end
match "/reports/*path", %{accept: [:json], layer: :api} do
Proxy.forward(conn, path, "http://resource/reports/")
end
###############
# LOGIN
###############
@@ -264,6 +285,10 @@ defmodule Dispatcher do
Proxy.forward(conn, path, "http://mocklogin/sessions/")
end
match "/sessions/*path", %{ reverse_host: ["dashboard" | _rest] } do
Proxy.forward conn, path, "http://dashboard-login/sessions/"
end
match "/sessions/*path", %{accept: [:any], layer: :api} do
Proxy.forward(conn, path, "http://login/sessions/")
end
@@ -428,6 +453,20 @@ defmodule Dispatcher do
# frontend layer
###############################################################
get "/assets/*path", %{ reverse_host: ["dashboard" | _rest], layer: :api } do
forward conn, path, "http://frontend-dashboard/assets/"
end
get "/@appuniversum/*path", %{ reverse_host: ["dashboard" | _rest], layer: :api} do
forward conn, path, "http://frontend-dashboard/@appuniversum/"
end
match "/*_path", %{ reverse_host: ["dashboard" | _rest], layer: :api } do
# *_path allows a path to be supplied, but will not yield
# an error that we don't use the path variable.
forward conn, [], "http://frontend-dashboard/index.html"
end
match "/assets/*path", %{layer: :api} do
Proxy.forward(conn, path, "http://frontend/assets/")
end

2
config/resources/domain.lisp
View File

@@ -20,3 +20,5 @@
(read-domain-file "files-domain.lisp")
(read-domain-file "dcat.json")
(read-domain-file "privacy-centric-service.json")
(read-domain-file "job.json")
(read-domain-file "report.json")

5
config/resources/files-domain.lisp
View File

@@ -7,7 +7,10 @@
(:created :datetime ,(s-prefix "nfo:fileCreated")))
:has-one `((file :via ,(s-prefix "nie:dataSource")
:inverse t
:as "download"))
:as "download")
(data-container :via ,(s-prefix "task:hasFile")
:inverse t
:as "data-container"))
:resource-base (s-url "http://data.lblod.info/files/")
:features `(no-pagination-defaults include-uri)
:on-path "files")

147
config/resources/job.json Normal file
View File

@@ -0,0 +1,147 @@
{
"version": "0.1",
"prefixes": {
"adms": "http://www.w3.org/ns/adms#",
"cogs": "http://vocab.deri.ie/cogs#",
"dct": "http://purl.org/dc/terms/",
"oslc": "http://open-services.net/ns/core#",
"nfo": "http://www.semanticdesktop.org/ontologies/2007/03/22/nfo#",
"task": "http://redpencil.data.gift/vocabularies/tasks/"
},
"resources": {
"jobs": {
"name": "job",
"class": "cogs:Job",
"attributes": {
"created": {
"type": "datetime",
"predicate": "dct:created"
},
"modified": {
"type": "datetime",
"predicate": "dct:modified"
},
"creator": {
"type": "url",
"predicate": "dct:creator"
},
"status": {
"type": "url",
"predicate": "adms:status"
},
"operation": {
"type": "url",
"predicate": "task:operation"
}
},
"relationships": {
"error": {
"predicate": "task:error",
"target": "job-error",
"cardinality": "one"
},
"tasks": {
"predicate": "dct:isPartOf",
"target": "group",
"cardinality": "many",
"inverse": true
}
},
"features": ["include-uri"],
"new-resource-base": "http://redpencil.data.gift/id/job/"
},
"tasks": {
"name": "task",
"class": "task:Task",
"attributes": {
"created": {
"type": "datetime",
"predicate": "dct:created"
},
"modified": {
"type": "datetime",
"predicate": "dct:modified"
},
"status": {
"type": "url",
"predicate": "adms:status"
},
"operation": {
"type": "url",
"predicate": "task:operation"
},
"index": {
"type": "string",
"predicate": "task:index"
}
},
"relationships": {
"error": {
"predicate": "task:error",
"target": "job-error",
"cardinality": "one"
},
"job": {
"predicate": "dct:isPartOf",
"target": "job",
"cardinality": "one"
},
"parent-tasks": {
"predicate": "cogs:dependsOn",
"target": "task",
"cardinality": "many"
},
"results-containers": {
"predicate": "task:resultsContainer",
"target": "data-container",
"cardinality": "many"
},
"input-containers": {
"predicate": "task:inputContainer",
"target": "data-container",
"cardinality": "many"
}
},
"features": ["include-uri"],
"new-resource-base": "http://redpencil.data.gift/id/task/"
},
"job-errors": {
"name": "job-error",
"class": "oslc:Error",
"attributes": {
"message": {
"type": "string",
"predicate": "oslc:message"
}
},
"features": ["include-uri"],
"new-resource-base": "http://redpencil.data.gift/id/jobs/error/"
},
"data-containers": {
"name": "data-container",
"class": "nfo:DataContainer",
"attributes": {
"has-graph": {
"type": "url",
"predicate": "task:hasGraph"
}
},
"relationships": {
"result-from-tasks": {
"predicate": "task:resultsContainer",
"target": "task",
"cardinality": "many",
"inverse": true
},
"input-from-tasks": {
"predicate": "task:inputContainer",
"target": "task",
"cardinality": "many",
"inverse":true
}
},
"features": ["include-uri"],
"new-resource-base": "http://redpencil.data.gift/id/dataContainers/"
}
}
}

36
config/resources/report.json Normal file
View File

@@ -0,0 +1,36 @@
{
"version": "0.1",
"prefixes": {
"dct": "http://purl.org/dc/terms/",
"prov": "http://www.w3.org/ns/prov#"
},
"resources": {
"reports": {
"name": "report",
"class": "http://lblod.data.gift/vocabularies/reporting/Report",
"attributes": {
"created": {
"type": "datetime",
"predicate": "dct:created"
},
"title": {
"type": "string",
"predicate": "dct:title"
},
"description": {
"type": "string",
"predicate": "dct:description"
}
},
"relationships": {
"file": {
"predicate": "prov:generated",
"target": "file",
"cardinality": "one"
}
},
"features": ["include-uri"],
"new-resource-base": "http://data.lblod.info/id/reports/"
}
}
}

14
docker-compose.yml
View File

@@ -147,6 +147,14 @@ services:
logging: *default-logging
links:
- db:database
dashboard-login:
image: semtech/mu-login-service:3.0.0
links:
- db:database
labels:
- "logging=true"
restart: always
logging: *default-logging
accountdetail:
image: lblod/account-detail-service:0.1.0
labels:
@@ -202,6 +210,12 @@ services:
- "logging=true"
restart: always
logging: *default-logging
frontend-dashboard:
image: lblod/frontend-dashboard:1.7.0
labels:
- "logging=true"
restart: always
logging: *default-logging
uri-info:
image: redpencil/mu-uri-info-service:0.2.1
links:

17
scripts/project/config.json
View File

@@ -16,6 +16,21 @@
"mounts": {
"app": "/data/app/"
}
},
{
"documentation": {
"command": "generate-dashboard-login",
"description": "generate a login account for the jobs dashboard",
"arguments": []
},
"environment": {
"image": "ruby",
"interactive": true,
"script": "generate-dashboard-login/run.sh"
},
"mounts": {
"app": "/project/"
}
}
]
}
}

80
scripts/project/generate-dashboard-login/run.sh Executable file
View File

@@ -0,0 +1,80 @@
#!/usr/bin/env ruby
require 'bundler/inline'
require 'yaml'
require 'securerandom'
require 'date'
$stdout.sync = true
print "installing dependencies..."
gemfile do
source 'https://rubygems.org'
gem 'bcrypt'
end
puts "done"
salt = ""
begin
File.open("/project/docker-compose.override.yml") do |file|
config = YAML.load(file)
salt = config.dig("services", "dashboard-login", "environment", "MU_APPLICATION_SALT")
throw "invalid salt" unless salt
end
rescue
puts "Tried to locate login salt in docker-compose.override.yml, but couldn't find it."
puts "Please make sure a salt is configured:"
puts <<EOF
login:
restart: "no"
environment:
MU_APPLICATION_SALT: "#{SecureRandom.hex}"
EOF
end
puts "this script will generate a new user account"
print "username: "
STDOUT.flush
account_name = STDIN.gets.chomp
print "password: "
STDOUT.flush
password = STDIN.gets.chomp
account_salt = SecureRandom.hex
person_uuid = SecureRandom.uuid
account_uuid = SecureRandom.uuid
domain = "data.gift"
hashed_password = BCrypt::Password.create(password + salt + account_salt)
now = DateTime.now.xmlschema
query = %(
PREFIX foaf: <http://xmlns.com/foaf/0.1/>
PREFIX people: <http://#{domain}/people/>
PREFIX accounts: <http://#{domain}/accounts/>
PREFIX dcterms: <http://purl.org/dc/terms/>
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX ext: <http://mu.semte.ch/vocabularies/ext/>
PREFIX account: <http://mu.semte.ch/vocabularies/account/>
PREFIX xsd: <http://www.w3.org/2001/XMLSchema#>
INSERT DATA {
GRAPH <http://mu.semte.ch/application> {
people:#{person_uuid} a foaf:Person ;
foaf:name "#{account_name}";
foaf:account accounts:#{account_uuid};
mu:uuid "#{person_uuid}";
dcterms:created "#{now}"^^xsd:datetime;
dcterms:modified "#{now}"^^xsd:datetime.
accounts:#{account_uuid} a foaf:OnlineAccount;
foaf:accountName "#{account_name}";
mu:uuid "#{account_uuid}";
account:password """#{hashed_password}""";
account:salt "#{account_salt}";
account:status <http://mu.semte.ch/vocabularies/account/status/active>;
dcterms:created "#{now}"^^xsd:datetime;
ext:sessionRole "dashboard-user";
dcterms:modified "#{now}"^^xsd:datetime.
}}
)
timestamp = DateTime.now.strftime("%Y%m%d%H%M%L")
filename = "#{timestamp}-create-user-#{account_name.gsub(" ", "-")}.sparql"
File.write("/project/config/migrations/local/#{filename}", query)
puts "generated migration ./config/migrations/local/#{filename} for #{account_name.inspect} : #{password.inspect}"