Operational runbook

The practical admin and debugging tasks around Atrium Actions and the runtime.

Endpoints that matter

Operational env vars

Set on the atrium Dokploy app (the web container). Sensitive values are NOT in this doc — see ~/.config/atrium/ and ~/.config/dokploy/secrets/ on the operator's box.

Before debugging a script

Check these in order:

1. Is the script ACTIVE?
2. Does another ACTIVE script already own the same exact trigger slot?
3. Does the matching ZITADEL Execution actually point at the intended Target?
4. Does the runtime list the script in /internal/scripts?
5. Do recent payloads exist for the exact trigger name you chose?

Slot conflicts

Atrium only routes one ACTIVE script per exact trigger slot.

Examples:

• FUNCTION:preuserinfo
• FUNCTION:preaccesstoken
• EVENT:user.human.added

If activation fails with a slot-ownership error, either:

• disable the current live owner
• or compose the missing behaviour into the current owner

Debug a script that "does nothing"

1. Open /admin/actions
2. Open the script
3. Check Recent runs
4. Check Recorded payloads
5. Compare the script's exact trigger to the payload's exact trigger

Most common causes:

• wrong trigger name (session.added vs oidc_session.added)
• claim logic placed on an Event instead of a Function
• expecting custom metadata in a session event payload

Test-run safety

Test-run is not a fully fake environment.

What stays local:

• the displayed return value
• the captured ctx.log output

What can still be real:

• ctx.fetch(...)
• ctx.atrium.groups.*
• ctx.zitadel.users.*

Use:

• placeholder secrets by default
• sample users
• guard clauses
• or deliberately safe payloads

when iterating on mutating scripts.

Recorded payloads

Recent payloads are stored as:

• structure-preserving
• sanitized examples
• last 10 per exact trigger

They are useful for:

• matching the real shape
• seeing which events actually occur in OMMAX
• seeding test payloads without copying raw production bodies

Add a new ZITADEL Target

TOKEN="atr_live_..."

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://atrium.ommax-intelligence.de/api/admin/zitadel/targets \
  -d '{
    "name": "atrium-actions-functions-preaccesstoken",
    "endpoint": "https://atrium.ommax-intelligence.de/actions/functions/preaccesstoken",
    "interruptOnError": false
  }'

Persist the returned signing key immediately.

Wire or replace an Execution

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://atrium.ommax-intelligence.de/api/admin/zitadel/executions \
  -d '{
    "condition": { "function": { "name": "preuserinfo" } },
    "targetIds": ["<target-id>"]
  }'

To remove the routing, pass:

{ "targetIds": [] }

Rotate a Target signing key

curl -X POST \
  -H "Authorization: Bearer $TOKEN" \
  https://atrium.ommax-intelligence.de/api/admin/zitadel/targets/<target-id>/rotate-key

Then:

• add the new key to ZITADEL_WEBHOOK_SIGNING_KEYS
• keep old + new during the cutover window
• redeploy the runtime
• remove the old key after traffic is clean

Push a schema change

1. Edit schema.prisma
2. Run cd src && npx prisma generate
3. Commit and deploy
4. Let container startup apply the additive db push

For destructive changes, do a real migration instead of relying on startup push.

Verify the running deploy

Don't trust Dokploy's "status=done" alone — orchestrator success ≠ container actually serving the new build. Instead probe the app:

curl -sS https://atrium.ommax-intelligence.de/api/health | jq
# {
#   "status": "ok",
#   "parentSubject": "feat(...): ...",
#   "parentSha": "abc1234",          ← parent of the deployed commit
#   "commitTime": "2026-...",        ← deployed commit's authoring time
#   ...
# }

To poll for a specific commit landing, watch for commitTime matching that commit's authoring time, OR the next commit landing on top (then parentSha becomes the one you pushed).

Email notifications

Email kinds, all routed through sendEmail (src/lib/email/send.ts), audited in AtriumEmailLog:

Debugging an email that didn't arrive:

SELECT kind, "toEmail", status, "lastError", "createdAt"
FROM "AtriumEmailLog"
WHERE "toEmail" = '<recipient>'
ORDER BY "createdAt" DESC
LIMIT 10;

status will be one of QUEUED (still in flight), SENT (Plunk accepted, providerId stored), FAILED (lastError populated), or SKIPPED (broadcast gate dropped it; reason in lastError).

Drift surface (per-app)

Live at /admin/apps/[slug] as an amber banner whenever there are direct-admin or external grants. Backed by GET /api/admin/apps/:slug/drift. See groups for the provenance contract.

To probe via API:

curl -sS -H "Authorization: Bearer $TOKEN" \
  https://atrium.ommax-intelligence.de/api/admin/apps/<slug>/drift | jq

Returns { totalGrants, groupManagedCount, driftCount, drift: [...] } with each drift entry tagged direct-admin (Atrium audit-row exists, just not via a group path) or manual-external (no audit row at all — created in Zitadel Console / SCIM / pre-audit-log).

Runtime restarts — silent provisioning outage if you don't repush

The runtime registry is in-memory and boots empty. The canonical script source remains in Atrium's DB. Atrium pushes scripts to the runtime via two paths:

1. Per-script push — every PATCH /api/admin/actions/{name} calls pushScript() in lib/runtime-client.ts. Used when you edit a script.
2. Periodic reconcile — maybeKickRuntimeRegistryReconcile() inside /api/health reconciles every ATRIUM_ACTIONS_RUNTIME_RECONCILE_INTERVAL_MS (default 5 minutes). Throttled — successive /api/health hits inside the window are no-ops.

The trap: between the moment the runtime restarts (deploy, OOM, host reboot) and the next reconcile-or-PATCH, every Zitadel webhook to /actions/events returns 404 silently. Events are NOT retried. The recent-payloads recorder still captures the body — but no script runs, no AtriumActionRun row is written, and the user-visible symptom is "the auto-provisioning just stopped working".

Workaround after every redeploy of atrium-actions:

source ~/.config/atrium/credentials.env  # ATRIUM_API_KEY
ATRIUM=https://atrium.ommax-intelligence.de

# List ACTIVE scripts to confirm what should be in the registry:
curl -sS -H "Authorization: Bearer $ATRIUM_API_KEY" \
  "$ATRIUM/api/admin/actions" | jq -r '.scripts[] | select(.state=="ACTIVE") | .name'

# Force re-push of each by name (forceVersion bumps version even if
# source is unchanged, which triggers pushScript regardless of
# bundleHash equality):
for name in sentry-team-sync auto-group-by-department dokploy-grant-sync \
            dokploy-grant-removed flatten-roles flatten-roles-access-token; do
  curl -sS -X PATCH -H "Authorization: Bearer $ATRIUM_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{"state":"ACTIVE","forceVersion":true}' \
    "$ATRIUM/api/admin/actions/$name" >/dev/null && echo "  pushed: $name"
done

This is BACKLOG #9 — runtime cold-start reconciliation. The proper fix is a runtime → atrium callback on boot that fetches every ACTIVE script + decrypted secrets and installs them in one shot. Until that lands, treat every atrium-actions deploy as a provisioning outage of unknown duration and run the loop above to close the window.

How to confirm the registry has a script loaded (after running the workaround, or any time you suspect drift):

• Test-run via POST /api/admin/actions/{name}/test-run is NOT a sufficient check — it sends the script source inline and bypasses the registry entirely. A successful test-run says nothing about whether live Zitadel webhooks would route to the script.
• The reliable check is: trigger a real event (sign in, add a user, etc.) and watch /admin/actions/{name} → recent payloads + the resulting Sentry/Dokploy/etc. side-effect. If the side-effect doesn't appear, the registry was empty.

Provisioning history we hit: