Config Reference¶
A copy-paste-ready annotated drakkar.yaml showing every configurable field. Each line carries a one-sentence explanation and the matching DK_* environment-variable override. The 📚 link beside each section header takes you to the deep tables and prose in Configuration.
How to read this page
- Pick only the sections you need – every field has a sane default, so an empty
drakkar.yamlis a valid config. - Comments mark reasonable values in
[brackets]so you know what’s typical vs. extreme. - Every field can be overridden by an env var. The pattern is
DK_<SECTION>__<FIELD>(double underscore for nesting). See Configuration Loading for the full precedence order. DK_CONFIG=/path/to/drakkar.yamlselects which YAML file to load (it’s not a config field itself).
Worker identity¶
Top-level fields that identify this worker in logs, metrics, and the operator UI.
# Name of the env var that holds this worker's display name.
# Empty / unset env var → falls back to "drakkar-<hex>".
worker_name_env: WORKER_ID # env: DK_WORKER_NAME_ENV · reasonable: WORKER_ID, HOSTNAME, K8S_POD_NAME
# Logical cluster name. Workers sharing this name are grouped in the operator UI
# and can cross-trace messages.
cluster_name: '' # env: DK_CLUSTER_NAME · reasonable: search-cluster, prod-east, ''
# Env var that holds the cluster name. If set and non-empty, overrides `cluster_name`.
# Useful when the cluster name comes from the deployment platform (k8s, ECS).
cluster_name_env: '' # env: DK_CLUSTER_NAME_ENV · reasonable: K8S_NAMESPACE, DEPLOY_ENV, ''
Kafka connection (kafka:)¶
Kafka cluster connection settings: brokers, transport security, raw client properties, Kafka-UI links. Shared by the Kafka source, the Kafka sinks, and the DLQ — brokers doubles as their fallback when their own brokers field is left empty.
kafka:
brokers: localhost:9092 # bootstrap servers, comma-separated. env: DK_KAFKA__BROKERS · reasonable: kafka-1:9092,kafka-2:9092
# Transport security. Default PLAINTEXT emits no client properties at all, so a worker
# that configures nothing here connects exactly as it always did. Kafka sinks and the DLQ
# inherit this block whenever their own `brokers` field is empty.
security:
protocol: PLAINTEXT # PLAINTEXT | SSL | SASL_PLAINTEXT | SASL_SSL. env: DK_KAFKA__SECURITY__PROTOCOL
sasl_mechanism: null # PLAIN | SCRAM-SHA-256 | SCRAM-SHA-512 | GSSAPI | OAUTHBEARER.
# Required for SASL_* protocols, rejected for the others.
# env: DK_KAFKA__SECURITY__SASL_MECHANISM
sasl_username: '' # env: DK_KAFKA__SECURITY__SASL_USERNAME
sasl_password: '' # prefer the env override over a YAML literal.
# env: DK_KAFKA__SECURITY__SASL_PASSWORD
ssl_ca_location: '' # PEM CA bundle; empty uses the system trust store.
# env: DK_KAFKA__SECURITY__SSL_CA_LOCATION
ssl_certificate_location: '' # client cert (mutual TLS). env: DK_KAFKA__SECURITY__SSL_CERTIFICATE_LOCATION
ssl_key_location: '' # client key (mutual TLS); requires ssl_certificate_location.
# env: DK_KAFKA__SECURITY__SSL_KEY_LOCATION
ssl_key_password: '' # passphrase for an encrypted client key.
# env: DK_KAFKA__SECURITY__SSL_KEY_PASSWORD
ssl_endpoint_identification_algorithm: null # unset = librdkafka default ("https", verification on).
# Set to "none" ONLY for internal CAs without matching SANs.
# Escape hatch: raw librdkafka properties, merged AFTER `security` so they win.
# For options the typed block does not model. These four keys are rejected at startup —
# each one backs a delivery invariant: enable.auto.commit, partition.assignment.strategy,
# group.id, bootstrap.servers.
client_config: {} # env: DK_KAFKA__CLIENT_CONFIG (JSON object)
# Kafka-UI deep-link integration: when both are set, the operator UI shows a clickable
# icon next to every <partition:offset>. Both empty = feature disabled silently.
ui_url: '' # env: DK_KAFKA__UI_URL · example: http://kafka-ui:8080
ui_cluster_name: '' # cluster name registered in Kafka-UI. env: DK_KAFKA__UI_CLUSTER_NAME
Input sources (sources:)¶
📚 Deep details · Input Sources · Staggered startup · Webapp page
The Kafka consumer and the synchronous HTTP ingress, as peers. Each is optional and enabled explicitly, but at least one of sources.kafka.enabled / sources.http.enabled must be true.
sources:
kafka:
enabled: false # consume the Kafka topic; off = no consumer group joined, arrange() never called. env: DK_SOURCES__KAFKA__ENABLED
topic: input-events # topic to consume from. env: DK_SOURCES__KAFKA__TOPIC
consumer_group: drakkar-workers # consumer-group ID; workers sharing it split partitions. env: DK_SOURCES__KAFKA__CONSUMER_GROUP
max_poll_records: 100 # messages per poll batch; ↑ throughput / ↓ latency. env: DK_SOURCES__KAFKA__MAX_POLL_RECORDS · reasonable: 50–500
max_poll_interval_ms: 300000 # max ms between polls before broker kicks us out. env: DK_SOURCES__KAFKA__MAX_POLL_INTERVAL_MS · raise for slow tasks
session_timeout_ms: 45000 # group-membership heartbeat window. env: DK_SOURCES__KAFKA__SESSION_TIMEOUT_MS
heartbeat_interval_ms: 3000 # heartbeat frequency; should be ≤ session_timeout_ms / 3. env: DK_SOURCES__KAFKA__HEARTBEAT_INTERVAL_MS
# Policy for messages whose value fails input_model parsing. env: DK_SOURCES__KAFKA__ON_PARSE_ERROR
# skip — message reaches arrange() with payload=None and msg.parse_error set (default)
# dlq — message is excluded from arrange() and written to the DLQ topic as a
# ParseFailurePayload; the offset commits only after the DLQ write is confirmed
# raise — fail fast: MessageParseError stops the partition processor (schema-broken deploys)
on_parse_error: skip
# Rolling-deploy alignment: serialize fleet-wide consumer-group rebalances.
startup_align_enabled: true # disable for snappy single-process dev. env: DK_SOURCES__KAFKA__STARTUP_ALIGN_ENABLED
startup_min_wait_seconds: 4.0 # min sleep before alignment (buffer for slow init). env: DK_SOURCES__KAFKA__STARTUP_MIN_WAIT_SECONDS
startup_align_interval_seconds: 10 # wake at next time%interval==0 boundary. env: DK_SOURCES__KAFKA__STARTUP_ALIGN_INTERVAL_SECONDS
http:
enabled: false # master switch; false = no FastAPI server. env: DK_SOURCES__HTTP__ENABLED
host: '0.0.0.0' # uvicorn bind interface. env: DK_SOURCES__HTTP__HOST
port: 8090 # uvicorn bind port. env: DK_SOURCES__HTTP__PORT
path: '/process' # single POST route; must start with '/'. env: DK_SOURCES__HTTP__PATH
sinks_enabled: false # when true, route on_message_complete payloads through SinkManager. env: DK_SOURCES__HTTP__SINKS_ENABLED
request_timeout_seconds: 30.0 # per-request budget; > 0. env: DK_SOURCES__HTTP__REQUEST_TIMEOUT_SECONDS
max_concurrent: 64 # per-worker in-flight cap; > 0. 65th concurrent request 503s. env: DK_SOURCES__HTTP__MAX_CONCURRENT
max_body_bytes: 10485760 # request-body cap in bytes; > 0 (413 beyond). env: DK_SOURCES__HTTP__MAX_BODY_BYTES
# Configured tenants. At least one entry is required; default is one
# anonymous client with rpm=4 so the HTTP source works out of the box.
clients:
- name: anonymous # tenant name; non-empty string
token: '' # bearer token; '' = anonymous slot (at most one client)
rpm: 4 # per-client requests/minute cap; > 0
- name: tenant-a
token: 'secret-tenant-a-token'
rpm: 60
When sources.http.enabled=true, the handler must declare HttpRequestT / HttpResponseT as the third and fourth generic parameters of BaseDrakkarHandler.
Executor pool (executor:)¶
📚 Deep details · Backpressure formula
Subprocess pool that runs user binaries. Tune max_executors and task_timeout_seconds first; everything else has good defaults.
executor:
binary_path: null # default binary for tasks. env: DK_EXECUTOR__BINARY_PATH · null = ExecutorTask must set its own
max_executors: 4 # concurrent subprocesses. env: DK_EXECUTOR__MAX_EXECUTORS · reasonable: 2–32
task_timeout_seconds: 120 # wall-clock cap per task. env: DK_EXECUTOR__TASK_TIMEOUT_SECONDS · reasonable: 30–600
max_stdout_bytes: 0 # stdout bytes retained per task (0 = unlimited). env: DK_EXECUTOR__MAX_STDOUT_BYTES
max_stderr_bytes: 0 # stderr bytes retained per task (0 = unlimited). env: DK_EXECUTOR__MAX_STDERR_BYTES
window_size: 100 # max messages collected per arrange() window. env: DK_EXECUTOR__WINDOW_SIZE
max_retries: 3 # retries per failed task (0 = no retries). env: DK_EXECUTOR__MAX_RETRIES
drain_timeout_seconds: 30 # graceful-shutdown wait for in-flight tasks. env: DK_EXECUTOR__DRAIN_TIMEOUT_SECONDS
# Backpressure: pause Kafka consumption when in-flight messages reach
# max_executors * high_multiplier; resume when down to max_executors * low_multiplier.
backpressure_high_multiplier: 32 # env: DK_EXECUTOR__BACKPRESSURE_HIGH_MULTIPLIER · default 32 = pause at 4*32=128 queued
backpressure_low_multiplier: 4 # env: DK_EXECUTOR__BACKPRESSURE_LOW_MULTIPLIER · default 4 = resume at max(1,4*4)=16 queued
# Subprocess environment.
env: {} # env vars passed to all subprocesses. env: DK_EXECUTOR__ENV (JSON)
env_inherit_parent: true # pass parent process env (filtered). env: DK_EXECUTOR__ENV_INHERIT_PARENT
env_inherit_deny: # fnmatch patterns NOT inherited (case-insensitive). env: DK_EXECUTOR__ENV_INHERIT_DENY (JSON list)
- 'DK_*' # framework internals
- '*PASSWORD*'
- '*PASSWD*'
- '*SECRET*'
- '*TOKEN*'
- '*_KEY'
- '*_DSN'
- '*CREDENTIAL*'
- '*SALT*'
Sinks (sinks:)¶
Where processed results are delivered. Each sink type maps instance names → config, so you can have multiple instances of the same type writing to different destinations. All instance keys (e.g. results, analytics) are user-chosen.
Kafka sink¶
sinks:
kafka:
results: # instance name (free-form, referenced from handler code)
topic: search-results # required. env: DK_SINKS__KAFKA__RESULTS__TOPIC
brokers: '' # empty = inherit kafka.brokers AND kafka.security.
# env: DK_SINKS__KAFKA__RESULTS__BROKERS
ui_url: '' # link to Kafka-UI/Kowl in the UI dashboard. env: DK_SINKS__KAFKA__RESULTS__UI_URL
security: {} # only consulted when `brokers` is set above; same fields as
# kafka.security. Leaving brokers empty is the usual choice.
client_config: {} # raw librdkafka overrides for this sink
flush_timeout_seconds: 30.0 # bound on the per-batch producer flush. env: DK_SINKS__KAFKA__RESULTS__FLUSH_TIMEOUT_SECONDS
# reasonable: 10-60; a last resort, not a latency target
PostgreSQL sink¶
sinks:
postgres:
main-db:
dsn: postgresql://user:pass@db:5432/myapp # required. env: DK_SINKS__POSTGRES__MAIN_DB__DSN
pool_min: 2 # min connections in asyncpg pool. env: DK_SINKS__POSTGRES__MAIN_DB__POOL_MIN · reasonable: 1–10
pool_max: 10 # max connections. env: DK_SINKS__POSTGRES__MAIN_DB__POOL_MAX · reasonable: 5–50
ui_url: '' # pgAdmin / Adminer URL. env: DK_SINKS__POSTGRES__MAIN_DB__UI_URL
statements: # optional. Operator-authored SQL by name
claim_job: | # env: DK_SINKS__POSTGRES__MAIN_DB__STATEMENTS__CLAIM_JOB
UPDATE jobs SET status = :status WHERE id = :id AND status = 'pending'
Statement names must match ^[a-z_][a-z0-9_]*$; placeholders are written
:name. Malformed SQL fails at startup, schema errors at delivery. See
Sink Write Operations.
MongoDB sink¶
sinks:
mongo:
logs:
uri: mongodb://mongo:27017 # required. env: DK_SINKS__MONGO__LOGS__URI
database: app_logs # required. env: DK_SINKS__MONGO__LOGS__DATABASE
ui_url: '' # Mongo Express URL. env: DK_SINKS__MONGO__LOGS__UI_URL
statements: # optional. Operator-authored MQL by name
record_attempt: # env: DK_SINKS__MONGO__LOGS__STATEMENTS__RECORD_ATTEMPT__COLLECTION
collection: jobs
op: update_one # one of: update_one, update_many, upsert, delete_one, delete_many
filter: { _id: ":id" } # required, and never empty
update: # required for the update ops, absent for the deletes
$set: { last_seen: ":now" }
$inc: { attempts: 1 }
Statement names must match ^[a-z_][a-z0-9_]*$. Values bind through ":name"
placeholders, whole values only and type-preserved; "::name" escapes a literal
leading colon. $where and $function are rejected at startup at any depth,
including inside aggregation-pipeline stages. Statements are not verified
against the live database. See
Sink Write Operations.
HTTP sink¶
Cloud metadata endpoints (169.254.169.254, metadata.google.internal, etc.) are rejected at config load to prevent IAM-credential leaks via misconfiguration.
sinks:
http:
webhook:
url: https://api.example.com/webhook # required, http(s) scheme. env: DK_SINKS__HTTP__WEBHOOK__URL
method: POST # env: DK_SINKS__HTTP__WEBHOOK__METHOD
timeout_seconds: 30 # request timeout. env: DK_SINKS__HTTP__WEBHOOK__TIMEOUT_SECONDS
max_retries: 3 # retries per failed request. env: DK_SINKS__HTTP__WEBHOOK__MAX_RETRIES
headers: {} # extra request headers. env: DK_SINKS__HTTP__WEBHOOK__HEADERS (JSON)
encoding: json # body format: json | form | multipart. env: DK_SINKS__HTTP__WEBHOOK__ENCODING
ui_url: '' # env: DK_SINKS__HTTP__WEBHOOK__UI_URL
Redis sink¶
sinks:
redis:
cache:
url: redis://localhost:6379/0 # env: DK_SINKS__REDIS__CACHE__URL
key_prefix: '' # prepended to every key, e.g. "result:". env: DK_SINKS__REDIS__CACHE__KEY_PREFIX
ui_url: '' # RedisInsight URL. env: DK_SINKS__REDIS__CACHE__UI_URL
scripts: # optional. Operator-authored Lua by name
push_and_cap: | # env: DK_SINKS__REDIS__CACHE__SCRIPTS__PUSH_AND_CAP
redis.call('LPUSH', KEYS[1], ARGV[1])
redis.call('LTRIM', KEYS[1], 0, tonumber(ARGV[2]) - 1)
Script names must match ^[a-z_][a-z0-9_]*$; values reach the script as KEYS
and ARGV, never interpolated. The Lua is not parsed at startup and not checked
against a live server, so a broken script fails at delivery. See
Sink Write Operations.
Filesystem sink¶
sinks:
filesystem:
archive:
base_path: /data/archive # required, payload paths are resolved under this. env: DK_SINKS__FILESYSTEM__ARCHIVE__BASE_PATH
ui_url: '' # env: DK_SINKS__FILESYSTEM__ARCHIVE__UI_URL
Custom sinks (plugins)¶
Instances of sink types registered by plugin packages via [project.entry-points."drakkar.sinks"]. The leaf config dict is plugin-defined and passed verbatim to the sink class constructor; an unregistered type name fails at startup.
sinks:
custom:
my_custom_type: # entry-point-registered sink type name
audit_trail_out: # instance name (free-form, referenced from handler code)
endpoint: https://audit.internal.example.com # plugin-defined fields — Drakkar passes
buffer_size: 500 # the dict through unvalidated
Delivery timeout (shared default)¶
sinks:
delivery_timeout_seconds: 30.0 # budget for one deliver() and one close(). env: DK_SINKS__DELIVERY_TIMEOUT_SECONDS · reasonable: 10–120
Circuit breaker (shared default)¶
📚 Deep details · State machine
Applies uniformly to every sink instance. Per-sink overrides are not supported in v1 – if one sink needs different thresholds, adjust the global default.
sinks:
circuit_breaker:
failure_threshold: 5 # consecutive terminal failures to trip. env: DK_SINKS__CIRCUIT_BREAKER__FAILURE_THRESHOLD · reasonable: 3–10
cooldown_seconds: 30.0 # seconds open before half-open probe. env: DK_SINKS__CIRCUIT_BREAKER__COOLDOWN_SECONDS · reasonable: 10–120
Dead letter queue (dlq:)¶
Failed sink deliveries (after retries are exhausted, or when a circuit breaker is open) are written to this Kafka topic.
dlq:
topic: '' # empty = auto-derive "{sources.kafka.topic}_dlq". env: DK_DLQ__TOPIC
# empty AND sources.kafka.enabled=false = no DLQ producer is
# built; DLQ sends are dropped and counted
brokers: '' # empty = inherit kafka.brokers AND kafka.security. env: DK_DLQ__BROKERS
security: {} # only consulted when `brokers` is set above; same fields as
# kafka.security (env: DK_DLQ__SECURITY__<FIELD>).
# Leaving brokers empty is the usual choice.
client_config: {} # raw librdkafka overrides for the DLQ producer, merged after
# security; only consulted when `brokers` is set
# Strategy when the DLQ write itself fails (payloads have nowhere safe to go):
# drop — log CRITICAL + tick drakkar_dlq_dropped_payloads_total, commit the offset,
# keep the pipeline moving; the payloads are lost (default)
# stall — leave the offset uncommitted and pause the partition; messages are
# redelivered after restart/rebalance (no loss, at the cost of lag)
flush_timeout_seconds: 30.0 # bound on the per-write producer flush. env: DK_DLQ__FLUSH_TIMEOUT_SECONDS
on_send_failure: drop # env: DK_DLQ__ON_SEND_FAILURE
Metrics (metrics:)¶
📚 Deep details · Observability
Prometheus scrape endpoint.
metrics:
enabled: true # env: DK_METRICS__ENABLED
port: 9090 # 1–65535. env: DK_METRICS__PORT
task_label_histograms: [] # task label keys observed into the
# drakkar_task_label_value histogram, one
# time series per key, e.g. [file_size_bytes]
# env: DK_METRICS__TASK_LABEL_HISTOGRAMS
Throughput (throughput:)¶
Opt-in task cost tracking: a numeric label becomes per-task speed and sliding-window throughput.
throughput:
cost_label: "" # task label key holding the numeric cost; "" disables.
# env: DK_THROUGHPUT__COST_LABEL
min_cost: 0.0 # smallest cost worth counting; smaller tasks carry no
# speed and enter no aggregate.
# env: DK_THROUGHPUT__MIN_COST
Runtime health (runtime_health:)¶
📚 Runtime health · Observability
Event-loop lag monitoring and stall introspection.
runtime_health:
enabled: true # env: DK_RUNTIME_HEALTH__ENABLED
tick_seconds: 0.25 # heartbeat interval. env: DK_RUNTIME_HEALTH__TICK_SECONDS
warn_lag_seconds: 0.1 # lag above this = degraded. env: DK_RUNTIME_HEALTH__WARN_LAG_SECONDS
stall_seconds: 1.0 # heartbeat silence above this = stalled + stack capture.
# env: DK_RUNTIME_HEALTH__STALL_SECONDS
max_stall_stacks: 10 # distinct stacks kept per stall. env: DK_RUNTIME_HEALTH__MAX_STALL_STACKS
sample_interval_seconds: 10.0 # recorder history sample cadence. env: DK_RUNTIME_HEALTH__SAMPLE_INTERVAL_SECONDS
history_window_seconds: 900 # in-memory sparkline window. env: DK_RUNTIME_HEALTH__HISTORY_WINDOW_SECONDS
episode_max_seconds: 300.0 # max length of one lag episode before it flushes.
# env: DK_RUNTIME_HEALTH__EPISODE_MAX_SECONDS
probe_interval_seconds: 0.0 # opt-in stack probe cadence; 0 disables.
# env: DK_RUNTIME_HEALTH__PROBE_INTERVAL_SECONDS
Blocking I/O pool (io:)¶
📚 Threads & Pools · Offloading CPU-bound hook work
asyncio’s default to_thread executor: every blocking call the handler (or the framework) sends
through asyncio.to_thread shares this one process-wide pool.
io:
max_threads: 0 # 0 = keep Python's default, min(32, cpu_count + 4).
# Set explicitly when blocking-I/O fan-out is capped by
# the pool (e.g. many-core hosts reading slow storage).
# env: DK_IO__MAX_THREADS
Offload pool (offload:)¶
📚 Offloading CPU-bound hook work · Runtime health
Thread pool behind handler.offload() — CPU-bound hook computations moved off the event loop so the worker stays responsive.
offload:
max_threads: 0 # concurrent offloaded computations before queueing.
# 0 = auto: ceil(executor.max_executors / 4), min 2.
# env: DK_OFFLOAD__MAX_THREADS
Logging (logging:)¶
Structured logging via structlog.
logging:
level: INFO # DEBUG | INFO | WARNING | ERROR | CRITICAL. env: DK_LOGGING__LEVEL
format: json # "json" for prod, "console" for dev. env: DK_LOGGING__FORMAT
output: stderr # stderr | stdout | file path. env: DK_LOGGING__OUTPUT
# File paths support {worker_id}, {cluster_name} templating
# e.g. /var/log/drakkar/{worker_id}.log
UI / Flight Recorder (ui:)¶
📚 Deep details · Authentication · Operator UI
The largest section. Provides the operator web UI, a SQLite-backed event log (ui.recorder.*), drakkar-ui bundle fetching (ui.release.*), WebSocket live streaming, worker autodiscovery, and Prometheus deep-links.
ui:
# --- Server ---
enabled: true # master switch for the whole UI feature. env: DK_UI__ENABLED
host: 127.0.0.1 # bind address. Use 0.0.0.0 for non-loopback. env: DK_UI__HOST
port: 8080 # 1–65535. env: DK_UI__PORT
public_url: '' # external URL advertised to peers (LB / ingress). env: DK_UI__PUBLIC_URL
# --- Auth (opt-in by default; most endpoints are read-only) ---
auth_token: '' # bearer token; empty = unauthenticated + startup warning. env: DK_UI__AUTH_TOKEN
# Generate: python -c "import secrets; print(secrets.token_urlsafe(32))"
# Never covers /healthz, /readyz and /api/v1/peer/ (the cache exchange).
allowed_ws_origins: [] # Origin allowlist for /ws and for the cross-origin reads of the UI fan-out
# (debug/trace, debug/trace-by-label, debug/databases). With auth_token set,
# list every worker origin. env: DK_UI__ALLOWED_WS_ORIGINS (JSON list)
# --- Side-effecting endpoints (gated independently of auth_token) ---
probe_enabled: true # POST /api/v1/debug/probe; runs caller bytes through the live handler
# and takes executor slots. false = 403. env: DK_UI__PROBE_ENABLED
merge_enabled: true # POST /api/v1/debug/merge; writes merged-<ts>-<token>.db into
# <shared.dir>/merged/, deleted after history.merged_retention_days.
# false = 403. env: DK_UI__MERGE_ENABLED
kafka_read_enabled: true # GET /api/v1/debug/kafka/*; ad-hoc reads of the configured topics
# (source/dlq/sink aliases only). false = 403. env: DK_UI__KAFKA_READ_ENABLED
# --- Timed consume pause (Live page; OPT-IN — pausing stops message intake) ---
consume_pause:
enabled: false # serve the pause API + Live-page control. env: DK_UI__CONSUME_PAUSE__ENABLED
durations_seconds: [15, 60, 300, 900] # preset buttons; API accepts any 1..3600s.
# env: DK_UI__CONSUME_PAUSE__DURATIONS_SECONDS (JSON list)
# --- Deployment metadata ---
expose_env_vars: [] # env vars captured into worker_config table. YAML only.
# e.g. ['GIT_SHA', 'DEPLOY_ENV', 'K8S_POD_NAME']
# --- Presentation ---
max_rows: 5000 # max rows returned by UI list endpoints. env: DK_UI__MAX_ROWS
# --- Duration thresholds (noise filters) ---
log_min_duration_ms: 500 # min ms to log slow_task_completed/failed. env: DK_UI__LOG_MIN_DURATION_MS
ws_min_duration_ms: 500 # min ms to broadcast over WebSocket. env: DK_UI__WS_MIN_DURATION_MS
# --- Prometheus deep-links in the UI ---
prometheus_url: '' # e.g. http://prometheus:9090. env: DK_UI__PROMETHEUS_URL
prometheus_rate_interval: 5m # rate() interval used in dashboard PromQL. env: DK_UI__PROMETHEUS_RATE_INTERVAL
prometheus_worker_label: '' # PromQL label for worker-scoped queries. env: DK_UI__PROMETHEUS_WORKER_LABEL
# Supports {worker_id}, {cluster_name}, {metrics_port}, {debug_port}
# e.g. 'worker_id="{worker_id}"'
prometheus_cluster_label: '' # PromQL label for cluster-wide queries. env: DK_UI__PROMETHEUS_CLUSTER_LABEL
# e.g. 'cluster="{cluster_name}"'
# --- Custom links shown in the dashboard nav ---
custom_links: [] # env: DK_UI__CUSTOM_LINKS (JSON list)
# Each entry: {name: "...", url: "..."}; url supports {worker_id} etc.
# --- Named URL bases for probe-details link templates (see ui-enrichment.md) ---
link_bases: {} # env: DK_UI__LINK_BASES (JSON object)
# e.g. {jira: 'https://jira.internal.example.com'}; a template base
# missing here logs a startup warning and renders as plain text.
# --- Deployment-provided custom cell renderers module (see ui-enrichment.md) ---
custom_renderers_path: '' # path to a JS module served at /api/v1/ui/renderers.js. env: DK_UI__CUSTOM_RENDERERS_PATH
# '' = off. Must exist at boot when set. Referenced from
# probe_field(view='custom', renderer=...) / Column(renderer=...).
# --- Flight recorder (persistence; files go to storage.dir, see Own storage below) ---
recorder:
store_events: true # write per-message events. env: DK_UI__RECORDER__STORE_EVENTS
store_config: true # write worker config into each file (history attribution). env: DK_UI__RECORDER__STORE_CONFIG
store_state: true # periodic worker-state snapshots. env: DK_UI__RECORDER__STORE_STATE
state_sync_interval_seconds: 10 # snapshot frequency. env: DK_UI__RECORDER__STATE_SYNC_INTERVAL_SECONDS
# --- Database rotation & archiving (see docs/local-databases.md#archiving) ---
rotation_interval_hours: 1 # how often to rotate the SQLite file; 1 = 1 hour. env: DK_UI__RECORDER__ROTATION_INTERVAL_HOURS
# Archiving settings live in history.archive (see History and archives below).
# --- Output (stdout/stderr) capture ---
store_output: true # include subprocess output in events. env: DK_UI__RECORDER__STORE_OUTPUT
store_stdin: false # store each task's stdin (capped) in task_started
# metadata; failures always store it. env: DK_UI__RECORDER__STORE_STDIN
stdin_max_bytes: 65536 # byte cap for stored stdin, 0 = unlimited.
# env: DK_UI__RECORDER__STDIN_MAX_BYTES
flush_interval_seconds: 5 # in-memory buffer → SQLite cadence. env: DK_UI__RECORDER__FLUSH_INTERVAL_SECONDS
max_buffer: 50000 # ring-buffer capacity. env: DK_UI__RECORDER__MAX_BUFFER
max_flush_retries: 3 # retries on transient SQLite errors. env: DK_UI__RECORDER__MAX_FLUSH_RETRIES
event_min_duration_ms: 0 # min ms to persist to SQLite (0 = persist all). env: DK_UI__RECORDER__EVENT_MIN_DURATION_MS
output_min_duration_ms: 500 # min ms to include stdout/stderr in event. env: DK_UI__RECORDER__OUTPUT_MIN_DURATION_MS
# --- Handler annotations (see annotations.md; 0 disables a byte cap) ---
annotations_enabled: true # record self.annotate(...) rows in the events table. env: DK_UI__RECORDER__ANNOTATIONS_ENABLED
annotation_max_bytes: 16384 # cap per annotation payload; oversize records drop whole.
# env: DK_UI__RECORDER__ANNOTATION_MAX_BYTES
annotation_max_bytes_per_call: 262144 # total annotation bytes one hook invocation may add.
# env: DK_UI__RECORDER__ANNOTATION_MAX_BYTES_PER_CALL
annotation_log_max_bytes: 2048 # cap on the payload copy in the drop-warning log line.
# env: DK_UI__RECORDER__ANNOTATION_LOG_MAX_BYTES
# --- drakkar-ui bundle fetching (never fatal; with nothing cached the worker runs API-only) ---
release:
enabled: true # resolve + serve the drakkar-ui bundle. env: DK_UI__RELEASE__ENABLED
repo: wlame/drakkar-ui # "owner/name" GitHub repo publishing UI bundles. env: DK_UI__RELEASE__REPO
pinned_version: '' # known-good UI release tag (e.g. v1.2.0); '' = unpinned. env: DK_UI__RELEASE__PINNED_VERSION
cache_dir: '' # bundle cache root; '' = $XDG_CACHE_HOME/drakkar/ui. env: DK_UI__RELEASE__CACHE_DIR
check_update: true # resolve the latest release tag on startup. env: DK_UI__RELEASE__CHECK_UPDATE
# --- Message Probe user-details write caps (see probe-user-details.md) ---
probe_details:
max_writes: 10000 # max probe.set/append/update calls per probe run; the first write
# past the cap records one ProbeError, further writes drop silently.
# env: DK_UI__PROBE_DETAILS__MAX_WRITES
max_total_bytes: 5000000 # total serialized bytes of probe-details writes per run; past it,
# writes drop like max_writes. env: DK_UI__PROBE_DETAILS__MAX_TOTAL_BYTES
# --- Live timeline tuning: history depth, bar color rules, label roles (see ui-timeline.md) ---
timeline:
history_factor: 100 # depth = history_factor x executor.max_executors (x8 if no pool), capped at 100000. env: DK_UI__TIMELINE__HISTORY_FACTOR
max_age_minutes: 60 # 1-1440; oldest task age the timeline shows. env: DK_UI__TIMELINE__MAX_AGE_MINUTES
color_rules: [] # first-match-wins bar-coloring rules; max 50. env: DK_UI__TIMELINE__COLOR_RULES (JSON list)
# e.g. [{name: failed, when: {field: status, op: eq, value: failed}, color: red}]
labels: # which task label fills each special role; '' = unbound
tag: '' # env: DK_UI__TIMELINE__LABELS__TAG
caption: '' # env: DK_UI__TIMELINE__LABELS__CAPTION
highlight: '' # env: DK_UI__TIMELINE__LABELS__HIGHLIGHT
filter: '' # env: DK_UI__TIMELINE__LABELS__FILTER
marker: '' # env: DK_UI__TIMELINE__LABELS__MARKER
events: [] # declared custom timeline event types (see ui-timeline-events.md); max 50.
# YAML only.
# e.g. [{name: deploy, kind: marker, color: purple, action: link, link: "{grafana}/d?from={ts_ms}"}]
# --- Trace search over sealed history (Trace page, history part) ---
trace:
history_max_files: 256 # most sealed files one trace request with history=1 opens. env: DK_UI__TRACE__HISTORY_MAX_FILES
history_max_seconds: 10 # time limit for that request. env: DK_UI__TRACE__HISTORY_MAX_SECONDS
# --- Operator docs site served at /docs/ + contextual anchors (see ui-docs.md) ---
docs:
site_dir: '' # directory of a PREBUILT static site (mkdocs site/, Sphinx html/, plain
# HTML); '' = feature off, /docs/ answers a hint-404. env: DK_UI__DOCS__SITE_DIR
title: Docs # nav-entry label + default drawer heading. env: DK_UI__DOCS__TITLE
anchors: [] # contextual links from UI surfaces into the site; max 200. YAML only.
# Each entry: {match: {...}, path: 'page.html#frag', title: '...'}; match sets
# exactly one of label (+optional value), sink, event, page.
# e.g. [{match: {sink: archive_results_db}, path: 'operations/#archive-database'}]
Cache (cache:)¶
Optional handler-accessible key/value cache. Disabled by default.
cache:
enabled: false # master switch; false = no-op stub. The cache file lives in storage.dir. env: DK_CACHE__ENABLED
flush_interval_seconds: 3.0 # write-behind flush cadence. env: DK_CACHE__FLUSH_INTERVAL_SECONDS
cleanup_interval_seconds: 60.0 # expired-row cleanup cadence. env: DK_CACHE__CLEANUP_INTERVAL_SECONDS
max_memory_entries: 10000 # in-memory LRU cap; null = unbounded (warns). env: DK_CACHE__MAX_MEMORY_ENTRIES
# Exchange of cluster/global rows with peers: fleet.cache_exchange below.
Own storage (storage:)¶
This worker’s own files: the live recorder file, the cache file, the crash marker. Only this worker opens them.
storage:
dir: /tmp/drakkar # created with mode 0700 when missing; '' = memory-only (no own files). env: DK_STORAGE__DIR
Shared folder (shared:)¶
The folder every worker of one fleet sees: registry cards, sealed history, daily archives, merge results.
History and archives (history:)¶
history:
archive:
enabled: true # fold recorder files into one .db.gz per cluster and window. env: DK_HISTORY__ARCHIVE__ENABLED
window_hours: 24 # archive window; must be >= ui.recorder.rotation_interval_hours. env: DK_HISTORY__ARCHIVE__WINDOW_HOURS
due_after_hours: 48 # a window is archived this long after it ends; must be >= rotation_interval_hours.
# env: DK_HISTORY__ARCHIVE__DUE_AFTER_HOURS
retention_days: 30 # delete archives older than this; 0 = keep forever. Must cover
# due_after_hours + window_hours. env: DK_HISTORY__ARCHIVE__RETENTION_DAYS
lock_stale_seconds: 7200 # archive lock / work files older than this belong to a dead archiver.
# env: DK_HISTORY__ARCHIVE__LOCK_STALE_SECONDS
merged_retention_days: 32 # delete UI merge results older than this; 0 = never. env: DK_HISTORY__MERGED_RETENTION_DAYS
Fleet registry and cache exchange (fleet:)¶
📚 Upgrading · Cache page
How workers find each other and exchange cache rows. Peers pull cluster/global cache rows over HTTP from the URL on each registry card. The pipeline itself does not depend on it.
fleet:
registry:
provider: filesystem # filesystem | static | none | <plugin name>; needs ui.enabled. env: DK_FLEET__REGISTRY__PROVIDER
heartbeat_seconds: 10 # card publish period. env: DK_FLEET__REGISTRY__HEARTBEAT_SECONDS
refresh_seconds: 15 # peer card read period. env: DK_FLEET__REGISTRY__REFRESH_SECONDS
offline_after_seconds: 30 # card older than this = offline; must be >= 2 x heartbeat_seconds.
# env: DK_FLEET__REGISTRY__OFFLINE_AFTER_SECONDS
prune_after_seconds: 86400 # delete cards older than this; must be >= offline_after_seconds.
# env: DK_FLEET__REGISTRY__PRUNE_AFTER_SECONDS
timeout_seconds: 5 # time limit for one registry operation. env: DK_FLEET__REGISTRY__TIMEOUT_SECONDS
filesystem:
dir: '' # '' = <shared.dir>/fleet. env: DK_FLEET__REGISTRY__FILESYSTEM__DIR
static:
peers: [] # [{name, cluster, url}]. YAML only.
plugin_options: {} # passed unchanged to a provider plugin. YAML only.
cache_exchange:
enabled: true # pull cluster/global cache rows from peers. env: DK_FLEET__CACHE_EXCHANGE__ENABLED
interval_seconds: 30 # cycle period. env: DK_FLEET__CACHE_EXCHANGE__INTERVAL_SECONDS
page_size: 500 # 1-1000 rows per page; pages until the peer says more=false. env: DK_FLEET__CACHE_EXCHANGE__PAGE_SIZE
timeout_seconds: 5 # time limit for one request to a peer (connect: 2 s). env: DK_FLEET__CACHE_EXCHANGE__TIMEOUT_SECONDS
cycle_deadline_seconds: null # hard cap on one cycle; null = interval x 0.9; must be < interval_seconds.
# env: DK_FLEET__CACHE_EXCHANGE__CYCLE_DEADLINE_SECONDS
max_concurrent_peers: 4 # peers pulled from in parallel. env: DK_FLEET__CACHE_EXCHANGE__MAX_CONCURRENT_PEERS
App config¶
The reserved top-level app: section carries user-defined application
config — the framework passes it through unvalidated to the
handler-declared Pydantic model and exposes the validated instance as
self.app_config. Env overrides use the handler’s own prefix (e.g.
MYAPP_SCORING__URL), never DK_APP__* (rejected at startup). The Debug
UI’s config reference renders the model as its own Application group,
secrets masked. See App Config for the full feature.
Environment-variable override cheatsheet¶
The pattern: DK_<SECTION>__<FIELD> – prefix DK_, double underscore between nesting levels, single underscore within a field name.
| Where | Path → env var |
|---|---|
| Top-level | cluster_name → DK_CLUSTER_NAME |
| One level deep | kafka.brokers → DK_KAFKA__BROKERS |
| Section, sub-section, field | sources.kafka.topic → DK_SOURCES__KAFKA__TOPIC |
| Two levels deep | fleet.cache_exchange.interval_seconds → DK_FLEET__CACHE_EXCHANGE__INTERVAL_SECONDS |
| Map key (sink instance) | sinks.postgres.main-db.dsn → DK_SINKS__POSTGRES__MAIN_DB__DSN |
| List element by index | sources.http.clients[0].rpm → DK_SOURCES__HTTP__CLIENTS__0__RPM |
| List value | ui.expose_env_vars → DK_UI__EXPOSE_ENV_VARS='["GIT_SHA","DEPLOY_ENV"]' (JSON) |
| Dict value | executor.env → DK_EXECUTOR__ENV='{"FOO":"bar"}' (JSON) |
Special cases:
DK_CONFIG– selects which YAML file to load; not a config field. Excluded from the env→config merge.- Hyphens in map keys –
main-dbbecomesMAIN_DB(single underscore) in the env var. Drakkar lowercases the env path on parse, then matches it against the YAML keys. - Booleans –
true/false(also1/0,yes/no). - Nulls – omit the env var, or use the literal
nullif the YAML value should be reset.
See Configuration Loading for the precedence order (defaults → YAML → env) and the deep-merge semantics.