Top-level configuration

An ArkFlow configuration describes the engine: logging, the health-check / control-plane server, the list of streams to run, and optional streaming jobs executed by the unified kernel. The file format is selected by extension — .yaml/.yml, .json, or .toml are all accepted.

logging:
  level: info

health_check:
  enabled: true
  address: "127.0.0.1:8080"

streams:
  - id: orders
    input:    { ... }
    pipeline: { ... }
    output:   { ... }

jobs: []      # optional declarative streaming jobs, see "job" below

Top-level fields

Field	Type	Required	Default	Description
streams	array<stream>	yes*	—	Streams to run.
jobs	array<job>	no	[]	Declarative streaming jobs (DAG + time + state + checkpoint) run by the unified kernel.
logging	object	no	see below	Logging configuration.
health_check	object	no	see below	Health-check and control-plane server.

* Both streams and jobs default to empty lists; a jobs-only configuration is valid (declare streams: [] or omit it).

logging

Field	Type	Required	Default	Description
level	string	no	info	Log level: debug, info, warn, error.
file_path	string	no	—	Write logs to this file instead of stdout.
format	string	no	plain	Log format: plain or json.

health_check

Runs an HTTP server with /health, /readiness, and /liveness endpoints (useful for Kubernetes). The same server also hosts the optional control-plane API and the Hub agent when hub_url is set (see Control plane).

Field	Type	Required	Default	Description
enabled	boolean	no	true	Start the health-check / control-plane server.
address	string	no	127.0.0.1:8080	Listen address.
health_path	string	no	/health	Overall health endpoint path.
readiness_path	string	no	/readiness	Readiness endpoint path.
liveness_path	string	no	/liveness	Liveness endpoint path.
api_prefix	string	no	/api/v1	Prefix for the versioned control-plane API.
api_token	string	no	—	Optional Bearer token protecting control-plane operations and configuration.
cors_origins	array<string>	no	[]	Browser origins allowed to call the control API. Empty denies cross-origin calls.
hub_url	string	no	—	Hub URL for compute-node agent mode. Absent ⇒ standalone mode.
node_id	string	no	—	Stable identity this process reports to its Hub.
node_token	string	no	—	Shared node registration credential. Never included in reports.
agent_lease_ttl_ms	integer	no	15000	Lease duration (ms) a compute node advertises to its Hub.
agent_session_ttl_ms	integer	no	3600000	Hard lifetime (ms) of a Hub-issued agent session credential; the Agent re-registers transparently when it elapses.

stream

Each entry in streams is one independent processing pipeline. Stream fields are documented in depth in the Components section; the shape is:

Field	Type	Required	Default	Description
id	string	no	stream-<index>	Stable stream identifier (must be unique; used for WAL identity and Hub reporting).
input	object	yes	—	Input component (source).
pipeline	object	yes	—	Processor pipeline.
output	object	yes	—	Output component (sink).
error_output	object	no	—	Output that receives batches a processor failed on.
buffer	object	no	—	Buffer / windowing strategy between input and processors.
durability	object	no	—	Per-stream WAL durability (see Delivery semantics).
temporary	array<object>	no	—	Temporary storage tables for joins.

pipeline

Field	Type	Required	Default	Description
thread_num	integer	no	1	Number of processor worker tasks.
processors	array<object>	yes	—	Ordered list of processor components.

job

Each entry in jobs is a declarative streaming job: an operator DAG with explicit event-time, state, checkpoint, and recovery settings. Jobs run locally through the same unified kernel as streams, and the same job shape is what the Hub distributes to compute nodes (see Distributed jobs).

jobs:
  - id: local-job
    version: 1
    parallelism: 1
    max_parallelism: 128
    operators:
      - { id: source, kind: source }
      - { id: sink, kind: sink }
    edges:
      - { id: e1, from: source, to: sink, partitioned: true }
    sources:
      - operator_id: source
        input_type: generate
        config: { type: generate, context: '{"value": 1}', interval: 1s, batch_size: 10 }
        time:
          mode: processing_time
    sinks:
      - operator_id: sink
        output_type: stdout
    recovery: latest_checkpoint

Field	Type	Required	Default	Description
id	string	yes	—	Stable job identifier; must be unique across jobs.
version	integer	yes	—	Job version; state-format compatibility on recovery is evaluated against it.
parallelism	integer	no	1	Default task parallelism.
max_parallelism	integer	no	128	Upper bound used for key-group partitioning.
operators	array<object>	yes	—	DAG nodes: id, kind (source, map, filter, aggregate, window, join, sink, udf), stateful, key_field, config.
edges	array<object>	no	[]	DAG edges: id, from, to, partitioned (key-group routing instead of same-subtask).
sources	array<object>	no	[]	Attach a component input to a source operator: operator_id, input_type, config, time.
sinks	array<object>	no	[]	Attach a component output to a sink operator: operator_id, output_type, config.
state	object	no	—	backend (e.g. embedded_kv), namespace, ttl_ms, format_version, max_pending_transactions (positive; default 4096; raise it when a window sees very high per-window key cardinality, since one transaction is held per open window group or unacknowledged output). Required by stateful operators.
checkpoint	object	no	—	interval_ms, retention, object_store_uri (e.g. file://... or s3://...).
recovery	string	no	latest_checkpoint	latest_checkpoint, latest_savepoint, or fail.

time (source event-time declaration)

Field	Type	Required	Default	Description
mode	string	yes	—	event_time or processing_time.
timestamp_field	string	no	—	Field read as the event timestamp when mode: event_time.
watermark	object	no	—	strategy (bounded_out_of_orderness (default) or monotonous), out_of_orderness_ms, idle_timeout_ms.
allowed_lateness_ms	integer	no	0	How far past the watermark late events are still accepted.
late_event_policy	string	no	drop	What happens to late events: drop, route, or update.
late_event_route	string	no	—	Operator receiving routed late events when the policy is route.

note

Intermediate operator kinds (map, filter, aggregate, window, join, udf) are primarily produced by the streaming SQL compiler and the console DAG orchestrator today. Always run --validate before deploying: it performs the same deep build checks as startup and rejects unsupported operators or state backends explicitly. ./target/release/arkflow schema emits the authoritative JSON Schema, including the jobs fields, for editor completion.

Validate before running

Always validate a config first:

./target/release/arkflow --config config.yaml --validate

Or emit the full JSON Schema and point your editor at it for field-level completion:

./target/release/arkflow schema > arkflow.schema.json

A pre-generated schema ships with the documentation at /config-schema.json; see IDE auto-completion for editor setup.