Build with ArkFlow

A pipeline in ArkFlow is a declarative YAML description of data flowing from an input, through processors (and optionally buffers), to an output — all backed by a columnar (Apache Arrow) data model. The same description compiles to one unified execution kernel whether it runs as a single stream, a local job DAG, or a slice of a distributed job.

Streams

The pipeline anatomy: input → processors → output, error outputs, and codecs.

Jobs

Multi-step DAGs with event time, keyed state, and checkpoints — locally or distributed.

Distributed jobs

How the control plane assigns, co-locates, and recovers job slices across a fleet.

Durability & delivery

At-least-once by default; exactly-once for Kafka outputs; WAL and checkpoint mechanics.

Architecture

The columnar data model, metadata columns, and how a config becomes a running graph.

Recipes

Task how-tos and end-to-end cases, each backed by a CI-validated example.

The full component catalog lives under Components in the sidebar, and the SQL dialect is documented separately under SQL.