Skip to main content
Apache-2.0Rust · Tokio · ArrowCNCF Landscape

Stream processing that never drops a record

High-performance Rust stream processing engine. Write declarative YAML pipelines, process with SQL, and rely on write-ahead-log durability, checkpointed state, and a built-in control plane for fleet operations.

Get startedGitHub

./arkflow --config config.yaml — one binary, no cluster required.

config.yaml

Built for production data pipelines

From a single binary on your laptop to an audited fleet rollout.

High performance

Rust on Tokio with a columnar Apache Arrow data model. Operators fuse into chains — no channel hop inside a pipeline stage.

Durable by default

Every message is fsynced to a write-ahead log before processing. Cursors advance only through the highest contiguous acknowledged sequence.

Exactly-once, opt-in

Kafka transactional outputs close the duplicate window end-to-end. Checkpointed state restores jobs to a consistent cut.

Process with SQL

DataFusion-powered SQL, window functions, Python UDFs, and VRL transforms — plus Protobuf, Debezium CDC, and Schema Registry codecs.

Fleet control plane

A Hub/Agent architecture with desired-state semantics, reconciliation, audited rollouts, and a web console with a job DAG editor.

One kernel, any topology

Streams and distributed jobs compile to the same JobSpec. Input → buffer → processors → output is all you declare; the kernel does the rest.

One config, the whole journey

Ingest
KafkaHTTPMQTTNATSPulsar
Process
SQLPythonVRLWindows
Deliver
SQLKafkaRedisInfluxDBS3 WAL
✓ WAL durability before processing✓ At-least-once by default, exactly-once opt-in✓ Checkpoint, crash, recover
Build your first pipeline

Under the hood

Streams and distributed jobs compile to the same JobSpec and run on one execution kernel. Batches stay durable, SQL does the heavy lifting, sinks stay ordered — while a Hub/Agent control plane keeps the fleet honest.

ArkFlow engine architectureSources feed the ArkFlow engine — input, WAL, buffer, processors, output over Arrow record batches — supervised by a Hub/Agent control plane, and write to sinks.CONTROL PLANE · ARKFLOW-SERVERWeb consoleHubAgentsdesired state · reconcileSOURCESSINKSARKFLOW ENGINEstreams & jobs compile to one JobSpec · unified execution kernelInputsource readerWALfsync · replayBufferwindows · joinsProcessSQL · UDF · VRLOutputordered sinkerror outputArrow MessageBatchbounded channels · backpressurecheckpointed stateKafkaconsumer groupsMQTTQoS subscriptionsHTTPREST pushNATScore · JetStreamMySQLupsert batchesKafkatransactionsRedishot path cacheInfluxDBmetrics
Hover any node to inspect it — the light wave is a batch moving through the kernel.
records in 128,940processed 128,940dropped 0

Where do you want to go?

Learn the concepts

Architecture, backpressure, delivery semantics, and the durability model.

Understand ArkFlow →

Ship a use case

Kafka → SQL, CDC with Debezium, windowed aggregation — each with a validated example.

Browse recipes →

Run a fleet

Control plane, web console, rollouts, observability, and a recovery runbook.

Operate ArkFlow →

Join the flow

ArkFlow is open source under Apache-2.0 and listed in the CNCF Landscape. Contributions, issues, and ideas are welcome.

Star on GitHubChat on Discord