Skip to main content
Apache-2.0Rust · Tokio · ArrowCNCF Landscape

Stream processing that never drops a record

High-performance Rust stream processing engine. Write declarative YAML pipelines, process with SQL, and rely on write-ahead-log durability, checkpointed state, and a built-in control plane for fleet operations.

Get startedGitHub

./arkflow --config config.yaml — one binary, no cluster required.

config.yaml
streams:
  - id: orders-to-mysql
    input:
      type: kafka
      brokers: [localhost:9092]
      topics: [shop.orders]
    pipeline:
      processors:
        - type: json_to_arrow
        - type: sql
          query: "SELECT * FROM flow WHERE status = 'PAID'"
        - type: arrow_to_json
    output:
      type: sql
      output_type:
        type: mysql
        uri: mysql://root@localhost:3306/arkflow
      table_name: orders
    error_output:
      type: stdout

Built for production data pipelines

From a single binary on your laptop to an audited fleet rollout.

High performance

Rust on Tokio with a columnar Apache Arrow data model. Operators fuse into chains — no channel hop inside a pipeline stage.

Durable by default

Every message is fsynced to a write-ahead log before processing. Cursors advance only through the highest contiguous acknowledged sequence.

Exactly-once, opt-in

Kafka transactional outputs close the duplicate window end-to-end. Checkpointed state restores jobs to a consistent cut.

Process with SQL

DataFusion-powered SQL, window functions, Python UDFs, and VRL transforms — plus Protobuf, Debezium CDC, and Schema Registry codecs.

Fleet control plane

A Hub/Agent architecture with desired-state semantics, reconciliation, audited rollouts, and a web console with a job DAG editor.

One kernel, any topology

Streams and distributed jobs compile to the same JobSpec. Input → buffer → processors → output is all you declare; the kernel does the rest.

One config, the whole journey

Ingest
KafkaHTTPMQTTNATSPulsar
Process
SQLPythonVRLWindows
Deliver
SQLKafkaRedisInfluxDBS3 WAL
✓ WAL durability before processing✓ At-least-once by default, exactly-once opt-in✓ Checkpoint, crash, recover
Build your first pipeline

Where do you want to go?

Learn the concepts

Architecture, backpressure, delivery semantics, and the durability model.

Understand ArkFlow →

Ship a use case

Kafka → SQL, CDC with Debezium, windowed aggregation — each with a validated example.

Browse recipes →

Run a fleet

Control plane, web console, rollouts, observability, and a recovery runbook.

Operate ArkFlow →

Join the flow

ArkFlow is open source under Apache-2.0 and listed in the CNCF Landscape. Contributions, issues, and ideas are welcome.

Star on GitHubChat on Discord