Skip to main content
Chen quan

Chen quan

Software Engineer
View all authors

ArkFlow v0.2.0 is now available!

· 3 min read
Chen quan
Chen quan
Software Engineer

We are excited to announce the official release of ArkFlow v0.2.0! ArkFlow is a high-performance Rust stream processing engine that provides powerful data flow processing capabilities. This version builds upon RC1/RC2 with further optimizations and improvements, delivering more stable performance and richer functionality.

Key Features​

Diverse Component Support​

Input Components​

  • Kafka: Consume messages from Kafka topics
  • MQTT: Receive data from MQTT brokers
  • HTTP: Receive data through HTTP endpoints
  • Memory: Read data from memory message queues
  • File: Read data from files using SQL queries
  • Generate: Generate test data
  • Database: Query data from databases (MySQL, PostgreSQL, SQLite, Duckdb)

Output Components​

  • Kafka: Write messages to Kafka topics
  • MQTT: Publish messages to MQTT brokers
  • HTTP: Send messages to HTTP servers
  • Drop: Discard messages
  • Stdout: Output messages to the console

Buffer Components​

  • Memory Buffer: Provides memory message queues for temporary message storage and buffering, with backpressure support

Powerful Processing Capabilities​

SQL Query Support​

Data Processing​

  • JSON processing
  • Protobuf encoding/decoding
  • Batch processing support

Extensibility​

  • Modular design
  • Easy to extend with new input, output, and processor components

v0.2.0 Improvements​

  • Performance optimization: Increased data processing throughput and reduced latency
  • Enhanced stability: Fixed issues found in RC1/RC2, improving system stability
  • Improved documentation: Updated component documentation and usage examples
  • Dependency updates: Updated core dependency library versions, enhancing security and compatibility

Quick Start​

  1. Create a configuration file config.yaml:
logging:
  level: info
streams:
  - input:
      type: "generate"
      context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
      interval: 1ms
      batch_size: 1000

    buffer:
      type: "memory"
      capacity: 10
      timeout: 10s
    pipeline:
      thread_num: 4
      processors:
        - type: "json_to_arrow"
        - type: "sql"
          query: "SELECT sum(value) as value ,avg(value) , 0 as x FROM flow"
        - type: "arrow_to_json"


    output:
      type: "http"
      url: "http://example.com/data"
      method: "POST"
      timeout_ms: 90
      retry_count: 1
  1. Run ArkFlow:
./arkflow -c config.yaml

Examples​

We provide various example configurations to help you quickly get started with ArkFlow:

Contribution​

We welcome contributions from the community! If you find any issues or have suggestions for improvements, please submit issues or pull requests on GitHub.

Next Steps​

We will continue to improve and optimize ArkFlow, adding more features and component support. Community contributions and feedback are welcome!

ArkFlow v0.2.0-rc1 is now available!

· 2 min read
Chen quan
Chen quan
Software Engineer

We are excited to announce the release of ArkFlow v0.2.0-rc1! ArkFlow is a high-performance Rust stream processing engine that provides powerful data stream processing capabilities. This version brings many important features and improvements.

Key Features​

Diverse Component Support​

Input Components​

  • Kafka: Support for consuming messages from Kafka topics
  • MQTT: Receive data from MQTT brokers
  • HTTP: Receive data through HTTP endpoints
  • Memory: Read data from memory message queues
  • File: Read data from files using SQL queries
  • Generate: Generate test data
  • Database: Query data from databases (MySQL, PostgreSQL, SQLite, Duckdb)

Output Components​

  • Kafka: Write messages to Kafka topics
  • MQTT: Publish messages to MQTT brokers
  • HTTP: Send messages to HTTP servers
  • Drop: Discard messages
  • Stdout: Output messages to the console

Buffer Components​

  • Memory Buffer: Provides memory message queues for temporary message storage and buffering, with backpressure support

Powerful Processing Capabilities​

SQL Query Support​

Data Processing​

  • JSON processing
  • Protobuf encoding/decoding
  • Batch processing support

Extensibility​

  • Modular design
  • Easy to extend with new input, output, and processor components

Quick Start​

  1. Create a configuration file config.yaml:
logging:
  level: info
streams:
  - input:
      type: "generate"
      context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
      interval: 1ms
      batch_size: 1000

    buffer:
      type: "memory"
      capacity: 10
      timeout: 10s
    pipeline:
      thread_num: 4
      processors:
        - type: "json_to_arrow"
        - type: "sql"
          query: "SELECT sum(value) as value ,avg(value) , 0 as x FROM flow"
        - type: "arrow_to_json"


    output:
      type: "http"
      url: "http://example.com/data"
      method: "POST"
      timeout_ms: 90
      retry_count: 1
  1. Run ArkFlow:
./arkflow -c config.yaml

Next Steps​

We will continue to improve and optimize ArkFlow, adding more features and component support. Community contributions and feedback are welcome!