We are thrilled to announce a significant update to ArkFlow! ArkFlow is a high-performance stream processing engine built on Rust, dedicated to providing powerful and easy-to-use data stream processing capabilities. The latest version brings many exciting new features and significant enhancements designed to help you process complex data streams more efficiently.

In this blog post, we will delve into these new features, from broader data source connectivity and more powerful processing capabilities with custom extensions, to landmark distributed computing experimental support, and continuous performance and usability optimizations.

We are excited to announce the official release of ArkFlow v0.2.0! ArkFlow is a high-performance Rust stream processing engine that provides powerful data flow processing capabilities. This version builds upon RC1/RC2 with further optimizations and improvements, delivering more stable performance and richer functionality.

Key Features​

Diverse Component Support​

Input Components​

• Kafka: Consume messages from Kafka topics
• MQTT: Receive data from MQTT brokers
• HTTP: Receive data through HTTP endpoints
• Memory: Read data from memory message queues
• File: Read data from files using SQL queries
• Generate: Generate test data
• Database: Query data from databases (MySQL, PostgreSQL, SQLite, Duckdb)

Output Components​

• Kafka: Write messages to Kafka topics
• MQTT: Publish messages to MQTT brokers
• HTTP: Send messages to HTTP servers
• Drop: Discard messages
• Stdout: Output messages to the console

Buffer Components​

• Memory Buffer: Provides memory message queues for temporary message storage and buffering, with backpressure support

Powerful Processing Capabilities​

SQL Query Support​

• Built-in SQL processor using DataFusion as the query engine
• Support for rich SQL features:
  • Aggregate Functions
  • Window Functions
  • Subqueries
  • Scalar Functions
  • Special Functions

Data Processing​

• JSON processing
• Protobuf encoding/decoding
• Batch processing support

Extensibility​

• Modular design
• Easy to extend with new input, output, and processor components

v0.2.0 Improvements​

• Performance optimization: Increased data processing throughput and reduced latency
• Enhanced stability: Fixed issues found in RC1/RC2, improving system stability
• Improved documentation: Updated component documentation and usage examples
• Dependency updates: Updated core dependency library versions, enhancing security and compatibility

Quick Start​

1. Create a configuration file config.yaml:

logging:
  level: info
streams:
  - input:
      type: "generate"
      context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
      interval: 1ms
      batch_size: 1000

    buffer:
      type: "memory"
      capacity: 10
      timeout: 10s
    pipeline:
      thread_num: 4
      processors:
        - type: "json_to_arrow"
        - type: "sql"
          query: "SELECT sum(value) as value ,avg(value) , 0 as x FROM flow"
        - type: "arrow_to_json"


    output:
      type: "http"
      url: "http://example.com/data"
      method: "POST"
      timeout_ms: 90
      retry_count: 1

2. Run ArkFlow:

./arkflow -c config.yaml

Examples​

We provide various example configurations to help you quickly get started with ArkFlow:

• Example Configurations

Contribution​

We welcome contributions from the community! If you find any issues or have suggestions for improvements, please submit issues or pull requests on GitHub.

Next Steps​

We will continue to improve and optimize ArkFlow, adding more features and component support. Community contributions and feedback are welcome!

We are excited to announce the release of ArkFlow v0.2.0-rc1! ArkFlow is a high-performance Rust stream processing engine that provides powerful data stream processing capabilities. This version brings many important features and improvements.

Key Features​

Diverse Component Support​

Input Components​

• Kafka: Support for consuming messages from Kafka topics
• MQTT: Receive data from MQTT brokers
• HTTP: Receive data through HTTP endpoints
• Memory: Read data from memory message queues
• File: Read data from files using SQL queries
• Generate: Generate test data
• Database: Query data from databases (MySQL, PostgreSQL, SQLite, Duckdb)

Output Components​

• Kafka: Write messages to Kafka topics
• MQTT: Publish messages to MQTT brokers
• HTTP: Send messages to HTTP servers
• Drop: Discard messages
• Stdout: Output messages to the console

Buffer Components​

• Memory Buffer: Provides memory message queues for temporary message storage and buffering, with backpressure support

Powerful Processing Capabilities​

SQL Query Support​

• Built-in SQL processor using DataFusion as the query engine
• Support for rich SQL features:
  • Aggregation functions
  • Window functions
  • Subqueries
  • Scalar functions
  • Special Functions

Data Processing​

• JSON processing
• Protobuf encoding/decoding
• Batch processing support

Extensibility​

• Modular design
• Easy to extend with new input, output, and processor components

Quick Start​

1. Create a configuration file config.yaml:

logging:
  level: info
streams:
  - input:
      type: "generate"
      context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
      interval: 1ms
      batch_size: 1000

    buffer:
      type: "memory"
      capacity: 10
      timeout: 10s
    pipeline:
      thread_num: 4
      processors:
        - type: "json_to_arrow"
        - type: "sql"
          query: "SELECT sum(value) as value ,avg(value) , 0 as x FROM flow"
        - type: "arrow_to_json"


    output:
      type: "http"
      url: "http://example.com/data"
      method: "POST"
      timeout_ms: 90
      retry_count: 1

2. Run ArkFlow:

./arkflow -c config.yaml

Next Steps​

We will continue to improve and optimize ArkFlow, adding more features and component support. Community contributions and feedback are welcome!

Related Links​

• GitHub
• Examples