The modern data landscape is characterized by continuous, high-velocity streams of data originating from a multitude of sources, including IoT devices, user interactions, financial trading systems, and sensor networks. While traditional batch processing methods are suitable for historical data analysis, they fall short in scenarios requiring immediate insights and rapid responses. Artificial intelligence (AI) and machine learning (ML) models possess powerful analytical capabilities on their own, but we firmly believe their business value is maximized when applied to real-time, flowing data. The ability to perform inference, detect anomalies, or understand language the moment data arrives has become a key competitive advantage across industries.
Although some discussions suggest that not all historical application scenarios strictly require true real-time processing, current trends and specific needs clearly point to the importance of real-time AI capabilities. As we agree, "data in motion can generate greater value." Particularly in fields like fraud detection, traditional batch processing methods are "no longer effective," and stream processing platforms (such as Kafka) play a central role in enabling real-time analytics.
The core of the real-time capability we pursue is not just about speed, but about the timeliness of insights. In dynamic operational environments such as fraud detection, algorithmic trading, or critical system monitoring, an insight delayed by even a few seconds can lose all its value. The value of predictions, classifications, or other actionable insights generated by AI models decays rapidly over time in many critical business and operational scenarios. Therefore, we are committed to enabling you to process data and apply AI algorithms simultaneously as the data flows in, which is crucial for maximizing the utility and impact of these insights. This places stringent demands on the underlying technology platform, and it is precisely why we created ArkFlow—to provide a solution that seamlessly integrates high-performance stream processing with the execution of complex AI models.
The Python processor significantly lowers the barrier for the large community of data scientists and machine learning engineers, who primarily use Python, to deploy their models and algorithms in high-throughput, low-latency stream processing applications. Previously, this often required developers to have deep expertise in systems languages used for building stream processors, such as Rust or Java/Scala, or to deal with complex integration layers. We believe this newfound accessibility will accelerate the adoption and innovation of real-time AI solutions across various industries. The use of PyArrow is a key technical pillar in achieving this goal, ensuring efficient data exchange between the Rust core and the Python processor, thereby maintaining overall performance. Traditionally, AI model development and real-time stream processing deployment have been complex, requiring expertise in both AI and the internal mechanisms of stream processing systems like Flink or Spark. As the dominant language for AI/ML development, Python boasts a vast talent pool and a rich ecosystem of libraries. By allowing Python to be used directly within a high-performance stream processor, ArkFlow reduces the need for developers to learn new languages (like Rust) or complex integration patterns to implement their AI logic, making it easier for a broader range of developers to build real-time AI applications. PyArrow plays a crucial role in this process by providing an efficient and standardized way to transfer data between ArkFlow's Rust core and the Python processor, minimizing the serialization overhead that typically degrades performance in multi-language systems.
A key aspect of this integration is our use of the PyArrow library for data exchange. Apache Arrow and its Python bindings, PyArrow, provide a language-agnostic columnar memory format. This format is designed for efficient data sharing between different processes and systems, often enabling zero-copy (or near-zero-copy) data access. In the context of ArkFlow (a Rust application) and its Python processors, the Arrow format allows structured data (such as record batches or data frames) to be passed from the Rust environment to a Python process (and vice versa) with minimal serialization and deserialization overhead. This is critical for maintaining performance when crossing language boundaries.
The choice of PyArrow for data exchange between ArkFlow's Rust core and its Python processors strongly demonstrates our project's commitment to maintaining high performance, even when introducing an external language runtime. Without an efficient data exchange mechanism like Apache Arrow, the overhead incurred from passing data between Rust and Python (e.g., by serializing to JSON or pickle and then deserializing) could easily negate the performance benefits of a fast, Rust-based core engine, especially when handling high-volume, low-latency data streams. Inter-process communication (IPC) or foreign function interfaces (FFI) between different programming languages like Rust and Python can introduce significant performance overhead if data needs to be repeatedly copied and converted between the different native memory layouts or data formats of each language. Apache Arrow defines a standardized, language-agnostic columnar memory format optimized for analytical data processing and efficient data movement. When both the producer (ArkFlow's Rust core) and the consumer (the Python processor) can read and write data in the Arrow format, data can often be shared or transferred with zero or minimal copying, significantly reducing the overhead across language boundaries. This data exchange efficiency is crucial for real-time AI applications where every millisecond of latency matters. Therefore, adopting Arrow makes Python suitable not just for trivial in-stream scripting tasks; it opens the door to executing complex AI model inference and other computationally intensive Python code with acceptable performance characteristics.
Our explicit strategic intent in introducing Python support in ArkFlow is to enable users to directly "call any Python machine learning/deep learning library (TensorFlow, PyTorch, etc.) and large models" within their stream processing pipelines. This highlights the immense strategic value of Python integration. It means that developers, data scientists, and machine learning engineers can now directly leverage their existing Python skills, familiar tools, pre-trained models, and the vast and mature Python AI ecosystem within ArkFlow's high-performance stream processing environment. We believe this capability will dramatically accelerate the development cycle and deployment of complex real-time AI applications.
The deep integration of Python within a high-performance stream processor like ArkFlow is poised to give rise to a new class of "stream-native" AI applications. In this paradigm, AI models are no longer just external components applied to data exported in batches from a stream, but are designed and deployed as integral, active components within the data stream itself. Traditionally, the worlds of AI model development (primarily in Python) and real-time stream processing (often in Java/Scala) have been somewhat separate. Deploying Python AI models in high-performance streams often involved data batching, using less efficient IPC, or requiring significant custom integration work. ArkFlow's native Python processor support allows AI logic written in Python to become a first-class citizen within the stream processing pipeline—a "processor" that interacts directly with the data flow. This tight integration facilitates the construction of systems where AI is not merely a passive consumer of stream data but an active participant in the processing, decision-making, and even real-time modification of the stream's behavior.
We have noted that user queries explicitly point to the need to run machine learning models (like TensorFlow, ONNX) directly within data streams to achieve millisecond-level prediction and analysis. In ArkFlow, a Python processor instantiated within a pipeline is configured to load a pre-trained machine learning model. These models can be in various standard formats, such as TensorFlow SavedModel, ONNX graph, or PyTorch (.pt or .pth) files. Input data records from the stream, possibly pre-processed by upstream ArkFlow native processors (e.g., for data cleansing, feature extraction, or format conversion), are efficiently passed to the Python process via PyArrow. The script within the Python processor utilizes the runtime of the corresponding AI framework (e.g., TensorFlow's predict method, ONNX Runtime session's run, PyTorch's forward pass) to perform inference on the received data. The inference results—such as predicted values, classification outcomes, embedding vectors, or other model outputs—are then passed back from the Python process to the ArkFlow pipeline using PyArrow for subsequent downstream processing, routing to an output, or triggering further actions.
This capability makes true millisecond-latency prediction and analysis on real-time data possible. This is crucial for a wide range of applications that require immediate responses, such as identifying objects or events in real-time video frames, flagging potentially fraudulent financial transactions before they are completed, making ultra-fast algorithmic trading decisions, or instantly personalizing user experiences. ArkFlow's intent to support these frameworks is clearly stated in our various information sources. To elaborate on how these frameworks are typically optimized for inference (which ArkFlow can now enable in-stream), one can refer to general concepts such as TensorFlow's graph freezing, inference optimization, quantization, JIT/AOT compilation, and ONNX Runtime's cross-platform execution, model export, and hardware acceleration capabilities. AWS Elastic Inference's support for TensorFlow and ONNX also indirectly corroborates the industry application of these technologies for accelerating inference.
Consider a manufacturing quality control system as an example: a stream of images from products on an assembly line is fed into ArkFlow. Each image is passed to a Python processor. This processor loads a pre-trained ONNX or TensorFlow computer vision model (e.g., a YOLO variant or a ResNet classifier) to detect defects. If a defect is identified, the model's output (e.g., defect type, location, confidence score) is passed back. ArkFlow can then immediately trigger an alert, divert the defective product, or log the issue within milliseconds of the image capture.
However, we must point out that achieving the sustained "millisecond-level" inference latency desired by users is a significant technical challenge. It largely depends on various factors such as the complexity of the AI model, the size of the input data, the efficiency of the Rust-Python interop layer, and the overall pipeline design. While ArkFlow's Rust core and the use of PyArrow lay the necessary foundation for low latency, users still need to rigorously optimize their AI models for inference speed. Millisecond-level latency sets a very tight time budget for the entire process of data transfer, pre-processing, model inference, and post-processing. Although ArkFlow's core engine is designed for speed, Python execution (especially for complex models) can be slower. Therefore, techniques such as model quantization (reducing numerical precision, e.g., to INT8), graph pruning, or fusion, and selecting lightweight model architectures are necessary. PyArrow is also critical in minimizing the data transfer overhead at the Rust-Python boundary. Consequently, achieving consistent millisecond-level predictions requires a holistic approach that includes model engineering and optimization from the user.
]]>In this blog post, we will delve into these new features, from broader data source connectivity and more powerful processing capabilities with custom extensions, to landmark distributed computing experimental support, and continuous performance and usability optimizations.
ArkFlow's core components [input, processing, output, and buffering] have all received significant enhancements and expansions. Let's explore them.
To enable ArkFlow to connect to a wider data ecosystem, we have introduced several new input components and strengthened the functionality of existing ones:
Websocket Input (New): ArkFlow can now directly subscribe to messages from WebSocket connections. This is an important addition for scenarios requiring the processing of data from web applications or other real-time frontend push data.
Nats Input (New): Added support for the Nats messaging system, allowing subscription to messages from Nats topics. Nats is popular in modern cloud-native and microservice architectures for its high performance and simplicity. Integrating Nats allows ArkFlow to better fit into these environments. Its configuration has also been further optimized.
Redis Input (New): ArkFlow can now also subscribe to messages from Redis channels or lists. As a widely used cache and message broker, this new input component allows users to easily leverage existing Redis data streams. Its configuration has also been refactored for improved usability.
The value of data lies in its flow and transformation. The new version also brings significant improvements in data processing:
Optimizations in JSON to Arrow conversion, significant enhancements to the SQL processor (especially UDF support), and improvements to the Protobuf processor indicate that ArkFlow is focusing on use cases involving structured and semi-structured data, where both transformation flexibility (SQL, VRL, UDF) and performance (Arrow, Protobuf filtering) are crucial.
Processed data needs to be reliably delivered to its destination, and error handling is equally important.
This symmetrical addition of Nats input, output, and error output, along with the improvement of overall error handling capabilities, demonstrates ArkFlow's thorough strategy in integrating new components and strengthening core functionalities, ensuring users can build more reliable and maintainable data processing links.
Buffering mechanisms in stream processing systems are crucial for handling backpressure, temporarily storing messages, and performing time-window aggregations.
Memory Buffer: ArkFlow continues to offer robust memory buffering capabilities, which are critical for high-throughput scenarios and basic window aggregation operations.
New Windowing Buffer Components! To meet more complex time-series analysis and event stream processing needs, ArkFlow has introduced three specialized windowing buffer components:
These new windowing buffer mechanisms greatly enhance the flexibility and power of ArkFlow in handling time-sensitive data, enabling users to define and execute complex data aggregation and analysis logic with greater precision. They complement the existing memory buffer, providing users with a richer toolset to tackle diverse stream processing challenges.
The most notable feature evolution in recent ArkFlow development is undoubtedly the experimental distributed computing support. It signifies that ArkFlow is evolving towards systems capable of handling larger data volumes and more demanding processing loads, allowing tasks to be distributed across multiple nodes or worker units.
In addition to introducing heavyweight new components, the ArkFlow team also continuously refines existing functionalities to provide better performance and user experience.
We re-emphasize the JSON to Arrow conversion optimization , a concrete example of performance improvement.
To enhance the performance of the final product, the build process has also been optimized, for instance, by enabling Link-Time Optimization (LTO), setting codegen-units to 1 and setting opt-level to 3.
The plugin mechanism has been refactored, paving the way for easier development and integration of custom extensions in the future, further reinforcing ArkFlow's commitment to extensibility. Documentation updates are also an area of continuous investment. To improve the clarity and usability of the documentation, we have split document versions. These ongoing documentation updates demonstrate our understanding that good documentation is crucial for product adoption and ease of use, especially as new features are constantly being added.
The project structure includes arkflow-plugin-examples , indicating our commitment to helping users extend the system. Modular design and extensibility are core features of ArkFlow. Providing examples is a practical way to lower the barrier for users to write custom plugins, which can greatly enhance the platform's capabilities beyond what the core team provides and foster a vibrant ecosystem.
We encourage you to try the latest version of ArkFlow immediately!
Using the standard Rust toolchain (cargo) and common YAML configuration makes ArkFlow relatively easy to get started with for developers familiar with the Rust ecosystem and common configuration practices.
Thanks to @xiaoziv for bringing us the VRL processor.
We believe that flowing data can generate greater value and are committed to building a vast data processing ecosystem to simplify the threshold for data processing. The development of ArkFlow relies on community support:
Discord: https://discord.gg/CwKhzb8pux
ArkFlow is committed to continuous improvement, with more exciting features in the pipeline. Stay tuned!
]]>config.yaml:logging:
level: info
streams:
- input:
type: "generate"
context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
interval: 1ms
batch_size: 1000
buffer:
type: "memory"
capacity: 10
timeout: 10s
pipeline:
thread_num: 4
processors:
- type: "json_to_arrow"
- type: "sql"
query: "SELECT sum(value) as value ,avg(value) , 0 as x FROM flow"
- type: "arrow_to_json"
output:
type: "http"
url: "http://example.com/data"
method: "POST"
timeout_ms: 90
retry_count: 1
./arkflow -c config.yaml
We provide various example configurations to help you quickly get started with ArkFlow:
We welcome contributions from the community! If you find any issues or have suggestions for improvements, please submit issues or pull requests on GitHub.
We will continue to improve and optimize ArkFlow, adding more features and component support. Community contributions and feedback are welcome!
]]>config.yaml:logging:
level: info
streams:
- input:
type: "generate"
context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
interval: 1ms
batch_size: 1000
buffer:
type: "memory"
capacity: 10
timeout: 10s
pipeline:
thread_num: 4
processors:
- type: "json_to_arrow"
- type: "sql"
query: "SELECT sum(value) as value ,avg(value) , 0 as x FROM flow"
- type: "arrow_to_json"
output:
type: "http"
url: "http://example.com/data"
method: "POST"
timeout_ms: 90
retry_count: 1
./arkflow -c config.yaml
We will continue to improve and optimize ArkFlow, adding more features and component support. Community contributions and feedback are welcome!