📄️ File
The File input reads JSON / CSV / Parquet / Avro / Arrow files via DataFusion. It supports local paths and cloud object storage (S3, GCS, Azure, HTTP, HDFS), with optional SQL over the file data or integration with the Ballista distributed engine.
📄️ Generate
The Generate input produces synthetic text messages at a fixed interval, primarily for testing, demos, and benchmarking. Each read returns batch_size copies of the same context string.
📄️ HTTP
The HTTP input runs as an Axum HTTP server and accepts POST requests sent to address+path. The request body (JSON) is decoded and forwarded into the stream processing pipeline. Optional CORS and Basic/Bearer authentication are supported. Bind failures surface synchronously dropping a pending read never loses an already-decoded message (see Input Delivery Semantics).
📄️ Kafka
The Kafka input consumes messages from one or more Apache Kafka topics using a consumer group. Offsets are only advanced after the downstream output acknowledges the write (enable.auto.offset.store=false), giving at-least-once delivery across crashes.
📄️ Memory
The Memory input reads messages from an in-memory queue that can be pre-seeded with initial messages in configuration. Mainly used for testing and development.
📄️ Modbus
The Modbus input polls a Modbus TCP device at a fixed interval and supports four register types: coils, discreteinputs, holdingregisters, and input_registers.
📄️ MQTT
The MQTT input connects to an MQTT broker, subscribes to one or more topics, and receives real-time messages. As a channel-based input it follows the engine's cancellation-safety contract: dropping a pending read never loses an already-decoded message (see Input Delivery Semantics).
📄️ Multiple Inputs
Multiple Inputs merges several independent input components into a single logical stream. All child inputs are read concurrently, and messages enter the same pipeline in arrival order. Each child input may carry a name, which is written to meta_source so downstream stages can distinguish the origin.
📄️ NATS
The NATS input connects to a NATS server and supports two modes dropping a pending read never loses an already-decoded message (see Input Delivery Semantics).
📄️ Pulsar
The Pulsar input subscribes to an Apache Pulsar topic and supports four subscription types — exclusive / shared / failover / key_shared — with optional Token or OAuth2 authentication. A fresh subscription starts from the earliest message (the existing backlog is consumed, matching the Kafka input default); acknowledgements always complete within a bounded time, even when the broker stops delivering new messages. As a channel-based input it follows the engine's cancellation-safety contract: dropping a pending read never loses an already-decoded message (see Input Delivery Semantics).
📄️ Redis
The Redis input reads from Redis with both standalone and cluster connection modes, and supports Subscribe (channels / patterns) and List consumption modes. As a channel-based input it follows the engine's cancellation-safety contract: dropping a pending read never loses an already-decoded message (see Input Delivery Semantics).
📄️ SQL
The SQL input executes a select_sql query through DataFusion to read from a database (MySQL, PostgreSQL, SQLite, DuckDB) or file format. Ballista is optional for distributed queries.
📄️ WebSocket
The WebSocket input connects to a remote WebSocket server as a client, decodes each inbound message, and forwards it into the pipeline. The current implementation supports client mode only. As a channel-based input it follows the engine's cancellation-safety contract: dropping a pending read never loses an already-decoded message (see Input Delivery Semantics).