Component inventory

This page is generated from component-inventory.json. It is the review point for component coverage, not a replacement for the detailed component pages.

Kind	Component	Description	Documentation
buffer	memory	In-memory buffer that releases a batch when it reaches capacity or after a timeout.	reference
buffer	session_window	Groups messages into sessions based on a maximum gap between messages. Supports SQL joins across sources.	reference
buffer	sliding_window	Overlapping time windows that slide forward by a fixed interval.	reference
buffer	tumbling_window	Fixed-size, non-overlapping time windows. Supports SQL joins across sources.	reference
codec	debezium_json	Decodes Debezium CDC Envelope JSON (before/after/op/source/ts_ms) into a columnar Arrow batch; attach to a Kafka input consuming a Debezium topic. CDC offset is the Kafka input's ack-gated offset.	reference
codec	json	Encodes/decodes Arrow RecordBatches as JSON byte payloads.	reference
codec	protobuf	Encodes/decodes Arrow RecordBatches using a Protobuf descriptor.	reference
codec	schema_registry	Decodes Confluent wire-format Protobuf messages by resolving the schema id from a Schema Registry.	reference
input	file	Reads records from local or remote object storage (S3, GCS, Azure, HDFS) in CSV/JSON/Parquet/Avro/Arrow formats.	reference
input	generate	Generates synthetic text messages on a fixed interval (useful for testing and load simulation).	reference
input	http	Receives data via HTTP. Can run as a server (POST/PUT on path) or poll a remote endpoint.	reference
input	kafka	Consumes messages from Apache Kafka topics with a consumer group.	reference
input	memory	In-memory input queue seeded with an initial list of messages. Primarily for tests and demos.	reference
input	modbus	Polls Modbus TCP devices on a fixed interval, reading coils, discrete inputs, or registers.	reference
input	mqtt	Subscribes to an MQTT broker and forwards messages from the configured topics.	reference
input	multiple_inputs	Combines multiple input sources into a single stream. Each source is tagged with __meta_source.	reference
input	nats	Consumes messages from NATS, supporting both regular subjects and JetStream consumers.	reference
input	pulsar	Subscribes to an Apache Pulsar topic with configurable subscription type and authentication.	reference
input	redis	Reads from Redis: list blocking pops, pub/sub subscriptions, or stream consumer groups.	reference
input	sql	Polls a SQL database (MySQL / PostgreSQL / SQLite / DuckDB) with a SELECT statement and emits rows as batches.	reference
input	websocket	Connects to a WebSocket server and forwards each incoming message as a batch.	reference
output	drop	Discards all messages. Useful for performance benchmarks and dead-end pipelines.	reference
output	http	Posts each batch to an HTTP endpoint. Supports custom headers, retry, and auth.	reference
output	influxdb	Writes time-series data to InfluxDB v2.x using the Line Protocol.	reference
output	kafka	Produces messages to Apache Kafka. Supports key-based partitioning and compression.	reference
output	mongodb	Writes Arrow rows to MongoDB as BSON documents.	reference
output	mqtt	Publishes messages to an MQTT broker topic.	reference
output	nats	Publishes to NATS, either to a regular subject or a JetStream stream.	reference
output	pulsar	Produces messages to an Apache Pulsar topic.	reference
output	redis	Writes messages to Redis: streams, lists, or pub/sub channels.	reference
output	sql	Batch-inserts records into a SQL database. Supports upsert and transaction management.	reference
output	stdout	Writes each message to the console. Useful for debugging and demos.	reference
processor	arrow_to_json	Converts an Arrow RecordBatch into JSON byte payloads (one per row).	reference
processor	arrow_to_protobuf	Serializes Arrow RecordBatches into Protobuf wire-format bytes.	reference
processor	batch	Batches messages by count, size, or time interval before forwarding.	reference
processor	json_to_arrow	Parses JSON byte payloads into an Arrow RecordBatch with inferred schema.	reference
processor	protobuf_to_arrow	Decodes Protobuf wire-format bytes into Arrow RecordBatches.	reference
processor	python	Runs a user-defined Python function (with PyArrow) against each batch.	reference
processor	sql	Runs a DataFusion SQL query against each batch. Supports window functions and joins against temporary tables.	reference
processor	vrl	Runs a Vector Remap Language (VRL) program against each batch for safe transformation and enrichment.	reference
temporary	redis	Redis-backed temporary lookup store (single node or cluster) read through a codec.	reference

When a component is added or removed, update the JSON inventory and its detailed page in the same pull request.