Component inventory

This page is generated from component-inventory.json. It is the review point for component coverage, not a replacement for the detailed component pages.

Kind	Component	Description	Documentation
buffer	memory	In-memory buffer that releases a batch when it reaches capacity or after a timeout.	reference
buffer	session_window	Groups messages into sessions based on a maximum gap between messages. Supports SQL joins across sources.	reference
buffer	sliding_window	Overlapping time windows that slide forward by a fixed interval.	reference
buffer	tumbling_window	Fixed-size, non-overlapping time windows. Supports SQL joins across sources.	reference
codec	debezium_json	Decodes Debezium CDC Envelope JSON (before/after/op/source/ts_ms) into a columnar Arrow batch; attach to a Kafka input consuming a Debezium topic. CDC offset is the Kafka input's ack-gated offset.	reference
codec	json	Encodes/decodes Arrow RecordBatches as JSON byte payloads.	reference
codec	protobuf	Encodes/decodes Arrow RecordBatches using a Protobuf descriptor.	reference
codec	schema_registry	Decodes Confluent wire-format Protobuf and Avro messages by resolving the schema id from a Schema Registry, with an optional subject compatibility gate.	reference
input	file	Reads records from local or remote object storage (S3, GCS, Azure, HDFS) in CSV/JSON/Parquet/Avro/Arrow formats.	reference
input	generate	Generates synthetic text messages on a fixed interval (useful for testing and load simulation).	reference
input	http	Receives data via HTTP. Can run as a server (POST/PUT on path) or poll a remote endpoint.	reference
input	kafka	Consumes messages from Apache Kafka topics with a consumer group.	reference
input	memory	In-memory input queue seeded with an initial list of messages. Primarily for tests and demos.	reference
input	modbus	Polls Modbus TCP devices on a fixed interval, reading coils, discrete inputs, or registers.	reference
input	mqtt	Subscribes to an MQTT broker and forwards messages from the configured topics.	reference
input	multiple_inputs	Combines multiple input sources into a single stream. Each source is tagged with __meta_source.	reference
input	nats	Consumes messages from NATS, supporting both regular subjects and JetStream consumers.	reference
input	pulsar	Subscribes to an Apache Pulsar topic with configurable subscription type and authentication.	reference
input	redis	Reads from Redis: list blocking pops, pub/sub subscriptions, or stream consumer groups.	reference
input	sql	Runs a one-shot SELECT against a SQL database (MySQL / PostgreSQL / SQLite / DuckDB) and emits the rows as batches.	reference
input	websocket	Connects to a WebSocket server and forwards each incoming message as a batch.	reference
output	drop	Discards all messages. Useful for performance benchmarks and dead-end pipelines.	reference
output	http	Posts each batch to an HTTP endpoint. Supports custom headers, retry, and auth.	reference
output	influxdb	Writes time-series data to InfluxDB v2.x using the Line Protocol.	reference
output	kafka	Produces messages to Apache Kafka. Supports key-based partitioning and compression.	reference
output	milvus	Upserts batch rows into a Milvus collection over the REST v2 vectordb API: a Float32 list column becomes the vector, other columns pack into a JSON payload field, and an optional id column keys the row.	reference
output	mongodb	Writes Arrow rows to MongoDB as BSON documents.	reference
output	mqtt	Publishes messages to an MQTT broker topic.	reference
output	nats	Publishes to NATS, either to a regular subject or a JetStream stream.	reference
output	pgvector	Upserts batch rows into a PostgreSQL table with the pgvector extension: a Float32 list column becomes the vector, other columns are packed into a jsonb payload, and an optional id column keys ON CONFLICT upserts.	reference
output	pulsar	Produces messages to an Apache Pulsar topic.	reference
output	qdrant	Upserts batch rows as Qdrant points over the REST API: a vector column, an optional id column, and all remaining columns as payload.	reference
output	redis	Writes messages to Redis via publish, list rpush, hashes, or plain strings.	reference
output	sql	Batch-inserts records into a MySQL or PostgreSQL database, with optional upsert (ON DUPLICATE KEY UPDATE / ON CONFLICT DO UPDATE) for idempotent writes.	reference
output	stdout	Writes each message to the console. Useful for debugging and demos.	reference
processor	arrow_to_json	Converts an Arrow RecordBatch into JSON byte payloads (one per row).	reference
processor	arrow_to_protobuf	Serializes Arrow RecordBatches into Protobuf wire-format bytes.	reference
processor	batch	Batches messages by row count with an idle timeout before forwarding.	reference
processor	embedding	Batch-embeds a text column through an OpenAI-compatible embeddings API and appends the vectors as a FixedSizeList(Float32) column.	reference
processor	json_to_arrow	Parses JSON byte payloads into an Arrow RecordBatch with inferred schema.	reference
processor	llm	Sends each row of a text column to an OpenAI-compatible chat completions API and appends the completion as a Utf8 column, with bounded ordered concurrency.	reference
processor	milvus_search	Searches a Milvus collection for the top-k nearest neighbors of each row's vector and appends the matches as a JSON array text column. Rows are searched with bounded, order-preserving concurrency.	reference
processor	pgvector_search	Searches a PostgreSQL table with the pgvector extension for the top-k nearest neighbors of each row's vector and appends the matches as a JSON array text column.	reference
processor	protobuf_to_arrow	Decodes Protobuf wire-format bytes into Arrow RecordBatches.	reference
processor	python	Runs a user-defined Python function (with PyArrow) against each batch.	reference
processor	sql	Runs a DataFusion SQL query against each batch. Supports window functions and joins against temporary tables.	reference
processor	vector_search	Searches a Qdrant collection for the top-k nearest neighbors of each row's vector and appends the matches as a JSON array text column.	reference
processor	vrl	Runs a Vector Remap Language (VRL) program against each batch for safe transformation and enrichment.	reference
temporary	redis	Redis-backed temporary lookup store (single node or cluster) read through a codec.	reference

When a component is added or removed, update the JSON inventory and its detailed page in the same pull request.