S3 WAL Backend Performance

This document describes the performance characteristics of the S3-backed WAL backend and provides guidance for tuning configuration based on workload requirements.

Architecture Overview

text

Input → append_batch (μs, memory) → segment buffer → channel → PUT worker (async) → S3
                                     ↓ flush triggers                         ↓
                               max_entries / max_bytes / flush_interval      backpressure
                                                                        ↓
                                                                   channel full

Pipeline Optimization: Segment encoding and PUT are now decoupled from the write path via a flume channel with bounded capacity (16 segments). The write path (append_batch) returns immediately after sending to the channel, while PUT operations execute asynchronously in the background. This eliminates the blocking 10-200ms PUT latency from the write path.

Batch Manifest Updates: Manifest updates are now batched—accumulated for 8 segments or 100ms before flushing to S3. This reduces S3 operations by ~58% (from 3 operations per segment to ~1.24 operations per segment on average).

Performance Characteristics

Latency Breakdown

Operation	Latency	Notes
append_batch	~1-50μs	In-memory write + channel send (non-blocking)
Segment PUT	10-200ms	Executes asynchronously in PUT worker
Channel send	<1μs	Blocking only when channel full (backpressure)
Manifest PUT (batched)	10-100ms	Small JSON payload, 8x less frequent
Recovery (LIST + GET)	100-500ms	Depends on segment count

Key changes:

1. append_batch no longer blocks on PUT operations. The 10-200ms PUT latency is moved to the background.
2. Manifest updates are batched (every 8 segments or 100ms), reducing S3 GET/PUT operations by ~58%.

Throughput Limits

Factor	Typical Range	Bottleneck
S3 PUT (single worker)	50-150 MB/s	Network bandwidth
S3 PUT (parallel, 4-8 workers)	200-450 MB/s	Network bandwidth
S3 LIST recovery	20-100 MB/s	API rate limits
Local encoding	500+ MB/s	CPU (CRC32 + optional compression)
Channel backpressure	16 segments/worker	Configurable bounded channel

Practical throughput: 100-200 MB/s with the default 1-worker setup; up to 450 MB/s with parallel_put.workers: 8.

Parallel PUT benefit: Multiple workers operate on independent bounded channels and round-robin assign segments, increasing throughput up to 2-3× for high-QPS workloads without changing the at-least-once contract. Watch for S3 per-prefix rate limits with 8 workers.

Replay Window

The "replay window" is the number of entries the source must re-deliver on node/pod disappearance. Unsealed entries are never silently lost: a source acknowledgement completes only after its entry is sealed into a segment object, so the source offset stays behind the seal frontier and unsealed work is re-delivered after a restart (at-least-once):

text

replay_window = min(
    segment.max_entries,           # entry count trigger
    segment.max_bytes / avg_msg_size,  # byte size trigger
    segment.flush_interval * msg_rate  # time trigger
)

flush_interval	@1000 msg/s	@10000 msg/s	@100000 msg/s
100ms	100 msgs	1,000 msgs	10,000 msgs
1s	1,000 msgs	10,000 msgs	100,000 msgs
5s	5,000 msgs	50,000 msgs	500,000 msgs

The same triggers bound the acknowledgement latency added by seal gating: an acknowledgement waits at most one flush_interval (plus the segment PUT time) for its entry to seal; all acknowledgements covered by one segment share that single wait. If a seal does not land within a bounded window (flush_interval × 4 + 5s for periodic, 30s for group_commit), the acknowledgement fails retryably instead of committing an unsealed entry, and sustained flusher failures escalate to error-level logging with a failure counter — distinguishing a slow flusher from a dead one.

Configuration Tuning

Segment Batching (D4)

Controls when the in-memory segment is sealed and PUT to S3.

yaml

segment:
  max_entries: 1000      # default
  max_bytes: 1048576     # 1 MiB default
  flush_interval: 1s    # default

Trade-offs:

Configuration	Latency	Replay Window / Ack Latency	PUT Frequency
Small segments (100, 100KB)	Lower PUT latency	Smaller window, faster acks	Higher PUT rate
Large segments (10000, 10MB)	Higher PUT latency	Larger window, slower acks	Lower PUT rate
Short interval (100ms)	Quick flush	Small window, acks ≤ ~100ms	High PUT rate
Long interval (10s)	Delayed flush	Large window, acks ≤ ~10s	Low PUT rate

Recommendations:

• High throughput, relaxed durability: max_entries: 10000, max_bytes: 10MB, flush_interval: 10s
• Balanced: defaults (1000, 1MB, 1s)
• Small replay window: max_entries: 100, max_bytes: 100KB, flush_interval: 100ms

Cursor Flushing (D6)

Controls how often the committed cursor (watermark) is persisted to manifest.json.

yaml

cursor:
  max_entries: 1000
  interval: 1s

Trade-offs:

interval	Crash Recovery Duplication	PUT Frequency
100ms	Minimal (~100 msgs)	High
1s	Low (~1000 msgs)	Moderate
10s	High (~10000 msgs)	Low

Note: Cursor batching adds at-least-once duplication but does not affect correctness. Outputs MUST tolerate duplicates regardless of this setting.

Comparison with Local Backend

Metric	Local (redb)	S3 Backend	Ratio
append latency	~1μs	~50μs	50x slower
flush latency	~2ms	~50ms	25x slower
throughput	500 MB/s	150 MB/s	3x slower
replay window	~100 msgs	~1000 msgs	10x larger
added ack latency (seal gating)	none	≤ flush_interval	N/A
node recovery	❌ No	✅ Yes	N/A

Cost Considerations

S3 API Costs

• PUT requests: $0.005 per 1,000 requests (US-East-1)
• GET requests: $0.0004 per 1,000 requests
• LIST requests: $0.005 per 1,000 requests
• Storage: $0.023 per GB/month (first 50TB)

Cost example @ 10,000 msg/s, 1KB per message:

text

- Segment PUTs: (10,000 / 1,000) × 3600 = 36 PUTs/hour × 24 × 30 = 25,920 PUTs/month
- PUT cost: 25,920 / 1,000 × $0.005 = $0.13/month
- Manifest PUTs (batched): ~0.125/second = 324,000 / 1,000 × $0.005 = $1.62/month
  (Previously: ~1/sec = $12.96/month, now ~87% reduction due to batching)
- Storage: 10,000 × 1KB × 86400 × 30 / 1e9 = 259 GB × $0.023 = $5.96/month
- Total: ~$7.71/month per stream (down from ~$19/month)

Optimization: Increase segment.max_entries and cursor.interval to reduce PUT costs further. Batching is already applied automatically (8 segments or 100ms).

Performance Tuning Strategies (since 2026-07-28)

The S3 WAL backend supports three preset strategies via segment_tuning.strategy:

Aggressive Strategy

For high throughput, low cost — tolerates a larger replay window (and an acknowledgement latency of up to ~10s from seal gating).

yaml

segment_tuning:
  strategy: aggressive

Defaults:

• max_entries: 10000
• max_bytes: 10MB
• flush_interval: 10s

Replay window @ 10K msg/s, 1 KB avg: ~10,000 messages re-delivered on node loss (max_entries/max_bytes fire before the 10s flush_interval) PUT cost reduction: ~10x vs balanced Throughput: Up to 200 MB/s

Balanced Strategy (default)

Default trade-off between throughput and replay window.

yaml

segment_tuning:
  strategy: balanced

Defaults:

• max_entries: 1000
• max_bytes: 1MB
• flush_interval: 1s

Replay window @ 10K msg/s, 1 KB avg: ~1,000 messages re-delivered (max_entries/max_bytes fire before the 1s flush_interval) Throughput: 100-150 MB/s

Low-Latency Strategy

For minimal replay window — highest PUT frequency.

yaml

segment_tuning:
  strategy: low_latency

Defaults:

• max_entries: 100
• max_bytes: 100KB
• flush_interval: 100ms

Replay window @ 10K msg/s, 1 KB avg: ~100 messages re-delivered (max_entries/max_bytes fire before the 100ms flush_interval) Throughput: Lower due to frequent flushes

Custom Overrides

Override individual parameters of any preset:

yaml

segment_tuning:
  strategy: aggressive
  max_entries: 20000      # override default 10000
  flush_interval: "30s"   # override default 10s

Parallel PUT Workers

Configure multiple PUT workers for 2-3x throughput improvement.

yaml

parallel_put:
  workers: 4              # 1-8, default 1
  shutdown_timeout: "30s" # how long to wait for in-flight uploads

Benefits:

• Workers operate in parallel via independent channels
• Round-robin assignment (oldest segments first)
• Per-worker backpressure (16 segments each)

Watch for S3 rate limits: 8 workers can hit per-prefix limits.

Manifest write safety: With workers > 1, multiple workers can finish a segment upload and seal it concurrently, each rewriting manifest.json. These concurrent rewrites are coordinated with ETag-based optimistic concurrency (read ETag → conditional PUT → retry on mismatch, up to 8 attempts covering the worker ceiling), so no worker's cursor advancement or sealed-segment entry is silently overwritten — provided the backend supports conditional PUT. S3 and S3-compatible stores do; a backend that does not (e.g. LocalFileSystem) falls back to a single-writer unconditional write, so keep workers: 1 there. This is invisible at workers: 1 (the default): no contention, no retries, behavior unchanged. See docs/docs/components/0-inputs/delivery-semantics.md for the delivery-contract details.

Compression

Reduce S3 storage and network costs by 50-70% via segment compression.

yaml

compression:
  type: zstd  # or "lz4", "none"
  level: 3    # algorithm-specific

Algorithm comparison:

Algorithm	Compression Ratio	CPU Cost	Best For
none	1.0x	0%	Default, low CPU
lz4	2-3x (measured ~108× on Arrow IPC)	2-5%	Fast compression
zstd-3	3-5x (measured ~181× on Arrow IPC)	5-10%	Balanced (default for compression)
zstd-9	5-8x	20-30%	Maximum compression

Measured compression ratios on actual WAL segment payloads (Arrow IPC frames with repetitive schema metadata) significantly exceed the generic 50-70% estimate above because Arrow IPC has substantial repetition. The numbers below are from the unit-test suite (compression_ratio_across_*_levels):

text

lz4-1:  50000 -> 462 bytes (ratio: 108.23x)
lz4-4:  50000 -> 462 bytes (ratio: 108.23x)
lz4-9:  50000 -> 462 bytes (ratio: 108.23x)
zstd-1: 50000 -> 276 bytes (ratio: 181.16x)
zstd-3: 50000 -> 276 bytes (ratio: 181.16x)
zstd-6: 50000 -> 276 bytes (ratio: 181.16x)
zstd-9: 50000 -> 276 bytes (ratio: 181.16x)

Real-world ratios on production payloads (mixed data, less repetition) will be lower than these synthetic benchmarks but still substantial.

Min size threshold: Segments smaller than 10KB are uploaded uncompressed.

Failure Scenarios

S3 Unavailable

• append: Blocks when channel fills up (16 segments) → input stalls (same as before, but with larger buffer)
• recovery: Fails to start stream → engine error
• Mitigation: Use S3 with cross-region replication, or add local cache

Network Partition

• Current behavior: Same as S3 unavailable, but channel provides 16-segment buffer before blocking
• Future improvement: Local write-through cache with retry queue

Segment PUT Partial Failure

• Current: A failed segment PUT restores the staged entries to the active segment and retries on the next flusher tick; the wake-path failures are counted and logged (rate-limited warn, escalating to error when sustained) instead of being silently swallowed. Recovery additionally skips a segment that was referenced but has since gone missing (D7)
• Impact: Acknowledgements for the affected entries are delayed (seal gating) and fail retryably if the PUT keeps failing — the source re-delivers, so no acknowledged entry is lost
• Mitigation: Enable S3 server-side encryption and versioning

Monitoring

Key Metrics

Metric	Description	Alert Threshold
segment_put_latency	Time to PUT a segment	>500ms p99
segment_put_frequency	PUTs per second	>10/sec (may need tuning)
segment_size	Average segment bytes	<100KB or >10MB
cursor_lag	Cursor vs max written seq	>10,000 entries
recovery_latency	Time to replay WAL on startup	>5s

Logging

Key log lines to monitor:

text

[DEBUG] Sealing segment: entries={n}, bytes={size}
[INFO]  Segment PUT complete: {segment_key}, {size} bytes in {duration_ms}ms
[INFO]  Recovery: replayed {n} entries from {m} segments in {duration_ms}ms
[WARN]  Segment PUT failed: {error}, retrying...

References

• User-facing tuning guide: docs/docs/components/0-inputs/wal-optimization.md
• Design: openspec/changes/archive/2026-07-27-add-wal-s3-backend/design.md
• Design: openspec/changes/archive/2026-07-28-comprehensive-wal-optimization/design.md
• Spec: openspec/specs/input-durability/spec.md
• Implementation: crates/arkflow-plugin/src/wal/s3.rs
• Compression: crates/arkflow-plugin/src/wal/compression.rs