Skip to main content
Version: 0.6.x

Recovery

ArkFlow has two independent recovery mechanisms, and knowing which one covers your failure is half the runbook:

MechanismProtectsTrigger
WAL replayEvery message that entered a durability-enabled streamAutomatic on startup
Checkpoint / savepoint restoreStateful job state: source positions, watermarks, keyed state, window stateAutomatic on restart (checkpoints) or explicit (savepoints)

What happens automatically​

Stream crash (WAL replay)​

When durability is enabled for a stream, every message is persisted and fsync'd to the write-ahead log before it enters the pipeline. The WAL cursor advances only through the highest contiguous acknowledged sequence, and the source commit happens only after the output confirms the write.

On restart the engine:

  1. Reconciles WAL entries already covered by the restored source position and advances the cursor past that covered prefix.
  2. Replays every WAL entry past the committed cursor before streams resume.

The result is at-least-once delivery: an outage window is neither skipped nor silently dropped — it is replayed. Duplicate handling, when you need end-to-end exactly-once, is covered in exactly-once delivery.

Out-of-order and fan-out acknowledgements never skip a gap: if sequence N+1 acks before N, the durable cursor stays before N until the gap closes.

Stateful job crash (checkpoint restore)​

A completed checkpoint identifies the job version, task assignments, source positions, watermark state, state snapshots, format versions, and integrity checksums — all from one acknowledged cut. In-flight records whose downstream acknowledgement had not completed at barrier injection are excluded from the cut and replayed from the recorded source position on restore.

On compute-node restart the job restores from the last valid checkpoint, replays the required source range, and reports recovery progress before becoming healthy. Incomplete, corrupt, checksum-invalid, or task-set- incomplete checkpoints are never selected; the previous valid checkpoint is retained until its replacement is durably committed (manifests are written through a temporary file and an atomic replace).

Source reconnects resume from the source's acknowledged cursor — explicitly carried offsets, not auto.offset.reset — so recovery is deterministic.

Operator procedures​

Trigger a checkpoint or savepoint​

# Barrier checkpoint (automatic recovery point)
curl -X POST -H "Authorization: Bearer $TOKEN" \
  https://hub.example/api/v1/jobs/{id}/checkpoints

# Savepoint (explicit, kept for upgrade/rollback)
curl -X POST -H "Authorization: Bearer $TOKEN" \
  https://hub.example/api/v1/jobs/{id}/savepoints

# List recovery artifacts
curl -H "Authorization: Bearer $TOKEN" \
  https://hub.example/api/v1/jobs/{id}/checkpoints

Roll a job back to a savepoint​

Upgrades require the job to be stopped and converged. If a new version misbehaves, roll back to a compatible savepoint:

curl -X POST -H "Authorization: Bearer $TOKEN" \
  https://hub.example/api/v1/jobs/{id}/upgrades/{upgrade_id}/rollback

Recovery artifacts are bound to the job version and state format version; an incompatible artifact is rejected before restore instead of corrupting state.

Verify recovery completed​

  1. Watch convergence on the job detail: the job reports recovery progress before becoming healthy.
  2. Check GET /api/v1/jobs/{id}/detail for restored source positions.
  3. Confirm checkpoint health — after a failed snapshot round the previous valid checkpoint remains selected and the job reports degraded checkpoint health; investigate rather than waiting for it to self-heal.

Failure scenarios​

SymptomCauseAction
Data repeats after restartAt-least-once replay from WAL/source cursorExpected; deduplicate downstream or enable exactly-once output.
Data lost after restartDurability was not enabled for the streamEnable durability; without a WAL the source commit is the only guarantee.
Job stuck "recovering"Checkpoint restore or source replay in progressInspect detail; large state or deep replay windows take time.
Checkpoint rounds keep failingA task cannot snapshot or checksums mismatchCheck node disk/object-store health; the last valid checkpoint is still in force.
Repeating WAL acknowledgement parked for >60s ... errorsThe lowest unacknowledged sequence's owner stalled or died without recording a failureInspect the task that owns the sequence named in the error; the delivery is retried or replayed once that owner settles or the stream restarts.
Rollback rejected as incompatibleSavepoint belongs to another version or state formatUse an artifact from the compatible version history.

Graceful shutdown​

On a graceful shutdown, a durability-enabled stream's WAL close no longer fails the stream just because an acknowledgement was parked behind an earlier in-flight source commit: the parked acknowledgement gets a bounded drain window (30 seconds) to settle once the earlier delivery completes. If the window expires without settling, the stream surfaces the WAL closed while acknowledgement was pending error and the usual WAL replay covers the unacknowledged delivery on restart — at-least-once semantics are unchanged either way.

Parked acknowledgements while the stream runs​

The close-time drain window above has a running-time counterpart. While the stream is up, an acknowledgement for sequence N+1 whose predecessor N has not settled yet is parked: it waits instead of skipping the gap, so the durable cursor never moves past an unacknowledged sequence. That wait is bounded by a 60-second park lease. If the lease expires — the owner of the lowest unsettled sequence stalled or died without recording a failure — the parked acknowledgement surfaces an explicit retryable error instead of waiting forever:

WAL acknowledgement parked for >60s waiting for sequence N; the gap owner appears stalled

The lease deliberately writes no failure fence: an owner that was merely slow (for example waiting out one object-store segment seal) is never blocked from settling, and once it does, a retry of the parked acknowledgement proceeds normally with no added delay. If the owner is truly dead, the error repeats on every retry — a visible periodic failure rather than a silent stall with no errors and no progress. Investigate the task that owns the sequence named in the error; on restart, the usual WAL replay covers the unsettled entries.

TLS support matrix​

Every network component's encrypted-transport enablement, in one place:

ComponentTLS enablement
kafka (input/output)rdkafka security.protocol / security.protocol config
mqtt (input/output)tls block (enabled, ca, client_cert, client_key)
nats (input/output)tls:// URL scheme (negotiated natively by async-nats)
pulsar (input/output)pulsar+ssl:// URL scheme
redis (input/output/temporary)rediss:// URL scheme
sql (output)sqlx TLS (Postgres sslmode, MySQL ssl-mode) via connection string
mongodb (output)mongodb+srv / tls=true connection string
http (input/output)https:// URL (TLS terminated by reqwest/hyper)
websocket (input)wss:// URL
qdrant / milvus / pgvector / sql (outputs) and embedding / llm / vector_search / pgvector_search / milvus_search (processors)https:// endpoint (reqwest / sqlx TLS)
secret references${secret:NAME} resolved from the Hub environment at dispatch; env:/file: resolve node-local

Components without network capability (memory, generate, drop, stdout, batch/json/sql/vrl/python processors) have no TLS surface.

WAL backends​

The local filesystem store is the default. For shared or remote durability, the object_store WAL backend writes to S3-compatible storage — see S3 WAL backend performance for the design and measured trade-offs.