How Plank writes your lake#

Plank turns source records into immutable Parquet files and staged table operations. The catalog applies those operations to Iceberg metadata, so a source run can publish a new snapshot without modifying a file that a reader already uses.

Files are immutable#

Rows enter a typed table stream. Plank validates each value against the closed lake kind set, forms bounded row groups, and writes canonical Parquet. Once a file is staged, its bytes do not change.

Updates therefore create new data or delete files and a new Iceberg snapshot. Old snapshots keep referring to their old files until retention proves those files unreachable.

Operations precede commits#

A completed stream produces an operation that names the lake, namespace, table, schema, files, and merge intent. Staging does not make the data visible. The catalog first validates identity and expected state, then publishes metadata through pointer arbitration.

This separation makes retries safe. A lost response can be reconciled against exact staged evidence. Unknown commit state is never reported as a clean conflict, because deleting files after an uncertain commit could destroy data that actually became live.

Merge by key#

Incremental source tables merge incoming rows by their declared key. The new snapshot retains the latest row for each key and carries deletion markers when the source proves absence.

Reconciliation can splice unchanged canonical Parquet row groups byte-for-byte and re-encode only dirty groups. This is an optimization, not a semantic branch: schema mismatch or unsupported layout falls back to a complete rewrite.

Checkpoints and recovery#

Long source reads seal bounded cycles. A crash abandons an uncommitted stream, while the next run resumes from the last durable source position. The current stream identity prevents two attempts from publishing through the same in-memory handle.

Staged operations remain replayable until the catalog accepts or definitively refuses them. Cleanup follows reachability from verified metadata, not file age alone.

Encryption and integrity#

Plank owns the file-format encryption boundary and records the metadata required for authorized readers. Readers receive scoped, expiring storage credentials rather than a bucket-wide identity.

Every persisted schema, key, statistic, and file reference is treated as untrusted input on read. Bounds are checked before allocation. A malformed file or metadata document is a refusal, never a reason to guess a type or skip integrity checks.

The result is a physical contract you can observe as atomic snapshots, durable history, merge-by-key behavior, and Parquet exports—without needing to coordinate readers with a source run.