Production Apache Iceberg in 2026: The Practitioner Playbook

Iceberg's interesting phase is over. The spec is solid, the ecosystem is mature, and most of the people I talk to don't need another piece on why the open lakehouse matters — they need to keep theirs running without getting paged at 3 a.m. because query planning suddenly takes forty seconds.

This is what we've learned from doing exactly that. Hard lessons, specific thresholds, and the operational patterns that actually work in 2026.

The current state (May 2026)#

Apache Iceberg has matured significantly. The V3 spec — "Extended Types and Capabilities" — is finalised, with Deletion Vectors, Row Lineage, and the VARIANT type as the headline additions. Engine support is rolling out unevenly though, and that matters for planning:

• Databricks has Deletion Vectors generally available (V2 reads break against tables with them on, so plan migrations).
• Snowflake has V3 in public preview — Variant and Row Lineage work today, default values and the geography type don't yet.
• Trino's Iceberg connector currently lists V3 as experimental: column default values and Deletion Vector reads aren't supported.
• Spark picks up V3 features release by release; check what your Iceberg version actually implements before committing a workload to it.

The features are real and worth planning for. Just check engine compatibility before you bet on them.

The REST Catalog has effectively won as the default for new deployments. It decouples your table format from any single vendor and gives you proper governance and multi-engine access without the old Hive Metastore headaches.

Branching and tags have been GA for several releases now and are mature enough for real workflows. We use them for feature development, safe experimentation, and controlled releases.

But the operational side is where most teams still struggle. The specification is solid. The ecosystem is mature. The thing that breaks is almost always how teams operate it.

The #1 production killer: small files + metadata bloat#

This is still the single biggest issue we see in 2026.

Every streaming or CDC write creates new Parquet files. Without regular compaction you end up with tables that have hundreds of thousands or millions of tiny files. Query planning time explodes. Metadata.json files grow into the hundreds of megabytes. Costs go up. Performance goes down.

Rules of thumb we use:

• Above ~1,000–1,500 files, start paying attention.
• Above 5,000 files, query performance usually starts degrading noticeably.
• Above 20,000 files, you're in pain.

Monitor SELECT COUNT(*) FROM my_table.files (or the equivalent metadata table in your engine). Set alerts at 2,000 and 8,000 files. iOmete's January 2026 write-up on Iceberg production anti-patterns — particularly the "Small Files Death Spiral" section — is worth reading if you want a deeper map of the failure mode.

Catalog strategy in 2026 — REST Catalog wins#

We used to see teams default to AWS Glue or Hive Metastore. In 2026, for most new deployments, we strongly recommend a proper REST Catalog implementation (Apache Polaris, Project Nessie, or a managed offering).

Why REST Catalog won:

• True multi-engine support without vendor lock-in
• Better security model (OAuth, fine-grained access)
• Cleaner separation between storage and metadata
• Easier to run in multi-cloud or hybrid environments

We helped one client migrate from Glue to a REST Catalog (Polaris). The biggest gain wasn't performance — it was operational simplicity. They could finally give different teams access to different tables without fighting IAM policies, and they stopped having mysterious "table not found" issues when switching between Spark and Trino.

Still using Glue? Reasonable for pure-AWS environments where the team is small and already deep in the AWS ecosystem. Even then, we often put a REST Catalog in front of it.

Compaction — the non-negotiable discipline#

Compaction is not optional. It's the single most important operational process for any serious Iceberg deployment. Conduktor's guide on maintaining Iceberg tables is a solid reference for the full set of operations (compaction + snapshot expiration + orphan cleanup).

What we actually do in production:

1. Bin-pack compaction for general file size management (target 128–512 MB files depending on workload).
2. Sort-based compaction on high-cardinality columns that are frequently filtered (e.g. user_id, event_date, customer_id).
3. Run it on a dedicated cluster or with resource limits so it doesn't starve interactive queries.
4. Use branch-specific compaction where possible.

A real example, Spark + Iceberg procedures:

-- Bin-pack compaction with a 256 MB target file size
CALL my_catalog.system.rewrite_data_files(
  table => 'prod.events',
  options => map(
    'target-file-size-bytes', '268435456',
    'max-concurrent-file-group-rewrites', '4'
  )
);
 
-- Sort-based compaction on high-cardinality filter columns.
-- sort_order and strategy are top-level arguments;
-- only file-level tuning goes inside options.
CALL my_catalog.system.rewrite_data_files(
  table       => 'prod.events',
  strategy    => 'sort',
  sort_order  => 'event_date DESC NULLS LAST, user_id ASC NULLS FIRST',
  options     => map(
    'target-file-size-bytes', '268435456'
  )
);

A note on PyIceberg: the Python client does not yet expose a data-file compaction call — that's tracked as planned work in the project. What it does expose, and what we use from orchestration tools like Dagster or Prefect, is the rest of the maintenance lifecycle. Snapshot expiration is the most common one to schedule:

from datetime import datetime, timedelta, timezone
from pyiceberg.catalog import load_catalog
 
catalog = load_catalog("prod")
table = catalog.load_table("prod.events")
 
# Expire snapshots older than 14 days as part of scheduled maintenance.
cutoff = datetime.now(timezone.utc) - timedelta(days=14)
table.maintenance.expire_snapshots().older_than(cutoff).commit()

For compaction itself, the pragmatic pattern in 2026 is to call the Spark procedure from your orchestrator, or use a managed table-maintenance service. The most common mistake at this point isn't technical, it's operational: running compaction on the same cluster as your main workloads. Compaction is I/O and CPU heavy. It will steal resources from your users and cause query timeouts. Always run it on a separate pool, or during low-traffic windows.

Branching and tags — git-style workflows for data#

This is one of the most under-used features in Iceberg, and it's now mature enough to rely on. lakeFS's guide to Iceberg branching is a good practitioner reference for the full pattern, including the difference between table-level branching and catalog-level coordination.

Patterns we actually use:

• Feature branches. A data engineer creates a branch for a major schema change or backfill, tests, validates, then merges.
• Release branches. Cut a release branch before a big deployment. If something goes wrong, point production at the previous branch.
• Experiment branches. Data scientists experiment with transformations without affecting main.
• Tags for compliance. Tag specific snapshots for audit or regulatory purposes (e.g. end-of-quarter-2026-Q1).

Keep branches short-lived. We've seen teams create a branch for a three-week project, then struggle with merge conflicts because main had moved on — new partitions, schema changes, compaction. The longer a branch lives, the more painful the merge becomes.

Two layers of branching are worth keeping straight. Iceberg's native, table-level branching is part of standard Spark DDL — useful when you want isolated work on a single table:

-- Create a branch on a single Iceberg table
ALTER TABLE prod.events CREATE BRANCH `feature_user_segmentation`;
 
-- Read or write against the branch by name
SELECT * FROM prod.events VERSION AS OF 'feature_user_segmentation';
 
-- Promote the branch to main by repointing main at its head
ALTER TABLE prod.events REPLACE BRANCH `main`
  AS OF VERSION 4892412345    -- snapshot id from the feature branch
  RETAIN 7 DAYS;
 
-- Tear it down when you're done
ALTER TABLE prod.events DROP BRANCH `feature_user_segmentation`;

Iceberg itself does not implement a MERGE BRANCH operation across tables; the spec gives you per-table snapshots, not a multi-table merge primitive. Catalog-level coordination across many tables — the actual git-style "branch the whole catalog, work, then merge back" workflow — is what Project Nessie and lakeFS provide on top of Iceberg. If your work spans more than one table, that's where to look.

Iceberg V3 features worth using now#

Deletion Vectors. Huge win for CDC and update-heavy workloads. Instead of rewriting whole files on delete or update, V3 represents deletes as compact bitmaps. Performance improvement is dramatic on large tables.

Row Lineage. Adds row-level identity and last-updated metadata automatically. Incredibly useful for debugging, audit, and regulatory work.

VARIANT type. Finally a clean way to store semi-structured data without exploding your schema or falling back to JSON strings.

One caveat: Deletion Vectors aren't magic. If you never compact, the delete files themselves can become a problem. Treat Deletion Vectors as a complement to, not a replacement for, regular compaction.

Streaming integration is the other thing that has matured#

If you're feeding Iceberg from Kafka, Kinesis or change-data-capture, the operational story has improved meaningfully in 2026. Engines like RisingWave now provide stable Iceberg sinks with exactly-once semantics, and the small-files / compaction story applies double for streaming workloads. RisingWave's April 2026 write-up is a useful reference for the streaming-to-Iceberg pattern, including the compaction and snapshot-expiration scheduling that needs to come with it.

Common 2026 anti-patterns we still see#

1. Streaming without automated compaction. The fastest way to create an unmanageable table.
2. Hive catalog in production. Still happens. Still causes pain with multi-engine access and permissions.
3. Ignoring manifest compaction. Small data files are obvious. Bloated manifest files are silent killers of query planning.
4. Mixing high-impact operations on long-lived branches. Schema evolution + heavy compaction + backfills on the same branch is merge-conflict hell.
5. No monitoring of metadata size. We've seen metadata.json files grow past 400 MB. Query planning suffers badly.

Recommended production stack (2026)#

• Catalog. REST Catalog (Polaris or Nessie), unless you have a very strong reason to use something else.
• Query engines. Trino for interactive and ad-hoc; Spark for heavy transforms; DuckDB for local development and small queries.
• Orchestration. Dagster or Prefect, with dedicated Iceberg maintenance jobs (compaction, snapshot expiration, orphan cleanup).
• Storage. S3 (or equivalent) with proper lifecycle policies.
• Observability. Track file count, metadata size, snapshot age, and average query planning time.

Final practical advice#

Iceberg is no longer the risky new thing. It's the solid foundation most serious lakehouse implementations are built on in 2026.

The teams that succeed are not the ones with the fanciest architecture diagrams. They're the ones who treat compaction, catalog strategy, and branching as first-class operational concerns — not afterthoughts.

Build the automation. Set the alerts. Keep the branches short. And never let a table grow past a few thousand files without intervention.