Why observability needs Apache Iceberg

"The breakthrough Iceberg achieves for observability is that it lets you keep logs, metrics, and traces as data sets in the same lakehouse that already holds business data. This enables you to explore telemetry with a SQL engine, a notebook, or your existing BI tools without transferring terabytes of data. The glue code and format translation steps that create drift go away."

"Iceberg's seamless schema evolution is well-matched with observability's frequent schema changes. Adding a new label, renaming a field there, or adding a late-arriving dimension from an upstream service is no longer a big deal. Iceberg's hidden partitioning features let you add or rename columns and adjust partition specifications over time without rewriting historical data or breaking queries. It's a much better fit for high-cardinality telemetry than rigid, pre-declared schemas."

Iceberg enables storing logs, metrics, and traces as datasets alongside business data in a lakehouse, allowing direct exploration via SQL engines, notebooks, or BI tools without moving terabytes. Seamless schema evolution handles frequent telemetry changes such as added labels, field renames, or late-arriving dimensions. Hidden partitioning lets teams add or rename columns and change partition specs without rewriting historical data, fitting high-cardinality telemetry. The manifest and snapshot model provides atomic commits, compaction, and data skipping to sustain write pressure while keeping reads predictable and supporting backfills, deletes, and retention. Time travel permits querying prior table states to compare revisions. The open format enables cross-team access and lake-level governance, removing export/import loops.

Read at InfoWorld