Delta Lake

Delta Lake is an open-source storage framework that enables building Lakehouse architectures with ACID transactions and scalable metadata handling on data lakes.

Use it when

• •ACID compliance needed: When you need Atomicity, Consistency, Isolation, and Durability for data lake operations.
• •Time travel requirements: When you need to query previous table versions, perform audits, or rollbacks.
• •Mixed batch/streaming: When you need unified batch and streaming data processing.
• •Schema evolution: When your data schemas change over time and you need enforcement and evolution capabilities.
• •Multi-engine access: When you need to access the same data with different compute engines (Spark, Flink, Trino, etc.).
• •Large-scale data operations: When working with massive datasets that require reliable, consistent updates.
• •Data reliability critical: When data corruption or partial updates could have significant business impact.

Watch out

• ⚠Multi-table transaction limitations: Delta Lake does not support multi-table transactions and foreign keys - transactions work only at the table level.
• ⚠Version compatibility issues: While backward compatible, forward compatibility may break with new features.
• ⚠Concurrency control problems: Metadata update exceptions occur during concurrent transactions.
• ⚠Storage dependency: ACID guarantees depend on the underlying storage system's atomicity and durability guarantees.
• ⚠S3 limitations: Delta Lake on S3 has several limitations not found on other storage systems.
• ⚠Spark dependency: Primarily designed for Spark ecosystem, though other engines are supported.

Available in stages

Installation

Example stacks