Apache Spark

Apache Spark 4.3 is a major update to the open-source unified analytics engine, now available in Databricks Runtime 19. It introduces stateful processing in Real-Time Mode for sub-150ms latency, history-preserving SCD Type 2 support for Auto CDC, and Project Feather for optimized small-data performance.

Apache Spark 4.3 is a major update to the open-source unified analytics engine, now available in Databricks Runtime 19. It introduces stateful processing in Real-Time Mode for sub-150ms latency, history-preserving SCD Type 2 support for Auto CDC, and Project Feather for optimized small-data performance.

• 01Stateful Real-Time Mode with sub-150ms P99 latency
• 02SCD Type 2 support in Auto CDC for automated history tracking
• 03Project Feather for reduced execution overhead on small datasets
• 04Expanded Spark Connect observability and JDBC integration
• 05New SQL features including VARIANT transformations and JSON_TABLE