AWS released AWS Glue 6.0 on August 21, making it generally available across all AWS Commercial, AWS GovCloud (US), and AWS China regions.
TL;DR
- -AWS Glue 6.0 is generally available across all AWS Commercial, GovCloud (US), and China regions with a 30% price reduction compared to previous versions.
- -The release adds Apache Iceberg v3 support, including VARIANT type with automatic shredding, deletion vectors, spatial types, and nanosecond-precision timestamps, alongside a runtime upgrade to Spark 4.1, Python 3.13, and Scala 2.13.
- -No API changes are required; teams can adopt Glue 6.0 via the existing --glue-version parameter through the AWS CLI, SDK, Glue Studio, or SageMaker Unified Studio.
What Changed
The most immediate change is pricing. Glue 6.0 carries a 30% price reduction compared to previous versions. Existing jobs billed by the second for crawlers and ETL processing will see that reduction applied when they run on the new version.
AWS says the release delivers the most complete Iceberg v3 implementation on any fully serverless managed Spark service, built on Iceberg 1.11.0. Iceberg v3 is an open table format specification that governs how data is stored, updated, and read in modern data lake and lakehouse architectures.
Runtime and Format Upgrades
The underlying runtime moves to Apache Spark 4.1, Python 3.13, and Scala 2.13. For teams running existing Glue jobs, those version jumps affect compatibility planning before migration.
On the Iceberg v3 side, Glue 6.0 adds several capabilities:
- VARIANT data type with automatic shredding stores and queries JSON, logs, and event data without flattening schemas, removing the need for custom parsing code and duplicate data copies. AWS says shredding achieves faster query read performance on semi-structured data compared to traditional string column storage.
- Deletion vectors support high-performance row-level updates.
- Geometry and geography types enable native spatial processing for GIS analytics, location intelligence, and geospatial data pipelines directly on managed Spark.
- Nanosecond-precision timestamps support IoT sensor data, scientific computing, and high-frequency financial workloads that need precision beyond standard milliseconds.
- UNKNOWN type and DEFAULT column values allow flexible schema evolution so pipelines can process data with unexpected or evolving schemas without failing.
The release also adds updated support for newer versions of Apache Hudi and Delta Lake.
Developer and Performance Features
Three additions target pipeline authoring and runtime performance:
Spark Declarative Pipelines change how engineers write ETL logic. Instead of scripting execution order manually, data engineers declare transformations and the engine determines execution order and optimization automatically, reducing orchestration overhead.
Real-Time Mode streaming achieves single-digit millisecond latency for stateless streaming use cases. AWS describes this as built on Spark 4.1's Real-Time Mode with Glue-optimized execution.
Arrow-native Python UDFs (user-defined functions) and UDTFs eliminate serialization overhead between Python and the JVM, improving PySpark performance for complex transformations.
Getting Started
No API changes are required. Teams can select Glue 6.0 using the existing --glue-version parameter through the AWS CLI, AWS SDK, AWS Glue Studio, or Amazon SageMaker Unified Studio. Existing jobs can be migrated using the Spark Upgrade Agent in AWS Glue Studio or the auto-upgrade feature. New jobs can select Glue 6.0 from the version dropdown when created.
Sources and supporting resources