AWS extended Glue Data Quality on July 27 to run ML-powered anomaly detection on Data Catalog tables and write evaluation results to Apache Iceberg tables queryable via standard SQL.
Back to NewsAWS extended Glue Data Quality on July 27 with two new capabilities. Anomaly detection now works on AWS Glue Data Catalog tables. Evaluation results can also be written to Data Catalog tables in Apache Iceberg format and queried with standard SQL.
Both features are available in all AWS commercial regions and AWS GovCloud (US) regions.
AWS Glue Data Quality now supports anomaly detection for Catalog-based evaluations. You enable it by setting ObservationScope: ALL. That flag applies ML-powered time-series forecasting to statistics on each Catalog table.
The model requires at least three data points before it flags anything. It learns patterns over time. It then computes upper and lower prediction bounds. When an actual statistic falls outside that range, the service creates an Anomaly Observation. Detectable shifts include sudden drops in distinct values or row-count spikes. No manually written threshold rule is needed.
How the model handles confirmed anomalies matters. When an anomaly is detected, it is treated as normal for subsequent runs unless explicitly rejected. As Metrotechs analysis, that behavior means uncorrected false positives can become baselines over time. AWS provides feedback tools in AWS Glue Studio and via API so teams can accept or reject anomalies to retrain the model.
On the results storage side, evaluation runs can now write four result types to Data Catalog tables. Those types are rule pass/fail outcomes, profiling metrics, row-level results, and observation results. Observation results include anomaly predictions with confidence bounds. Each result type writes to a separate Iceberg table. The rule results, profiling, and observation tables are partitioned by catalog ID, database name, table name, and day. That structure supports efficient time-range queries.
Results storage is disabled by default. Each result type requires explicit enablement. You use the --additional-run-options parameter of the StartDataQualityRulesetEvaluationRun API. ETL jobs use the additional_options parameter instead. The announcement does not state whether enabling results storage changes the cost of evaluation runs. Verify pricing against current AWS Glue Data Quality documentation before broad rollout.
Data engineering teams managing hundreds of Catalog tables face a real problem. Rule-based quality checks go stale. When business conditions change, fixed thresholds become outdated. Fixed criteria stop detecting issues even as data quality deteriorates. Pipeline failures can then go undetected until downstream users notice bad data.
Anomaly detection addresses this directly. It learns from historical statistics. It does not rely on a human to set and update every bound.
The Catalog results storage capability closes a different gap. Quality outcomes were not automatically saved in a queryable, central location. Teams can now build dashboards with visualization tools and maintain a centralized history of quality outcomes across their account. The Iceberg tables support standard SQL. The documentation does not state whether extra connectors are needed for Athena, BI tools, or notebook environments.
These capabilities work across both ETL jobs and Catalog evaluations. A team does not need separate monitoring for each workflow type. That consistency matters for estates that mix both.
Data engineering, analytics platform, and governance leaders should work through several questions before enabling these features across a production estate.
IAM permissions. The IAM role used for evaluation must have permission to create and update databases and tables in the Data Catalog. It must also have permission to write to the S3 location where Iceberg table data is stored. Audit existing evaluation roles against these requirements. The documentation does not state whether these permissions are added to existing roles automatically.
S3 storage location and lifecycle. Each result type writes Iceberg data to S3. If no CatalogTableConfig is provided, AWS Glue Data Quality derives default values automatically, including the S3 path. For production, specify locations explicitly and define retention policies. The announcement does not state default retention periods for results data.
Anomaly detection training history. The model needs at least three runs before producing anomaly signals. Tables with infrequent evaluation schedules take longer to reach useful detection. Identify which Catalog tables have enough run history before enabling ObservationScope: ALL estate-wide.
False-positive feedback loop. The ML algorithm treats a detected anomaly as a normal value in future runs unless explicitly excluded. Establish a review process so teams can reject false positives promptly. Without that process, the model may drift on busy or irregular tables.
Row-level results volume. Row-level result configuration accepts a MaxRowsToWrite parameter and a ResultType filter of FAILED_ONLY, PASSED_ONLY, or ALL. On large tables with high failure rates, uncapped row-level results can generate significant S3 write volume. Set explicit caps that match your storage and cost tolerances.
Watch signal. Run Catalog evaluations on a fixed schedule. Monitor the first three to five runs after enabling anomaly detection. Track observation volume and false-positive rate. Adjust analyzer scope and feedback before expanding to additional tables.
