Skip to content

table daily recurring checks

TableTimelinessDailyRecurringChecksSpec

Container of table level daily recurring for timeliness data quality checks.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
daily_data_freshness Daily calculating the number of days since the most recent event timestamp (freshness) TableDataFreshnessCheckSpec
daily_data_staleness Daily calculating the time difference in days between the current date and the most recent data ingestion timestamp (staleness) TableDataStalenessCheckSpec
daily_data_ingestion_delay Daily calculating the time difference in days between the most recent event timestamp and the most recent ingestion timestamp TableDataIngestionDelayCheckSpec

TableAvailabilityDailyRecurringChecksSpec

Container of built-in preconfigured data quality checks on a table level that are detecting the table availability.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
daily_table_availability Verifies availability on table in database using simple row count. Stores the most recent table availability status for each day when the data quality check was evaluated. TableAvailabilityCheckSpec

TableComparisonDailyRecurringChecksSpec

Container of built-in comparison (accuracy) checks on a table level that are using a defined comparison to identify the reference table and the data grouping configuration. Contains the daily recurring comparison checks.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
daily_row_count_match Verifies that the row count of the tested (parent) table matches the row count of the reference table. Compares each group of data with a GROUP BY clause. Stores the most recent captured value for each day when the data quality check was evaluated. TableComparisonRowCountMatchCheckSpec

TableDailyRecurringCheckCategoriesSpec

Container of table level daily recurring. Contains categories of daily recurring.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
volume Daily recurring volume data quality checks TableVolumeDailyRecurringChecksSpec
timeliness Daily recurring timeliness checks TableTimelinessDailyRecurringChecksSpec
accuracy Daily recurring accuracy checks TableAccuracyDailyRecurringChecksSpec
sql Daily recurring custom SQL checks TableSqlDailyRecurringChecksSpec
availability Daily recurring table availability checks TableAvailabilityDailyRecurringChecksSpec
schema Daily recurring table schema checks TableSchemaDailyRecurringChecksSpec
comparisons Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. TableComparisonDailyRecurringChecksSpecMap
custom Dictionary of custom checks. The keys are check names. CustomCheckSpecMap

TableSqlDailyRecurringChecksSpec

Container of built-in preconfigured data quality checks on a table level that are using custom SQL expressions (conditions).

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
daily_sql_condition_passed_percent_on_table Verifies that a set percentage of rows passed a custom SQL condition (expression). Stores the most recent captured value for each day when the data quality check was evaluated. TableSqlConditionPassedPercentCheckSpec
daily_sql_condition_failed_count_on_table Verifies that a set number of rows failed a custom SQL condition (expression). Stores the most recent captured value for each day when the data quality check was evaluated. TableSqlConditionFailedCountCheckSpec
daily_sql_aggregate_expr_table Verifies that a custom aggregated SQL expression (MIN, MAX, etc.) is not outside the set range. Stores the most recent captured value for each day when the data quality check was evaluated. TableSqlAggregateExprCheckSpec

TableVolumeDailyRecurringChecksSpec

Container of table level daily recurring for volume data quality checks.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
daily_row_count Verifies that the number of rows in a table does not exceed the minimum accepted count. Stores the most recent captured value for each day when the data quality check was evaluated. TableRowCountCheckSpec
daily_row_count_anomaly_differencing_30_days Verifies that the total row count of the tested table changes in a rate within a percentile boundary during last 30 days. TableAnomalyDifferencingRowCount30DaysCheckSpec
daily_row_count_anomaly_differencing Verifies that the total row count of the tested table changes in a rate within a percentile boundary during last 90 days. TableAnomalyDifferencingRowCountCheckSpec
daily_row_count_change Verifies that the total row count of the tested table has changed by a fixed rate since the last readout. TableChangeRowCountCheckSpec
daily_row_count_change_yesterday Verifies that the total row count of the tested table has changed by a fixed rate since the last readout from yesterday. Allows for exact match to readouts from yesterday or past readouts lookup. TableChangeRowCountSinceYesterdayCheckSpec
daily_row_count_change_7_days Verifies that the total row count of the tested table has changed by a fixed rate since the last readout from last week. Allows for exact match to readouts from 7 days ago or past readouts lookup. TableChangeRowCountSince7DaysCheckSpec
daily_row_count_change_30_days Verifies that the total row count of the tested table has changed by a fixed rate since the last readout from last month. Allows for exact match to readouts from 30 days ago or past readouts lookup. TableChangeRowCountSince30DaysCheckSpec

TableSchemaDailyRecurringChecksSpec

Container of built-in preconfigured volume data quality checks on a table level that are executed as a daily recurring (checkpoint) checks.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
daily_column_count Detects if the number of column matches an expected number. Retrieves the metadata of the monitored table, counts the number of columns and compares it to an expected value (an expected number of columns). Stores the most recent column count for each day when the data quality check was evaluated. TableSchemaColumnCountCheckSpec
daily_column_count_changed Detects if the count of columns has changed since the most recent day. Retrieves the metadata of the monitored table, counts the number of columns and compares it the last known column count that was captured when this data quality check was executed the last time. Stores the most recent column count for each day when the data quality check was evaluated. TableSchemaColumnCountChangedCheckSpec
daily_column_list_changed Detects if new columns were added or existing columns were removed since the most recent day. Retrieves the metadata of the monitored table and calculates an unordered hash of the column names. Compares the current hash to the previously known hash to detect any changes to the list of columns. TableSchemaColumnListChangedCheckSpec
daily_column_list_or_order_changed Detects if new columns were added, existing columns were removed or the columns were reordered since the most recent day. Retrieves the metadata of the monitored table and calculates an ordered hash of the column names. Compares the current hash to the previously known hash to detect any changes to the list of columns or their order. TableSchemaColumnListOrOrderChangedCheckSpec
daily_column_types_changed Detects if new columns were added, removed or their data types have changed since the most recent day. Retrieves the metadata of the monitored table and calculates an unordered hash of the column names and the data types (including the length, scale, precision, nullability). Compares the current hash to the previously known hash to detect any changes to the list of columns or their types. TableSchemaColumnTypesChangedCheckSpec

TableComparisonDailyRecurringChecksSpecMap

Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains the daily recurring comparison checks for each configured reference table.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
access_order boolean
size integer
mod_count integer
threshold integer

TableAccuracyDailyRecurringChecksSpec

Container of built-in preconfigured data quality checks on a table level that are verifying the accuracy of the table, comparing it with another reference table.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
daily_total_row_count_match_percent Verifies the total ow count of a tested table and compares it to a row count of a reference table. Stores the most recent captured value for each day when the data quality check was evaluated. TableAccuracyTotalRowCountMatchPercentCheckSpec