Last updated: July 22, 2025
DQOps YAML file definitions
The definition of YAML files used by DQOps to configure the data sources, monitored tables, and the configuration of activated data quality checks.
TableDailyMonitoringCheckCategoriesSpec
Container of table level daily monitoring. Contains categories of daily monitoring.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
volume |
Daily monitoring volume data quality checks | TableVolumeDailyMonitoringChecksSpec | |||
timeliness |
Daily monitoring timeliness checks | TableTimelinessDailyMonitoringChecksSpec | |||
accuracy |
Daily monitoring accuracy checks | TableAccuracyDailyMonitoringChecksSpec | |||
custom_sql |
Daily monitoring custom SQL checks | TableCustomSqlDailyMonitoringChecksSpec | |||
availability |
Daily monitoring table availability checks | TableAvailabilityDailyMonitoringChecksSpec | |||
schema |
Daily monitoring table schema checks | TableSchemaDailyMonitoringChecksSpec | |||
uniqueness |
Daily monitoring uniqueness checks on a table level. | TableUniquenessDailyMonitoringChecksSpec | |||
comparisons |
Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | TableComparisonDailyMonitoringChecksSpecMap | |||
custom |
Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
TableVolumeDailyMonitoringChecksSpec
Container of table level daily monitoring for volume data quality checks.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
daily_row_count |
Verifies that the tested table has at least a minimum accepted number of rows. The default configuration of the warning, error and fatal severity rules verifies a minimum row count of one row, which ensures that the table is not empty. Stores the most recent captured row count value for each day when the row count was evaluated. | TableRowCountCheckSpec | |||
daily_row_count_anomaly |
Detects when the row count has changed too much since the previous day. It uses time series anomaly detection to find the biggest volume changes during the last 90 days. | TableRowCountAnomalyDifferencingCheckSpec | |||
daily_row_count_change |
Detects when the volume's (row count) change since the last known row count exceeds the maximum accepted change percentage. | TableRowCountChangeCheckSpec | |||
daily_row_count_change_1_day |
Detects when the volume's change (increase or decrease of the row count) since the previous day exceeds the maximum accepted change percentage. | TableRowCountChange1DayCheckSpec | |||
daily_row_count_change_7_days |
This check verifies that the percentage of change in the table's volume (row count) since seven days ago is below the maximum accepted percentage. Verifying a volume change since a value a week ago overcomes the effect of weekly seasonability. | TableRowCountChange7DaysCheckSpec | |||
daily_row_count_change_30_days |
This check verifies that the percentage of change in the table's volume (row count) since thirty days ago is below the maximum accepted percentage. Comparing the current row count to a value 30 days ago overcomes the effect of monthly seasonability. | TableRowCountChange30DaysCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableTimelinessDailyMonitoringChecksSpec
Container of table level daily monitoring for timeliness data quality checks.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
daily_data_freshness |
Daily calculating the number of days since the most recent event timestamp (freshness) | TableDataFreshnessCheckSpec | |||
daily_data_freshness_anomaly |
Verifies that the number of days since the most recent event timestamp (freshness) changes in a rate within a percentile boundary during the last 90 days. | TableDataFreshnessAnomalyCheckSpec | |||
daily_data_staleness |
Daily calculating the time difference in days between the current date and the most recent data ingestion timestamp (staleness) | TableDataStalenessCheckSpec | |||
daily_data_ingestion_delay |
Daily calculating the time difference in days between the most recent event timestamp and the most recent ingestion timestamp | TableDataIngestionDelayCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableAccuracyDailyMonitoringChecksSpec
Container of built-in preconfigured data quality checks on a table level that are verifying the accuracy of the table, comparing it with another reference table.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
daily_total_row_count_match_percent |
Verifies the total ow count of a tested table and compares it to a row count of a reference table. Stores the most recent captured value for each day when the data quality check was evaluated. | TableAccuracyTotalRowCountMatchPercentCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableCustomSqlDailyMonitoringChecksSpec
Container of built-in preconfigured data quality checks on a table level that are using custom SQL expressions (conditions).
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
daily_sql_condition_failed_on_table |
Verifies that a custom SQL expression is met for each row. Counts the number of rows where the expression is not satisfied, and raises an issue if too many failures were detected. This check is used also to compare values between columns: `{alias}.col_price > {alias}.col_tax`. Stores the most recent count of failed rows for each day when the data quality check was evaluated. | TableSqlConditionFailedCheckSpec | |||
daily_sql_condition_passed_percent_on_table |
Verifies that a minimum percentage of rows passed a custom SQL condition (expression). Reference the current table by using tokens, for example: `{alias}.col_price > {alias}.col_tax`. Stores the most recent captured percentage for each day when the data quality check was evaluated. | TableSqlConditionPassedPercentCheckSpec | |||
daily_sql_aggregate_expression_on_table |
Verifies that a custom aggregated SQL expression (MIN, MAX, etc.) is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated. | TableSqlAggregateExpressionCheckSpec | |||
daily_sql_invalid_record_count_on_table |
Runs a custom query that retrieves invalid records found in a table and returns the number of them, and raises an issue if too many failures were detected. This check is used for setting testing queries or ready queries used by users in their own systems (legacy SQL queries). For example, when this check is applied on a age column, the condition can find invalid records in which the age is lower than 18 using an SQL query: `SELECT age FROM {table} WHERE age < 18`. | TableSqlInvalidRecordCountCheckSpec | |||
daily_import_custom_result_on_table |
Runs a custom query that retrieves a result of a data quality check performed in the data engineering, whose result (the severity level) is pulled from a separate table. | TableSqlImportCustomResultCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableAvailabilityDailyMonitoringChecksSpec
Container of built-in preconfigured data quality checks on a table level that are detecting the table availability.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
daily_table_availability |
Verifies availability of a table in a monitored database using a simple query. Stores the most recent table availability status for each day when the data quality check was evaluated. | TableAvailabilityCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableSchemaDailyMonitoringChecksSpec
Container of built-in preconfigured volume data quality checks on a table level that are executed as a daily monitoring (checkpoint) checks.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
daily_column_count |
Detects if the number of column matches an expected number. Retrieves the metadata of the monitored table, counts the number of columns and compares it to an expected value (an expected number of columns). Stores the most recent column count for each day when the data quality check was evaluated. | TableSchemaColumnCountCheckSpec | |||
daily_column_count_changed |
Detects if the count of columns has changed since the most recent day. Retrieves the metadata of the monitored table, counts the number of columns and compares it the last known column count that was captured when this data quality check was executed the last time. Stores the most recent column count for each day when the data quality check was evaluated. | TableSchemaColumnCountChangedCheckSpec | |||
daily_column_list_changed |
Detects if new columns were added or existing columns were removed since the most recent day. Retrieves the metadata of the monitored table and calculates an unordered hash of the column names. Compares the current hash to the previously known hash to detect any changes to the list of columns. | TableSchemaColumnListChangedCheckSpec | |||
daily_column_list_or_order_changed |
Detects if new columns were added, existing columns were removed or the columns were reordered since the most recent day. Retrieves the metadata of the monitored table and calculates an ordered hash of the column names. Compares the current hash to the previously known hash to detect any changes to the list of columns or their order. | TableSchemaColumnListOrOrderChangedCheckSpec | |||
daily_column_types_changed |
Detects if new columns were added, removed or their data types have changed since the most recent day. Retrieves the metadata of the monitored table and calculates an unordered hash of the column names and the data types (including the length, scale, precision, nullability). Compares the current hash to the previously known hash to detect any changes to the list of columns or their types. | TableSchemaColumnTypesChangedCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableUniquenessDailyMonitoringChecksSpec
Container of table level daily monitoring for uniqueness data quality checks.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
daily_duplicate_record_count |
Verifies that the number of duplicate record values in a table does not exceed the maximum accepted count. | TableDuplicateRecordCountCheckSpec | |||
daily_duplicate_record_percent |
Verifies that the percentage of duplicate record values in a table does not exceed the maximum accepted percentage. | TableDuplicateRecordPercentCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableComparisonDailyMonitoringChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains the daily monitoring comparison checks for each configured reference table.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
self |
Dict[string, TableComparisonDailyMonitoringChecksSpec] |
TableComparisonDailyMonitoringChecksSpec
Container of built-in comparison (accuracy) checks on a table level that are using a defined comparison to identify the reference table and the data grouping configuration. Contains the daily monitoring comparison checks.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
daily_row_count_match |
Verifies that the row count of the tested (parent) table matches the row count of the reference table. Compares each group of data with a GROUP BY clause. Stores the most recent captured value for each day when the data quality check was evaluated. | TableComparisonRowCountMatchCheckSpec | |||
daily_column_count_match |
Verifies that the column count of the tested (parent) table matches the column count of the reference table. Only one comparison result is returned, without data grouping. Stores the most recent captured value for each day when the data quality check was evaluated. | TableComparisonColumnCountMatchCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |