Last updated: July 22, 2025
DQOps YAML file definitions
The definition of YAML files used by DQOps to configure the data sources, monitored tables, and the configuration of activated data quality checks.
TableMonthlyPartitionedCheckCategoriesSpec
Container of table level monthly partitioned checks. Contains categories of monthly partitioned checks.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
volume |
Volume monthly partitioned data quality checks that verify the quality of every month of data separately | TableVolumeMonthlyPartitionedChecksSpec | |||
timeliness |
Monthly partitioned timeliness checks | TableTimelinessMonthlyPartitionedChecksSpec | |||
custom_sql |
Custom SQL monthly partitioned data quality checks that verify the quality of every month of data separately | TableCustomSqlMonthlyPartitionedChecksSpec | |||
uniqueness |
Monthly partitioned uniqueness checks on a table level. | TableUniquenessMonthlyPartitionChecksSpec | |||
comparisons |
Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | TableComparisonMonthlyPartitionedChecksSpecMap | |||
custom |
Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
TableVolumeMonthlyPartitionedChecksSpec
Container of table level monthly partitioned volume data quality checks.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_row_count |
Verifies that each monthly partition in the tested table has at least a minimum accepted number of rows. The default configuration of the warning, error and fatal severity rules verifies a minimum row count of one row, which ensures that the partition is not empty. | TableRowCountCheckSpec | |||
monthly_partition_row_count_change |
Detects when the partition's volume (row count) change between the current monthly partition and the previous partition exceeds the maximum accepted change percentage. | TableRowCountChangeCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableTimelinessMonthlyPartitionedChecksSpec
Container of table level monthly partitioned timeliness data quality checks.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_data_ingestion_delay |
Monthly partitioned check calculating the time difference in days between the most recent event timestamp and the most recent ingestion timestamp | TableDataIngestionDelayCheckSpec | |||
monthly_partition_reload_lag |
Monthly partitioned check calculating the longest time a row waited to be loaded, it is the maximum difference in days between the ingestion timestamp and the event timestamp column on any row in the monitored partition | TablePartitionReloadLagCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableCustomSqlMonthlyPartitionedChecksSpec
Container of built-in preconfigured data quality checks on a table level that are using custom SQL expressions (conditions).
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_sql_condition_failed_on_table |
Verifies that a custom SQL expression is met for each row. Counts the number of rows where the expression is not satisfied, and raises an issue if too many failures were detected. This check is used also to compare values between columns: `{alias}.col_price > {alias}.col_tax`. Stores a separate data quality check result for each monthly partition. | TableSqlConditionFailedCheckSpec | |||
monthly_partition_sql_condition_passed_percent_on_table |
Verifies that a minimum percentage of rows passed a custom SQL condition (expression). Reference the current table by using tokens, for example: `{alias}.col_price > {alias}.col_tax`. Stores a separate data quality check result for each monthly partition. | TableSqlConditionPassedPercentCheckSpec | |||
monthly_partition_sql_aggregate_expression_on_table |
Verifies that a custom aggregated SQL expression (MIN, MAX, etc.) is not outside the expected range. Stores a separate data quality check result for each monthly partition. | TableSqlAggregateExpressionCheckSpec | |||
monthly_partition_import_custom_result_on_table |
Runs a custom query that retrieves a result of a data quality check performed in the data engineering, whose result (the severity level) is pulled from a separate table. | TableSqlImportCustomResultCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableUniquenessMonthlyPartitionChecksSpec
Container of table level monthly partition for uniqueness data quality checks
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_duplicate_record_count |
Verifies that the number of duplicate record values in a table does not exceed the maximum accepted count. | TableDuplicateRecordCountCheckSpec | |||
monthly_partition_duplicate_record_percent |
Verifies that the percentage of duplicate record values in a table does not exceed the maximum accepted percentage. | TableDuplicateRecordPercentCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |
TableComparisonMonthlyPartitionedChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains the monthly partitioned comparison checks for each configured reference table.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
self |
Dict[string, TableComparisonMonthlyPartitionedChecksSpec] |
TableComparisonMonthlyPartitionedChecksSpec
Container of built-in comparison (accuracy) checks on a table level that are using a defined comparison to identify the reference table and the data grouping configuration. Contains the monthly partitioned comparison checks, comparing each month of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_row_count_match |
Verifies that the row count of the tested (parent) table matches the row count of the reference table, for each monthly partition (grouping rows by the time period, truncated to the month). Compares each group of data with a GROUP BY clause. Stores the most recent captured value for each monthly partition and optionally data groups. | TableComparisonRowCountMatchCheckSpec | |||
custom_checks |
Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. | CustomCategoryCheckSpecMap |