Skip to content

Last updated: July 22, 2025

DQOps YAML file definitions

The definition of YAML files used by DQOps to configure the data sources, monitored tables, and the configuration of activated data quality checks.

TableMonthlyPartitionedCheckCategoriesSpec

Container of table level monthly partitioned checks. Contains categories of monthly partitioned checks.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
volume Volume monthly partitioned data quality checks that verify the quality of every month of data separately TableVolumeMonthlyPartitionedChecksSpec
timeliness Monthly partitioned timeliness checks TableTimelinessMonthlyPartitionedChecksSpec
custom_sql Custom SQL monthly partitioned data quality checks that verify the quality of every month of data separately TableCustomSqlMonthlyPartitionedChecksSpec
uniqueness Monthly partitioned uniqueness checks on a table level. TableUniquenessMonthlyPartitionChecksSpec
comparisons Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. TableComparisonMonthlyPartitionedChecksSpecMap
custom Dictionary of custom checks. The keys are check names within this category. CustomCheckSpecMap

TableVolumeMonthlyPartitionedChecksSpec

Container of table level monthly partitioned volume data quality checks.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
monthly_partition_row_count Verifies that each monthly partition in the tested table has at least a minimum accepted number of rows. The default configuration of the warning, error and fatal severity rules verifies a minimum row count of one row, which ensures that the partition is not empty. TableRowCountCheckSpec
monthly_partition_row_count_change Detects when the partition's volume (row count) change between the current monthly partition and the previous partition exceeds the maximum accepted change percentage. TableRowCountChangeCheckSpec
custom_checks Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. CustomCategoryCheckSpecMap

TableTimelinessMonthlyPartitionedChecksSpec

Container of table level monthly partitioned timeliness data quality checks.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
monthly_partition_data_ingestion_delay Monthly partitioned check calculating the time difference in days between the most recent event timestamp and the most recent ingestion timestamp TableDataIngestionDelayCheckSpec
monthly_partition_reload_lag Monthly partitioned check calculating the longest time a row waited to be loaded, it is the maximum difference in days between the ingestion timestamp and the event timestamp column on any row in the monitored partition TablePartitionReloadLagCheckSpec
custom_checks Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. CustomCategoryCheckSpecMap

TableCustomSqlMonthlyPartitionedChecksSpec

Container of built-in preconfigured data quality checks on a table level that are using custom SQL expressions (conditions).

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
monthly_partition_sql_condition_failed_on_table Verifies that a custom SQL expression is met for each row. Counts the number of rows where the expression is not satisfied, and raises an issue if too many failures were detected. This check is used also to compare values between columns: `{alias}.col_price > {alias}.col_tax`. Stores a separate data quality check result for each monthly partition. TableSqlConditionFailedCheckSpec
monthly_partition_sql_condition_passed_percent_on_table Verifies that a minimum percentage of rows passed a custom SQL condition (expression). Reference the current table by using tokens, for example: `{alias}.col_price > {alias}.col_tax`. Stores a separate data quality check result for each monthly partition. TableSqlConditionPassedPercentCheckSpec
monthly_partition_sql_aggregate_expression_on_table Verifies that a custom aggregated SQL expression (MIN, MAX, etc.) is not outside the expected range. Stores a separate data quality check result for each monthly partition. TableSqlAggregateExpressionCheckSpec
monthly_partition_import_custom_result_on_table Runs a custom query that retrieves a result of a data quality check performed in the data engineering, whose result (the severity level) is pulled from a separate table. TableSqlImportCustomResultCheckSpec
custom_checks Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. CustomCategoryCheckSpecMap

TableUniquenessMonthlyPartitionChecksSpec

Container of table level monthly partition for uniqueness data quality checks

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
monthly_partition_duplicate_record_count Verifies that the number of duplicate record values in a table does not exceed the maximum accepted count. TableDuplicateRecordCountCheckSpec
monthly_partition_duplicate_record_percent Verifies that the percentage of duplicate record values in a table does not exceed the maximum accepted percentage. TableDuplicateRecordPercentCheckSpec
custom_checks Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. CustomCategoryCheckSpecMap

TableComparisonMonthlyPartitionedChecksSpecMap

Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains the monthly partitioned comparison checks for each configured reference table.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
self Dict[string, TableComparisonMonthlyPartitionedChecksSpec]

TableComparisonMonthlyPartitionedChecksSpec

Container of built-in comparison (accuracy) checks on a table level that are using a defined comparison to identify the reference table and the data grouping configuration. Contains the monthly partitioned comparison checks, comparing each month of data.

The structure of this object is described below

 Property name   Description                       Data type   Enum values   Default value   Sample values 
monthly_partition_row_count_match Verifies that the row count of the tested (parent) table matches the row count of the reference table, for each monthly partition (grouping rows by the time period, truncated to the month). Compares each group of data with a GROUP BY clause. Stores the most recent captured value for each monthly partition and optionally data groups. TableComparisonRowCountMatchCheckSpec
custom_checks Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check. CustomCategoryCheckSpecMap