Last updated: July 22, 2025

DQOps YAML file definitions

The definition of YAML files used by DQOps to configure the data sources, monitored tables, and the configuration of activated data quality checks.

ColumnDailyMonitoringCheckCategoriesSpec

Container of column level daily monitoring checks. Contains categories of daily monitoring checks.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
nulls	Daily monitoring checks of nulls in the column	ColumnNullsDailyMonitoringChecksSpec
uniqueness	Daily monitoring checks of uniqueness in the column	ColumnUniquenessDailyMonitoringChecksSpec
accepted_values	Configuration of accepted values checks on a column level	ColumnAcceptedValuesDailyMonitoringChecksSpec
text	Daily monitoring checks of text values in the column	ColumnTextDailyMonitoringChecksSpec
whitespace	Configuration of column level checks that detect blank and whitespace values	ColumnWhitespaceDailyMonitoringChecksSpec
conversions	Configuration of conversion testing checks on a column level.	ColumnConversionsDailyMonitoringChecksSpec
patterns	Daily monitoring checks of pattern matching on a column level	ColumnPatternsDailyMonitoringChecksSpec
pii	Daily monitoring checks of Personal Identifiable Information (PII) in the column	ColumnPiiDailyMonitoringChecksSpec
numeric	Daily monitoring checks of numeric values in the column	ColumnNumericDailyMonitoringChecksSpec
anomaly	Daily monitoring checks of anomalies in numeric columns	ColumnAnomalyDailyMonitoringChecksSpec
datetime	Daily monitoring checks of datetime in the column	ColumnDatetimeDailyMonitoringChecksSpec
bool	Daily monitoring checks of booleans in the column	ColumnBoolDailyMonitoringChecksSpec
integrity	Daily monitoring checks of integrity in the column	ColumnIntegrityDailyMonitoringChecksSpec
accuracy	Daily monitoring checks of accuracy in the column	ColumnAccuracyDailyMonitoringChecksSpec
custom_sql	Daily monitoring checks of custom SQL checks in the column	ColumnCustomSqlDailyMonitoringChecksSpec
datatype	Daily monitoring checks of datatype in the column	ColumnDatatypeDailyMonitoringChecksSpec
schema	Daily monitoring column schema checks	ColumnSchemaDailyMonitoringChecksSpec
comparisons	Dictionary of configuration of checks for table comparisons at a column level. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table.	ColumnComparisonDailyMonitoringChecksSpecMap
custom	Dictionary of custom checks. The keys are check names within this category.	CustomCheckSpecMap

---

ColumnNullsDailyMonitoringChecksSpec

Container of nulls data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_nulls_count	Detects incomplete columns that contain any null values. Counts the number of rows having a null value. Raises a data quality issue when the count of null values is above a max_count threshold. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNullsCountCheckSpec
daily_nulls_percent	Detects incomplete columns that contain any null values. Measures the percentage of rows having a null value. Raises a data quality issue when the percentage of null values is above a max_percent threshold. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNullsPercentCheckSpec
daily_nulls_percent_anomaly	Detects day-to-day anomalies in the percentage of null values. Raises a data quality issue when the rate of null values increases or decreases too much during the last 90 days.	ColumnNullPercentAnomalyStationaryCheckSpec
daily_not_nulls_count	Verifies that a column contains a minimum number of non-null values. The default value of the min_count parameter is 1 to detect at least one value in a monitored column. Raises a data quality issue when the count of non-null values is below min_count. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNotNullsCountCheckSpec
daily_not_nulls_percent	Detects columns that contain too many non-null values. Measures the percentage of rows that have non-null values. Raises a data quality issue when the percentage of non-null values is above max_percentage.	ColumnNotNullsPercentCheckSpec
daily_empty_column_found	Detects empty columns that contain only null values. Counts the number of rows that have non-null values. Raises a data quality issue when the column is empty. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnEmptyColumnFoundCheckSpec
daily_nulls_percent_change	Verifies that the null percent value in a column changed in a fixed rate since the last readout.	ColumnNullPercentChangeCheckSpec
daily_nulls_percent_change_1_day	Verifies that the null percent value in a column changed in a fixed rate since the last readout from yesterday.	ColumnNullPercentChange1DayCheckSpec
daily_nulls_percent_change_7_days	Verifies that the null percent value in a column changed in a fixed rate since the last readout from the last week.	ColumnNullPercentChange7DaysCheckSpec
daily_nulls_percent_change_30_days	Verifies that the null percent value in a column changed in a fixed rate since the last readout from the last month.	ColumnNullPercentChange30DaysCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnUniquenessDailyMonitoringChecksSpec

Container of uniqueness data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_distinct_count	Verifies that the number of distinct values stays within an accepted range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnDistinctCountCheckSpec
daily_distinct_percent	Verifies that the percentage of distinct values in a column does not fall below the minimum accepted percent. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnDistinctPercentCheckSpec
daily_duplicate_count	Verifies that the number of duplicate values in a column does not exceed the maximum accepted count. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnDuplicateCountCheckSpec
daily_duplicate_percent	Verifies that the percentage of duplicate values in a column does not exceed the maximum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnDuplicatePercentCheckSpec
daily_distinct_count_anomaly	Verifies that the distinct count in a monitored column is within a two-tailed percentile from measurements made during the last 90 days.	ColumnDistinctCountAnomalyDifferencingCheckSpec
daily_distinct_percent_anomaly	Verifies that the distinct percent in a monitored column is within a two-tailed percentile from measurements made during the last 90 days.	ColumnDistinctPercentAnomalyStationaryCheckSpec
daily_distinct_count_change	Verifies that the distinct count in a monitored column has changed by a fixed rate since the last readout.	ColumnDistinctCountChangeCheckSpec
daily_distinct_count_change_1_day	Verifies that the distinct count in a monitored column has changed by a fixed rate since the last readout from yesterday.	ColumnDistinctCountChange1DayCheckSpec
daily_distinct_count_change_7_days	Verifies that the distinct count in a monitored column has changed by a fixed rate since the last readout from last week.	ColumnDistinctCountChange7DaysCheckSpec
daily_distinct_count_change_30_days	Verifies that the distinct count in a monitored column has changed by a fixed rate since the last readout from last month.	ColumnDistinctCountChange30DaysCheckSpec
daily_distinct_percent_change	Verifies that the distinct percent in a monitored column has changed by a fixed rate since the last readout.	ColumnDistinctPercentChangeCheckSpec
daily_distinct_percent_change_1_day	Verifies that the distinct percent in a monitored column has changed by a fixed rate since the last readout from yesterday.	ColumnDistinctPercentChange1DayCheckSpec
daily_distinct_percent_change_7_days	Verifies that the distinct percent in a monitored column has changed by a fixed rate since the last readout from last week.	ColumnDistinctPercentChange7DaysCheckSpec
daily_distinct_percent_change_30_days	Verifies that the distinct percent in a monitored column has changed by a fixed rate since the last readout from last month.	ColumnDistinctPercentChange30DaysCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnAcceptedValuesDailyMonitoringChecksSpec

Container of accepted values data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_text_found_in_set_percent	The check measures the percentage of rows whose value in a tested column is one of values from a list of expected values or the column value is null. Verifies that the percentage of rows having a valid column value does not exceed the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextFoundInSetPercentCheckSpec
daily_number_found_in_set_percent	The check measures the percentage of rows whose value in a tested column is one of values from a list of expected values or the column value is null. Verifies that the percentage of rows having a valid column value does not exceed the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNumberFoundInSetPercentCheckSpec
daily_expected_text_values_in_use_count	Verifies that the expected string values were found in the column. Raises a data quality issue when too many expected values were not found (were missing). Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnExpectedTextValuesInUseCountCheckSpec
daily_expected_texts_in_top_values_count	Verifies that the top X most popular column values contain all values from a list of expected values. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnExpectedTextsInTopValuesCountCheckSpec
daily_expected_numbers_in_use_count	Verifies that the expected numeric values were found in the column. Raises a data quality issue when too many expected values were not found (were missing). Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnExpectedNumbersInUseCountCheckSpec
daily_text_valid_country_code_percent	Verifies that the percentage of valid country codes in a text column does not fall below the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextValidCountryCodePercentCheckSpec
daily_text_valid_currency_code_percent	Verifies that the percentage of valid currency codes in a text column does not fall below the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextValidCurrencyCodePercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnTextDailyMonitoringChecksSpec

Container of text data quality monitoring checks on a column level that are monitoring tables at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_text_min_length	This check finds the length of the shortest text in a column. Then, it verifies that the minimum length is within an accepted range. It detects that the shortest text is too short. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextMinLengthCheckSpec
daily_text_max_length	This check finds the length of the longest text in a column. Then, it verifies that the maximum length is within an accepted range. It detects that the texts are too long or not long enough. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextMaxLengthCheckSpec
daily_text_mean_length	Verifies that the mean (average) length of texts in a column is within an accepted range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextMeanLengthCheckSpec
daily_text_length_below_min_length	The check counts the number of text values in the column that is below the length defined by the user as a parameter. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextLengthBelowMinLengthCheckSpec
daily_text_length_below_min_length_percent	The check measures the percentage of text values in the column that is below the length defined by the user as a parameter. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextLengthBelowMinLengthPercentCheckSpec
daily_text_length_above_max_length	The check counts the number of text values in the column that is above the length defined by the user as a parameter. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextLengthAboveMaxLengthCheckSpec
daily_text_length_above_max_length_percent	The check measures the percentage of text values in the column that is above the length defined by the user as a parameter. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextLengthAboveMaxLengthPercentCheckSpec
daily_text_length_in_range_percent	The check measures the percentage of those text values with length in the range provided by the user in the column. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextLengthInRangePercentCheckSpec
daily_min_word_count	This check finds the lowest word count of text in a column. Then, it verifies that the minimum length is within an accepted range. It detects that the text contains too less words.	ColumnTextMinWordCountCheckSpec
daily_max_word_count	This check finds the highest word count of text in a column. Then, it verifies that the maximum length is within an accepted range. It detects that the text contains too many words.	ColumnTextMaxWordCountCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnWhitespaceDailyMonitoringChecksSpec

Container of whitespace value detection data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_empty_text_found	Detects empty texts (not null, zero-length texts). This check counts empty and raises a data quality issue when their count exceeds a max_count parameter value. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnWhitespaceEmptyTextFoundCheckSpec
daily_whitespace_text_found	Detects texts that contain only spaces and other whitespace characters. It raises a data quality issue when their count exceeds a max_count parameter value. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnWhitespaceWhitespaceTextFoundCheckSpec
daily_null_placeholder_text_found	Detects texts that are well-known placeholders of null values, such as None, null, n/a. It counts null placeholders and raises a data quality issue when their count exceeds a max_count parameter value. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnWhitespaceNullPlaceholderTextFoundCheckSpec
daily_empty_text_percent	Detects empty texts (not null, zero-length texts) and measures their percentage in the column. This check verifies that the rate of empty strings in a column does not exceed the maximum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnWhitespaceEmptyTextPercentCheckSpec
daily_whitespace_text_percent	Detects texts that contain only spaces and other whitespace characters and measures their percentage in the column. It raises a data quality issue when their rate exceeds a max_percent parameter value. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnWhitespaceWhitespaceTextPercentCheckSpec
daily_null_placeholder_text_percent	Detects texts that are well-known placeholders of null values, such as None, null, n/a, and measures their percentage in the column. It raises a data quality issue when their rate exceeds a max_percent parameter value. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnWhitespaceNullPlaceholderTextPercentCheckSpec
daily_text_surrounded_by_whitespace_found	Detects text values that are surrounded by whitespace characters on any side. This check counts whitespace-surrounded texts and raises a data quality issue when their count exceeds the max_count parameter value. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnWhitespaceTextSurroundedByWhitespaceFoundCheckSpec
daily_text_surrounded_by_whitespace_percent	This check detects text values that are surrounded by whitespace characters on any side and measures their percentage. This check raises a data quality issue when their percentage exceeds the max_percent parameter value. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnWhitespaceTextSurroundedByWhitespacePercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnConversionsDailyMonitoringChecksSpec

Container of conversion test checks that are monitoring if text values are convertible to a target data type at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_text_parsable_to_boolean_percent	Verifies that the percentage of text values that are parsable to a boolean value does not fall below the minimum accepted percentage, text values identified as boolean placeholders are: 0, 1, true, false, t, f, yes, no, y, n. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextParsableToBooleanPercentCheckSpec
daily_text_parsable_to_integer_percent	Verifies that the percentage text values that are parsable to an integer value in a column does not fall below the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextParsableToIntegerPercentCheckSpec
daily_text_parsable_to_float_percent	Verifies that the percentage text values that are parsable to a float value in a column does not fall below the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextParsableToFloatPercentCheckSpec
daily_text_parsable_to_date_percent	Verifies that the percentage text values that are parsable to a date value in a column does not fall below the minimum accepted percentage. DQOps uses a safe_cast when possible, otherwise the text is verified using a regular expression. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTextParsableToDatePercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnPatternsDailyMonitoringChecksSpec

Container of built-in preconfigured daily monitoring checks on a column level that are checking for values matching patterns (regular expressions) in text columns.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_text_not_matching_regex_found	Verifies that the number of text values not matching the custom regular expression pattern does not exceed the maximum accepted count.	ColumnTextNotMatchingRegexFoundCheckSpec
daily_texts_not_matching_regex_percent	Verifies that the percentage of strings not matching the custom regular expression pattern does not exceed the maximum accepted percentage.	ColumnTextsNotMatchingRegexPercentCheckSpec
daily_invalid_email_format_found	Verifies that the number of invalid emails in a text column does not exceed the maximum accepted count.	ColumnInvalidEmailFormatFoundCheckSpec
daily_invalid_email_format_percent	Verifies that the percentage of invalid emails in a text column does not exceed the maximum accepted percentage.	ColumnInvalidEmailFormatPercentCheckSpec
daily_text_not_matching_date_pattern_found	Verifies that the number of texts not matching the date format regular expression does not exceed the maximum accepted count.	ColumnTextNotMatchingDatePatternFoundCheckSpec
daily_text_not_matching_date_pattern_percent	Verifies that the percentage of texts not matching the date format regular expression in a column does not exceed the maximum accepted percentage.	ColumnTextNotMatchingDatePatternPercentCheckSpec
daily_text_not_matching_name_pattern_percent	Verifies that the percentage of texts not matching the name regular expression does not exceed the maximum accepted percentage.	ColumnTextNotMatchingNamePatternPercentCheckSpec
daily_invalid_uuid_format_found	Verifies that the number of invalid UUIDs in a text column does not exceed the maximum accepted count.	ColumnInvalidUuidFormatFoundCheckSpec
daily_invalid_uuid_format_percent	Verifies that the percentage of invalid UUID in a text column does not exceed the maximum accepted percentage.	ColumnInvalidUuidFormatPercentCheckSpec
daily_invalid_ip4_address_format_found	Verifies that the number of invalid IP4 addresses in a text column does not exceed the maximum accepted count.	ColumnInvalidIp4AddressFormatFoundCheckSpec
daily_invalid_ip6_address_format_found	Verifies that the number of invalid IP6 addresses in a text column does not exceed the maximum accepted count.	ColumnInvalidIp6AddressFormatFoundCheckSpec
daily_invalid_usa_phone_format_found	Verifies that the number of invalid USA phone numbers in a text column does not exceed the maximum accepted count.	ColumnInvalidUsaPhoneFoundCheckSpec
daily_invalid_usa_zipcode_format_found	Verifies that the number of invalid zip codes in a text column does not exceed the maximum accepted count.	ColumnInvalidUsaZipcodeFoundCheckSpec
daily_invalid_usa_phone_format_percent	Verifies that the percentage of invalid USA phones number in a text column does not exceed the maximum accepted percentage.	ColumnInvalidUsaPhonePercentCheckSpec
daily_invalid_usa_zipcode_format_percent	Verifies that the percentage of invalid USA zip code in a text column does not exceed the maximum accepted percentage.	ColumnInvalidUsaZipcodePercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnPiiDailyMonitoringChecksSpec

Container of PII data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_contains_usa_phone_percent	Detects USA phone numbers in text columns. Verifies that the percentage of rows that contains a USA phone number in a column does not exceed the maximum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPiiContainsUsaPhonePercentCheckSpec
daily_contains_email_percent	Detects emails in text columns. Verifies that the percentage of rows that contains emails in a column does not exceed the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPiiContainsEmailPercentCheckSpec
daily_contains_usa_zipcode_percent	Detects USA zip codes in text columns. Verifies that the percentage of rows that contains a USA zip code in a column does not exceed the maximum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPiiContainsUsaZipcodePercentCheckSpec
daily_contains_ip4_percent	Detects IP4 addresses in text columns. Verifies that the percentage of rows that contains IP4 address values in a column does not fall below the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPiiContainsIp4PercentCheckSpec
daily_contains_ip6_percent	Detects IP6 addresses in text columns. Verifies that the percentage of rows that contains valid IP6 address values in a column does not fall below the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPiiContainsIp6PercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnNumericDailyMonitoringChecksSpec

Container of built-in preconfigured data quality monitoring on a column level that are checking numeric values at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_number_below_min_value	The check counts the number of values in the column that are below the value defined by the user as a parameter. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNumberBelowMinValueCheckSpec
daily_number_above_max_value	The check counts the number of values in the column that are above the value defined by the user as a parameter. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNumberAboveMaxValueCheckSpec
daily_negative_values	Verifies that the number of negative values in a column does not exceed the maximum accepted count. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNegativeCountCheckSpec
daily_negative_values_percent	Verifies that the percentage of negative values in a column does not exceed the maximum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNegativePercentCheckSpec
daily_number_below_min_value_percent	The check counts the percentage of values in the column that are below the value defined by the user as a parameter. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNumberBelowMinValuePercentCheckSpec
daily_number_above_max_value_percent	The check counts the percentage of values in the column that are above the value defined by the user as a parameter. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNumberAboveMaxValuePercentCheckSpec
daily_number_in_range_percent	Verifies that the percentage of values from range in a column does not exceed the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNumberInRangePercentCheckSpec
daily_integer_in_range_percent	Verifies that the percentage of values from range in a column does not exceed the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnIntegerInRangePercentCheckSpec
daily_min_in_range	Verifies that the minimum value in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnMinInRangeCheckSpec
daily_max_in_range	Verifies that the maximum value in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnMaxInRangeCheckSpec
daily_sum_in_range	Verifies that the sum of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnSumInRangeCheckSpec
daily_mean_in_range	Verifies that the average (mean) of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnMeanInRangeCheckSpec
daily_median_in_range	Verifies that the median of all values in a column is not outside the expected range. Stores the most recent value for each month when the data quality check was evaluated.	ColumnMedianInRangeCheckSpec
daily_percentile_in_range	Verifies that the percentile of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPercentileInRangeCheckSpec
daily_percentile_10_in_range	Verifies that the percentile 10 of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPercentile10InRangeCheckSpec
daily_percentile_25_in_range	Verifies that the percentile 25 of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPercentile25InRangeCheckSpec
daily_percentile_75_in_range	Verifies that the percentile 75 of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPercentile75InRangeCheckSpec
daily_percentile_90_in_range	Verifies that the percentile 90 of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPercentile90InRangeCheckSpec
daily_sample_stddev_in_range	Verifies that the sample standard deviation of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnSampleStddevInRangeCheckSpec
daily_population_stddev_in_range	Verifies that the population standard deviation of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPopulationStddevInRangeCheckSpec
daily_sample_variance_in_range	Verifies that the sample variance of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnSampleVarianceInRangeCheckSpec
daily_population_variance_in_range	Verifies that the population variance of all values in a column is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnPopulationVarianceInRangeCheckSpec
daily_invalid_latitude	Verifies that the number of invalid latitude values in a column does not exceed the maximum accepted count. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnInvalidLatitudeCountCheckSpec
daily_valid_latitude_percent	Verifies that the percentage of valid latitude values in a column does not fall below the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnValidLatitudePercentCheckSpec
daily_invalid_longitude	Verifies that the number of invalid longitude values in a column does not exceed the maximum accepted count. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnInvalidLongitudeCountCheckSpec
daily_valid_longitude_percent	Verifies that the percentage of valid longitude values in a column does not fall below the minimum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnValidLongitudePercentCheckSpec
daily_non_negative_values	Verifies that the number of non-negative values in a column does not exceed the maximum accepted count. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNonNegativeCountCheckSpec
daily_non_negative_values_percent	Verifies that the percentage of non-negative values in a column does not exceed the maximum accepted percentage. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnNonNegativePercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnAnomalyDailyMonitoringChecksSpec

Container of built-in preconfigured data quality checks on a column level for detecting anomalies.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_sum_anomaly	Verifies that the sum in a column changes in a rate within a percentile boundary during the last 90 days.	ColumnSumAnomalyDifferencingCheckSpec
daily_mean_anomaly	Verifies that the mean value in a column changes in a rate within a percentile boundary during the last 90 days.	ColumnMeanAnomalyStationaryCheckSpec
daily_median_anomaly	Verifies that the median in a column changes in a rate within a percentile boundary during the last 90 days.	ColumnMedianAnomalyStationaryCheckSpec
daily_min_anomaly	Detects new outliers, which are new minimum values, much below the last known minimum value. If the minimum value is constantly changing, detects outliers as the biggest change of the minimum value during the last 90 days.	ColumnMinAnomalyDifferencingCheckSpec
daily_max_anomaly	Detects new outliers, which are new maximum values, much above the last known maximum value. If the maximum value is constantly changing, detects outliers as the biggest change of the maximum value during the last 90 days.	ColumnMaxAnomalyDifferencingCheckSpec
daily_mean_change	Verifies that the mean value in a column changed in a fixed rate since the last readout.	ColumnMeanChangeCheckSpec
daily_mean_change_1_day	Verifies that the mean value in a column changed in a fixed rate since last readout from yesterday.	ColumnMeanChange1DayCheckSpec
daily_mean_change_7_days	Verifies that the mean value in a column changed in a fixed rate since last readout from last week.	ColumnMeanChange7DaysCheckSpec
daily_mean_change_30_days	Verifies that the mean value in a column changed in a fixed rate since last readout from last month.	ColumnMeanChange30DaysCheckSpec
daily_median_change	Verifies that the median in a column changed in a fixed rate since the last readout.	ColumnMedianChangeCheckSpec
daily_median_change_1_day	Verifies that the median in a column changed in a fixed rate since the last readout from yesterday.	ColumnMedianChange1DayCheckSpec
daily_median_change_7_days	Verifies that the median in a column changed in a fixed rate since the last readout from the last week.	ColumnMedianChange7DaysCheckSpec
daily_median_change_30_days	Verifies that the median in a column changed in a fixed rate since the last readout from the last month.	ColumnMedianChange30DaysCheckSpec
daily_sum_change	Verifies that the sum in a column changed in a fixed rate since the last readout.	ColumnSumChangeCheckSpec
daily_sum_change_1_day	Verifies that the sum in a column changed in a fixed rate since the last readout from yesterday.	ColumnSumChange1DayCheckSpec
daily_sum_change_7_days	Verifies that the sum in a column changed in a fixed rate since the last readout from the last week.	ColumnSumChange7DaysCheckSpec
daily_sum_change_30_days	Verifies that the sum in a column changed in a fixed rate since the last readout from the last month.	ColumnSumChange30DaysCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnDatetimeDailyMonitoringChecksSpec

Container of date-time data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_date_values_in_future_percent	Detects dates in the future in date, datetime and timestamp columns. Measures a percentage of dates in the future. Raises a data quality issue when too many future dates are found. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnDateValuesInFuturePercentCheckSpec
daily_date_in_range_percent	Verifies that the dates in date, datetime, or timestamp columns are within a reasonable range of dates. The default configuration detects fake dates such as 1900-01-01 and 2099-12-31. Measures the percentage of valid dates and raises a data quality issue when too many dates are found. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnDateInRangePercentCheckSpec
daily_text_match_date_format_percent	Verifies that the values in text columns match one of the predefined date formats, such as an ISO 8601 date. Measures the percentage of valid date strings and raises a data quality issue when too many invalid date strings are found. Creates a separate data quality check (and an alert) for each daily monitoring.	ColumnTextMatchDateFormatPercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnBoolDailyMonitoringChecksSpec

Container of boolean data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_true_percent	Measures the percentage of true values in a boolean column and verifies that it is within the accepted range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnTruePercentCheckSpec
daily_false_percent	Measures the percentage of false values in a boolean column and verifies that it is within the accepted range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnFalsePercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnIntegrityDailyMonitoringChecksSpec

Container of integrity data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_lookup_key_not_found	Detects invalid values that are not present in a dictionary table using an outer join query. Counts the number of invalid keys. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnIntegrityLookupKeyNotFoundCountCheckSpec
daily_lookup_key_found_percent	Measures the percentage of valid values that are present in a dictionary table. Joins this table to a dictionary table using an outer join. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnIntegrityForeignKeyMatchPercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnAccuracyDailyMonitoringChecksSpec

Container of accuracy data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_total_sum_match_percent	Verifies that the percentage of difference in total sum of a column in a table and total sum of a column of another table does not exceed the set number. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnAccuracyTotalSumMatchPercentCheckSpec
daily_total_min_match_percent	Verifies that the percentage of difference in total min of a column in a table and total min of a column of another table does not exceed the set number. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnAccuracyTotalMinMatchPercentCheckSpec
daily_total_max_match_percent	Verifies that the percentage of difference in total max of a column in a table and total max of a column of another table does not exceed the set number. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnAccuracyTotalMaxMatchPercentCheckSpec
daily_total_average_match_percent	Verifies that the percentage of difference in total average of a column in a table and total average of a column of another table does not exceed the set number. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnAccuracyTotalAverageMatchPercentCheckSpec
daily_total_not_null_count_match_percent	Verifies that the percentage of difference in total not null count of a column in a table and total not null count of a column of another table does not exceed the set number. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnAccuracyTotalNotNullCountMatchPercentCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnCustomSqlDailyMonitoringChecksSpec

Container of built-in preconfigured data quality checks on a column level that are using custom SQL expressions (conditions).

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_sql_condition_failed_on_column	Verifies that a custom SQL expression is met for each row. Counts the number of rows where the expression is not satisfied, and raises an issue if too many failures were detected. This check is used also to compare values between the current column and another column: `{alias}.{column} > col_tax`. Stores the most recent captured count of failed rows for each day when the data quality check was evaluated.	ColumnSqlConditionFailedCheckSpec
daily_sql_condition_passed_percent_on_column	Verifies that a minimum percentage of rows passed a custom SQL condition (expression). Reference the current column by using tokens, for example: `{alias}.{column} > {alias}.col_tax`. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnSqlConditionPassedPercentCheckSpec
daily_sql_aggregate_expression_on_column	Verifies that a custom aggregated SQL expression (MIN, MAX, etc.) is not outside the expected range. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnSqlAggregateExpressionCheckSpec
daily_sql_invalid_value_count_on_column	Runs a custom query that retrieves invalid values found in a column and returns the number of them, and raises an issue if too many failures were detected. This check is used for setting testing queries or ready queries used by users in their own systems (legacy SQL queries). For example, when this check is applied on a column. The condition can find invalid values in the column which have values lower than 18 using an SQL query: `SELECT {column} FROM {table} WHERE {column} < 18`.	ColumnSqlInvalidValueCountCheckSpec
daily_import_custom_result_on_column	Runs a custom query that retrieves a result of a data quality check performed in the data engineering, whose result (the severity level) is pulled from a separate table.	ColumnSqlImportCustomResultCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnDatatypeDailyMonitoringChecksSpec

Container of datatype data quality monitoring checks on a column level that are checking at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_detected_datatype_in_text	Detects the data type of text values stored in the column. The sensor returns the code of the detected type of column data: 1 - integers, 2 - floats, 3 - dates, 4 - datetimes, 5 - timestamps, 6 - booleans, 7 - strings, 8 - mixed data types. Raises a data quality issue when the detected data type does not match the expected data type. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnDetectedDatatypeInTextCheckSpec
daily_detected_datatype_in_text_changed	Detects that the data type of texts stored in a text column has changed since the last verification. The sensor returns the detected type of column data: 1 - integers, 2 - floats, 3 - dates, 4 - datetimes, 5 - timestamps, 6 - booleans, 7 - strings, 8 - mixed data types. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnDatatypeDetectedDatatypeInTextChangedCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnSchemaDailyMonitoringChecksSpec

Container of built-in preconfigured data quality checks on a column level that are checking the column schema at a daily level.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_column_exists	Checks the metadata of the monitored table and verifies if the column exists. Stores the most recent value for each day when the data quality check was evaluated.	ColumnSchemaColumnExistsCheckSpec
daily_column_type_changed	Checks the metadata of the monitored column and detects if the data type (including the length, precision, scale, nullability) has changed since the last day. Stores the most recent hash for each day when the data quality check was evaluated.	ColumnSchemaTypeChangedCheckSpec
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---

ColumnComparisonDailyMonitoringChecksSpecMap

Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains configuration of column level comparison checks. Each column level check container also defines the name of the reference column name to which we are comparing.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
self		Dict[string, ColumnComparisonDailyMonitoringChecksSpec]

---

ColumnComparisonDailyMonitoringChecksSpec

Container of built-in preconfigured column level comparison checks that compare min/max/sum/mean/nulls measures between the column in the tested (parent) table and a matching reference column in the reference table (the source of truth). This is the configuration for daily monitoring checks that are counted in KPIs.

The structure of this object is described below

Property name	Description	Data type	Enum values	Default value	Sample values
daily_sum_match	Verifies that percentage of the difference between the sum of values in a tested column in a parent table and the sum of a values in a column in the reference table. The difference must be below defined percentage thresholds. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnComparisonSumMatchCheckSpec
daily_min_match	Verifies that percentage of the difference between the minimum value in a tested column in a parent table and the minimum value in a column in the reference table. The difference must be below defined percentage thresholds. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnComparisonMinMatchCheckSpec
daily_max_match	Verifies that percentage of the difference between the maximum value in a tested column in a parent table and the maximum value in a column in the reference table. The difference must be below defined percentage thresholds. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnComparisonMaxMatchCheckSpec
daily_mean_match	Verifies that percentage of the difference between the mean (average) value in a tested column in a parent table and the mean (average) value in a column in the reference table. The difference must be below defined percentage thresholds. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnComparisonMeanMatchCheckSpec
daily_not_null_count_match	Verifies that percentage of the difference between the count of not null values in a tested column in a parent table and the count of not null values in a column in the reference table. The difference must be below defined percentage thresholds. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnComparisonNotNullCountMatchCheckSpec
daily_null_count_match	Verifies that percentage of the difference between the count of null values in a tested column in a parent table and the count of null values in a column in the reference table. The difference must be below defined percentage thresholds. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnComparisonNullCountMatchCheckSpec
daily_distinct_count_match	Verifies that percentage of the difference between the count of distinct values in a tested column in a parent table and the count of distinct values in a column in the reference table. The difference must be below defined percentage thresholds. Stores the most recent captured value for each day when the data quality check was evaluated.	ColumnComparisonDistinctCountMatchCheckSpec
reference_column	The name of the reference column name in the reference table. It is the column to which the current column is compared to.	string
custom_checks	Dictionary of additional custom checks within this category. The keys are check names defined in the definition section. The sensor parameters and rules should match the type of the configured sensor and rule for the custom check.	CustomCategoryCheckSpecMap

---