column monthly partitioned checks
ColumnNullsMonthlyPartitionedChecksSpec
Container of nulls data quality partitioned checks on a column level that are checking monthly partitions or rows for each day of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_nulls_count | Verifies that the number of null values in a column does not exceed the set count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNullsCountCheckSpec | |||
monthly_partition_nulls_percent | Verifies that the percentage of null values in a column does not exceed the set percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNullsPercentCheckSpec | |||
monthly_partition_not_nulls_count | Verifies that the number of not null values in a column does not exceed the set count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNotNullsCountCheckSpec | |||
monthly_partition_not_nulls_percent | Verifies that the percentage of not null values in a column does not exceed the set percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNotNullsPercentCheckSpec |
ColumnComparisonMonthlyPartitionedChecksSpec
Container of built-in preconfigured column level comparison checks that compare min/max/sum/mean/nulls measures between the column in the tested (parent) table and a matching reference column in the reference table (the source of truth). This is the configuration for daily partitioned checks that are counted in KPIs.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_sum_match | Verifies that percentage of the difference between the sum of values in a tested column in a parent table and the sum of a values in a column in the reference table. The difference must be below defined percentage thresholds. Compares each monthly partition (each month of data) between the compared table and the reference table (the source of truth). | ColumnComparisonSumMatchCheckSpec | |||
monthly_partition_min_match | Verifies that percentage of the difference between the minimum value in a tested column in a parent table and the minimum value in a column in the reference table. The difference must be below defined percentage thresholds. Compares each monthly partition (each month of data) between the compared table and the reference table (the source of truth). | ColumnComparisonMinMatchCheckSpec | |||
monthly_partition_max_match | Verifies that percentage of the difference between the maximum value in a tested column in a parent table and the maximum value in a column in the reference table. The difference must be below defined percentage thresholds. Compares each monthly partition (each month of data) between the compared table and the reference table (the source of truth). | ColumnComparisonMaxMatchCheckSpec | |||
monthly_partition_mean_match | Verifies that percentage of the difference between the mean (average) value in a tested column in a parent table and the mean (average) value in a column in the reference table. The difference must be below defined percentage thresholds. Compares each monthly partition (each month of data) between the compared table and the reference table (the source of truth). | ColumnComparisonMeanMatchCheckSpec | |||
monthly_partition_not_null_count_match | Verifies that percentage of the difference between the count of not null values in a tested column in a parent table and the count of not null values in a column in the reference table. The difference must be below defined percentage thresholds. Compares each monthly partition (each month of data) between the compared table and the reference table (the source of truth). | ColumnComparisonNotNullCountMatchCheckSpec | |||
monthly_partition_null_count_match | Verifies that percentage of the difference between the count of null values in a tested column in a parent table and the count of null values in a column in the reference table. The difference must be below defined percentage thresholds. Compares each monthly partition (each month of data) between the compared table and the reference table (the source of truth). | ColumnComparisonNullCountMatchCheckSpec | |||
reference_column | The name of the reference column name in the reference table. It is the column to which the current column is compared to. | string |
ColumnMonthlyPartitionedCheckCategoriesSpec
Container of data quality partitioned checks on a column level that are checking numeric values at a monthly level.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
nulls | Monthly partitioned checks of nulls values in the column | ColumnNullsMonthlyPartitionedChecksSpec | |||
numeric | Monthly partitioned checks of numeric values in the column | ColumnNumericMonthlyPartitionedChecksSpec | |||
strings | Monthly partitioned checks of strings values in the column | ColumnStringsMonthlyPartitionedChecksSpec | |||
uniqueness | Monthly partitioned checks of uniqueness values in the column | ColumnUniquenessMonthlyPartitionedChecksSpec | |||
datetime | Monthly partitioned checks of datetime values in the column | ColumnDatetimeMonthlyPartitionedChecksSpec | |||
pii | Monthly partitioned checks of Personal Identifiable Information (PII) in the column | ColumnPiiMonthlyPartitionedChecksSpec | |||
sql | Monthly partitioned checks using custom SQL expressions and conditions on the column | ColumnSqlMonthlyPartitionedChecksSpec | |||
bool | Monthly partitioned checks for booleans in the column | ColumnBoolMonthlyPartitionedChecksSpec | |||
integrity | Monthly partitioned checks for integrity in the column | ColumnIntegrityMonthlyPartitionedChecksSpec | |||
accuracy | Monthly partitioned checks for accuracy in the column | ColumnAccuracyMonthlyPartitionedChecksSpec | |||
datatype | Monthly partitioned checks for datatype in the column | ColumnDatatypeMonthlyPartitionedChecksSpec | |||
anomaly | Monthly partitioned checks for anomaly in the column | ColumnAnomalyMonthlyPartitionedChecksSpec | |||
comparisons | Dictionary of configuration of checks for table comparisons at a column level. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | ColumnComparisonMonthlyPartitionedChecksSpecMap | |||
custom | Dictionary of custom checks. The keys are check names. | CustomCheckSpecMap |
ColumnDatatypeMonthlyPartitionedChecksSpec
Container of datatype data quality partitioned checks on a column level that are checking monthly partitions or rows for each month of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_date_match_format_percent | Verifies that the percentage of date values matching the given format in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnDatatypeDateMatchFormatPercentCheckSpec | |||
monthly_partition_string_datatype_changed | Detects that the data type of texts stored in a text column has changed when compared to an earlier not empty partition. The sensor returns the detected data type of a column: 1 - integers, 2 - floats, 3 - dates, 4 - timestamps, 5 - booleans, 6 - strings, 7 - mixed data types. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnDatatypeStringDatatypeChangedCheckSpec |
ColumnStringsMonthlyPartitionedChecksSpec
Container of strings data quality partitioned checks on a column level that are checking monthly partitions or rows for each month of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_string_max_length | Verifies that the length of string in a column does not exceed the maximum accepted length. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringMaxLengthCheckSpec | |||
monthly_partition_string_min_length | Verifies that the length of string in a column does not fall below the minimum accepted length. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringMinLengthCheckSpec | |||
monthly_partition_string_mean_length | Verifies that the length of string in a column does not exceed the mean accepted length. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringMeanLengthCheckSpec | |||
monthly_partition_string_length_below_min_length_count | The check counts the number of strings in the column that is below the length defined by the user as a parameter. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringLengthBelowMinLengthCountCheckSpec | |||
monthly_partition_string_length_below_min_length_percent | The check counts the percentage of strings in the column that is below the length defined by the user as a parameter. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringLengthBelowMinLengthPercentCheckSpec | |||
monthly_partition_string_length_above_max_length_count | The check counts the number of strings in the column that is above the length defined by the user as a parameter. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringLengthAboveMaxLengthCountCheckSpec | |||
monthly_partition_string_length_above_max_length_percent | The check counts the percentage of strings in the column that is above the length defined by the user as a parameter. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringLengthAboveMaxLengthPercentCheckSpec | |||
monthly_partition_string_length_in_range_percent | The check counts the percentage of those strings with length in the range provided by the user in the column. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringLengthInRangePercentCheckSpec | |||
monthly_partition_string_empty_count | Verifies that the number of empty strings in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringEmptyCountCheckSpec | |||
monthly_partition_string_empty_percent | Verifies that the percentage of empty strings in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringEmptyPercentCheckSpec | |||
monthly_partition_string_whitespace_count | Verifies that the number of whitespace strings in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringWhitespaceCountCheckSpec | |||
monthly_partition_string_whitespace_percent | Verifies that the percentage of whitespace strings in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringWhitespacePercentCheckSpec | |||
monthly_partition_string_surrounded_by_whitespace_count | Verifies that the number of strings surrounded by whitespace in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringSurroundedByWhitespaceCountCheckSpec | |||
monthly_partition_string_surrounded_by_whitespace_percent | Verifies that the percentage of strings surrounded by whitespace in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringSurroundedByWhitespacePercentCheckSpec | |||
monthly_partition_string_null_placeholder_count | Verifies that the number of null placeholders in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringNullPlaceholderCountCheckSpec | |||
monthly_partition_string_null_placeholder_percent | Verifies that the percentage of null placeholders in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringNullPlaceholderPercentCheckSpec | |||
monthly_partition_string_boolean_placeholder_percent | Verifies that the percentage of boolean placeholder for strings in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringBooleanPlaceholderPercentCheckSpec | |||
monthly_partition_string_parsable_to_integer_percent | Verifies that the percentage of parsable to integer string in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringParsableToIntegerPercentCheckSpec | |||
monthly_partition_string_parsable_to_float_percent | Verifies that the percentage of parsable to float string in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringParsableToFloatPercentCheckSpec | |||
monthly_partition_expected_strings_in_use_count | Verifies that the expected string values were found in the column. Raises a data quality issue when too many expected values were not found (were missing). Creates a separate data quality check (and an alert) for each monthly partition. | ColumnExpectedStringsInUseCountCheckSpec | |||
monthly_partition_string_value_in_set_percent | The check measures the percentage of rows whose value in a tested column is one of values from a list of expected values or the column value is null. Verifies that the percentage of rows having a valid column value does not exceed the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringValueInSetPercentCheckSpec | |||
monthly_partition_string_valid_dates_percent | Verifies that the percentage of valid dates in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringValidDatesPercentCheckSpec | |||
monthly_partition_string_valid_country_code_percent | Verifies that the percentage of valid country code in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringValidCountryCodePercentCheckSpec | |||
monthly_partition_string_valid_currency_code_percent | Verifies that the percentage of valid currency code in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringValidCurrencyCodePercentCheckSpec | |||
monthly_partition_string_invalid_email_count | Verifies that the number of invalid emails in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringInvalidEmailCountCheckSpec | |||
monthly_partition_string_invalid_uuid_count | Verifies that the number of invalid UUID in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringInvalidUuidCountCheckSpec | |||
monthly_partition_valid_uuid_percent | Verifies that the percentage of valid UUID in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringValidUuidPercentCheckSpec | |||
monthly_partition_string_invalid_ip4_address_count | Verifies that the number of invalid IP4 address in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringInvalidIp4AddressCountCheckSpec | |||
monthly_partition_string_invalid_ip6_address_count | Verifies that the number of invalid IP6 address in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringInvalidIp6AddressCountCheckSpec | |||
monthly_partition_string_not_match_regex_count | Verifies that the number of strings not matching the custom regex in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringNotMatchRegexCountCheckSpec | |||
monthly_partition_string_match_regex_percent | Verifies that the percentage of strings matching the custom regex in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringMatchRegexPercentCheckSpec | |||
monthly_partition_string_not_match_date_regex_count | Verifies that the number of strings not matching the date format regex in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringNotMatchDateRegexCountCheckSpec | |||
monthly_partition_string_match_date_regex_percent | Verifies that the percentage of strings matching the date format regex in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringMatchDateRegexPercentCheckSpec | |||
monthly_partition_string_match_name_regex_percent | Verifies that the percentage of strings matching the name format regex in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringMatchNameRegexPercentCheckSpec | |||
monthly_partition_expected_strings_in_top_values_count | Verifies that the top X most popular column values contain all values from a list of expected values. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnExpectedStringsInTopValuesCountCheckSpec | |||
monthly_partition_string_datatype_detected | Detects the data type of text values stored in the column. The sensor returns the code of the detected data type of a column: 1 - integers, 2 - floats, 3 - dates, 4 - timestamps, 5 - booleans, 6 - strings, 7 - mixed data types. Raises a data quality issue when the detected data type does not match the expected data type. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnStringDatatypeDetectedCheckSpec |
ColumnIntegrityMonthlyPartitionedChecksSpec
Container of integrity data quality partitioned checks on a column level that are checking monthly partitions or rows for each month of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_foreign_key_not_match_count | Verifies that the number of values in a column that does not match values in another table column does not exceed the set count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnIntegrityForeignKeyNotMatchCountCheckSpec | |||
monthly_partition_foreign_key_match_percent | Verifies that the percentage of values in a column that matches values in another table column does not exceed the set count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnIntegrityForeignKeyMatchPercentCheckSpec |
ColumnComparisonMonthlyPartitionedChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains configuration of column level comparison checks. Each column level check container also defines the name of the reference column name to which we are comparing.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
access_order | boolean | ||||
size | integer | ||||
mod_count | integer | ||||
threshold | integer |
ColumnUniquenessMonthlyPartitionedChecksSpec
Container of uniqueness data quality partitioned checks on a column level that are checking at a monthly level.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_distinct_count | Verifies that the number of distinct values in a column does not fall below the minimum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnDistinctCountCheckSpec | |||
monthly_partition_distinct_percent | Verifies that the percentage of distinct values in a column does not fall below the minimum accepted percent. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnDistinctPercentCheckSpec | |||
monthly_partition_duplicate_count | Verifies that the number of duplicate values in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnDuplicateCountCheckSpec | |||
monthly_partition_duplicate_percent | Verifies that the percent of duplicate values in a column does not exceed the maximum accepted percent. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnDuplicatePercentCheckSpec |
ColumnBoolMonthlyPartitionedChecksSpec
Container of boolean data quality partitioned checks on a column level that are checking monthly partitions or rows for each month of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_true_percent | Verifies that the percentage of true values in a column does not exceed the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnTruePercentCheckSpec | |||
monthly_partition_false_percent | Verifies that the percentage of false values in a column does not exceed the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnFalsePercentCheckSpec |
ColumnAnomalyMonthlyPartitionedChecksSpec
Container of built-in preconfigured data quality checks on a column level for detecting anomalies.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_mean_change | Verifies that the mean value in a column changed in a fixed rate since last readout. | ColumnChangeMeanCheckSpec | |||
monthly_partition_median_change | Verifies that the median in a column changed in a fixed rate since last readout. | ColumnChangeMedianCheckSpec | |||
monthly_partition_sum_change | Verifies that the sum in a column changed in a fixed rate since last readout. | ColumnChangeSumCheckSpec |
ColumnDatetimeMonthlyPartitionedChecksSpec
Container of date-time data quality partitioned checks on a column level that are checking monthly partitions or rows for each month of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_date_values_in_future_percent | Verifies that the percentage of date values in future in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnDateValuesInFuturePercentCheckSpec | |||
monthly_partition_datetime_value_in_range_date_percent | Verifies that the percentage of date values in the range defined by the user in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnDatetimeValueInRangeDatePercentCheckSpec |
ColumnAccuracyMonthlyPartitionedChecksSpec
Container of accuracy data quality partitioned checks on a column level that are checking monthly partitions or rows for each month of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
ColumnNumericMonthlyPartitionedChecksSpec
Container of numeric data quality partitioned checks on a column level that are checking at a monthly level.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_negative_count | Verifies that the number of negative values in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNegativeCountCheckSpec | |||
monthly_partition_negative_percent | Verifies that the percentage of negative values in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNegativePercentCheckSpec | |||
monthly_partition_non_negative_count | Verifies that the number of non-negative values in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNonNegativeCountCheckSpec | |||
monthly_partition_non_negative_percent | Verifies that the percentage of non-negative values in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNonNegativePercentCheckSpec | |||
monthly_partition_expected_numbers_in_use_count | Verifies that the expected numeric values were found in the column. Raises a data quality issue when too many expected values were not found (were missing). Creates a separate data quality check (and an alert) for each monthly partition. | ColumnExpectedNumbersInUseCountCheckSpec | |||
monthly_partition_number_value_in_set_percent | The check measures the percentage of rows whose value in a tested column is one of values from a list of expected values or the column value is null. Verifies that the percentage of rows having a valid column value does not exceed the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnNumberValueInSetPercentCheckSpec | |||
monthly_partition_values_in_range_numeric_percent | Verifies that the percentage of values from range in a column does not exceed the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnValuesInRangeNumericPercentCheckSpec | |||
monthly_partition_values_in_range_integers_percent | Verifies that the percentage of values from range in a column does not exceed the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnValuesInRangeIntegersPercentCheckSpec | |||
monthly_partition_value_below_min_value_count | The check counts the number of values in the column that is below the value defined by the user as a parameter. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnValueBelowMinValueCountCheckSpec | |||
monthly_partition_value_below_min_value_percent | The check counts the percentage of values in the column that is below the value defined by the user as a parameter. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnValueBelowMinValuePercentCheckSpec | |||
monthly_partition_value_above_max_value_count | The check counts the number of values in the column that is above the value defined by the user as a parameter. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnValueAboveMaxValueCountCheckSpec | |||
monthly_partition_value_above_max_value_percent | The check counts the percentage of values in the column that is above the value defined by the user as a parameter. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnValueAboveMaxValuePercentCheckSpec | |||
monthly_partition_max_in_range | Verifies that the maximal value in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnMaxInRangeCheckSpec | |||
monthly_partition_min_in_range | Verifies that the minimal value in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnMinInRangeCheckSpec | |||
monthly_partition_mean_in_range | Verifies that the average (mean) of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnMeanInRangeCheckSpec | |||
monthly_partition_percentile_in_range | Verifies that the percentile of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPercentileInRangeCheckSpec | |||
monthly_partition_median_in_range | Verifies that the median of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnMedianInRangeCheckSpec | |||
monthly_partition_percentile_10_in_range | Verifies that the percentile 10 of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPercentile10InRangeCheckSpec | |||
monthly_partition_percentile_25_in_range | Verifies that the percentile 25 of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPercentile25InRangeCheckSpec | |||
monthly_partition_percentile_75_in_range | Verifies that the percentile 75 of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPercentile75InRangeCheckSpec | |||
monthly_partition_percentile_90_in_range | Verifies that the percentile 90 of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPercentile90InRangeCheckSpec | |||
monthly_partition_sample_stddev_in_range | Verifies that the sample standard deviation of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnSampleStddevInRangeCheckSpec | |||
monthly_partition_population_stddev_in_range | Verifies that the population standard deviation of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPopulationStddevInRangeCheckSpec | |||
monthly_partition_sample_variance_in_range | Verifies that the sample variance of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnSampleVarianceInRangeCheckSpec | |||
monthly_partition_population_variance_in_range | Verifies that the population variance of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPopulationVarianceInRangeCheckSpec | |||
monthly_partition_sum_in_range | Verifies that the sum of all values in a column is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnSumInRangeCheckSpec | |||
monthly_partition_invalid_latitude_count | Verifies that the number of invalid latitude values in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnInvalidLatitudeCountCheckSpec | |||
monthly_partition_valid_latitude_percent | Verifies that the percentage of valid latitude values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnValidLatitudePercentCheckSpec | |||
monthly_partition_invalid_longitude_count | Verifies that the number of invalid longitude values in a column does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnInvalidLongitudeCountCheckSpec | |||
monthly_partition_valid_longitude_percent | Verifies that the percentage of valid longitude values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnValidLongitudePercentCheckSpec |
ColumnSqlMonthlyPartitionedChecksSpec
Container of built-in preconfigured data quality checks on a column level that are using custom SQL expressions (conditions).
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_sql_condition_passed_percent_on_column | Verifies that a minimum percentage of rows passed a custom SQL condition (expression). Creates a separate data quality check (and an alert) for each monthly partition. | ColumnSqlConditionPassedPercentCheckSpec | |||
monthly_partition_sql_condition_failed_count_on_column | Verifies that a number of rows failed a custom SQL condition(expression) does not exceed the maximum accepted count. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnSqlConditionFailedCountCheckSpec | |||
monthly_partition_sql_aggregate_expr_column | Verifies that a custom aggregated SQL expression (MIN, MAX, etc.) is not outside the set range. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnSqlAggregateExprCheckSpec |
ColumnPiiMonthlyPartitionedChecksSpec
Container of PII data quality partitioned checks on a column level that are checking monthly partitions or rows for each month of data.
The structure of this object is described below
Property name | Description | Data type | Enum values | Default value | Sample values |
---|---|---|---|---|---|
monthly_partition_valid_usa_phone_percent | Verifies that the percentage of valid USA phone values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiValidUsaPhonePercentCheckSpec | |||
monthly_partition_contains_usa_phone_percent | Verifies that the percentage of rows that contains USA phone number in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiContainsUsaPhonePercentCheckSpec | |||
monthly_partition_valid_usa_zipcode_percent | Verifies that the percentage of valid USA zip code values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiValidUsaZipcodePercentCheckSpec | |||
monthly_partition_contains_usa_zipcode_percent | Verifies that the percentage of rows that contains USA zip code in a column does not exceed the maximum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiContainsUsaZipcodePercentCheckSpec | |||
monthly_partition_valid_email_percent | Verifies that the percentage of valid emails values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiValidEmailPercentCheckSpec | |||
monthly_partition_contains_email_percent | Verifies that the percentage of rows that contains emails in a column does not exceed the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiContainsEmailPercentCheckSpec | |||
monthly_partition_valid_ip4_address_percent | Verifies that the percentage of valid IP4 address values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiValidIp4AddressPercentCheckSpec | |||
monthly_partition_contains_ip4_percent | Verifies that the percentage of rows that contains IP4 address values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiContainsIp4PercentCheckSpec | |||
monthly_partition_valid_ip6_address_percent | Verifies that the percentage of valid IP6 address values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiValidIp6AddressPercentCheckSpec | |||
monthly_partition_contains_ip6_percent | Verifies that the percentage of rows that contains valid IP6 address values in a column does not fall below the minimum accepted percentage. Creates a separate data quality check (and an alert) for each monthly partition. | ColumnPiiContainsIp6PercentCheckSpec |