columns
ColumnComparisonDailyMonitoringChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains configuration of column level comparison checks. Each column level check container also defines the name of the reference column name to which we are comparing.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, ColumnComparisonDailyMonitoringChecksSpec] |
CustomCheckSpecMap
Dictionary of custom checks indexed by a check name.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, CustomCheckSpec] |
ColumnDailyMonitoringCheckCategoriesSpec
Container of column level daily monitoring checks. Contains categories of daily monitoring checks.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| nulls | Daily monitoring checks of nulls in the column | ColumnNullsDailyMonitoringChecksSpec |
| numeric | Daily monitoring checks of numeric in the column | ColumnNumericDailyMonitoringChecksSpec |
| strings | Daily monitoring checks of strings in the column | ColumnStringsDailyMonitoringChecksSpec |
| uniqueness | Daily monitoring checks of uniqueness in the column | ColumnUniquenessDailyMonitoringChecksSpec |
| datetime | Daily monitoring checks of datetime in the column | ColumnDatetimeDailyMonitoringChecksSpec |
| pii | Daily monitoring checks of Personal Identifiable Information (PII) in the column | ColumnPiiDailyMonitoringChecksSpec |
| sql | Daily monitoring checks of custom SQL checks in the column | ColumnSqlDailyMonitoringChecksSpec |
| bool | Daily monitoring checks of booleans in the column | ColumnBoolDailyMonitoringChecksSpec |
| integrity | Daily monitoring checks of integrity in the column | ColumnIntegrityDailyMonitoringChecksSpec |
| accuracy | Daily monitoring checks of accuracy in the column | ColumnAccuracyDailyMonitoringChecksSpec |
| datatype | Daily monitoring checks of datatype in the column | ColumnDatatypeDailyMonitoringChecksSpec |
| anomaly | Daily monitoring checks of anomaly in the column | ColumnAnomalyDailyMonitoringChecksSpec |
| schema | Daily monitoring column schema checks | ColumnSchemaDailyMonitoringChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons at a column level. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | ColumnComparisonDailyMonitoringChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
ColumnComparisonDailyPartitionedChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains configuration of column level comparison checks. Each column level check container also defines the name of the reference column name to which we are comparing.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, ColumnComparisonDailyPartitionedChecksSpec] |
ColumnDailyPartitionedCheckCategoriesSpec
Container of data quality partitioned checks on a column level that are checking numeric values at a daily level.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| nulls | Daily partitioned checks of nulls in the column | ColumnNullsDailyPartitionedChecksSpec |
| numeric | Daily partitioned checks of numeric in the column | ColumnNumericDailyPartitionedChecksSpec |
| strings | Daily partitioned checks of strings in the column | ColumnStringsDailyPartitionedChecksSpec |
| uniqueness | Daily partitioned checks of uniqueness in the column | ColumnUniquenessDailyPartitionedChecksSpec |
| datetime | Daily partitioned checks of datetime in the column | ColumnDatetimeDailyPartitionedChecksSpec |
| pii | Daily partitioned checks of Personal Identifiable Information (PII) in the column | ColumnPiiDailyPartitionedChecksSpec |
| sql | Daily partitioned checks using custom SQL expressions evaluated on the column | ColumnSqlDailyPartitionedChecksSpec |
| bool | Daily partitioned checks for booleans in the column | ColumnBoolDailyPartitionedChecksSpec |
| integrity | Daily partitioned checks for integrity in the column | ColumnIntegrityDailyPartitionedChecksSpec |
| accuracy | Daily partitioned checks for accuracy in the column | ColumnAccuracyDailyPartitionedChecksSpec |
| datatype | Daily partitioned checks for datatype in the column | ColumnDatatypeDailyPartitionedChecksSpec |
| anomaly | Daily partitioned checks for anomaly in the column | ColumnAnomalyDailyPartitionedChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons at a column level. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | ColumnComparisonDailyPartitionedChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
PhysicalTableName
Physical table name that is a combination of a schema name and a physical table name (without any quoting or escaping).
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| schema_name | Schema name | string |
| table_name | Table name | string |
ColumnListModel
Column list model that returns the basic fields from a column specification, excluding nested nodes like a list of activated checks.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| connection_name | Connection name. | string |
| table | Physical table name including the schema and table names. | PhysicalTableName |
| column_name | Column names. | string |
| sql_expression | SQL expression. | string |
| column_hash | Column hash that identifies the column using a unique hash code. | long |
| disabled | Disables all data quality checks on the column. Data quality checks will not be executed. | boolean |
| has_any_configured_checks | True when the column has any checks configured. | boolean |
| has_any_configured_profiling_checks | True when the column has any profiling checks configured. | boolean |
| has_any_configured_monitoring_checks | True when the column has any monitoring checks configured. | boolean |
| has_any_configured_partition_checks | True when the column has any partition checks configured. | boolean |
| type_snapshot | Column data type that was retrieved when the table metadata was imported. | ColumnTypeSnapshotSpec |
| run_checks_job_template | Configured parameters for the "check run" job that should be pushed to the job queue in order to run all checks within this column. | CheckSearchFilters |
| run_profiling_checks_job_template | Configured parameters for the "check run" job that should be pushed to the job queue in order to run profiling checks within this column. | CheckSearchFilters |
| run_monitoring_checks_job_template | Configured parameters for the "check run" job that should be pushed to the job queue in order to run monitoring checks within this column. | CheckSearchFilters |
| run_partition_checks_job_template | Configured parameters for the "check run" job that should be pushed to the job queue in order to run partition partitioned checks within this column. | CheckSearchFilters |
| collect_statistics_job_template | Configured parameters for the "collect statistics" job that should be pushed to the job queue in order to run all statistics collector within this column. | StatisticsCollectorSearchFilters |
| data_clean_job_template | Configured parameters for the "data clean" job that after being supplied with a time range should be pushed to the job queue in order to remove stored results connected with this column. | DeleteStoredDataQueueJobParameters |
| can_edit | Boolean flag that decides if the current user can update or delete the column. | boolean |
| can_collect_statistics | Boolean flag that decides if the current user can collect statistics. | boolean |
| can_run_checks | Boolean flag that decides if the current user can run checks. | boolean |
| can_delete_data | Boolean flag that decides if the current user can delete data (results). | boolean |
ColumnModel
Table model that returns the specification of a single column in the REST Api.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| connection_name | Connection name. | string |
| table | Physical table name including the schema and table names. | PhysicalTableName |
| column_name | Column name. | string |
| column_hash | Column hash that identifies the column using a unique hash code. | long |
| spec | Full column specification. | ColumnSpec |
| can_edit | Boolean flag that decides if the current user can update or delete this object. | boolean |
| yaml_parsing_error | Optional parsing error that was captured when parsing the YAML file. This field is null when the YAML file is valid. If an error was captured, this field returns the file parsing error message and the file location. | string |
ColumnComparisonMonthlyMonitoringChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains configuration of column level comparison checks. Each column level check container also defines the name of the reference column name to which we are comparing.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, ColumnComparisonMonthlyMonitoringChecksSpec] |
ColumnMonthlyMonitoringCheckCategoriesSpec
Container of column level monthly monitoring checks. Contains categories of monthly monitoring checks.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| nulls | Monthly monitoring checks of nulls in the column | ColumnNullsMonthlyMonitoringChecksSpec |
| numeric | Monthly monitoring checks of numeric in the column | ColumnNumericMonthlyMonitoringChecksSpec |
| strings | Monthly monitoring checks of strings in the column | ColumnStringsMonthlyMonitoringChecksSpec |
| uniqueness | Monthly monitoring checks of uniqueness in the column | ColumnUniquenessMonthlyMonitoringChecksSpec |
| datetime | Monthly monitoring checks of datetime in the column | ColumnDatetimeMonthlyMonitoringChecksSpec |
| pii | Monthly monitoring checks of Personal Identifiable Information (PII) in the column | ColumnPiiMonthlyMonitoringChecksSpec |
| sql | Monthly monitoring checks of custom SQL checks in the column | ColumnSqlMonthlyMonitoringChecksSpec |
| bool | Monthly monitoring checks of booleans in the column | ColumnBoolMonthlyMonitoringChecksSpec |
| integrity | Monthly monitoring checks of integrity in the column | ColumnIntegrityMonthlyMonitoringChecksSpec |
| accuracy | Monthly monitoring checks of accuracy in the column | ColumnAccuracyMonthlyMonitoringChecksSpec |
| datatype | Monthly monitoring checks of datatype in the column | ColumnDatatypeMonthlyMonitoringChecksSpec |
| anomaly | Monthly monitoring checks of anomaly in the column | ColumnAnomalyMonthlyMonitoringChecksSpec |
| schema | Monthly monitoring column schema checks | ColumnSchemaMonthlyMonitoringChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons at a column level. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | ColumnComparisonMonthlyMonitoringChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
ColumnComparisonMonthlyPartitionedChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains configuration of column level comparison checks. Each column level check container also defines the name of the reference column name to which we are comparing.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, ColumnComparisonMonthlyPartitionedChecksSpec] |
ColumnMonthlyPartitionedCheckCategoriesSpec
Container of data quality partitioned checks on a column level that are checking numeric values at a monthly level.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| nulls | Monthly partitioned checks of nulls values in the column | ColumnNullsMonthlyPartitionedChecksSpec |
| numeric | Monthly partitioned checks of numeric values in the column | ColumnNumericMonthlyPartitionedChecksSpec |
| strings | Monthly partitioned checks of strings values in the column | ColumnStringsMonthlyPartitionedChecksSpec |
| uniqueness | Monthly partitioned checks of uniqueness values in the column | ColumnUniquenessMonthlyPartitionedChecksSpec |
| datetime | Monthly partitioned checks of datetime values in the column | ColumnDatetimeMonthlyPartitionedChecksSpec |
| pii | Monthly partitioned checks of Personal Identifiable Information (PII) in the column | ColumnPiiMonthlyPartitionedChecksSpec |
| sql | Monthly partitioned checks using custom SQL expressions and conditions on the column | ColumnSqlMonthlyPartitionedChecksSpec |
| bool | Monthly partitioned checks for booleans in the column | ColumnBoolMonthlyPartitionedChecksSpec |
| integrity | Monthly partitioned checks for integrity in the column | ColumnIntegrityMonthlyPartitionedChecksSpec |
| accuracy | Monthly partitioned checks for accuracy in the column | ColumnAccuracyMonthlyPartitionedChecksSpec |
| datatype | Monthly partitioned checks for datatype in the column | ColumnDatatypeMonthlyPartitionedChecksSpec |
| anomaly | Monthly partitioned checks for anomaly in the column | ColumnAnomalyMonthlyPartitionedChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons at a column level. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | ColumnComparisonMonthlyPartitionedChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
ColumnComparisonProfilingChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains configuration of column level comparison checks. Each column level check container also defines the name of the reference column name to which we are comparing.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, ColumnComparisonProfilingChecksSpec] |
ColumnProfilingCheckCategoriesSpec
Container of column level, preconfigured checks.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| nulls | Configuration of column level checks that verify nulls and blanks. | ColumnNullsProfilingChecksSpec |
| numeric | Configuration of column level checks that verify negative values. | ColumnNumericProfilingChecksSpec |
| strings | Configuration of strings checks on a column level. | ColumnStringsProfilingChecksSpec |
| uniqueness | Configuration of uniqueness checks on a column level. | ColumnUniquenessProfilingChecksSpec |
| datetime | Configuration of datetime checks on a column level. | ColumnDatetimeProfilingChecksSpec |
| pii | Configuration of Personal Identifiable Information (PII) checks on a column level. | ColumnPiiProfilingChecksSpec |
| sql | Configuration of SQL checks that use custom SQL aggregated expressions and SQL conditions in data quality checks. | ColumnSqlProfilingChecksSpec |
| bool | Configuration of booleans checks on a column level. | ColumnBoolProfilingChecksSpec |
| integrity | Configuration of integrity checks on a column level. | ColumnIntegrityProfilingChecksSpec |
| accuracy | Configuration of accuracy checks on a column level. | ColumnAccuracyProfilingChecksSpec |
| datatype | Configuration of datatype checks on a column level. | ColumnDatatypeProfilingChecksSpec |
| anomaly | Configuration of anomaly checks on a column level. | ColumnAnomalyProfilingChecksSpec |
| schema | Configuration of schema checks on a column level. | ColumnSchemaProfilingChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons at a column level. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | ColumnComparisonProfilingChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
ColumnStatisticsModel
Column model that returns the basic fields from a column specification and a summary of the most recent statistics collection.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| connection_name | Connection name. | string |
| table | Physical table name including the schema and table names. | PhysicalTableName |
| column_name | Column name. | string |
| column_hash | Column hash that identifies the column using a unique hash code. | long |
| disabled | Disables all data quality checks on the column. Data quality checks will not be executed. | boolean |
| has_any_configured_checks | True when the column has any checks configured. | boolean |
| type_snapshot | Column data type that was retrieved when the table metadata was imported. | ColumnTypeSnapshotSpec |
| collect_column_statistics_job_template | Configured parameters for the "collect statistics" job that should be pushed to the job queue in order to run all statistics collectors for this column | StatisticsCollectorSearchFilters |
| can_collect_statistics | Boolean flag that decides if the current user can collect statistics. | boolean |
TableColumnsStatisticsModel
Model that returns a summary of the column statistics (the basic profiling results) for a single table, showing statistics for all columns.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| connection_name | Connection name. | string |
| table | Physical table name including the schema and table names. | PhysicalTableName |
| collect_column_statistics_job_template | Configured parameters for the "collect statistics" job that should be pushed to the job queue in order to run all statistics collectors for all columns on this table. | StatisticsCollectorSearchFilters |
| can_collect_statistics | Boolean flag that decides if the current user can collect statistics. | boolean |