tables
TableComparisonDailyMonitoringChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains the daily monitoring comparison checks for each configured reference table.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, TableComparisonDailyMonitoringChecksSpec] |
CustomCheckSpecMap
Dictionary of custom checks indexed by a check name.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, CustomCheckSpec] | |
| self | Dict[string, CustomCheckSpec] |
TableDailyMonitoringCheckCategoriesSpec
Container of table level daily monitoring. Contains categories of daily monitoring.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| volume | Daily monitoring volume data quality checks | TableVolumeDailyMonitoringChecksSpec |
| timeliness | Daily monitoring timeliness checks | TableTimelinessDailyMonitoringChecksSpec |
| accuracy | Daily monitoring accuracy checks | TableAccuracyDailyMonitoringChecksSpec |
| sql | Daily monitoring custom SQL checks | TableSqlDailyMonitoringChecksSpec |
| availability | Daily monitoring table availability checks | TableAvailabilityDailyMonitoringChecksSpec |
| schema | Daily monitoring table schema checks | TableSchemaDailyMonitoringChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | TableComparisonDailyMonitoringChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
TableComparisonDailyPartitionedChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains the daily partitioned comparison checks for each configured reference table.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, TableComparisonDailyPartitionedChecksSpec] |
TableDailyPartitionedCheckCategoriesSpec
Container of table level daily partitioned checks. Contains categories of daily partitioned checks.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| volume | Volume daily partitioned data quality checks that verify the quality of every day of data separately | TableVolumeDailyPartitionedChecksSpec |
| timeliness | Daily partitioned timeliness checks | TableTimelinessDailyPartitionedChecksSpec |
| sql | Custom SQL daily partitioned data quality checks that verify the quality of every day of data separately | TableSqlDailyPartitionedChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | TableComparisonDailyPartitionedChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
ProfilingTimePeriod
The time period for profiling checks (millisecond, daily, monthly, weekly, hourly). The default profiling check stores one value per month. When profiling checks is re-executed during the month, the previous profiling checks value is overwritten and only the most recent value is stored.
The structure of this object is described below
| Data type | Enum values |
|---|---|
| string | one_per_week all_results one_per_hour one_per_month one_per_day |
TableListModel
Table list model returned by the rest api that is limited only to the basic fields, excluding nested nodes.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| connection_name | Connection name. | string |
| table_hash | Table hash that identifies the table using a unique hash code. | long |
| target | Physical table details (a physical schema name and a physical table name). | PhysicalTableName |
| disabled | Disables all data quality checks on the table. Data quality checks will not be executed. | boolean |
| stage | Stage name. | string |
| filter | SQL WHERE clause added to the sensor queries. | string |
| priority | Table priority (1, 2, 3, 4, ...). The tables could be assigned a priority level. The table priority is copied into each data quality check result and a sensor result, enabling efficient grouping of more and less important tables during a data quality improvement project, when the data quality issues on higher priority tables are fixed before data quality issues on less important tables. | integer |
| owner | Table owner information like the data steward name or the business application name. | TableOwnerSpec |
| profiling_checks_result_truncation | Defines how many profiling checks results are stored for the table monthly. By default, DQOps will use the 'one_per_month' configuration and store only the most recent profiling checks result executed during the month. By changing this value, it is possible to store one value per day or even store all profiling checks results. | ProfilingTimePeriod |
| has_any_configured_checks | True when the table has any checks configured. | boolean |
| has_any_configured_profiling_checks | True when the table has any profiling checks configured. | boolean |
| has_any_configured_monitoring_checks | True when the table has any monitoring checks configured. | boolean |
| has_any_configured_partition_checks | True when the table has any partition checks configured. | boolean |
| partitioning_configuration_missing | True when the table has missing configuration of the "partition_by_column" column, making any partition checks fail when executed. | boolean |
| run_checks_job_template | Configured parameters for the "check run" job that should be pushed to the job queue in order to run all checks within this table. | CheckSearchFilters |
| run_profiling_checks_job_template | Configured parameters for the "check run" job that should be pushed to the job queue in order to run profiling checks within this table. | CheckSearchFilters |
| run_monitoring_checks_job_template | Configured parameters for the "check run" job that should be pushed to the job queue in order to run monitoring checks within this table. | CheckSearchFilters |
| run_partition_checks_job_template | Configured parameters for the "check run" job that should be pushed to the job queue in order to run partition partitioned checks within this table. | CheckSearchFilters |
| collect_statistics_job_template | Configured parameters for the "collect statistics" job that should be pushed to the job queue in order to run all statistics collectors within this table. | StatisticsCollectorSearchFilters |
| data_clean_job_template | Configured parameters for the "data clean" job that after being supplied with a time range should be pushed to the job queue in order to remove stored results connected with this table. | DeleteStoredDataQueueJobParameters |
| can_edit | Boolean flag that decides if the current user can update or delete this object. | boolean |
| can_collect_statistics | Boolean flag that decides if the current user can collect statistics. | boolean |
| can_run_checks | Boolean flag that decides if the current user can run checks. | boolean |
| can_delete_data | Boolean flag that decides if the current user can delete data (results). | boolean |
| yaml_parsing_error | Optional parsing error that was captured when parsing the YAML file. This field is null when the YAML file is valid. If an error was captured, this field returns the file parsing error message and the file location. | string |
TableModel
Full table model that returns the specification of a single table in the REST Api.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| connection_name | Connection name. | string |
| table_hash | Table hash that identifies the table using a unique hash code. | long |
| spec | Full table specification including all nested information, the table name is inside the 'target' property. | TableSpec |
| can_edit | Boolean flag that decides if the current user can update or delete this object. | boolean |
| yaml_parsing_error | Optional parsing error that was captured when parsing the YAML file. This field is null when the YAML file is valid. If an error was captured, this field returns the file parsing error message and the file location. | string |
TableComparisonMonthlyMonitoringChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains the monthly monitoring comparison checks for each configured reference table.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, TableComparisonMonthlyMonitoringChecksSpec] |
TableMonthlyMonitoringCheckCategoriesSpec
Container of table level monthly monitoring checks. Contains categories of monthly monitoring checks.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| volume | Monthly monitoring of volume data quality checks | TableVolumeMonthlyMonitoringChecksSpec |
| timeliness | Monthly monitoring of timeliness checks | TableTimelinessMonthlyMonitoringChecksSpec |
| accuracy | Monthly monitoring accuracy checks | TableAccuracyMonthlyMonitoringChecksSpec |
| sql | Monthly monitoring of custom SQL checks | TableSqlMonthlyMonitoringChecksSpec |
| availability | Daily partitioned availability checks | TableAvailabilityMonthlyMonitoringChecksSpec |
| schema | Monthly monitoring table schema checks | TableSchemaMonthlyMonitoringChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | TableComparisonMonthlyMonitoringChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
TableComparisonMonthlyPartitionedChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table. Contains the monthly partitioned comparison checks for each configured reference table.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, TableComparisonMonthlyPartitionedChecksSpec] |
TableMonthlyPartitionedCheckCategoriesSpec
Container of table level monthly partitioned checks. Contains categories of monthly partitioned checks.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| volume | Volume monthly partitioned data quality checks that verify the quality of every month of data separately | TableVolumeMonthlyPartitionedChecksSpec |
| timeliness | Monthly partitioned timeliness checks | TableTimelinessMonthlyPartitionedChecksSpec |
| sql | Custom SQL monthly partitioned data quality checks that verify the quality of every month of data separately | TableSqlMonthlyPartitionedChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | TableComparisonMonthlyPartitionedChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
TablePartitioningModel
Rest model that returns the configuration of table partitioning information.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| connection_name | Connection name. | string |
| target | Physical table details (a physical schema name and a physical table name) | PhysicalTableName |
| timestamp_columns | Column names that store the timestamps that identify the event (transaction) timestamp and the ingestion (inserted / loaded at) timestamps. Also configures the timestamp source for the date/time partitioned data quality checks (event timestamp or ingestion timestamp). | TimestampColumnsSpec |
| incremental_time_window | Configuration of time windows for executing partition checks incrementally, configures the number of recent days to analyze for daily partitioned tables or the number of recent months for monthly partitioned data. | PartitionIncrementalTimeWindowSpec |
| can_edit | Boolean flag that decides if the current user can update or delete this object. | boolean |
TableComparisonProfilingChecksSpecMap
Container of comparison checks for each defined data comparison. The name of the key in this dictionary must match a name of a table comparison that is defined on the parent table.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| self | Dict[string, TableComparisonProfilingChecksSpec] |
TableProfilingCheckCategoriesSpec
Container of table level checks that are activated on a table level.
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| result_truncation | Defines how many profiling checks results are stored for the table monthly. By default, DQOps will use the 'one_per_month' configuration and store only the most recent profiling checks result executed during the month. By changing this value, it is possible to store one value per day or even store all profiling checks results. | ProfilingTimePeriod |
| volume | Configuration of volume data quality checks on a table level. | TableVolumeProfilingChecksSpec |
| timeliness | Configuration of timeliness checks on a table level. Timeliness checks detect anomalies like rapid row count changes. | TableTimelinessProfilingChecksSpec |
| accuracy | Configuration of accuracy checks on a table level. Accuracy checks compare the tested table with another reference table. | TableAccuracyProfilingChecksSpec |
| sql | Configuration of data quality checks that are evaluating custom SQL conditions and aggregated expressions. | TableSqlProfilingChecksSpec |
| availability | Configuration of the table availability data quality checks on a table level. | TableAvailabilityProfilingChecksSpec |
| schema | Configuration of schema (column count and schema) data quality checks on a table level. | TableSchemaProfilingChecksSpec |
| comparisons | Dictionary of configuration of checks for table comparisons. The key that identifies each comparison must match the name of a data comparison that is configured on the parent table. | TableComparisonProfilingChecksSpecMap |
| custom | Dictionary of custom checks. The keys are check names within this category. | CustomCheckSpecMap |
TableStatisticsModel
Model that returns a summary of the table level statistics (the basic profiling results).
The structure of this object is described below
| Property name | Description | Data type |
|---|---|---|
| connection_name | Connection name. | string |
| table | Physical table name including the schema and table names. | PhysicalTableName |
| collect_table_statistics_job_template | Configured parameters for the "collect statistics" job that should be pushed to the job queue in order to run all statistics collectors within this table, limited only to the table level statistics (row count, etc). | StatisticsCollectorSearchFilters |
| collect_table_and_column_statistics_job_template | Configured parameters for the "collect statistics" job that should be pushed to the job queue in order to run all statistics collectors within this table, including statistics for all columns. | StatisticsCollectorSearchFilters |
| can_collect_statistics | Boolean flag that decides if the current user can collect statistics. | boolean |