Last updated: July 22, 2025
List of table level timeliness data quality checks
This is a list of timeliness table data quality checks supported by DQOps and a brief description of what data quality issued they detect.
table-level timeliness checks
Assesses the freshness and staleness of data, as well as data ingestion delay and reload lag for partitioned data.
data freshness
A table-level check that calculates the time difference between the most recent row in the table and the current time. The timestamp column that is used for comparison is defined as the timestamp_columns.event_timestamp_column on the table configuration. This check is also known as "Data Freshness".
| Data quality check name | Friendly name | Check type | Description | Standard |
|---|---|---|---|---|
profile_data_freshness |
Data freshness (Maximum age of the most recent row) | profiling | Calculates the number of days since the most recent event timestamp (freshness) | |
daily_data_freshness |
Data freshness (Maximum age of the most recent row) | monitoring | Daily calculating the number of days since the most recent event timestamp (freshness) | |
monthly_data_freshness |
Data freshness (Maximum age of the most recent row) | monitoring | Monthly monitoring calculating the number of days since the most recent event timestamp (freshness) |
data freshness anomaly
This check calculates the most recent rows value and the current time and detects anomalies in a time series of previous averages. The timestamp column that is used for comparison is defined as the timestamp_columns.event_timestamp_column on the table configuration. It raises a data quality issue when the mean is in the top anomaly_percent percentage of the most outstanding values in the time series. This data quality check uses a 90-day time window and requires a history of at least 30 days.
| Data quality check name | Friendly name | Check type | Description | Standard |
|---|---|---|---|---|
profile_data_freshness_anomaly |
Data freshness anomaly (Abnormal delay in data delivery) | profiling | Verifies that the number of days since the most recent event timestamp (freshness) changes in a rate within a percentile boundary during the last 90 days. | |
daily_data_freshness_anomaly |
Data freshness anomaly (Abnormal delay in data delivery) | monitoring | Verifies that the number of days since the most recent event timestamp (freshness) changes in a rate within a percentile boundary during the last 90 days. |
data staleness
A table-level check that calculates the time difference between the last timestamp when any data was loaded into a table and the current time. This check can only be use when a data pipeline, ETL process, or trigger in the data warehouse is filling an extra column with the timestamp when the data loading job was loaded. The ingestion column used for comparison is defined as the timestamp_columns.ingestion_timestamp_column on the table configuration. This check is also known as "Data Staleness".
| Data quality check name | Friendly name | Check type | Description | Standard |
|---|---|---|---|---|
profile_data_staleness |
Data staleness (Maximum number of days since the recent ingestion) | profiling | Calculates the time difference in days between the current date and the most recent data ingestion timestamp (staleness) | |
daily_data_staleness |
Data staleness (Maximum number of days since the recent ingestion) | monitoring | Daily calculating the time difference in days between the current date and the most recent data ingestion timestamp (staleness) | |
monthly_data_staleness |
Data staleness (Maximum number of days since the recent ingestion) | monitoring | Monthly monitoring calculating the time difference in days between the current date and the most recent data ingestion timestamp (staleness) |
data ingestion delay
A table-level check that calculates the time difference between the most recent row in the table and the most recent timestamp when the last row was loaded into the data warehouse or data lake. To identify the most recent row, the check finds the maximum value of the timestamp column that should contain the last modification timestamp from the source. The timestamp when the row was loaded is identified by the most recent (maximum) value a timestamp column that was filled by the data pipeline, for example: "loaded_at", "updated_at", etc. This check requires that the data pipeline is filling an extra column with the timestamp when the data loading job has been executed. The names of both columns used for comparison should be specified in the "timestamp_columns" configuration entry on the table.
| Data quality check name | Friendly name | Check type | Description | Standard |
|---|---|---|---|---|
profile_data_ingestion_delay |
Data ingestion delay (Maximum number of days between the last record has been created and loaded) | profiling | Calculates the time difference in days between the most recent event timestamp and the most recent ingestion timestamp | |
daily_data_ingestion_delay |
Data ingestion delay (Maximum number of days between the last record has been created and loaded) | monitoring | Daily calculating the time difference in days between the most recent event timestamp and the most recent ingestion timestamp | |
monthly_data_ingestion_delay |
Data ingestion delay (Maximum number of days between the last record has been created and loaded) | monitoring | Monthly monitoring calculating the time difference in days between the most recent event timestamp and the most recent ingestion timestamp | |
daily_partition_data_ingestion_delay |
Data ingestion delay (Maximum number of days between the last record has been created and loaded) | partitioned | Daily partitioned check calculating the time difference in days between the most recent event timestamp and the most recent ingestion timestamp | |
monthly_partition_data_ingestion_delay |
Data ingestion delay (Maximum number of days between the last record has been created and loaded) | partitioned | Monthly partitioned check calculating the time difference in days between the most recent event timestamp and the most recent ingestion timestamp |
reload lag
A table-level check that calculates the maximum difference in days between ingestion timestamp and event timestamp values on any row. This check should be executed only as a partitioned check because this check finds the longest delay between the time that the row was created in the data source and the timestamp when the row was loaded into its daily or monthly partition. This check detects that a daily or monthly partition was reloaded, setting also the most recent timestamps in the created_at, loaded_at, inserted_at or other similar columns filled by the data pipeline or an ETL process during data loading.
| Data quality check name | Friendly name | Check type | Description | Standard |
|---|---|---|---|---|
daily_partition_reload_lag |
Reload lag (Maximum delay to load the last record for each partition) | partitioned | Daily partitioned check calculating the longest time a row waited to be loaded, it is the maximum difference in days between the ingestion timestamp and the event timestamp column on any row in the monitored partition | |
monthly_partition_reload_lag |
Reload lag (Maximum delay to load the last record for each partition) | partitioned | Monthly partitioned check calculating the longest time a row waited to be loaded, it is the maximum difference in days between the ingestion timestamp and the event timestamp column on any row in the monitored partition |