Data observability for Data Ingestion

Ensure that your data sources are pulled correctly

How often do you hear that the data received from external sources is wrong, but it was correct in the past?

Data Observability is a way to define Data Quality rules to monitor your ingestion tables. Detect schema changes, data format changes, missing data or inconsistent delays in the data delivery.

Data format and ranges

Data format and ranges

Detect data format and data ranges issues in source data before the data pipeline fails on the transformation steps.


Validity checks like the data format, not null, data ranges or uniqueness checks are defined for each source table. Data Quality checks are executed after the source data was loaded into your ingestion tables. Any Data Quality issues are easy to understand. Order of columns in a CSV file has changes and the data was loaded into different columns? Data format or distinct column count checks will detect it.

  • Define data format and data range rules for source data
  • Run Data Quality checks at the end of the data ingestion pipeline
  • Detect data format issues by detecting unexpected changes to distinct counts of column values

Schema change detection

Schema change detection

Monitor unexpected changes to the table schema that will affect the downstream. Find out that a column size was increased and some rows may not load to downstream tables.


DQO.ai captures the table schema when the table metadata is retrieved for the first time. The list of columns is hashed to detect schema changes. Additionally column data types are compared every time when the schema checks are executed.

  • Detect column data type changes
  • Detect column count changes, added or removed columns
  • Detect a change to the order of columns that may turn SELECT * queries unusable

Data delays and stale tables

Data delays and stale tables

Analyze the growth (row count increase) and data lags (most recent rows) to detect tables that are not refreshed recently.


DQO.ai can analyze the timeliness of the data to measure the average lag of data. When your rows do not have a timestamp column, DQO.ai can analyze the growth of the row count by measuring the row count every day and learning the average table growth rate.

  • Detect tables that are not refreshed recently
  • Find out which tables are not updated frequently in their data sources (inconsistent refresh)
  • Detect tables that receive less data than the average

Ground Truth Checks

Ground Truth Checks

Compare the source data with other trusted sources to ensure that your database shows accurate data.


DQO.ai can run accuracy checks that will compare the table with a real world reference data. Just define the other table or a query that returns the same aggregated data (sum, count, etc.) grouped by the same business dimension. Also compare the data with flat files that you can load to the Data Quality database.

  • Compare the data with the real world, reference data
  • Detect issues at a business relevant dimension (date, country, city, department, state, etc.)
  • Ensure that you really trust the source data and you can proof it

No one can understand your data like we do!