Data completeness means that all of the information that is needed is there and all required values are present. It is a way to measure how good the data is and if it is useful for things like analysis, modeling, and making decisions. Data completeness is important because if the data is incomplete, it can lead to biased or inaccurate results, and it can be hard to draw conclusions from it. Data completeness is also one of the data quality dimensions, which are categories of data quality problems that can affect data.
Organizations that are actively improving data completeness are tracking all new incidents related to missing or incomplete data by raising, tracking, and fixing data quality issues. By assigning roles and responsibilities to the teams and departments that can fix these issues, the organization can react quickly to incidents related to incomplete data and fix the problems before the incidents affect the business and are visible to users or customers.
Table of Contents
You can monitor data completeness for free
Before you keep reading, DQOps Data Quality Operations Center is a data observability platform that measures and improves data completeness. Please refer to the DQOps documentation to learn how to get started.
Examples of data completeness issues
The best way to understand data completeness is to learn about the most common completeness issues and why they occur. These issues are divided into two categories: incomplete data at a record level and incomplete or missing data at a dataset or table level.
The simplest and also the most severe record-level completeness issue is a missing value in a required field that stores a business identifier that is required for querying, analysis, data matching, or further data transformation. More subtle examples of missing values are related to some dependencies between values inside a single record. A customer record that contains the street name should also contain the city name, otherwise the address is incomplete without providing the city. There is another unique category of incomplete data. The record can contain invalid values that were used as placeholders for missing or out-of-range values. For example, Microsoft Excel shows a “#n/a” placeholder instead of out-of-range values.
Record-level issues are caused by human errors during data collection, especially when the user interface of the system-of-record does not perform instant validation of invalid values. The next source of these issues comes from the data transformation code. Incorrectly configured field mappings in ETL processes or incorrect transformations in custom data pipelines can turn valid values into null values. This problem happens when the data types between the source and destination systems differ and a value cannot be correctly converted, so the data transformation system chooses to discard too big values.
Table-level issues are simpler. They happen when the tables are empty or too small because data processing failed during a data refresh operation. Usually, it happens when the system runs out of disk space, the credentials expire, or some data files are abandoned due to timeouts in data processing.
Consequences of missing data
Issues related to missing data can have several adverse effects. One of the primary concerns is inaccurate results. Data analysis with missing values in a dataset can yield biased statistical results. This can lead to misinterpretation of data and incorrect conclusions. For example, a study analyzing the effectiveness of a new medical treatment may have missing data on patient outcomes. This missing data could potentially skew the results of the analysis, leading to an inaccurate conclusion about the treatment’s efficacy.
Another consequence of completeness issues is wasted time and resources. Data analysts may spend a significant amount of time attempting to correct a dataset with numerous missing or invalid values. This can be a time-consuming and frustrating process, diverting resources that could be better spent on other tasks. Furthermore, the effort expended in correcting the data may not always be successful, resulting in further wasted time and resources.
How to prevent from missing or lost data
You can avoid missing data in the system or record only when the system provides data validation logic embedded in these platforms. Adding a data validation step at data entry effectively prevents all record-level missing values. Some applications, such as CRM, ERP, or line-of-business applications, may not provide configuration options to enforce data completeness validation at data entry, limiting this option to post-validation.
The next stage where you can prevent incomplete data is inside the data loading and transformation pipelines, which replicate the data to downstream data stores, such as data lakes and data warehouses. If we detect incomplete data in the data source, the data transformation can be stopped until the problem is resolved. The data pipeline code must test and assess the data completeness of the data source by measuring if the data completeness level is satisfactory to proceed.
The data transformation code could also be the source of data completeness issues, especially when the data types did not match. In that situation, missing data will be loaded to downstream data stores. The only solution to detect and fix this problem is to periodically analyze the data for completeness issues and raise data quality incidents that are forwarded to respective data teams for resolution.
How to measure
Data completeness is one of the core data quality dimensions. It is measured by running data quality checks on the data sources. These checks analyze the record-level and table-level completeness metrics, detecting missing values and datasets.
Applications that analyze data for completeness and other data quality issues are called Data Observability platforms. One such platform is DQOps Data Quality Operations Center, acombined data quality and data observability solution. A data observability platform continuously monitors data sources, evaluates data quality rules, identifies data quality issues, and manages the resolution workflow of data quality incidents.
Data observability platforms can also detect non-obvious data completeness issues by performing anomaly detection, and applying machine learning to detect when the data completeness is decreasing.
By running data quality checks and counting the data quality issues, DQOps measures the completeness of data sources with a data quality KPI score. It is a percentage measure that proves the reliability and trustworthiness of the data source. The goal is to achieve a 100% data completeness score.
How to improve data completeness
A data observability platform that monitors data sources will raise data quality incidents related to missing data or null values. The platform will notify the respective data teams or data asset owners about identified issues. It is now the responsibility of the data team to fix invalid data transformation code that truncated values or the owner of the source system to activate additional data validation in the source system. As soon as the root cause of the problem is identified, the data teams can apply corrections, reload the data and revalidate the data completeness again.
You can also use a data observability platform to prevent those issues from spreading to downstream systems. A data observability platform, such as DQOps, can expose a REST API to execute data quality checks on the data source or to ask for the current data completeness status of the data. The data pipeline can perform a pre-validation step to verify in a data observability platform that the source data is free from completeness issues.
By measuring data completeness with data quality KPI scores, the data teams can prioritize the affected dataset and focus on the most valuable data assets first. Within a few weeks, it is possible to remediate all sources of issues and improve data quality to reach almost a 100% data completeness KPI score.
If you are interested in our experience of applying data observability across different data domains, you can download our free eBook “A step-by-step guide to improve data quality“. This eBook is the reference of our business process that we follow to clean data sources.
Data quality best practices - a step-by-step guide to improve data quality
- Learn the best practices in starting and scaling data quality
- Learn how to find and manage data quality issues
How to start
The Data Observability market is occupied by many vendors, and most solutions are closed-source SaaS platforms. You can start a trial period on these platforms and expose access to your data sources from the cloud to run data monitoring on your systems.
Another option is faster and avoids exposing your data to a SaaS vendor. You can try DQOps, our source-available data quality platform. You can set up DQOps locally or in your on-premise environment to learn how data observability can help detect missing data.
Follow the DQOps documentation, go through the DQOps getting started guide to learn how to set up DQOps locally, and try it.
