The data platform lifecycle is transitioning the platform and its data through the development, deployment, and operations stages.
A data platform is the backbone of a modern, data-driven organization. It’s a comprehensive software system that brings together the tools, technologies, and processes needed to collect, store, manage, process, analyze, and visualize data from various sources. Think of it as the central hub that allows businesses to harness the power of their information, whether it’s uncovering customer trends, optimizing operations, or making strategic decisions. In essence, a data platform is the engine room that transforms raw data into actionable insights, fueling innovation and growth.
Data platforms are often built upon a foundation of either data lakes or traditional data warehouses. In both cases, the data platform functions as a sophisticated pipeline, ingesting raw data from diverse sources, loading it into the chosen storage infrastructure (the lake or warehouse), and then transforming that data into a usable, structured format. This entire process is orchestrated by data engineers, who ensure that all requested data is properly loaded and transformed, making it ready for consumption. Once the platform is primed, it’s handed off to data analysts and data scientists who then have the freedom to select and utilize the specific data sets they require for their analytical and machine learning projects.
Data platform maturity stages
The lifecycle of a data platform is a dynamic journey, evolving through distinct stages from its initial creation to eventual decommissioning. The first phase, aptly named “development,” is primarily the domain of data engineers. Their expertise is focused on connecting disparate data sources, configuring intricate data ingestion jobs, orchestrating complex transformations, and ensuring the smooth loading of data into the platform. It’s a period of intensive construction, where the platform’s architecture and capabilities take shape.
Once the platform is populated with data and made accessible, it transitions into the “consumption” stage. Here, data consumers, including analysts and scientists, take center stage. They begin to explore the available data assets, identify those relevant to their projects, and potentially request adjustments to the data model to better align with their needs. In a mature organization with established data governance practices, data stewards play a crucial role in this phase, rigorously testing and validating the quality of the data assets. The final stage, “operations,” sees the data platform integrated into the wider ecosystem of applications that draw upon its data. A dedicated data operations team takes the reins, monitoring the platform’s health, maintaining its performance, and swiftly responding to any incidents that may arise. Over time, as technologies evolve or business needs shift, a data platform may become obsolete. At this point, it’s typically archived and decommissioned, marking the end of its lifecycle.
Why ensuring data quality at all stages is essential
Data quality is the cornerstone of any successful data platform. It refers to the overall reliability, accuracy, completeness, consistency, and timeliness of data. High-quality data is essential for making informed business decisions, driving accurate analysis, and ensuring the success of machine learning models. Data quality issues, such as missing data (e.g., an address field not populated in new customer records), inaccurate data, or inconsistent formats, can lead to erroneous conclusions and undermine the entire purpose of the data platform.
The responsibility for managing data quality shifts throughout the data platform lifecycle, involving different teams and departments at each stage. During the development phase, data engineers primarily handle data quality by implementing data contracts. These contracts detect changes in table schemas, data types, or other basic issues like duplicate data or values in incorrect formats. Once the platform transitions to the consumption stage, data consumers focus on thorough data quality profiling and testing to validate the data’s usability, accuracy, and reasonableness. This stage often involves more advanced data quality checks, ideally performed using a dedicated data quality platform that can automate the process and continuously monitor data quality over time. As the data platform matures and enters the operations phase, the operations team takes over, occasionally addressing data quality issues that may arise. A data quality platform equipped with incident workflow management and a user interface for rerunning checks can significantly enhance the efficiency of their response to such issues.
How to ensure data quality when developing a data platform
The development stage is where the data platform comes to life, primarily through the efforts of data engineers. Their tasks encompass everything from establishing connections to diverse data sources and configuring data ingestion jobs, to orchestrating intricate data transformations and ensuring the seamless loading of data into the platform. A critical aspect of this phase is implementing data quality checks.
There are various approaches to handling data quality checks during development. One common, yet short-sighted, method involves hardcoding checks directly into the data pipeline code using Python data quality libraries or configuring basic checks within the data transformation engine (e.g., Dbt tests). While this may seem expedient initially, it has several drawbacks. Firstly, the built-in data quality checks offered by data transformation platforms are often limited to basic structural validations like data types, required columns, and duplicate rows. Secondly, hardcoding these checks within the source code makes it difficult for non-coding personnel, such as data consumers or stewards, to modify existing rules or add new ones. Moreover, any change to the data quality checks necessitates a time-consuming and potentially risky redeployment of the entire data platform.
A more forward-thinking approach involves storing data quality check configurations as code within the source code repository, alongside other data platform components. This approach not only promotes version control and collaboration but also allows for greater flexibility and maintainability of data quality rules throughout the platform’s lifecycle.
How to verify data quality when users connect to the platform
The consumption stage marks a significant shift in the data platform lifecycle. The data pipelines, diligently crafted during development, are now actively feeding data into the target tables, essentially transforming the platform into a bustling data marketplace. Data consumers, including data analysts, data scientists, and data governors, step onto the stage, ready to explore and utilize this wealth of information.
In organizations adhering to robust software development practices, the transition to consumption may involve a formal handover process. A dedicated data quality team, platform owners, or data stewards meticulously test and verify all data assets, often comparing them against trusted third-party sources to identify any discrepancies. Regardless of the specific handover process, data consumers typically begin by conducting basic profiling of tables to gain a comprehensive understanding of their structure and the typical format of data within.
Modern data teams well-versed in data quality practices go beyond basic profiling. They define more sophisticated data quality checks that evaluate the data from a business and usage perspective. For instance, a data analyst might create a custom check that mirrors the filtering and aggregation logic used in a dashboard’s semantic data model. A data steward might implement checks that compare data across multiple sources to uncover discrepancies, indicating potential issues like missing or duplicated data, or incorrect mappings.
Ideally, these custom data quality checks are stored within a dedicated data quality platform and incorporated into automated daily or monthly test suites. This ensures ongoing monitoring for data drifts that could cause these checks to fail in the future. Given that many data consumers, such as analysts and stewards, are not typically proficient in coding, a no-code data quality solution that enables them to configure and execute checks through a user interface is often the preferred choice.
Data quality best practices - a step-by-step guide to improve data quality
- Learn the best practices in starting and scaling data quality
- Learn how to find and manage data quality issues
How to monitor data quality after the data platform has transitioned to the operations stage
The operations stage represents the culmination of the data platform lifecycle. The platform is now fully integrated into the organization’s data ecosystem, feeding a myriad of downstream systems such as dashboards and machine learning pipelines. In this critical phase, the data operations team plays a pivotal role in ensuring the smooth and efficient functioning of the platform. Their primary focus is on streamlining operational processes to minimize response times to problems and incidents.
The issues they encounter can vary widely, ranging from hardware and infrastructure failures to bugs in the data pipeline code. However, a significant portion of their workload often revolves around data quality issues. These can be detected automatically by a data quality monitoring platform or reported manually by data consumers. Some data quality issues, like a required column in recent records suddenly becoming empty, may be triggered by changes in source systems or business processes governing data collection and entry. These issues are not caused by technical malfunctions within the data platform itself, but rather by the complex interplay of processes that generate and handle data upstream.
To proactively identify and address such issues, a modern data operations team leverages a data quality monitoring platform capable of regularly rerunning all previously defined data quality checks. Given the time-sensitive nature of their work, minimizing the time spent confirming and validating data quality issues is paramount. This requires effective tooling.
One option is to equip the team with a data observability platform that monitors data distribution and detects anomalies within each dataset. However, such platforms often have limited data quality checking capabilities. A more robust approach involves adopting a professional data quality platform that combines data observability with traditional data quality checks and data reconciliation (comparison) capabilities. To further streamline their workflow, the data operations team ideally uses the same data quality platform as the data engineers and all other data consumers, fostering consistency and collaboration across the entire data lifecycle.
How to achieve an end-to-end data quality monitoring
While data quality responsibilities are distributed across various roles throughout the data platform lifecycle, the data operations team ultimately bears the burden of managing and maintaining the platform’s overall health, including its data quality. They must react swiftly to any data quality issues that arise, often necessitating more than just fixing source data or adding extra transformations.
In reality, many data quality issues stem from unexpected changes in business processes, requiring the data platform to adapt accordingly. For example, a data quality check might fail because the data distribution has shifted due to evolving regulations or market trends. The solution might not involve rectifying the data itself but rather recalibrating data quality rules to match the new reality.
Consider a scenario where a data completeness rule mandates that at least 80% of customer records contain an email address. Due to GDPR regulations, email collection is discontinued for certain channels, causing the percentage to drop below the threshold and triggering a data quality alert. In this case, the business decision is valid, but the data quality rule needs adjustment.
The data operations team must be able to quickly modify the rule’s threshold parameter to a lower value, rerun the check, and confirm that the issue is resolved. If the rules were hardcoded into the data pipeline, this change would necessitate a time-consuming and potentially disruptive redeployment of the platform.
To support these dynamic scenarios, which become increasingly common as a platform matures, a comprehensive data quality platform is essential. This platform should facilitate an end-to-end data quality management process, encompassing automation and configuration storage in a format accessible to data engineers during development. It should also provide a user-friendly interface for all data consumers, especially data stewards and operations teams. This interface should empower them to configure new checks, modify existing ones (even those initially defined by engineers), and execute checks on demand. By democratizing data quality management, the platform ensures that everyone involved in the data lifecycle can contribute to maintaining the platform’s integrity and value.
The whole process is presented on the following diagram.
What is the DQOps Data Quality Operations Center
DQOps is a data quality platform designed for end-to-end data quality monitoring. DQOps provides extensive support for configuring data quality checks, applying configuration by data quality policies, detecting anomalies, and managing the data quality incident workflow.
The extensive API provided by DQOps allows full automation of all aspects of the platform, allowing the data engineers to integrate the platform into data pipelines by calling actions to run data quality checks instead of hardcoding their configuration within the data pipeline code. DQOps stores the configuration in developer-friendly YAML files but provides a full user interface over these files, allowing non-coding users such as data stewards or data operations teams to configure, run, and alert any configured data quality checks.
You can set up DQOps locally or in your on-premises environment to learn how DQOps can monitor data sources and ensure data quality within a data platform. Follow the DQOps documentation, go through the DQOps getting started guide to learn how to set up DQOps locally, and try it.
You may also be interested in our free eBook, “A step-by-step guide to improve data quality.” The eBook documents our proven process for managing data quality issues and ensuring a high level of data quality over time.
