Skip to content

Last updated: September 24, 2024

DQOps YAML files reference

DQOps stores all configuration of data sources, and the configuration of activated data quality checks in YAML files. The list of supported files is below.

DQOps file types

The list of supported files, their file extension and a link to the full reference of all fields is below. All file name patterns are relative to the DQOps user home folder.

Type File location Description
Data source connection sources/<connection>/connection.dqoconnection.yaml The data source configuration parameters, including the CRON schedule for running data quality checks, and all data source connection parameters.
Table schema sources/<connection>/*.dqotable.yaml The table metadata with the configuration of activated data quality checks.
Custom dashboard list settings/dashboardslist.dqodashboards.yaml The configuration of custom data quality dashboards organized as a dashboard folder tree shown in the Data quality dashboards section of the user interface.
Default notification webhooks settings/defaultnotifications.dqonotifications.yaml The configuration of the default webhooks used for data quality incidents' notifications when custom webhook urls are not assigned in the configuration file of a data source.
Table-level data quality policy patterns/<pattern_name>.dqotablepattern.yaml The selection and the parameters of the data observability checks that are activated on all imported tables.
Column-level data quality policy pattern/<pattern_name>.dqocolumnpattern.yaml The selection and the parameters of the data observability checks that are activated on all imported tables.
Default CRON schedules settings/defaultschedules.dqoschedules.yaml The default CRON schedules for running data quality checks. DQOps does not use these schedules directly and changing the file has no effect on data sources. DQOps only uses this file to copy the default CRON schedules to the data source connection file, when the data source is imported.
Local settings .localsettings.dqosettings.yaml The local configuration settings that are specific only to the current instance of DQOps. Mainly used to store the DQOps Cloud Pairing API key.
Sensor definition sensors/**/*.dqosensor.yaml The definition of a custom data quality sensor, with the configuration of the parameters that will be shown in on data quality check editor screen.
Database specific sensor definition sensors/**/<data_source_type>.dqoprovidersensor.yaml The definition of a database specific custom data quality sensor.
Data quality rule definition rules/*/*.dqorule.yaml The definition of a custom data quality rules.
Custom check definition checks/**/*.dqocheck.yaml The definition of a custom data quality check. Specifies a pair of a data quality sensor and a data quality rule that make up a data quality check.

Incident notification's webhook message

DQOps supports integration with external ticketing systems and messaging platforms, publishing notification messages of new or acknowledged data quality incidents using webhooks.

DQOps makes a HTTP POST operation, sending a json object that is a data quality incident notification message.

What's next