column exists

column exists checks

Description
Column level check that reads the metadata of the monitored table and verifies that the column still exists in the data source. The data quality sensor returns 1.0 when the column was found or 0.0 when the column was not found.

---

profile column exists

Check description
Checks the metadata of the monitored table and verifies if the column exists.

Check name	Check type	Time scale	Sensor definition	Quality rule
profile_column_exists	profiling		column_exists	equals_integer

Enable check (Shell)
To enable this check provide connection name and check name in check enable command

dqo> check enable -c=connection_name -ch=profile_column_exists

Run check (Shell)
To run this check provide check name in check run command

dqo> check run -ch=profile_column_exists

It is also possible to run this check on a specific connection. In order to do this, add the connection name to the below

dqo> check run -c=connection_name -ch=profile_column_exists

It is additionally feasible to run this check on a specific table. In order to do this, add the table name to the below

dqo> check run -c=connection_name -t=table_name -ch=profile_column_exists

It is furthermore viable to combine run this check on a specific column. In order to do this, add the column name to the below

dqo> check run -c=connection_name -t=table_name -col=column_name -ch=profile_column_exists

Check structure (Yaml)

      profiling_checks:
        schema:
          profile_column_exists:
            warning:
              expected_value: 1
            error:
              expected_value: 1
            fatal:
              expected_value: 1

Sample configuration (Yaml)

# yaml-language-server: $schema=https://cloud.dqo.ai/dqo-yaml-schema/TableYaml-schema.json
apiVersion: dqo/v1
kind: table
spec:
  timestamp_columns:
    event_timestamp_column: col_event_timestamp
    ingestion_timestamp_column: col_inserted_at
  incremental_time_window:
    daily_partitioning_recent_days: 7
    monthly_partitioning_recent_months: 1
  columns:
    target_column:
      profiling_checks:
        schema:
          profile_column_exists:
            warning:
              expected_value: 1
            error:
              expected_value: 1
            fatal:
              expected_value: 1
      labels:
      - This is the column that is analyzed for data quality issues
    col_event_timestamp:
      labels:
      - optional column that stores the timestamp when the event/transaction happened
    col_inserted_at:
      labels:
      - optional column that stores the timestamp when row was ingested

---

daily column exists

Check description
Checks the metadata of the monitored table and verifies if the column exists. Stores the most recent value for each day when the data quality check was evaluated.

Check name	Check type	Time scale	Sensor definition	Quality rule
daily_column_exists	recurring	daily	column_exists	equals_integer

Enable check (Shell)
To enable this check provide connection name and check name in check enable command

dqo> check enable -c=connection_name -ch=daily_column_exists

Run check (Shell)
To run this check provide check name in check run command

dqo> check run -ch=daily_column_exists

It is also possible to run this check on a specific connection. In order to do this, add the connection name to the below

dqo> check run -c=connection_name -ch=daily_column_exists

It is additionally feasible to run this check on a specific table. In order to do this, add the table name to the below

dqo> check run -c=connection_name -t=table_name -ch=daily_column_exists

It is furthermore viable to combine run this check on a specific column. In order to do this, add the column name to the below

dqo> check run -c=connection_name -t=table_name -col=column_name -ch=daily_column_exists

Check structure (Yaml)

      recurring_checks:
        daily:
          schema:
            daily_column_exists:
              warning:
                expected_value: 1
              error:
                expected_value: 1
              fatal:
                expected_value: 1

Sample configuration (Yaml)

# yaml-language-server: $schema=https://cloud.dqo.ai/dqo-yaml-schema/TableYaml-schema.json
apiVersion: dqo/v1
kind: table
spec:
  timestamp_columns:
    event_timestamp_column: col_event_timestamp
    ingestion_timestamp_column: col_inserted_at
  incremental_time_window:
    daily_partitioning_recent_days: 7
    monthly_partitioning_recent_months: 1
  columns:
    target_column:
      recurring_checks:
        daily:
          schema:
            daily_column_exists:
              warning:
                expected_value: 1
              error:
                expected_value: 1
              fatal:
                expected_value: 1
      labels:
      - This is the column that is analyzed for data quality issues
    col_event_timestamp:
      labels:
      - optional column that stores the timestamp when the event/transaction happened
    col_inserted_at:
      labels:
      - optional column that stores the timestamp when row was ingested

---

monthly column exists

Check description
Checks the metadata of the monitored table and verifies if the column exists. Stores the most recent value for each month when the data quality check was evaluated.

Check name	Check type	Time scale	Sensor definition	Quality rule
monthly_column_exists	recurring	monthly	column_exists	equals_integer

Enable check (Shell)
To enable this check provide connection name and check name in check enable command

dqo> check enable -c=connection_name -ch=monthly_column_exists

Run check (Shell)
To run this check provide check name in check run command

dqo> check run -ch=monthly_column_exists

It is also possible to run this check on a specific connection. In order to do this, add the connection name to the below

dqo> check run -c=connection_name -ch=monthly_column_exists

It is additionally feasible to run this check on a specific table. In order to do this, add the table name to the below

dqo> check run -c=connection_name -t=table_name -ch=monthly_column_exists

It is furthermore viable to combine run this check on a specific column. In order to do this, add the column name to the below

dqo> check run -c=connection_name -t=table_name -col=column_name -ch=monthly_column_exists

Check structure (Yaml)

      recurring_checks:
        monthly:
          schema:
            monthly_column_exists:
              warning:
                expected_value: 1
              error:
                expected_value: 1
              fatal:
                expected_value: 1

Sample configuration (Yaml)

# yaml-language-server: $schema=https://cloud.dqo.ai/dqo-yaml-schema/TableYaml-schema.json
apiVersion: dqo/v1
kind: table
spec:
  timestamp_columns:
    event_timestamp_column: col_event_timestamp
    ingestion_timestamp_column: col_inserted_at
  incremental_time_window:
    daily_partitioning_recent_days: 7
    monthly_partitioning_recent_months: 1
  columns:
    target_column:
      recurring_checks:
        monthly:
          schema:
            monthly_column_exists:
              warning:
                expected_value: 1
              error:
                expected_value: 1
              fatal:
                expected_value: 1
      labels:
      - This is the column that is analyzed for data quality issues
    col_event_timestamp:
      labels:
      - optional column that stores the timestamp when the event/transaction happened
    col_inserted_at:
      labels:
      - optional column that stores the timestamp when row was ingested

---