Data Governance and Data Quality for Analysts

What Is Data Governance?

Data governance is the set of policies, processes, roles, and standards that ensure data is accurate, consistent, secure, and used appropriately across an organization. It answers fundamental questions: Who owns this data? Who can access it? How should it be defined? How do we know it's correct? Without governance, organizations accumulate conflicting definitions ("revenue" means different things to Finance and Sales), siloed datasets, compliance risks, and decisions based on data nobody trusts.

Data quality is governance in practice — the ongoing measurement and improvement of data's fitness for use. The two disciplines are inseparable: governance defines the rules; data quality enforces them.

The Six Dimensions of Data Quality

Data Governance Roles and Responsibilities

Effective governance requires clearly defined human roles, not just technical systems.

The Business Glossary and Data Catalog

The business glossary is a centralized list of agreed-upon definitions for business terms. Without it, "active customer," "churn," and "monthly recurring revenue" mean different things to different teams, making cross-functional reports impossible to reconcile. A good glossary entry includes the term name, definition, owner, related terms, and examples of correct and incorrect usage.

The data catalog extends the glossary to the technical layer — it inventories datasets, tables, and columns, and links each to its business definition, lineage, owner, and quality metrics. Modern data catalogs (Alation, Collibra, Atlan, dbt docs, Google Data Catalog) allow analysts to search for data, understand its history, and trust its provenance before using it in a report.

Data Lineage

Data lineage tracks the full journey of data from its source through every transformation to its final use. It answers: Where did this number come from? What transformations changed it? Which reports depend on this table? Lineage is critical for impact analysis (if we change this source table, what breaks?), debugging (why did this metric change?), and compliance (can we trace every value in this GDPR report back to its source?).

Tools like dbt automatically generate column-level lineage for SQL transformations. Data observability platforms (Monte Carlo, Acceldata, Bigeye) extend lineage to runtime monitoring, alerting when table row counts, distributions, or freshness deviate from expectations.

Implementing Data Quality Checks in Code

Quality checks belong in code, not spreadsheets. They should run automatically at ingestion, at transformation, and before delivery to downstream consumers.

import pandas as pd
import numpy as np
from datetime import datetime

def run_quality_checks(df: pd.DataFrame, table_name: str) -> dict:
    """
    Run a standard suite of data quality checks on a DataFrame.
    Returns a dict of check results with pass/fail status.
    """
    results = {}
    total_rows = len(df)

    # 1. Completeness: check required columns
    required_cols = ['customer_id', 'order_date', 'amount']
    for col in required_cols:
        if col in df.columns:
            null_rate = df[col].isna().mean()
            results[f'completeness_{col}'] = {
                'passed': null_rate == 0,
                'null_rate': round(null_rate, 4),
                'threshold': 0.0
            }

    # 2. Uniqueness: check primary key
    if 'order_id' in df.columns:
        dup_rate = df['order_id'].duplicated().mean()
        results['uniqueness_order_id'] = {
            'passed': dup_rate == 0,
            'duplicate_rate': round(dup_rate, 4)
        }

    # 3. Validity: check amount is positive
    if 'amount' in df.columns:
        invalid_rate = (df['amount'] <= 0).mean()
        results['validity_amount_positive'] = {
            'passed': invalid_rate == 0,
            'invalid_rate': round(invalid_rate, 4)
        }

    # 4. Timeliness: check freshness
    if 'order_date' in df.columns:
        max_date = pd.to_datetime(df['order_date']).max()
        lag_days = (datetime.now() - max_date).days
        results['timeliness_order_date'] = {
            'passed': lag_days <= 1,
            'lag_days': lag_days,
            'threshold_days': 1
        }

    # 5. Row count anomaly: flag unexpected drop
    results['row_count'] = {
        'row_count': total_rows,
        'passed': total_rows > 0
    }

    # Summary
    passed = sum(1 for r in results.values() if r.get('passed', False))
    results['summary'] = {
        'table': table_name,
        'total_checks': len(results) - 1,
        'passed': passed,
        'failed': len(results) - 1 - passed,
        'run_at': datetime.now().isoformat()
    }

    return results

# Usage
df = pd.read_parquet('s3://my-bucket/orders/2024-03/')
results = run_quality_checks(df, 'orders')
failed = [k for k, v in results.items() if not v.get('passed', True) and k != 'summary']
if failed:
    print(f"QUALITY FAILURES: {failed}")

dbt Tests for Data Quality

dbt (data build tool) has built-in test types that run automatically after each model build. Generic tests check common patterns across any column; singular tests are custom SQL assertions for complex business rules.

# schema.yml — dbt generic tests
models:
  - name: orders
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('customers')
              field: customer_id
      - name: status
        tests:
          - accepted_values:
              values: ['pending', 'shipped', 'delivered', 'cancelled']
      - name: amount
        tests:
          - not_null
          - dbt_utils.expression_is_true:
              expression: ">= 0"

Run dbt test after every build. Failed tests block downstream models from deploying, preventing bad data from reaching dashboards.

Data Quality Metrics and SLAs

Data quality should be measured, tracked over time, and tied to service level agreements (SLAs). A data quality SLA might state: "The orders table will be refreshed by 6am UTC every day, with zero null order_ids and a duplicate rate below 0.01%." Breaches trigger alerts to the data owner. Tracking quality metrics over time reveals systemic issues — a gradual increase in null rates often signals an upstream schema change.