Data Governance Without the Bureaucracy: Quality Built Into the Pipeline

Most data governance programs do not survive contact with reality. They start as a 40-page framework deck, get a kickoff with the heads of every domain, and end as a Confluence space nobody opens. Six months later the same data quality problems are still landing in the same Slack channels, and the next governance initiative is being scoped by a different VP.

This is not because governance is a bad idea. It is because traditional governance tries to enforce policy after data is in the warehouse, through committees and review boards. By that point the bad data has already shipped, and the review boards become bottlenecks that the rest of the org learns to route around.

Why traditional governance fails

Top-down policy without enforcement is theater. If the rule is "all PII columns must be tagged" but nothing breaks when they are not tagged, the rule is decorative.

Centralized review creates dependencies. When every schema change requires a governance committee sign-off, teams either wait weeks for trivial changes or work around the process entirely.

Quality issues are caught downstream. The classic failure mode: a producer ships a schema change, ingest succeeds, the silver layer transforms cleanly, the dashboard renders — and is silently wrong because the change broke a join key. Stakeholders find out before the data team does.

The shift: governance as code, embedded in the pipeline

The model that actually works is the inverse of the committee model. Instead of governance as policy reviewed by humans after the fact, governance becomes quality gates that run with every pipeline execution. If the gate fails, the pipeline stops. The bad data never reaches the next stage.

Five gates, in order of when they run, will catch roughly 95 percent of the issues that show up in production.

1. Schema contract at the source

A schema contract is a producer-owned, machine-readable description of the data the producer is promising to send: column names, types, nullability, and acceptable enum values. It can be JSON Schema, an Avro spec, a Protobuf definition, or even a dbt source YAML. The point is that it is checked before ingest, not after, and the producer is on the hook to update it before changing the data.

In practice: every source has a contract checked in alongside the ingestion job. The first thing the job does is validate incoming records against the contract. Mismatches block ingestion and page the producer team, not the data team.

2. Freshness and volume

The two cheapest, most useful checks: did the data arrive on time, and is the volume in the expected range? These catch upstream outages (no data) and silent failures (data arrives but at 10 percent of normal volume because a partition got truncated). Most modern observability tools (Monte Carlo, dbt source freshness, Lightup) handle this out of the box. The mistake teams make is treating these as nice-to-haves rather than blocking gates.

3. Type and null checks at stage

Once data is in your bronze/staging layer, run mechanical checks: are the types what you expect, are required columns non-null, are foreign keys actually present in the dimension table? dbt source tests covers most of this declaratively. Great Expectations is the heavyweight option when you need cross-row assertions.

4. Business rules at transform

These are the assertions that depend on domain knowledge: revenue should never be negative, churn rate should be between 0 and 1, the sum of cohort counts should equal the total user count. Encode them as dbt tests or custom SQL assertions that run on every model build. Make them block deploys, not just emit warnings.

5. SLO monitoring at serve

For tables that power dashboards or APIs, define and monitor SLOs: 99 percent of queries return in under 5 seconds, the table is fresh within 30 minutes, the row count is within ±5 percent of the prior period. Wire these to PagerDuty so an SLO breach pages someone, the same way a service outage would.

Federated ownership: who owns what

Quality gates are necessary but not sufficient. Without clear ownership, every gate failure becomes a fire drill where nobody knows who should fix it. The model that holds up over time is federated.

Domain teams own their data products. The team that produces orders data owns the orders schema contract, the orders source-level SLA, and the orders quality gates. They are on call when those gates fail. This is the "you build it, you own it" principle from operational software, applied to data.

Platform team owns the enablers. The framework for declaring contracts, the test runner, the lineage tooling, the catalog. The platform team does not own anyone else's data — they make it cheap and obvious for domain teams to do the right thing.

Stewards council handles the cross-domain stuff. PII classification, regulatory requirements (GDPR, CCPA, PIPEDA), conflict resolution between domains. The council does not approve every change — it sets the standards and adjudicates when domains disagree.

Quality metrics that actually matter

The classic five — completeness, accuracy, consistency, timeliness, validity — are still the right framework. The trick is to instrument them, not just talk about them.

Completeness: percentage of expected rows that arrived. Anomaly-detected per table.

Accuracy: percentage of rows that pass business-rule tests. Tracked over time per table.

Consistency: referential integrity across tables. Foreign keys, dimension joins, cross-source reconciliation.

Timeliness: latency from source event to availability in serving table. Compared against the SLO for that table.

Validity: percentage of rows that conform to the schema contract. Should be 100 percent if the upstream gate is doing its job.

The toolchain

You do not need to buy everything. A reasonable starting stack:

dbt for tests and source freshness — covers gates 2, 3, and 4 declaratively. Great Expectations or Soda when you need cross-row, distribution-shape assertions that dbt cannot easily express. Monte Carlo, Lightup, or Bigeye for ML-based anomaly detection on freshness and volume — gate 2.

OpenMetadata, DataHub, or Atlan for lineage and catalog — required for stewards to do their job. PagerDuty or Opsgenie for SLO breach paging, wired to the same on-call rotation that owns the data product.

How to start

Do not try to roll out all five gates across all pipelines at once. The fastest way to kill a governance program is to make it a 12-month transformation project. Pick the pipeline that is causing the most pain — usually the one whose dashboards your CEO complains about — and instrument it end to end. Get all five gates green. Use that pipeline as the reference implementation when domain teams ask "what does this look like in practice?"

Then expand. Within six months you can have most of the critical pipelines under quality gates. Within a year, all of them. Bureaucratic governance programs spend the same year arguing about org charts.