Data Engineers Are Not Data Catalogs
Your data has never been this accessible. Your analysts have never done this much guessing.
You're unlikely to find "answer fifteen Teams messages a week about which table to use for this report" in any analyst or developer job description. And yet it happens every day. Analysts aren't clairvoyants. They need reliable data to do their job, and they need to draw from the right source.
The gap between data availability and cataloging didn't appear by accident. It's the byproduct of three pressures at once: move faster, decentralize data by business domain or department, and build increasingly self-service data platforms.
The V6_Final_RealFinal Problem
Office professionals have known this for decades. A good analysis goes into circulation. Twenty people pick it up, massage it, adapt it to their audience. Six months later, only an echo of the original intent remains, and the file that carries authority is named V6_Final_RealFinal.
The same thing happens in data management. Everyone is trying to do the right thing — support internal clients, deliver analyses, integrate new sources — but everyone is moving fast to hit deadlines. One morning you look up and you have six near-identical data models (tables, views, semantic models, and so on), and nobody knows which one is current, or which field to use.
The costs are real:
time spent pointing people to the right source, over and over
data quality that degrades gradually
time wasted repairing analyses built on the wrong tables
trust eroding across the organization
more bugs and more data downtime
When trust drops, organizations compensate by widening the margins of error in their forecasts. It's an understandable reflex, and an expensive one: when demand and supply chains are uncertain, an imprecise forecast costs real money in inventory, in halted production, and in missed opportunities.
What Does It Look Like at Your Company?
Take the production tracking spreadsheet three supervisors maintain in parallel, each with a slightly different definition of a billable hour. The concrete consequence is that production figures differ depending on which system you query: the scale system, the shift report, the ERP. You end up with project costs that won't reconcile between field reports and what was invoiced to the client. Think too of the purchase order that exists in the procurement system, but where nobody is sure where to find the final approved version.
At a 200-person company, it's rare for anyone to hold the title of data manager. What happens instead is that several different people become the company's human data catalog. It's inefficient, but it works. The trouble surfaces when those people leave the company or go on vacation.
Look at the bright side: at that scale, you don't need a big governance program that takes three years to implement. You just need a few validated, officially certified data sources. That's the essence of a data catalog: a central, accessible directory listing the official tables and data sources, ready to be pulled into a report, into an Excel file, or queried directly with SQL.
Which brings up the central question: as data becomes more accessible, how does a downstream consumer know what's been prepared, transformed, and approved? How do you make sure a common foundation of standards, ownership, and communication holds up across the organization?
For many teams, the answer is certification.
On the left, what circulates and what it costs. On the right, the same subject published once — with a name, an SLA, and monitors.
Certifying Your Data
Certification is the process by which a data source is approved for use across the organization, after meeting mutually agreed-upon SLAs covering quality, observability, ownership, issue resolution, and communication.
Like validation or verification, it layers processes that align people, frameworks, and technology to business policy. Requirements vary with the needs of the business, the capacity of the team, and the availability of data, but they generally include automated checks for freshness, volume, schema, and distribution; delivery SLAs with a defined uptime figure; owners accountable for alerts; automatic alert routing to Teams or to mailboxes; and a communication process settled in advance for outages.
Seven Steps to Set the Company Up Properly
A good certification program has to grow naturally with the company and the volume of data. To do that, it needs to apply the same winning approach across every business domain. Here are the seven ground rules behind a certification practice that's solid and holds up over time.
1. Start with observability. You can't certify what you can't see. It takes a baseline on current performance, plus a systemic end-to-end approach to detecting, alerting, and triaging events. Typically an incident dashboard automatically surfaces anomalies, schema changes, deleted tables, and other breaks. When something in the pipeline breaks — and it will, guaranteed — you know about it before the data consumers do.
2. Name owners. Every certified data source needs a person responsible for its entire life cycle, from ingestion through the analytics layer. Modern metadata tools let you assign an owner directly on the table, alongside the other tags. How you implement it depends on your needs and the tools you have. Some teams roll out a full RACI matrix; others simply name the owner in written procedures.
3. Define what "good" means. Ask your stakeholders the who, what, when, where, and why, and you'll learn what quality means to them and which data actually matters. Concrete indicators follow: freshness ("refreshed by 8 a.m., every day"), distribution ("this column must never be empty"), volume, schema, downtime, query speed, and so on. It's also what lets you configure alerts that are actually useful, tuned to what matters to the business.
4. Write specific SLAs. "Reliable data at all times" is too vague to be useful. A good SLA is precise and detailed: it documents why it matters to each domain or function, what the expectations are, when they need to be met, how, where the data lives, and who the stakeholders are. It also sets out a clear course of action when things fail. "Table X is refreshed daily by 8 a.m." becomes: "Team Z guarantees the refresh by 8 a.m. Within two hours of an anomaly alert, the team verifies, communicates to affected parties, and begins root cause analysis. Within one business day a ticket is created and the team is updated on progress."
Note that this contains two different promises, and they need to be kept apart. The first is the target: the data is refreshed by 8 a.m. The second is how often you actually hit it: 99% of the time, which in practice means allowing yourself to miss the refresh about three mornings a year.
There's no need to start by trying to cover everything. Begin with the five most-used tables or data sources.
5. Formalize communication and incident management. Who receives the alerts? How are next steps communicated, internally and externally? It may seem elementary, but clear and transparent communication is the foundation of an accountability culture. Many teams keep alerts and triage in Teams or Slack. That allows rapid coordination and full visibility for the wider team. Plan for the major outage too — who announces what to the rest of the company, where, and how often.
6. Choose your tagging mechanism. You now have measurable SLAs, transparency, communication processes, and clear resolution expectations. What's left is making the approved data sources available. The company decides once and for all what "approved" means: the commitments, the named owner, the response time. From there, each department decides which of its tables earn the label. Nobody has to run the decision past a committee.
The marking itself doesn't need to be sophisticated. It can be a tag in your reporting tool, a prefix on the table name, or a simple SharePoint page listing the approved tables with the owner's name beside each one. What matters is that whoever uses the data can tell it's certified without having to ask.
7. Train your teams and your consumers. The label isn't magic on its own. Someone will inevitably keep opening the old table, because it's been bookmarked for two years, or because it has a column the new table doesn't, or because their report already works and they don't feel like rebuilding it. Those are good reasons. Listen to them: more often than not, something is missing from the approved table. When nothing is missing, say so plainly — that one is being retired, and here's the date.
Make sure you tune the alert volume as well. Occasionally receiving an alert that requires no action is normal. Not every alert points to a real problem. There will always be some noise to manage.
That said, alerts can quickly cause fatigue if they carry too much noise. If the team starts ignoring them, that's the sign to tighten them up or split the channels so only what matters stands out.
And with the people who use your data, don't be shy. You've just built something solid for them. Help them move from "the numbers feel off" to something measurable. Give them the vocabulary to be part of the solution.
The seven steps of a certification program, in three moves: see and own, agree, operate and spread.
The Payoff
When the seven steps are executed well, it's a good thing to watch. The engineer tags the table as certified, attaches its owner, surfaces it in the warehouse. The analyst picks it up and uses it in their dashboard. Questions and downtime are minimized.
The main thing to take away is this: without the right processes and a strong data culture, certifying reliability and building trust in the organization becomes extremely difficult. Technology helps, seriously — but it will never replace good data hygiene or a good governance culture.
The first steps rarely begin with a big tool or a highly sophisticated technology platform. They begin with picking the five tables that genuinely matter to the company and giving them a quality label.
Adapted from ideas presented in the book Data Quality Fundamentals (Barr Moses, Lior Gavish, and Molly Vorwerck, O'Reilly, 2022).