One dbt: Data collaboration built on trust with dbt Explorer

At dbt, we believe a large part of trust is aligning teams that are invested in data. This can be challenging, as different data stakeholders approach data from different perspectives.

Consider a developer, Dante. He's a data producer. He really wants to know how he can better understand, troubleshoot, and improve the quality of his data pipelines so that he can effectively serve his stakeholders.

On the other side, we have Kathy. She's a consumer of data. She's really wondering how she can gather context, build trust in the data, and reuse it so she can make decisions quickly.

Even if you’re using a platform like dbt to build, transform, and harden data pipelines and code, that doesn’t mean that your data is always easy to maintain or draw insights from. Used as part of One dbt, dbt Explorer bridges this gap by supporting data discovery, transparency, and collaboration - not just for data engineers but for all data stakeholders.

We’ll discuss how dbt Explorer fits into a modern analytics workflow and the key features that enable teams to collaborate on data, regardless of their needs or perspectives.

dbt as the standard

Before we get into the details of dbt Explorer or even dbt Cloud, it's useful to talk about the future that we envision for dbt as a standard.

Last year at Coalesce, we shared the vision for what it means for dbt to be the standard. By “standard,” we mean the best and chosen framework for data transformation.

Standardizing means that different teams aren't solving the same problems in slightly different ways. We can actually share solutions, align faster, and collaborate more seamlessly because we all speak the same language, the language of dbt. And that's regardless of your cloud provider data platform, what team you sit on, or whether your team uses dbt Cloud or dbt Core.

We call this vision One dbt. With One dbt, for example:

• A product team in Seattle can deploy and run dbt on AWS
• Meanwhile, their colleagues in engineering over in Madrid run dbt on Azure - One dbt operating in the cloud environment or environments that make the most sense for the business
• A data science team running Databricks can directly reference a dbt project that the finance team manages over in Snowflake
• A central data team can build data models in a CLI using DBT Core, and those data models can then be investigated and built upon by a downstream marketing ops team that uses dbt Explorer and the new visual editing experience in dbt Cloud

It’s all just One dbt - one central platform disseminating knowledge throughout the business.

The Analytics Development Lifecycle (ADLC)

Providing better collaboration across data stakeholders first requires making sure that everyone is working together as part of the same team. That’s why dbt has heavily promoted what we call the Analytics Development Lifecycle (ADLC).

The ADLC is our recommended and standardized approach for a mature analytics process. It’s a vendor-agnostic framework designed to help organizations of any size mature their analytics workflows.

The ADLC encourages collaboration among various stakeholders. It’s designed to help data producers, data consumers, and - ultimately - the business ship and use trusted data products at speed and scale.

The ADLC has eight distinct phases:

• Plan
• Develop
• Test
• Deploy
• Operate and Observe
• Discover and Analyze

The ADLC borrows heavily from the Software Development Lifecycle (SDLC), which became popular in the early 2000s to help cross-functional teams work better together with more agility, velocity and ultimately impact to the business.

The SDLC was very successful at breaking down the barriers the industry had built between software engineers who were building applications and the IT professionals who maintain the systems that those applications ran on. The new approach gave both roles a standardized, repeatable framework by which to work better together.

It’s high time that analytics professionals had a similar revolution - one that accelerates and hardens data workflows for the data engineers, analysts, and data stakeholders who turn business requirements into data-driven reality.

The data control plane

However, the ADLC isn’t enough. Powering it requires a single, uniform way to access your data.

The modern data stack looks like an eye chart, with a myriad of solutions - data orchestration, observability, data catalogs, semantic stores, etc. - springing up over the pats decade. While all this has been great progress for the industry and our maturity, all of these add-ons ultimately create data silos.

Centralizing these metadata silos will make or break your analytics workflow. We believe that the solution that accomplishes this is a data control plane.

A data control plane sits across your data stack, unifying capabilities for orchestration, observability, and more. dbt’s data control plane centralizes this metadata across the business, giving you signals on what's happening in your data estate - all supercharged with AI.

The data control plane helps you understand:

• Is your data fresh?
• Is your data platform cost-optimized?
• Is everyone running from a common understanding of how business metrics are defined?

While the ADLC is vendor-agnostic, the data control plane is a vendor-backed technology solution built to embrace various phases of the ADLC. A good data control plane implementation should have three defining characteristics:

1. It should be flexible and cross platform so that it can power distributed teams, help organizations avoid vendor lock-in, and manage data platform costs
2. It should make data streamlined, accessible, and governed to more types of users - not just your data engineering team
3. It must produce trustworthy outputs - i.e., stakeholders need to understand where data comes from, how to improve it, how to troubleshoot it, and trust that the data that they're receiving is fresh and error-free.

How dbt Explorer bridges the gap

To trust something, you have to understand it. Consider, for example, cooking. You trust a recipe because you understand the ingredients, the cooking process, and the expected outcome. You might not know every chemical reaction, but you know the basics of heat, timing, and seasoning well enough to trust what comes out of it.

(This analogy only works for us for cooking. We still haven’t figured out baking.)

Data is the same way. For an organization to foster a culture of data collaboration, all the people expected to collaborate around the data - both the producers and consumers - need to understand where the data comes from, how it's used, and what its current quality is. Teams also need a standardized way to troubleshoot problems and fine-tune the overall workflow.

This is where dbt Explorer comes in.

dbt Explorer is the catalog for your dbt ecosystem. It can help not only your producers of data, but also your consumers of data. Its goal is to help all data stakeholders discover existing assets, view lineage, be able to troubleshoot and optimize your pipelines, and build knowledge and context about the data estate.

dbt Explorer collaboration features

dbt Explorer has many features that help foster this part of the process foundationally. Let’s dig into a few of the key ones in detail.

Resource pages

Resource pages are a foundational part of Explorer because every asset has a page. This can be an important starting point when you're doing discovery - whether it's a dbt model, a source, or a metric.

Here you can understand the contextual information about an asset, like what it means. Through its description, you can validate its health. It shows overall health scores (which you can see at the top of the page here in the image) as well as specifics on quality and test results (which you can see in the middle).

Most importantly, these resource pages help you view the data lineage around the asset so you can see where the data comes from and in general how it flows and its dependencies.

Auto-exposures

One of our newer features, called auto-exposures, integrates natively with your BI tools to add dashboards that are actually built off of your dbt models. These are generated automatically and added to your data lineage.

This is an incredibly valuable feature because you can see in what downstream assets the models you have are consumed in. This helps you improve assets that are high visibility. After all, you don't want to break your CFO's dashboard - you want to make it better! You can also identify what’s not consumed in any way downstream and either remove or find a different use for them.

Model query history

Continuing our theme of understanding, model query history is another important feature that helps with trust because it tells users how frequently models are being consumed. This is visible, not only in the Performance section of our interface, but also through Lineage Lenses, where a heat map allows you to zoom in and out on the hotspots.

Similarly to auto-exposures, this can help you understand what are your most popular assets, so you can continue to maintain them and make sure the quality is good. Also like auto-exposures, it can help you prune unused models and save costs.

Model query history is currently supported for Bigquery and Snowflake, and we're working on Redshift and Databricks next.

Data health tiles

Finally, to tie this story on understanding together, we have data health tiles and in-app trust signals.

Data health tiles are tied to your exposures, since that’s the unit of how things are being consumed in dashboards and reports for your downstream assets. But these are incredibly useful because they're just iframes you can embed directly into the dashboard where they're being consumed.

Data health tiles meet users where they are, signaling to them whether the asset is healthy. In the application, we also have high level trust signals based on a number of factors, such as test status, freshness, and usability.

In other words, whether you're in application or outside, you'll be able to understand the relative health of the data you're looking at.

See dbt Explorer in action

A picture’s worth a thousand words. A video may be worth even more. To see more of dbt Explorer in action, view the full webinar showing how dbt Explorer works as part of One dbt to drive a mature analytics workflow and enable data discovery, transparency, and collaboration.