More scale, less chaos: dbt Mesh is now generally available

At our recent Product Launch Showcase we announced that support for multi-project collaboration patterns, known as dbt Mesh, is now generally available for customers of dbt Cloud Enterprise.

Since introducing dbt Mesh at Coalesce last year, we’ve seen a rapidly growing number of organizations adopt this deployment pattern across their many data teams and domains. In fact, we're thrilled to share that this number has just crossed one hundred — and it includes some of the largest enterprises in the world.

The general availability of dbt Mesh support is a huge milestone in unlocking more seamless collaboration around data, at scale, for more teams and more organizations.

Today we want to share why we're so excited about this, a few of the things we’ve learned from customers so far, and a bit about what’s coming next.

But first…

What the heck is dbt Mesh?

dbt Mesh is a pattern for collaboration across multiple teams and domains of data. It is the culmination of several new dbt capabilities, and was motivated by a key product development priority: helping data teams handle complexity at scale.

As a dbt project naturally grows in size and scale, navigating it successfully can become challenging. It’s not clear who is responsible for maintaining which models, and stakeholders’ trust in data (and the data team) suffers. Within the team, development bogs down, for fear of breaking downstream uses of dbt models. Onboarding to a monolithic project is too high a barrier for many less-technical contributors. This can be an ugly picture: the data team on the back foot, “shadow” analyses competing with vetted data products.

At sufficient scale, one team of data and analytics engineers cannot do it all themselves — no matter their talent, no matter the tools on their utility belt. They need a safe, scalable way to invite others into the work of building and maintaining data products. The big ideas introduced by the dbt Mesh pattern are thus:

• Teams should own their data, and as such their own dbt projects, from development through deployment.
• Those teams must not operate in silos.
• dbt models should define the interface for data sharing across teams. But not every dbt model is so worthy — these cross-boundary interfaces must be designated explicitly as such.
• The teams maintaining those interfaces should treat them as mature APIs, with a stable set of guarantees that do not break without prior warning — and even then, a careful versioning and deprecation window.
• The central data team must retain the ability to set global governance standards, and keep visibility on global lineage across teams.

The dbt Mesh pattern allows teams to accomplish these things with a handful of dbt features, rolled out over the past year. Now, data teams can:

• define explicit governance rules for dbt models (contracts, versions, access levels)
• enable collaborative development with cross-project references
• see cross-project lineage in dbt Explorer

“dbt Mesh enables us to make data mesh a reality by offering a simple, cohesive way to integrate and manage data pipelines & products across the enterprise using a single platform.” —Marc Johnson, Data Strategy & Architecture, Fifth Third Bank

What’s new

Alongside the general availability of these capabilities, we announced some new product features that we’re confident will allow even more customers to adopt the dbt Mesh pattern in production.

(New!) Cross-project job triggers

Now your team can schedule and trigger jobs across projects, so they can own their data models from development through deployment, informed by their upstream dependencies.

Beyond this, a major priority for us is to continue building extensions of dbt Cloud’s built-in orchestration, allowing it scale to more complex cross-project topologies. Look out for more on this over the coming months.

(New!) Staging environments

This is a new environment type in dbt Cloud that allows for improved data isolation. It enables developers to contribute in dbt Cloud — with all the cost-saving benefits of more efficient development and cross-project references — but without requiring access to production data. Using it allows organizations to more easily adopt dbt best practices and follow global governance policies.

We want to enable customers to adopt scalable, governed approaches to their multi-project deployments in dbt Cloud. To that end, we’re rolling out two related capabilities in the next few months:

• Environment-level permissions (coming soon): Enable developers to edit and trigger jobs in specific environments (such as Staging), while restricting them from accessing others (Production).
• Environment-level connections (coming soon): Need to use different authentication, or hit a different database/project/warehouse in your cloud data platform in Development vs. Staging vs. Production? Not a problem.

(New!) Azure Single Tenant Support

The dbt Mesh pattern is now supported for dbt Cloud customers deployed on Microsoft Azure — on Single Tenant deployments today, and Multi-Tenant deployments later this year. We’re glad to offer the same compelling dbt Cloud experiences to more of our largest customers.

“It’s now easier to ref the models from other projects, and we keep this in one single tool, without having to … identify the database/schema and create the source.” — Renato Sousa, IT Solutions Expert, Siemens

What’s next

Beyond these enhancements, we have a robust roadmap in place for improving the dbt Mesh experience even further.

We’re refining a bundle of enhancements to existing model governance features in dbt — making them more intuitive and capable, and making it easier for teams to manage and maintain their project interfaces.

We’re exploring ways to provide a first-class cross-project orchestration experience that easily accommodates even more deployment architectures.

And we’re thinking now about how to strengthen the connection between the dbt Semantic Layer and dbt Mesh concepts, so that data teams can easily curate and extend semantic objects across projects.

As the kids say, “watch this space” for news on these and other product enhancements!

How should you use dbt Mesh?

dbt Mesh is not one feature — it’s a new way of working, and of architecting how data teams should work together. We take seriously our responsibility to provide you with both product experiences and hard-won guidance to take “data mesh” from theory to practice.

To date, we’ve published a detailed guide, FAQs, and a hands-on quickstart. We're excited to update those resources in the coming weeks with lessons learned from successful Mesh implementations at customers large and small.

Going forward

Analytics engineering, at its heart, is about solving data problems and people problems. Tooling is a start, but it’s just a start. Any solution that’s going to be effective at tackling problems of collaboration and complexity around data needs to acknowledge these problems are socio-technical.

The dbt Mesh pattern is our attempt to give teams a set of tools that they can adapt and employ to reflect their organizational structure, instead of trying to force it to be the other way around. It’s still highly opinionated — there are patterns we encourage and discourage, best practices and anti-patterns — but we firmly believe this is the only kind of solution that will scale.

This journey’s not done; it’s only just begun. Alongside our first hundred dbt Mesh customers, we will continue to innovate and to push the boundaries of what's possible with this pattern, supported by the platform we’re building in dbt Cloud.