Introducing release tracks for dbt version upgrades
Dec 17, 2024
ProductToday, we’re announcing release tracks as the new-and-improved way to manage dbt version upgrades across your dbt Cloud environments. Now, when you configure your dbt Cloud environments to one of the three release tracks, your dbt versions will be upgraded automatically—ensuring that you’re always enjoying the latest (or very recent) capabilities without any added maintenance overhead.
The three release tracks are:
- Latest (GA, available to all accounts): Always run the latest and greatest version of dbt, updated daily (this was formerly known as “versionless” dbt).
- Compatible (in Preview, available to Team and Enterprise accounts): Version of dbt that includes all changes from final dbt Core OSS releases, updated once per month.
- Extended (in Preview, available to Enterprise accounts): The previous “Compatible” version (one month delay), updated once per month.
dbt Cloud customers can configure different release tracks across different environments to support a range of patterns for development, testing, and deployment. Release tracks mark the next evolution of our platform, helping our customers simplify maintenance, enjoy ready-access to our latest capabilities, and more seamlessly support hybrid dbt Core and dbt Cloud architectures.
How it started: Delivering latest and greatest dbt with “versionless”
So, how did we get here? As an open-core company, historically we had different development cadences for “core” dbt functionality and our dbt Cloud platform. Unfortunately, this meant that:
- dbt Cloud customers would wait for core dbt functionality to land in final releases of dbt Core
- Then, they’d try upgrading some of their environments, from v1.X to v1.Y
- And hope it worked
When we introduced an exciting new capability into the dbt framework—like model contracts in v1.5, or “retry from point of failure” in v1.6—we had to tell our dbt Cloud customers that they needed to upgrade their environments before they could access those features. This wasn’t a great user experience, and it was difficult to scale across many teams. It blocked valuable new capabilities from reaching the people who would most benefit from them, for months or years. And it just isn’t the way you expect SaaS to work.
This is the problem we set out to solve a year ago. Observing that the plurality of dbt projects were still running versions from 6-12 months ago, Grace and I wrote: "There is a lot of value locked up in the features we’ve already released in 2023, and we want to lower the barrier for tens of thousands of existing projects who are still on older versions."
Enter: automatic dbt upgrades, most recently known as “versionless” dbt, and generally available since May 2024. When enabled, this configuration automatically upgrades dbt environments to the latest and greatest version of dbt, including all new functionality from dbt-core and adapters.
Today, 85% of dbt Cloud customers are running “versionless” dbt. They get access to the latest features, fixes, and performance enhancements—automatically, without the manual overhead of coordinating upgrades across all their users and teams—and always before those features are available in dbt Core. Plus, they get access to new and exciting capabilities that are exclusively available in dbt Cloud.
To enable this, we made strong commitments to our customers about compatibility, and put in the work to see those commitments through: decoupling our adapter interface, introducing behavior-change flags, and making our robust testing and release pipelines more robust. Thanks to that work, the number of functional regressions in core dbt functionality has decreased by 60+% year over year.
This is vital work for defending dbt’s position as a standard for the industry, and its place at the heart of mission-critical workflows for some of the largest data organizations in the world. The strong organic adoption of “versionless” dbt validated our work on stability and ease of upgrades. We’ve consistently heard that customers value the ability to stay up to date with feature releases, bugfixes, and security updates. Still, some of those customers asked for a bit more control over when and how to receive those ongoing updates—trading off some of the latest-and-greatest functionality, in favor of predictability and Core/Cloud compatibility.
How it’s going: New naming and more flexible options
Based on feedback from our customers, from today forward, “versionless” is being crowned with a new name (or really an old one… third time’s the charm): “Latest.” This change is in response to customer confusion, and also better reflects how this configuration option really works. (“Versionless,” like “serverless,” is a misleading term; there’s always a version/server, so the real question is, do you need to manage it yourself?)
And, we’re broadening beyond “Latest” and including two new configuration options—“Compatible” and “Extended”—as part of a coherent set of “release tracks.” Compatible is the version of dbt that includes all changes from final dbt Core releases, updated once per month. Extended is the previous cycle’s Compatible version (on a one-month delay), updated once per month.
We’re broadening to release tracks because there are certain scenarios in which our customers don’t always want the hot-off-the-press version of dbt; there are real use cases for deploying slightly older versions:
- With Compatible, you can support a hybrid pattern (combining dbt Core + dbt Cloud) for developing and deploying in the same project. This use case requires a consistent set of functionality in dev and deployment environments, regardless if the team is on Core or Cloud. This is a step towards our vision for “One dbt”, as a turnkey way to ensure interoperability for hybrid deployments.
- With Extended, you can run production environments on versions that have been in the wild for more than one month.
- By using both in combination, you can try newer versions in a lower (dev or test) environment (on Compatible) before they land in your production environment (on Extended).
Below is a table of generalized customer architecture recommendations, by environment, based on your priorities:
It’s important to note that choosing Compatible or Extended means that you’ll get a slightly older version of dbt—but you’ll never be grossly out-of-date with your version upgrades. Your friends on Latest will have access to some good stuff sooner—and you can too, in a separate environment or user-level override to try out a beta feature and give us feedback.
It’s also important to note that it will no longer be possible to pick a years-old dbt Core version and get stuck on it, left behind while missing out on all the features and fixes of the subsequent years. This is a good thing.
Understanding the Compatible and Extended release tracks
By aligning to the functionality available in the most recent open source releases of dbt Core and adapters, the Compatible release track meets the needs of customers with hybrid Core/Cloud deployments, or customers who (for whatever reason) prefer less-frequent release cadences.
The Extended release track—which will always include the exact same versions as Compatible, just one month later—will meet the needs of enterprise customers who want the ability to test changes in development or staging environments, before those changes land in production environments.
These release track options make it possible for customers to get automated upgrades on a less-frequent cadence—after that version has been live in dbt Core for a period of time, and later than “Latest.”
To offer one example: Microbatch incremental models are an exciting new feature just released in dbt Core v1.9 last week. This feature has already been available on the Latest release track for months. The same goes for improvements to snapshots, state:modified, foreign-key constraints in model contracts, and more. Release tracks enable you to opt into trying new functionality on your terms—and to participate in discussions and feedback sessions that help shape the product and framework—with fewer barriers than before. Now that microbatch models have become available in a final release of dbt Core (v1.9.0), they have also become available in the subsequent release of the Compatible track.
Both Compatible and Extended are now available in Preview for eligible customers. For the very first release (December 2024), they will be identical, and they both include all functionality from dbt Core v1.9—meaning that the functionality they contain is very close to what’s in “Latest.” Over the coming months, they will diverge:
- Latest will include the very latest changes as we make, test, and release them—just the same as how “versionless” dbt has worked all year long. The “Latest” release track will include early access (always opt-in) to all new framework features before they land in dbt Core v1.10 prereleases or final releases.
- Compatible will have its next update in mid-January, including fixes in dbt-core v1.9.X patches and any adapter updates. It will not include new dbt Core features until the v1.10 release ~6 months from now.
- Extended will have its next update in mid-February, unless there's a critical security issue we need to hotfix.
Looking ahead
The evolution of our platform to release tracks puts us on a path to be able to regularly and continuously deliver new framework capabilities that dbt Cloud customers can use immediately. And to do this in a way that our customers never even have to think about—it just works.
We’re going to keep investing in the reliability of our framework and platform, across all release tracks—proactively detecting functional and performance regressions before they roll out to customers, and ensuring ongoing compatibility with more of the popular open source dbt packages.
If you’re not yet on release tracks, or are interested in using Compatible or Extended, below are important milestones:
- Latest release track is currently GA to all dbt Cloud customers. Compatible (Team & Enterprise) and Extended (Enterprise) release tracks are currently in Preview. We will also be introducing self-managed rollbacks for Business Critical customers in February 2025.
- All release tracks are expected to be GA in March 2025.
- Once all release tracks are GA, dbt Core v1.7 in dbt Cloud (which had its End Of Life date extended) is no longer going to receive any updates. At that point, we will be encouraging all customers to move to release tracks.
Finally, I’d like to extend my personal gratitude to the dbt Cloud customers and dbt Community members who have taken the time to speak with us about dbt upgrades over the past year. This is a big, nuanced topic—and it is thanks to your good-faith engagement and thoughtful feedback that we have been able to get this far. I’m excited about the balance we’re striking with release tracks— providing our customers with more control and flexibility, while ensuring they stay close to the latest and greatest that dbt has to offer.
Last modified on: Dec 18, 2024
Set your organization up for success. Read the business case guide to accelerate time to value with dbt Cloud.