Advancing the vision for the data control plane with SDF and dbt

Earlier today, we announced that dbt Labs has acquired SDF Labs. If you missed the announcement, you can learn more about it from Tristan here.

Since launching into GA in mid-2024, SDF has quickly emerged as a powerful tool in the data analytics space. The technology offers a multi-dialect SQL compiler and software toolset that results in remarkably fast developer feedback. This means that users can write and deploy data models orders of magnitude faster as compared to dbt Core alone. This extremely responsive development experience not only boosts developer productivity and pipeline velocity, but allows teams to “shift left” to catch data quality issues earlier, supports improved governance, and helps organizations optimize data platform costs.

What’s more, integrating SDF into dbt brings additional rich metadata into the “consciousness” of dbt. This will give dbt Cloud’s data control plane improved table- and column-level awareness, enabling users to confidently address new advanced use cases, such as for PII and PHI.

Folding this technology into the dbt engine will give our users unprecedented data velocity and efficiency while organizations can enjoy improved data quality and optimized data costs. We look forward to integrating the SDF team and technology into our company and are excited to work with them in cementing dbt as the industry standard for data transformation.

Advancing our vision of a data control plane

Last year at Coalesce, we introduced our vision for dbt Cloud as a data control plane that supports users across every stage of the analytics development lifecycle (ADLC)—regardless of their title, technical aptitude, chosen data platform, or where they build and consume data. Central to this vision is empowering all users to collaborate on data. For a data control plane to truly become a central component of any organization’s data stack, it needs to be built in a way that meets all users where they are, regardless of the role they play in the organization.

SDF helps us advance this vision in a number of ways. First, when developers are more efficient, it makes the platform that much more sticky. The performance and data quality benefits SDF will bring to the dbt workflow will help developers do more with dbt and cement it as the common framework for how data work gets done.

Second, enabled by capabilities like the visual editing experience and dbt Explorer, as more collaborators standardize on dbt, SDF’s tech will supercharge their efficiency. SDF helps everyone in the collaboration workflow “shift left,” making it more efficient for them to come to a common understanding on what's happening and what needs to get done. Less time dissecting why data pipelines don’t look right, more time delivering value to the business.

And third, SDF will add a new type of detailed metadata to the dbt consciousness—that is, metadata about the deep semantic understanding of the SQL itself—which enriches data lineage (both table- and column-level) and opens up an array of new complex use cases to be solved with dbt. The ability to traverse the data pipeline with this detail allows organizations to confidently embrace nuanced governance use cases (for example, that require tracing of PII or PHI to ensure compliance), helping business leaders build data trust and reduce business risk.

Enhancing the business value of dbt

While the combination of SDF and dbt brings a lot of productivity benefits to developers, the improvement in pipeline performance and data quality will deliver meaningful downstream impact to the business. Customers will be able to improve data quality while building trust in data through these new capabilities. Data quality is driven by the producer of data assets and trust is the primary interest of those that consume the data. Since good data quality leads to high trust, integrating SDF’s tech into dbt will ensure everyone can make decisions with confidence.

Customers will also benefit from how SDF will help them drive cost optimization. SDF and dbt will provide organizations the visibility and tools required to optimize infrastructure, operational, and people costs in data. “Shifting left” to catch data quality issues earlier in the development process—immediately as the code is being written—not only makes developers more productive, but organizations can avoid unnecessary warehouse compute by validating data models without materializing anything in the warehouse. This capability not only improves velocity of trustworthy data products, but does so in a way that optimizes data platform costs.

The road ahead

Over the next several weeks and months our team will be working to bring the SDF functionality into dbt. In the meantime, be sure to join our upcoming webinar where Tristan and SDF’s co-founder and CEO will talk more about the benefits of incorporating SDF’s tech into dbt.

We look forward to working with you, our customers, as we bring these new capabilities to your workloads. We hope you are as excited about this next chapter as we are, and look forward to working closely with you to help unlock the value and impact that SDF and dbt can have in your organization.