dbt Labs on dbt: How our training team uses dbt Cloud

For those of us on customer education teams, we know how to make an impact on customers, but how do we tell that story to the greater business that isn’t directly working with customers or processing the feedback? How do we go beyond NPS and completion rates to demonstrate value to the business and real ROI? We’ll talk through how we’ve tackled this at dbt Labs to communicate the impact our on-demand learning.

Measuring the impact of on-demand learning

As a training team, we launched our first on-demand learning experience through an LMS in October 2020. In our world of customer education, there are some essential tools to launching this many offerings for educational content:

• An LMS is for building educational content, enrolling learners, and monitoring course progress.
• Credentialing software issues credentials for course completion and passing certifications.
• Survey software collects learner responses throughout the course and at the end of the course.

As we saw signups and course completions take off, we started to monitor our work across the following metrics:

• Quality
  • NPS on courses and training sessions
  • Completion rate of on-demand courses
• Reach
  • Enrolled — started or attended at least one learning experience
  • Trained — received the dbt fundamentals badge
  • Certified — passed the developer exam
• Revenue
  • How is training impacting our GTM efforts?
  • What direct revenue are we generating from the site? (Note: N/A as all content is currently free)

These metrics allow us to know where we can improve and communicate the ROI of our training programming.

Connecting learning data to business outcomes

As a team that doesn’t directly generate revenue, we’ve often faced the challenge of finding effective ways to demonstrate the return on investment of our programming. From talking to other customer education leaders, this is a common challenge in our space.

As outlined above, a customer education stack has multiple tools with various data sets. In isolation, this data doesn’t tell a compelling story as it is typically used to drive improvements on the content itself, not the larger business impact. The impact above that is hardest to measure is how is on-demand training impacting our GTM efforts?

The impact comes when we can connect the learner experience to business outcomes.

• How does course completion for customers correlate with churn?
• How do we use learner enrollment data to drive pipeline for marketing?
• How does learner engagement impact win rates for our sales team?

One method for doing this is the all too common approach of exporting to CSV, creating multiple tabs in an Excel workbook, and joining it all together with vlookups (or index + match if you prefer that like me) and aggregations.

Every time we run our report, we need to get the latest version of the data, bring it into the Excel workbook, and run the same analysis. If we want to compare changes over time, we are then likely comparing multiple versions of the same notebook. This is a ton of complexity as we change versions and iterate on your reporting.

Bringing it all together with dbt Cloud

Luckily, we work at dbt Labs with customers, so we know what the dream state can look like. Here is what we can build with dbt Cloud:

• Model our data to be in the shape that we want (not stuck in in-app reporting and CSVs)
• Enrich existing business data sets with learner data (I see you Salesforce 👀)
• Automate the refreshing of our reporting on a cadence of our choice
• Robustly manage the changes to our logic over time with version control

To get there, we need some world-class, modern data practices. Let’s talk through that journey together. Let’s take one of our questions above

How does learner engagement impact win rates for our sales team?

After doing some digging with the data team and using dbt Explorer, I’ve discovered the data sets that I need

1. Salesforce data - all the information about our sales cycle, specifically Land opportunities where are talking to a new potential customer.
2. LMS data - all the data about learner engagement, specifically sign-ups and completions

Next, let’s go on a quick analytics development cycle.

Mise en Place: Prepping ingredients for data excellence

Like a professional chef, we need to get all our ingredients in front of us and prepared. For us, that means getting all of our application data into a data platform like Snowflake or Databricks. Luckily, our data team has already loaded (and modeled) our Salesforce data. We used Fivetran to grab the data from our LMS and load it directly into our Snowfake instance. This is similar to exporting a CSV and uploading it to a database, however, Fivetran helps us run this on a schedule without the manual effort and clicks. Now we have everything we need to cook up some insights.

How do you wish your data looked: slicing and serving

When we look at any raw data, we notice that it’s actually pretty messy and not very ready for business analysis. There are all sorts of cryptic IDs and fields that were likely designed to make the application work well but is less optimized for analytics.

This is where staging your data comes in — we call this “cleaning the data to make it look like how we wished it looked”. This is like the food prep we do to chop our onions and dice our carrots before we really get cooking.

Mapping original columns to match our style guide, making them more descriptive, and creating a new column that we wish we had.

Once I’ve staged the data, then we can start to create the business concepts that I want. For learner data, I’ve created two models:

• dim_all_learners - a comprehensive model where each row is a learner with information like is_trained and courses_complete
• fct_training_enrollments - a single table that shows one row for each unique learner enrolling in each unique course with information like is_complete and percent_complete

By publishing these two models, we’ve already created a single place for our team to go for learner information.

However, we still haven’t quite told the story of business impact yet.

Enrich your business data

Our data team has created a model called fct_opportunities that lists every single sales opportunity in the history of the business.

Coming back to the question:

How does learner engagement impact win rates for our sales team?

We need to show learner engagement mapped against win rates. Let’s take the following approach:

• How does learner enrollments in dbt Learn before close correlate with win rates?
• How does learner completion of dbt Fundamentals before close correlate with win rates?

We need to map these learner stats against each of the opportunities. Here is how we did this

• The fct_opportunities model already has email_domain extracted as a column
• We add an email domain column to the dim_all_learners model along with enrolled_date and trained_date
• Using email domain (think vlookup) we can then roll up the number of enrolled learners and completed learners before close date.

Now these stats live on the fct_opportunities model in our data platform. Keep reading for what we discovered.

Automate refreshing of reports

The spreadsheet workflow I outlined above is extremely cumbersome. The beauty of SQL in dbt is that we can write all this logic once and refresh all the models on cadence that we choose.

In dbt, we do the following:

• Push changes to our production environment with a review from our data team. By moving our code to production, the rest of the org knows this data can be trusted
• Extract / load data on a cadence using Fivetran and other loading tools (no more CSV exports!)
• Transform data on a cadence to refresh the models like dim_all_learner and fct_opportunities (no more checking and re-writing formulas!)

This means we can show up to work on Monday morning with a coffee and our reporting is up to date.

Manage changes like a boss

We might update our logic as our business and learning program evolves over time. If we update our logic in spreadsheets, it might look like the following:

• training_win_rate_analysis_jan_2024_v1.xlsx
• training_win_rate_analysis_apr_2024_v2.xlsx
• training_win_rate_analysis_jun_2024_v3.xlsx

Oof! That’s really hard to navigate across files and compare cells to see how logic has changed over time and wrangle / review the changes in a robust way.

dbt Cloud leverages git for version control and here is the impact:

1. We can compare changes in logic between any two points in time. Think of this like a highly scalable version of version history in Google Sheets.

1. We can ensure the right people review the code before changes get pushed to production. Just send it doesn’t have to be the way we work with the right governance in place.

Git can be intimidating at first, but with a little practice, you can unlock this super power.

Turning insights into action

Through this data workflow, we were able to show a correlation such that if just one or more learners simply enroll in dbt Fundamentals, our win rate increases by 4% on average. Similarly, if we can get two or more learners to complete the fundamentals course, our win rate jumps by 3% on average. There are undoubtedly other factors at play here, but being able to point to this correlation speaks to the ROI of our on-demand learning efforts.

Why does that matter? Three things:

• Prospects can learn dbt and see the value of dbt Cloud before buying
• Sales can leverage our one to many content to win more deals with less time on calls
• Training gets more enrollments and completion to drive our KPIs

It’s a win-win-win for the everyone. Data speaks volumes—make sure yours is telling the right story. Unlock the full impact of your customer education efforts with dbt Cloud. Automate reporting, connect learning data to business outcomes, and prove ROI with confidence. Start learning today.