Hello! And About the Cloud IDE...

Hello everyone — my name is Nate Nunta. I want to write to you today about the Cloud IDE and share some updates on how we’re planning to improve your experience in the coming months.

Briefly about myself — I joined dbt Labs as the product manager for experience (including the dbt Cloud IDE) about 10 weeks ago. I’m very food-motivated, I love big-budget TV shows/movies, and my happy place is to host dinners with friends, especially using GBBO recipes. In my prior role, I led a small data team and built out a medium-sized dbt project, and like many of you, I developed my models locally on VS Code.

I’ve been lurking in the dbt Slack community for a few years now. I never actually posted anything but always appreciated that someone out there had already asked my very specific data modeling questions, even if I needed to search deep in the threads. While it feels a little weird to be jumping in with my official dbt Labs hat, I’m grateful that the internal conversations at dbt Labs look pretty similar to the discussions that I know are top of mind in the community. And rest assured, everything you have been sharing through Slack, customer support, account teams, Twitter, and Substack are being communicated to me and has consumed the first couple months on the job. I hear you. I know the IDE isn’t delivering the experience that many of us had hoped for. I feel the frustrations daily too as I use the IDE for my own analytics work. That brings us to this post—let’s talk about the Cloud IDE.

The Problem

The dbt Cloud IDE (integrated development environment) was first launched in 2020 as a way for anyone who knows SQL to develop dbt models in the browser easily and safely. It has been a productivity boon for hundreds of data professionals just wading into this way of work. But, it has seen better days. As you may have learned from Julia’s recent post on scheduler improvements, some things in dbt Cloud weren’t built to support the scale of users we’ve amassed in the last 2 years—including the IDE. Today, with thousands of you logging in weekly to build, review, and learn dbt, it’s incredibly frustrating for all of us at dbt Labs to know we’re delivering a sub-optimal experience.

I’ve spent my first two months here working closely with our engineering team to catch up on our product and infrastructure. Here’s a quick download on the architectural root causes of our performance problems with the IDE today:

Environment startup (causing long and unreliable startup of the IDE)

When you click Develop to start the IDE, a couple things are happening behind the scenes. We provision a dedicated server to power your environment, we fetch all the contents of your files, and we start a dbt-rpc server to help compile and run queries within the environment. Only when all these steps complete can you actually start development. The majority of sessions take 15-30 seconds, which is already not so great, but some of you have experienced upwards of 2 minutes or much longer, which is horrible!

File re-syncing (causing slow interaction during the IDE session)

The current architecture requires the IDE to have the latest versions of all files. This means that any time you perform an action—be it saving a file, committing your code, or running a dbt command—the IDE is put in a temporary read-only mode to re-sync all the files from the server. In other words, the IDE becomes slower and less interactive as you have more files in the project (including dbt-generated log files and packages).

The Near-Term Plan

Our priority is to address the performance and reliability issues that have been degrading your experience developing dbt models on the cloud. Our primary metrics that we have started to track include startup time (time from starting the IDE to being ready to develop), interaction time (time to save, commit, or perform an action), and failure rate (percentage of time that the IDE fails due to non-user error).

We have worked very hard to identify incremental ways to improve our current product to be more performant using our existing architecture—hopefully you have seen some of those incremental changes in the past few months. Unfortunately, to truly fix the underlying problems, our team has concluded that the best way forward is to rearchitect a fairly major part of our app.

In the next low single-digit number of months (classic PM’s ambiguous timeline), we are working to refresh the app in a way that supports more scalability. This includes 1) allowing you to access the IDE sooner even while some components are still getting ready and 2) redesigning the way that we work with the filesystem to avoid file re-syncing in every action you take.

I’ve already seen some of this work start to come together in the lab, and I’m confident that you’ll agree that the updated experience is a meaningful improvement to where we are today. Much of the experience of any IDE is its responsiveness… a snappy IDE minimizes the distractions so you can just get into flow. The near-term plan is to get a lot more of you into that flow state ASAP.

The Long-Term Plan

Once we nail down performance issues, we will focus our attention on new functionality we know you want in the IDE. I am well aware that many of you build dbt models locally on tools like VS Code, Atom, or other editors, and there are essential developer ergonomics and dbt-specific development features that we would like to bring to the Cloud IDE. Those are probably still a couple of quarters away, but these are the things that get me excited. Linting, native integration with documentation, intellisense, and more—I want these things too, and we’ll get there.

The dbt Cloud IDE and the associated Developer (free forever) plan are incredibly critical to our vision for the dbt Community. As more and more humans get exposed to analytics engineering and dbt, it becomes exponentially more important that we make the onboarding and developer experience as smooth and accessible as possible. The more accessible this experience becomes, the more humans will adopt the analytics engineering workflow, and the more diverse this group of humans will be. It's important that we build an experience that works for you, and I’ll be working with all of you to get this right.

Stay Tuned

I’m excited for the changes that we’re making to improve the IDE. While I do wish I had that product magic wand to fix all these problems today, this will take some time. We’re on it. Stay tuned on #dbt-cloud channel for further announcements when we’re ready to launch a beta.

In the meantime, if you have any questions or feedback, please DM me: I’m @Nate Nunta. Comments, questions, and venting are all welcome.