2024 State of Analytics Engineering

Welcome to 2024

Do you feel it? If you work in data, the ground is shifting.

This year, the State of Analytics Engineering highlights key trends: the ever-growing importance of data quality, the challenge of making data accessible and meaningful across organizations, and reduced budgets. And yes, the undeniable impact of AI.

Amid these dynamics, the essence of analytics engineering remains the same—applying software engineering best practices to harness the power of data.

Data practitioners are more essential than ever, but it's also a time of introspection for us in the data community. How do we grow through these seismic changes? How do we ensure the work we do remains compelling, relevant, and aligned with the evolving needs of our organizations?

The 2024 State of Analytics Engineering, sponsored by dbt Labs, examines how data practitioners and teams are answering these questions, shipping data products, and equipping their organizations for success.

Abstract & methodology

The 2024 State of Analytics Engineering survey report highlights the complex landscape faced by data professionals and their organizations.

What you will learn in this report

• The top priorities and challenges of data teams
• How data team objectives align to those of the broader organization
• How data teams are investing for the future—including how they’re thinking about AI

dbt Labs collected survey responses Dec. 22nd, 2023 - March 2nd, 2024 from 456 data practitioners and leaders. The margin of error for this study is +/- 5.6% at the 95% confidence level.

Respondent Breakout

• 29% of respondents are analytics engineers
• 23% of respondents are data engineers
• 10% of respondents are data analysts
• 24% of respondents are managers, directors, or VPs in the data space
• 14% of respondents are other types of data professionals

It starts with well-organized data:‌ Unsurprisingly, like last year, most respondents spend most of their time organizing data sets for analysis (over 50% selected it as their #1 task). This critical task is the bedrock that enables all downstream analytics to be done with accuracy.

Data quality on the mind: Meanwhile, the survey highlights critical challenges that persist in data management, with poor data quality emerging as a predominant issue for 57% of professionals, an increase from 41% in 2022. The importance of data quality and data trust were consistent themes throughout this year’s responses.

AI is coming: A significant and growing number of data teams are investing in AI. 57% of respondents indicate that they currently—or soon expect to—manage data for AI training, and many practitioners report already adopting generative AI in their day-to-day workflows.

Key Findings

Salary

Because let’s be honest, this is what everyone wants to see first.

North America

In a trend that holds up across regions, analytics engineers can expect to earn considerably more than their data analyst counterparts.

In North America, 78% of all analytics engineers earned over $100k/year, compared to a slightly lower 61% of all data analysts and 66% of all data engineers earning over $100K/year.

However, data engineers were the most likely practitioners (6% of data engineers) to earn over $200K/year.

Europe

In Europe, meanwhile, 37% of analytics engineers earned over $100k/year, compared with 29% of data analysts and 31% of data engineers.

Rest of world*

In the rest of the world (South America, Africa, Asia and Oceania), 21% of analytics engineers earned over $100k+/year, while no data analysts or data engineers reported doing so.

*We recognize “rest of world” is not a great term; we’ve grouped it this way because of a limited sample size of respondents from these regions.

The day-to-day of a data practitioner

Data practitioners carry a wide range of responsibilities, and the variety of those tasks expanded from last year. The top ways data practitioners spend their time:

1. 55% Maintaining or organizing data sets
2. 26% Maintaining platforms or infrastructure
3. 13% Building reports and dashboards

As organizations scale, so must their data teams. Once organizations cross the 500+ employee mark, data teams start to cross the “two-pizza” threshold, and it becomes the norm to have 6+ team members responsible for data infrastructure and data analytics, respectively.

How work is distributed

Work is distributed among data teams in a variety of ways—by function (data engineering, data science, etc.), by business area (marketing, sales, finance, etc.), by project, or by a hybrid approach.

1. Hybrid: both business area and function 36%
2. Function (e.g. data engineering, data science, business intelligence) 26%
3. Project-based 17%

The number of respondents reporting their work was distributed via a hybrid model increased from 24% last year to 36% this year, likely signaling that data team members of all specialties are embedding more deeply within their organizations.

Challenges for data teams

We asked respondents to tell us the top three challenges they faced when preparing data for analysis. Here’s what we learned:

Poor data quality is the most prevalent problem today. As data-related complexity increases, maintaining quality is becoming a greater struggle for teams. Already a concern in the last edition of this report, this has escalated as a significant hurdle for data practitioners: 57% of respondents now highlight it as one of their chief obstacles in preparing data for analysis, up from 41%. (Note that respondents were asked to select three challenges, so total percentages add up to more than 100%.)

Unclear data ownership persists as an issue. “Ambiguous data ownership” was the second most frequently cited challenge, highlighted by almost 50% of respondents. This number also increased from last year’s 43%, again pointing to potential obstacles with navigating growing data complexity.

Improved stakeholder data literacy would be a boon for data teams. Stakeholder data literacy was cited as an ongoing challenge by almost 50% of respondents, suggesting that there remains a considerable opportunity to enhance ‌data understanding among data consumers.

Most data practitioners feel they have a handle on building data transformations. 12% of respondents identified building data transformations as a challenge, down from 20% in 2022.

The value of a data team

What does “good” look like for data teams? Survey respondents cited non-quantitative goals as the predominant measure of their success. 72% indicated that they’re primarily evaluated against either enablement of other teams or project completion, with just a fraction primarily using progress to financial goals or SLA metrics.

How data teams measure success:

1. Enablement of other teams 42%
2. Project completion 31%
3. Qualitative evaluations 12%
4. SLA metrics are a key metric used by only 5% of organizations

Many data team members want clearer goals

Responses were polarized on whether goals are clearly defined for their data teams.

About one-third of data team members (33%) strongly or somewhat disagree that their organization sets clear enough goals for them. Worryingly, this number holds steady across individual contributors and data team managers. Only 14% of data professionals “strongly agree” that their organization sets clear goals for their team.

Investment areas for 2024

Amid recent macroeconomic changes, many teams report that they’re being asked to do more with less—a substantial number of respondents report reductions in budget and/or headcount.

However, most teams expect to maintain (rather than grow or cut) their investments in data tooling across the board.

As teams navigate challenges with data quality and complexity in this environment, organizational design is just as important as tooling. Decentralized data architectures like data mesh ‌continue to garner consideration across the industry, and at large enterprises in particular.

The economy has affected budgets for many data teams

Although “no change” was still the most prevalent single answer, 41% of data professionals say that recent changes to the macroeconomic environment have negatively affected the size of their data team’s budget.

And the economy has affected headcount

33% of data professionals say that recent changes to the macroeconomic environment have led to reduced headcount on their data team, although half of respondents reported no change.

Investments are mostly holding steady

Although respondents are generally not planning major changes to their investment in data tooling, some areas were more likely to be prioritized for increased investment in 2024 than others. The areas most frequently cited for increased investment were:

1. Data quality/observability
2. Data platforms (data warehouse, lake, or lakehouse)
3. Data catalogs

Data teams seek to build more trust

Asked to evaluate how important some potential priorities were to their own organizations, an emphasis on increasing data trust emerged as the #1 focus area.

Data mesh draws serious interest at the enterprise

While the concept of data mesh burst onto the industry’s radar only recently, decentralized architectures have been in vogue for some time. Today, these architectures draw consideration across companies of various sizes, but the largest enterprises are far more likely to be using them.

Over 60% ‌of data teams from large companies (5,001+ employees) are currently using or considering a decentralized data architecture such as data mesh.

AI adoption

There are signals of a growing trend in AI adoption among data practitioners, with an increasing number managing data for AI model training and incorporating generative AI into their daily workflows.

A third of respondents currently use generative AI in some form, and there's a strong expectation of future benefits it might help create—including for self-serve data exploration.

Teams are starting to manage data for AI training

While a large gulf remains, the majority of data teams currently manage data for AI model training or expect to in the next 12 months.

AI is part of everyday work…for some

Despite its relatively limited availability in analytics tooling today, one in three data professionals already ‌use generative AI as a part of their day-to-day data workflow.

AI is only growing in usage

Today, only 33% of data professionals use generative AI, but 55% soon expect AI to reap benefits for self-serve data exploration—among other use cases.

Now the really important stuff

10% of respondents have the incorrect opinion, respectfully. We may consider excluding them from future surveys.

The State of Analytics Engineering editorial team refuses to reach a consensus. The debate got out of hand, frankly. Things were said that can’t be taken back.

Most of you are underselling yourselves.

Our favorite charts are the ones about charts.

Download the report

The 2024 State of Analytics Engineering, sponsored by dbt Labs, provides a comprehensive overview of the pains, gains, and areas of investment for global data teams.

Complete the form below to download the report.