Overcoming data bottlenecks to power smarter DataOps
Enpal, Germany’s leading solar and heat pump installer, faced challenges with their legacy data infrastructure. Bottlenecks, poor data quality, and slow processing times were holding back their vision of empowering employees with reliable, real-time insights. Operating across Europe with a team of several thousand employees, Enpal knew their legacy systems couldn’t keep up with the growing demands of their data-driven operations. To overcome these hurdles, they set out to transform their data stack and workflows, ensuring scalability, efficiency, and cost-effectiveness.
A vision for centralized analytics across the organization
Enpal’s operations—from customer acquisition to energy generation and supply chain management—depend on seamless access to trustworthy, reliable data. Dashboards and integrations with tools like Salesforce and Pipedrive ensure that business users can make data-driven decisions in their daily workflows.
To support this, Enpal’s central data team (~ 30 data professionals) focuses on providing the infrastructure for raw data and integrations. Collaborating with three groups of domain expert analysts, they develop data models designed to empower employees across business and fulfillment functions. While the central data team currently supports 1,000 employees, their long-term vision is to scale these capabilities and deliver actionable insights to the entire organization.
Tackling poor data quality and low velocity
Due to Enpal’s rapid organisational growth, their data infrastructure struggled to keep pace. The legacy setup, built on five separate databases dependent on a single larger one, lacked the consistency and scalability needed to meet growing demands. This fragmented design created l issues:
- Scattered transformations and data silos: Scripts were spread across multiple Microsoft SQL Server clusters and Azure Data Factory databases, creating duplicate datasets that were hard to maintain and inconsistent thus eroding trust of the users .
- Slow performance: Queries were inefficient due to the fragmented structure, hindering analytical workflows.
- Low data velocity: Dependencies on different teams with varying permissions caused delays
- Data quality issues: Frequent raw data changes and slow processing times caused pipeline malfunctions, affecting metrics and reports.
With their aging data warehouse, the data team took a critical decision: investing in a scalable, modern infrastructure.
“Mornings would start with ‘Alex, can you fix this?’,” said Alexander Novikov, Director of Data and BI at Enpal. “. We had to choose between firefighting and investing in a data structure that would meet our growing needs”.
Choosing simplicity and scalability
To address these challenges, Enpal took a bold step toward transformation. They opted to move away from disconnected platforms and adopt dbt’s unified data control plane. This approach centralized all transformations in a single, collaborative environment where analysts and engineers could efficiently work together, eliminating redundancy, and ensuring consistent data quality. Leveraging CI/CD and adhering to software development best practices, the team streamlined operations, improved collaboration, and established a scalable foundation for modern data analytics.
“Work before dbt was a rollercoaster. We had multiple places where transformation was happening”, shared Alex.
Shedding legacy systems: seamless migration with dbt Cloud
Recognized as the "state of the art data collaboration tool," dbt quickly became the solution of choice for Enpal’s data team. After testing dbt Core, they adopted dbt Cloud for its user-friendly interface and minimal onboarding effort, which was especially critical as they transitioned from their reliance on Azure Data Factory.
With dbt Cloud in place, Enpal set an ambitious goal: decommission their legacy server within three months while continuing to meet all business data needs. This required a strategic and phased migration process:
- Introducing analytics engineering concepts: Business In analysts, were onboarded to dbt Cloud. They learned the fundamentals of code management and CI/CD, enabling them to adopt modern data engineering workflows.
- Cleaning and structuring data: The migration provided an opportunity to organize thousands of legacy models into a streamlined data warehousing structure. Layers were created for raw data, cleaned data, core models, and reporting. dbt’s macros significantly reduced duplication, enhancing efficiency.
- Ensuring a seamless transition: Over 1,400 models were successfully migrated to the new infrastructure. Dashboards and reporting tools continued functioning smoothly, even as the legacy server was decommissioned.
This migration not only met Enpal’s immediate needs but also set the foundation for a scalable and collaborative data ecosystem, empowering the team to achieve their long-term vision.
Achieving data efficiency, stability, and 70% cost savings
Enpal’s data transformation delivered tangible results, driving operational improvements and cost reductions:
- Enhanced collaboration and sustainability: dbt’s built-in best practices, combined with Enpal’s new ground rules for naming conventions and pipelines, established a strong foundation for scalable, sustainable data development.
- Dramatically faster processing: Engineering optimizations such as modularity, macros, and the migration to Snowflake reduced job processing times from up to 36 hours to just one hour, significantly accelerating insights and decision-making.
- Improved system stability: Pipeline breaks became rare. What were once daily disruptions transformed into infrequent, quickly resolvable issues, ensuring reliable operations.
- Major cost savings: Migrating to Snowflake, dbt, and Fivetran reduced infrastructure costs by 70%. Enpal now spends just 30% of their legacy system’s six-digit monthly costs, with additional savings from decreased maintenance requirements.
Through this transformation, Enpal achieved not only cost-efficiency but also the operational resilience needed to scale their data capabilities across the organization.
Paving the way for broader data accessibility with dbt Semantic Layer
Enpal is looking ahead to make data more accessible across the organization. While dbt Cloud is currently used by the Data and BI team, plans are underway to extend access to domain experts, empowering them to leverage governed data without compromising compliance. The dbt Semantic Layer and MetricFlow will play a critical role, enabling stakeholders to interact with accurate, reliable data through tools they already use, such as Google Sheets and Excel.
“The path to data accessibility isn’t just dashboards but meeting people where they are, in the tools they already use,” said Alexander Novikov, Director of Data and BI at Enpal.
By building a modern, streamlined data stack, Enpal has positioned itself for scalable, efficient analytics that not only enhance internal operations but also support its mission of accelerating the adoption of renewable energy. The future holds even greater potential as Enpal continues to democratize data access and foster collaboration across its growing organization.
