Leveraging data-driven behavioral science
Identifying and reaching financially at-risk customers
Founded in 2016, Symend helps companies, like Canadian telco Telus, with digital customer engagement “treatment strategies” to collect debt from customers. Millions of consumers have been reached by Symend, helping them avoid negative credit outcomes.
A new vision with data at its core
Although Symend started with a single channel—emails—over the years, data’s role expanded from results reporting to shaping the product. Today, the company uses Artificial Intelligence, and machine learning to identify, segment and create content and journeys aligned to drive desired outcomes. The longer a customer is with Symend, the more value they receive thanks to a deeper understanding of their data and their behavior, which allows for further personalization.
A legacy data stack unaligned with the business’ new data vision
Symend’s “V1” data warehouse was originally built on a tier-1 cloud data warehouse SQL offering to run weekly reporting. However, the company’s growing needs were leading to issues, such as:
- Poor data quality: with more time-sensitive data needs, the pipeline was pushed from weekly to daily operations, which resulted in regular incidents.
- Limited accessibility: more team members needed access to data— to expose it, the data team had to create and maintain a complex third-tier system as a workaround for technological limitations.
- High data latency: although not an issue historically, Symend’s new data products required lower data latency.
- High cost: the operating cost of the “V1” data warehousing solution proved to be much pricier than initially estimated by the data team.
With the legacy stack, Symend was building its advanced analytics on top of a shaky foundation—a risk to scaling further. Since data had become mission-critical, it became clear to the team that data needed a better home.
The search for a new data stack, designed for engineers
Back to the drawing board, starting with Snowflake
Symend settled on Snowflake as the anchor of their data system. During their evaluation phase, one of the companies’ core considerations was that the new stack should fit their workflow as an engineering organization: the software development lifecycle (SDLC). Code shouldn’t be written in production and large SQL stored procedures should be avoided, replaced by CI/CD and modular development instead.
Evaluating, selecting, and migrating to dbt Cloud
dbt Cloud plugged directly into Symend’s SDLC, with a direct connection to GitHub. The product also offered critical features important to Symend: Jinja and macros support for modular models, built-in observability, and concurrent pipeline runs.
The migration to dbt Cloud and Snowflake was completed in 1.5 quarters, marked by the first data set going live. The team went through dbt’s on-demand training alongside support from a consultant to understand the product and build comprehensive documentation.
Delivering value faster, at a lower cost
Improved accessibility
Symend’s first live bronze data set on dbt Cloud and Snowflake was made immediately available to analysts. In under 2 weeks, analysts with no previous dbt experience were onboarded and exploring the data sets. Whereas previously analysts had to query production directly, they now had unlimited (yet governed) freedom in the Snowflake environment to uncover valuable insights for the business.
Productivity gains and data quality improvements
The ease of use of dbt Cloud, paired with out-of-box features like the job scheduler, data testing, modularity, and documentation, increased the efficiency of the data team:
“The way dbt is designed and documented makes it very easy to follow and comprehend. If something goes wrong, it’s easy to debug,” said Raman Singh, Tech Lead at Symend.
“With dbt at the heart of our data transformations, we were able to do the work of 8 people with a team of 4 FTEs.” added Ziko Rajabali, VP of Engineering at Symend.
The built-in development best practices provided scalability to Symend’s data infrastructure; the solution now seamlessly loads tens of millions of records daily.
Decreased cost for higher frequency data
As part of the migration from their legacy stack, Symend decreased data latency from one week to 12 hours. As part of the migration from their legacy stack, Symend decreased data latency from one week to 12 hours.
With dbt, the team was able to create very modular systems, because they could choose how they materialize the data—not everything needed to be materialized to be reused. dbt allowed the team to embrace engineering best practices to build a system that was scalable, version-controlled, and accessible to more stakeholders. All in a way that didn’t break the bank.
At a higher level, the paradigm shift to the cloud allowed the team to develop a more streamlined solution, increase data velocity with a simpler architecture, and allowed them to improve costs with more efficient executions of the full data pipelines. This amounted to drastically reduced costs compared to what they were paying for their legacy stack, totaling 89% less:
“Even though we are loading more data, at a higher frequency, higher quality, and in a more accessible manner, our new costs were only a fraction of the previous price. There was no loss in the equation,” shared Ziko.
Lower data latency opens new product development capabilities
Symend’s improved 12-hour latency from moving to the cloud was still deemed too lengthy for product analytics needs. They were ultimately able to reduce latency even further, to only two hours, by revamping the design of their dbt models to make them run incrementally.
They leveraged dbt features like built-in macros and dynamic queries, enabling them to run 200 models every 2 hours, instead of every 12 hours. All while setting themselves up for future scale, and notably, without driving up compute costs. In fact, they were able to save 70% of daily credits on Snowflake.
This improved latency opened opportunities to build new capabilities based on data. With data pipelines refreshing 12 times a day, they were able to ship three new data-driven product features, improving both customer retention and Symend’s competitive advantage.
“I fell in love with dbt when we were decreasing data latency,” said David Petiot, Senior Manager of Enterprise Analytics. “We were previously doing full loads, twice a day. With dbt’s built-in macros, we implemented an incremental solution that significantly increased Snowflake's speed so we could get latency down to two hours.”
Looking ahead: BI migration and a semantic layer
Symend will soon be incorporating a new BI component, Sisense, into their platform—a move simplified by all data models and logic now living in dbt Cloud. The team is also exploring how to best use additional dbt Cloud features, such as leveraging dbt Semantic Layer to ensure metric governance and increased data trust.
