Connecting Health Practitioners with Prescriptions
Founded in Ottawa, Canada, in 2011, Fullscript works to help more than 90,000 health practitioners deliver better care to their patients through its online prescription platform. With Fullscript, health practitioners can create personalized plans for their patients and access over 20,000 practitioner-grade products from more than 300 brands.
Any operation of this scale generates a vast amount of information, and Fullscript is no exception.
“Data is fundamental to Fullscript,” said Amit Jain, Data Team Technical Director. “We rely on data to report on both the financial and operational aspects of our business.”
The team tracks customer engagement through clickstream data on its front-end applications as well as back-end operations for warehousing and shipping. These insights boost customer service and allow the business to make decisions based on accurate, reliable numbers.
As Fullscrip grew through 2021, the organization's development structure presented challenges when it came to effective collaboration between data engineering and data analysts, who used separate workflows and tools.
“Data engineers were inclined to use software engineering development practices, which meant they didn't speak the same language as data analysts,” explained Amit.
Over time, the initial team members with a deep understanding of the legacy data engineering systems underwent role transitions, leaving the data warehouse team devoid of crucial historical knowledge.
“Almost at a Standstill”: The Need to Upgrade a Legacy Data Stack
Fullscript’s legacy data stack consisted of Postgres, SQL, and Python scripts embedded in Argo—a Kubernetes-based orchestration tool. However, it soon became clear that the organization needed to update these systems to a more accessible, reliable data stack in order to scale.
“From a data analytics point of view, our legacy systems were tough to manage when things went wrong, and the data didn’t match between source and target,” explained Jain. “We couldn’t identify faulty sources to apply the appropriate fixes.”
The fact that all of Fullscript’s tools were custom-developed further complicated matters.
“We were almost at a standstill,” said Jain. “The data pipeline was written using complex orchestration mechanisms and had SQL and Python wrapped inside the code. Our team was spending all its time just making things run.”
This massive drain of time and resources meant the team didn’t have the capacity to improve the systems or build new products. Fullscript wanted to bring in data from their CRM, NetSuite ERP, Google Analytics, and Facebook Ads, but there was no time to add new sources.
“Our teams were at capacity just managing issues as they came in. We didn’t have time to add new requirements or sources.”
Modernizing Online Prescriptions with dbt Cloud and Secoda
Jain and his team knew their legacy stack was limiting the team's potential. They needed to find reliable, scalable, accessible tools to transform and catalog their data.
“We estimated that our modernization project would take almost nine months,” said Jain. “We wanted a way to track how the migration was progressing because we had to have both systems running in parallel.”
After researching and experimenting with several potential tools, the Fullscript team decided that dbt Cloud was the best candidate for data transformation, while Secoda—an AI-powered data search, cataloging, lineage, and documentation platform—could help them keep track of information as it moved through their systems.
According to Jain, “Fullscript chose dbt—and dbt Cloud in particular—because it offers excellent integration of data engineering and analytics. It’s also very scalable, has great data analytics capabilities, and is easy to adopt and ramp up quickly.”
“At the same time, we found Secoda was the most flexible cataloging tool available. We looked at many other options, but there were always some gaps or source types they didn't support. I can’t imagine anyone running a modern data stack without Secoda.”
Soon, the team was able to run a proof-of-concept project during one of the company’s regular hackathons. Almost immediately, they delivered impressive results.
“I worked with a few data analysts on our team and used dbt Cloud to create a report within a couple of days,” said Jain. “In the old setup, where everything was custom developed, this would have taken months.”
Starting Small and Scaling Up
Transforming an organization's data stack isn’t a small or simple project. That’s why Fullstack’s team took advantage of dbt's ability to easily scale up as required.
“You could start with the open source version and then explore options to add dbt Cloud and then dbt Enterprise for the added support,” said Jain.
“It was a very flexible model. You could easily start small and prove your concept; this was especially useful when getting executive buy-in.”
With no engineers left from the team that had designed Fullstack’s legacy tools, the business needed a tool that was easy to onboard and adopt.
“That's what we found with dbt—a solution easy to teach and use,” said Jain. “Using just SQL, somebody with a data analytics background could use the tool fairly easily. For the initial setup, much of the tools just ran out of the box.”
Solid Support Leads To An Easy Implementation
Clear documentation and accessible support are vital in getting projects off the ground. The Fullscript team found dbt Cloud could deliver on both fronts when modernizing their data stack.
“We used dbt Cloud's documentation to create a reference architecture for the initial implementation,” said Jain. “From there, we were able to work with the dbt enterprise team to refine it according to our needs."
“The support and training capabilities dbt Cloud offered were almost as important as the tool itself.”
Ease of Use Delivers Cost Savings
While it’s easy to think of ease-of-use and well-designed support packages as fringe benefits, the combination of features allowed dbt to deliver when it came to Fullscript’s bottom line.
Jain noted that even though the business had moved from free software to a paid product with dbt Cloud, the switch still helped save on costs.
“A Kubernetes-based solution required a lot of engineer support and a wide array of specialized engineers,” he explained. “Once we moved to dbt Cloud, we didn't need engineers with very high Docker and Kubernetes expertise. Knowledge of SQL and analytics engineering capabilities were enough to use the platform.”
Tackling an Acquisition Mid-Platform-Migration
The flexibility of dbt Cloud, Secoda, and Fullscript’s new data stack was tested mid-implementation. The team was only a few months into the migration when the business merged with Natural Partners—a supplements provider operating on roughly the same scale as Fullscript.
“Overnight, our data sources doubled,” emphasized Jain, “The acquisition plans had been under wraps, so teams weren't even aware it was coming.”
Fortunately, the Fullscript team had already started making full use of Secoda, which was able to help them integrate the metadata from Natural Partners’ data warehouse.
“Secoda was very helpful,” said Jain. “We were able to quickly connect to these new sources and bring in metadata from their new source databases.”
While the acquisition represented a significant challenge for the data team, it also provided an opportunity to showcase the power of their modern data stack. Leveraging dbt Cloud, the team could quickly extend its existing data model and effectively map the data.
“Mapping the data took a lot of upfront work, but once it was done, we could extend our existing models,” said Jain. “All of that was powered by dbt Cloud, and it proved the scalability of the modern data stack. We did it quickly and finished the migration within the planned nine-month period.”
“I can't imagine how we would have been able to scale so easily or double in size without dbt Cloud and Secoda. I know we made the right choice with this set of tooling, and if I were to go back in time, I would choose it again.”
Looking Ahead: AI and Expansion
With a modern, powerful data stack now in place, Fullscript plans to continue building its capabilities over the coming months and years.
One of the key goals is to extend its data models to support the new requirements expected to come in from across the business. For example, the ability to support more Artificial Intelligence Markup Language (AIML) use cases, anticipating increased use of AI across both Fullscript and the broader data ecosystem.
“We’d like to use AI/ML around the recommendations engine,” explained Jain. “Specifically, for product recommendations using large language models, ChatGPT-style.”
“For example, the ability to develop guidelines for product recommendations would greatly help our practitioners. Something like that would have a major impact on their day-to-day and we now have to tools and foundation to make it happen.”
