FutureLearn Data Platform Optimisation: Reducing Snowflake Operational Costs and Streamlining Access to Data.
Published: March 2024
Client Name: FutureLearn
Industry: Online education and educational technology (EdTech) provider.
Executive Summary
The main challenge was to reduce the cost and time of running the Client’s Data Platform and to update ways of working between the business stakeholders and technical teams, to enable alignment of objectives and constructive dialogue.
Client Background
Launched in 2012, FutureLearn has become a leading online learning platform, democratising access to education through high-quality courses from top universities, such as King’s College London and University of Edinburgh, and organisations worldwide. Since launch, the company has grown to over 20 million registered users, a catalogue of over 2,000 courses, and partnerships with over 260 institutions.
Recently acquired by Global University Systems, FutureLearn continues to expand their offerings and innovate in online learning formats.
FutureLearn relies heavily on its Data Platform for making product decisions, business performance monitoring and client reporting.
Challenges
Since inception, FutureLearn’s Data Platform and the data models grew in complexity, eventually becoming disproportionately expensive to maintain and difficult to manage and extend.
The main challenges were to reduce the operational costs of running the Data platform, without completely refactoring the platform, respecting the strict timeline to deliver the cost reduction. Other challenges included keeping the data platform up-to-date with a small number of engineers, and driving continuous alignment between the stakeholders - the internal consumers of data, with the data engineers, data analysts and data scientists.
Solutions
Irysan Data Audit team engaged with the Client to analyse the data models that were being processed daily, and propose efficiency-focused changes, reducing the redundant reprocessing of data, and adjust the computational resources used in Snowflake and AWS for all data processing.
Furthermore, we resolved the upkeep problems with the platform with an analysis of the current state of continuous integration and delivery (CI/CD) and subsequent improvements to them.
We facilitated a constructive dialogue between the Growth team, the Product Engineering and Data teams, via a number of workshops, which focused on objective clarification and ways of working.
The outcomes achieved
Our bespoke focused approach allowed Irysan to suggest a range of realistic and achievable optimisations and cost savings in a short period of time.
The key outcomes were as follows":
Irysan identified models the results of which were not being used, disabled them, removing altogether at a later date;
Identified models that were reprocessing data without changing the end result, which could be modified as an “incremental model”, keeping data that was already processed and only processing new data. This change reduced runtime for the modified models by 50%-90%. The models had original runtimes ranging from 10 minutes to 40 minutes and after optimisations that changed to 1- 4 minutes.
Irysan managed to reduce the Snowflake usage by 25%, within the first 3 months of consultation, through efficiency-driven optimisations. That brought down the cost by USD$150,000.00/annum, and prevented the near-exponential YoY cost increase.
Projected Snowflake usage reduced by 55%, within 1 year of consultation.
With this, the end of the contract term was reached with credit left to spare. In contrast, before Irysan engaged on the project, FutureLearn’s Snowflake credit consumption was projected to exceed annual limits 3 months before the contract end.
The Irysan team also updated the automated tests and integrations to the latest versions, improving security and stability, and restructured tests for faster execution. We also migrated some of the infrastructure to infrastructure as a service (IaaS) framework using Terraform and Terragrunt, providing better control over infrastructure-related costs and facilitating the management of multiple systems in one single place.
Last but not least, before the Irysan teams started on the project, the Data Platform and Data teams and business stakeholders were working in silos, and were not benefiting from the expertise of their respective teams. This impacted not only FutureLearn’s ability to lean on their in-house Data expertise to power business growth, but also led to tensions between business and technology departments.
The Irysan team took a holistic approach to updating FutureLearn’s ways of working, facilitating a dialogue between the business and the tech, propagating this change all the way to the C-suite. This change resulted in happier, aligned and higher-performing teams, working together to take FutureLearn towards its strategic success.
Results and Benefits
The key results and benefits of this project were as follows:
Streamlined Platform: as a result of removing unused models.
Inefficient models refactored as “incremental models”, reducing runtime for the modified models by 50%-90%.
Reduction in Snowflake usage by 25%, within the first 3 months of consultation, bringing the operating cost down by USD$150,000.00/annum.
Projected Snowflake usage reduced by 55%, within 1 year of consultation.
Conclusion
Increase in technological complexity and platform spend are a normal occurrence for evolving businesses, but untamed, they significantly hinder the ability of the business to evolve and drain the budget. It is also a common occurrence for Data teams to work in silos from the rest of the business, when actually, they should be involved in strategic conversations and help the business make data-driven decisions.
In this instance, going “back to basics” and methodically reviewing necessary and obsolete resources, ruthlessly driving efficiencies resulted in massive financial savings in Data Infrastructure costs, improving FutureLearn’s financial health.
Updating ways of working, facilitating a dialogue between business stakeholders and tech teams solved the problem of misalignment of objectives, and drove significant operational efficiencies, funnelling resources and expertise of the business towards areas of strategic growth.
