Unlocking Data’s Hidden Treasure:

Cost Savings in Data Engineering

Published: November 2023
Client Name: OVO Energy
Industry: Energy supplier. Gas and electricity provider.

The main challenges for the client were high costs associated with BigQuery and the consumption of the Google Cloud Platform resources, and cost inconsistency - for example, the costs were higher in the Non-Production Environment than in the Production Environment. After the project briefing, the Irysan team performed a 10-day audit of the client’s systems, which was carried out without interruptions to the clients’ team and their day-to-day work.

At the end of the assessment, the Irysan team found and suggested cost savings of 63% of the monthly spend on BigQuery and GCP resources. Additionally, we suggested engineering practices that strengthen the security of the platform.

At the implementation stage, Irysan works in partnership with the Client’s teams - Irysan engineers are specifically trained for pair-programming and upskilling. This approach allows us to not only reduce the technology spend for the Client, but also to equip the Client’s teams with the necessary skillsets and best practices required for maintaining and operating lean and efficient platforms after Irysan has left the building.

Background

Founded in 2009, OVO Energy has grown to become the second-largest energy supply company in the United Kingdom, with around 5 million customers and over 8,000 employees and an annual revenue of over £5 billion (2022). OVO Energy is regarded for its innovative approach and commitment to sustainability: specifically, for the dedication to promoting green and renewable energy sources and the emphasis on a reduced carbon footprint. The company advocates and implements smart metre technology to empower consumers with real-time energy usage insights and improve energy efficiency. OVO Energy's focus on sustainability, innovative solutions, and customer satisfaction has positioned it as a notable player in the global energy market.

Challenges

This project focused on analysing over-utilisation and high costs of BigQuery and GCP resources in the Payments team of the OVO Energy Group. 

Irysan’s Data Audit team engaged with the Client to analyse the key challenges and propose a solution. During the project, we inspected and analysed trends in consumption and costs of BigQuery and GCP resources. 

Specifically, we examined trends of consumption and spend across the following GCP services: Compute Engine, Cloud Composer, Support, Cloud Run, Networking, Cloud SQL, Cloud Logging and Cloud Storage. We looked at the costs in both Non-Production and Production Environments, as well as the costs of Continuous Integration and configurations of the Cloud functions and K8s clusters.

Our analysis covered patterns of data processing and utilisation, a thorough inspection of GCP configuration, an analysis of BigQuery costs - especially compute and storage and a detailed, query-specific consumer analysis.
The team performed slot capacity calculations and also analysed the billing options and configurations for their suitability to the client’s use case. 

The outcomes achieved

Our bespoke focused approach allowed Irysan to suggest a range of realistic and achievable cost savings which resulted in the 63% reduction of the monthly spend on BigQuery and GCP resources. Some savings could be realised almost immediately, without significant development effort - such as changes to the billing strategies. Of course, while easy to implement, arriving at this recommendation required a clear understanding of the real resource consumption needs.

Other savings were more involved and included refactoring of antipatterns found in query construction, fixes of broken code, introduction of slot limits, BigQuery Edition-selection strategies for different types of workloads (e.g. using Standard Edition for scheduled queries), a set of recommendations for upgrades to cloud functions and refactoring of K8s cluster configuration, as well as a recommendation to change the approach for memory allocation - all of which resulted in significant efficiencies for the Client. We also suggested model versioning, to strengthen the security of the platform.

Finally, we discovered that some subscriptions were not used or used very sparingly, so we offered recommendations for restructuring those subscriptions for greater efficiency.

Results and Benefits

The key benefit was that without any disruption and upfront investment, the Client received a comprehensive overview of the key pain points of their data platform and cloud infrastructure, in only 10 days, with tangible recommendations of how to bring the cost down and drive efficiencies. This is rarely achievable with in-house teams, who are often busy supporting the day-to-day business operations.

For instance, just the recommendation to use slots with limits allowed to bring a monthly cost of £1,000 to just £1 in a non-production environment. Furthermore, switching test and development workloads to BigQuery standard edition brought down the costs from £3,116 to £82 per month.

The audit offers a clear visibility of the ROI and confidence that the value of gained efficiencies would far outweigh the cost of implementation of the proposed efficiencies.

Conclusion

The key takeaway of this case study was that the problems encountered by our Client are not unique to them - on the contrary, they are common in matrixed and successful organisations with the fast rate of growth and multiple product lines. Platform inefficiencies can impede productivity, innovation, and the ability to adapt to changing business needs. Some key factors contributing to platform inefficiencies include: silos and fragmentation, lack of integration, scalability challenges, lack of standardisation, embracing the status quo, insufficient performance monitoring, and many others.

Overcoming platform inefficiencies in large organisations often involves a combination of technology upgrades, process optimization, data management improvements, employee training, and a commitment to continuous improvement. It is often impossible to find time and resources for a holistic assessment of inefficiencies and develop strategies to address them effectively, especially in complex and fast-paced environments and when specialist skills are required. 

Outsourcing this task to a dedicated team of “Inefficiency Detectives” will save a lot of time and money and bring an additional benefit of equipping your team with necessary skills to design and operate more efficient, leaner platforms.