Waikato Milking Systems embrace the power of Data Lake

To support its growth due to international demand and overcome increasing levels of staff frustration with using multiple systems, WMS selected Fusion5 to implement Microsoft Dynamics 365 Finance and Operations into the business in 2022.

“Before Dynamics 365, we were running five ERPs due to various business acquisitions,” says Josh Norquay, ICT Service Delivery Manager for Waikato Milking Systems. “That meant our data was stored all around the business, on different interfaces – and it was often replicated, making it hard to report on anything with confidence. The introduction of Power BI was transformative for the business.”

While moving to Dynamics 365 Finance and Operations solved many problems for WMS, the business needed more than just standard reporting – which wasn’t possible without better access to data.

“We wanted to be able to generate far more complex reporting and metrics, as well as gain deeper insights into our customers and markets. And, of course, to be able to make better business decisions. Due to using Power BI extensively to consolidate, dedupe product details, and rationalise our data to improve inventory accuracy with our past ERP systems, we wanted to be able to replicate the often-complex dashboards. This necessitated the adoption of Azure Data Lake,”

Josh Norquay | ICT Service Delivery Manager, Waikato Milking Systems

WMS considered several business partners for its data project. “We chose Fusion5 based on our previous experience working with them, their obvious data and analytics expertise and capability with Microsoft solutions, and their ability to maintain strong ongoing partnerships,” says Norquay.

To enable WMS to generate more in-depth reporting and analytics, it was agreed to stage the company’s data in a separate storage area, using Azure Data Lake.

A cloud storage technology in the Microsoft Azure cloud platform, Data Lake lets users store and work with "big data" for analytics. As it supports any type of data (structured or unstructured), it provides that essential ‘one source of truth’ for analytics.

Fusion5 moved WMS’s data to their new Data Lake using the Synapse Link for Dataverse solution.

WMS users can now select up to 350 tables in Dynamics 365 Finance and Operations in one go and store them in their data lake. Unlike a BYOD service, using Azure Data Lake means WMS doesn't have to monitor and manage data exports. The integration manages the continuous near real-time export of data to the data lake as it changes in the Dynamics 365 Finance and Operations apps.​

The WMS team can select the data they want using standard or custom tables or entities, and then work with data in the data lake using Microsoft Azure Synapse Analytics as their base data analytics platform. The data can be refined, transformed, or manipulated using Power BI, PowerApps or Power Automate.

Using Data Lake has reduced WMS’s cost of data storage. It’s not only less expensive than relational databases, but because the data is stored as .csv and no initial transformations are applied, the read time is also faster. Data Lake provides additional metadata like table structure, descriptions, and data types so downstream tools like Power BI can determine which data is related.

Fusion5 structured, aggregated, and presented WMS’s data in Data Lake to allow users to generate near real-time insights into the company’s sales, inventory, and procurement activities. As WMS runs a predominantly Microsoft environment, users found the familiar look and feel of the data lake and data analytics platform and tools easy and intuitive. 

"Our data is consistently named and searchable, so we can find what we are looking for far more easily. It's easy to share data as well."

Josh Norquay | ICT Service Delivery Manager, Waikato Milking Systems

And now that they can see all product numbers in one place, WMS has been able to clean up all and any instances of duplicated data. Norquay says this, in turn, has enabled them to rationalise their inventory, confident they finally have 100% visibility of stock.

Before the move to the data lake, WMS employees had mainly relied on standard ERP reporting imported into spreadsheets. They had little clarity of the accurate and up-to-date information they needed to help them to do their jobs.

The introduction of the data lake allowed Norquay and Fusion5 to create custom dashboards for each role in the business to serve up the exact data that team members require for their decision-making processes.

“The dashboards have enabled us to improve efficiency throughout the business. We can track what’s going on where – from purchase orders to the warehouse and inventory levels. We can easily see where we have slow and obsolete products and general inventory turnover – which is essential to our survival if there’s an industry downturn.”

Josh Norquay | ICT Service Delivery Manager, Waikato Milking System

“The dashboards serve up all the relevant data for decision-making from our data lake,” says Norquay, “whereas before, we had to pull it through from multiple tables and positions. The dashboards have definitely improved our ability to make good decisions, more quickly.”

To accelerate WMS’s time to value, Fusion5 used its Fast Start accelerator methodology to set up the data lake environment, says Norquay. “They also used their expertise to guide us through applying best practice governance to how we manage our data lake going forward.”

“The functionality of Azure Data Lake combined with the capabilities of Fusion5 means we’ve got less housekeeping and administration tasks on our plate​”, says Norquay, “while having access to all the data we need to power up our reporting output.”