Luma Analytics Logo .png

Tel: +64 274 670912

Auckland Chancery, 41 Shortland Street, 


New Zealand, 1010

  • LinkedIn - Grey Circle

© 2018 by Luma Analytics Ltd. 

Be the Fastest Fish in the Data Lake

Updated: Aug 30, 2018

Businesses can no longer sit back and be complacent in the knowledge that an established brand and existing customer base is enough to guarantee future success. Do so, and small and nimble start-ups, which are able to innovate at a much greater speed, will jump in and take your place. We’re all well versed in the stories of Uber, Air BnB and Netflix - whose innovation has disrupted the larger well-established competition who weren’t able to move quick enough to meet their customers shifting expectations.

At the heart of all such tales is the fast fish consistently winning with their superior customer experience by accelerating the use of their most important asset, data.

Making decisions faster

Traditional data and analytics architectures have been centred around the principle of batch processing. Data is collected by the systems running a business and is transferred, typically once a day, week or month, to a centralised data warehouse where analysts and data scientists can turn it into reporting, insights or use it as a trigger for action. Analytics teams and data consumers have at best been living in yesterdays data and at worst in last month's.

Over the last few years, the transition to what’s known as Fast Data architecture has begun to materialise, with the emergence of largely open source technologies that make it far easier and more cost effective to process and analyse data in real-time. This opens doors for analysts to provide in-the-moment insights and drive immediate actions and responses. Decision makers no longer have to wait for slow batch jobs to run before they can see results from yesterday or even last month. They can live in the now and react to the present.

Batch is a handbrake to innovation

Throughout my analytics career, I’ve seen first-hand how batch processing can be a hindrance on optimal business performance. Customers are offered products they’ve already bought, no longer making the offer relevant, or they’re asked for feedback on interactions that happened a week ago and have had multiple interactions since. Worse still, are customers failing credit checks after being conditionally approved for finance or businesses leaking revenue due to system issues that have gone weeks without being noticed. All a symptom of the limitations of batch data.

The diagram below, outlines how data driven conversations for a personalised customer offer can be immediate versus their batch counterpart.

Fast Data vs. Batch Processing

Batch processing might see the customer communication 3+ days after the transaction happened. A lot can and will change in that time. Out of date data defeats the purpose of having a data driven conversation. The risk of error is also far greater in the batch process due to the higher number of data jumps that require manual analyst intervention. Fast Data principles allow the transaction to trigger a reaction immediately though real-time processing and analysis enabling the customer communication to happen when it’s relevant and needed.

The possibilities are endless and valuable

The ability to analyse and react to events as they are happening is an exciting opportunity for any business that wants to be truly data driven. In a recently released report from Streamlio and Dimensional Research, 99% of the business decision makers surveyed said that they had existing applications that would benefit from faster data. However, 74% said that they still mostly processed data in conventional batch fashion.

The survey also showed how business leaders are awakening to the opportunities Fast Data can unleash across many different areas.

There’s going to be no slowing down in the exponential growth of data that is produced. According to Gartner connected devices are expected to boom to $20.4b by 2020. There will be a staggering amount of data capturing real-time events and behaviour detailing every aspect of society. Data will continue to flood in a greater volume and at a greater speed.

The race will be on to turn it into insight and actions to gain a competitive advantage.

So, make sure you don’t lose out in the race, be the fast fish.

Sources (Streamlio and Dimensional Research) (Gartner)