Data Lakes and Machine Learning

Models are trained on history — ideally raw, granular, and covering behaviour nobody thought significant at the time.

A short history worth knowing

The term "data lake" was coined by Pentaho CTO James Dixon in October 2010. His analogy was deliberate: a warehouse holds bottled water, cleaned and packaged for a known purpose, while a lake holds the water in its natural state, available for whatever purpose arises.

Fifteen years on, that distinction is what makes lakes the default foundation for machine learning.

Why machine learning needs the lake

Models are trained on history — ideally raw, granular, and covering behaviour nobody thought significant at the time.

That is precisely what warehouse design removes. Structuring data on ingestion means aggregating it, and aggregation discards the detail a model would have learned from. A lake keeps structured and unstructured data in native format, so data scientists can work with what actually happened rather than a summary of it.

A lake keeps structured and unstructured data in native format, so data scientists can work with what actually happened rather than a summary of it.

The lakehouse

The more recent development is lakehouse architecture, which brings data science, traditional analytics and machine learning under one roof.

It matters because the two-system split was expensive in a way that was rarely acknowledged. Organisations maintained a lake for data science and a warehouse for BI, copied data between them, and then spent time reconciling why the board pack and the model disagreed. A lakehouse adds warehouse-style structure, transactions and performance over lake storage — one copy, both uses.

What you get from doing this properly

• Structured and unstructured data centralised safely
• Retrieval fast enough for exploratory work, not just scheduled reporting
• Scalability that does not require re-platforming each time volume grows
• A foundation that supports analytics and ML from the same source

Keeping it a lake, not a swamp

A data lake with no access controls and no way to find anything in it is just an expensive place to lose data — the usual name for that is a data swamp. Two things keep it from happening. First, access control at the level of what a given user or system can see, not an all-or-nothing key to the whole store. Second, anything holding personal data gets pseudonymised so the lake stays usable for analytics without becoming a GDPR liability sitting in one place.

Getting ready

The preparation is unglamorous and decides the outcome. Catalogue what you hold. Assign ownership by domain. Separate raw from curated data so analysts know what they are looking at. Control access at ingestion. Track lineage, so a model's inputs can be explained later — which is increasingly not optional.

Organisations that skip this reach the same place every time: a large, expensive store of data that no one is willing to make a decision on.