How Big Data became the precursor to the Internet of Things

By: Jon Kennard

30, May, 2017


Big Data - Connected Living - Data - Featured - Interoperability - IoT -


If technology is the skeleton of the IoT, data is the brain, distributed. As early as the 1990s Big Data emerged as a concept, but it wasn’t an idea that could be harnessed, leveraged and used for a common good until much more recently. Using the prefix ‘Big’ sometimes has negative connotations, but while Big Data may sound similar to Oil, Pharma or Tobacco, it’s a term that signifies possibility and organisation as well as scalability of the petabytes of information that we generate through our everyday on- and offline activity.

Jeremy Hindle is CTO and Co-founder of Headstart the AI-based recruitment app, and details a few important technological breakthroughs which have allowed us to use Big Data to some of its potential:

“It’s a culmination of fantastic hardware, software and infrastructure improvements which have resulted in more advanced usages –  tonnes of strong analytics tools allowing exploration of that data and UX improvements allowing non-data scientists to explore data naturally – Tableau, Quicksight, Superset etc., the rise in popularity of machine learning pushing more people into an industry which is almost entirely dependent on Big Data (at least for more complex problems and deep learning) and the ever-increasing number of open APIs and connected devices allowing for effective amalgamation and association of data sets.”

In enterprise, logistics and retail, monitoring of storage conditions along the supply chain and product tracking for trace-ability purposes will become ever more commonplace.

Geoffrey Roberts is managing director of Democrata, a predictive analytics company and he says the Big Data evolution is down to two things: first, the concepts that have democratised Big Data such as cloud development platforms like Microsoft, Amazon, Oracle etc. as well as open source software like Apache, and second, there is the technology that has enabled Big Data e.g. MapReduce (for the processing of large amounts of quickly changing data), Hive (a Facebook-developed SQL-like bridge that allows conventional business information applications to run queries against a Hadoop cluster) and PIG (another bridge, developed by Yahoo and open sourced, with similar functionality to Hive). Here’s his take on how important Big Data is to IoT:

“The IOT relies on the ability to make sense of data on the move. It is constantly changing and needs re-evaluating. Without the big data tools mentioned earlier, none of this would be possible. Also, we realise now that to capture data (and we can because of big data tools) we can create the relevant sensor technologies to capture the data.”

Jeremy Hindle adds to this:

“Initially I see the Internet of Things as something which was consumer product driven rather than the other way around. That being said this has now come around and most IoT devices have their highest value in the data which they collect.”

It’s this simple – without Big Data there would be no Internet of Things, and although the two concepts may not have developed concurrently, they are inextricably linked. The success of IoT is predicated on the use and understanding of Big Data.

We have evolved from simple storage of data to active usage of data, for analysis and prediction of behaviour. While industrial designers were busy developing hardware, data scientists were developing ever better ways of understanding the data that came with it. And, even though IoT hardware has existed in iterative but similar forms for decades, this new understanding of data has given our things their Internet. According to Geoffrey Roberts, here’s how enterprise and government could be (and in some cases already are) using Big Data and IoT in future:

In smart cities, among other things we’ll see widespread monitoring of parking spaces availability in the city, monitoring of vibrations and material conditions in buildings, bridges and historical monuments, sound monitoring in bar areas and centric zones in real time. Intelligent highways will display warning messages and diversions according to climate conditions and unexpected events like accidents or traffic jams.

In enterprise, logistics and retail, monitoring of storage conditions along the supply chain and product tracking for trace-ability purposes will become ever more commonplace. NFC Payment processing will soon be standard, and tracking customer and employee habits and preferences will become the norm.

Outside urban centres there’ll be more monitoring of combustion gases and preemptive fire conditions to define alert zones, snow level measurement to know in real time the quality of ski tracks and allow avalanche prevention, analysis of soil moisture, vibrations and earth density to detect dangerous patterns in land conditions, as well as chemical leakage detection in rivers.

The list goes on, and every IoT element is the result of the application of Big Data. However far we’ve come, it’s obvious we’ll be able to do so much more as our software, hardware and understanding improve.

(c) iStock: solarseven, thitivong