Big insight into big data

Internationally, almost everyone is talking about big data and there is a distinct possibility that it will change the world. Basically, big data refers to our ability to collect and analyse the vast amounts of data we are generating today.

The ability to harness the ever-expanding amounts of data is completely transforming our ability to understand the world and everything within it. The advances in analysing big data allow us to, for example, decode human DNA in minutes. Take this business example that was quoted on one of the sites I visited.

"Walmart is able to take data from your past buying patterns, their internal stock information, your mobile phone location data, social media as well as external weather information and analyse all of this in seconds so it can send you a voucher for a BBQ cleaner to your phone - but only if you own a barbeque, the weather is nice and you currently within a 5km radius of a Walmart store that has the BBQ cleaner in stock."

Why so much data?

That's scary stuff, but one step at a time; let's first look at why we have so much more data than ever before.

One of the research articles on big data mentions 'datafication of the world'. This 'datafication' is caused by a number of things including the adoption of social media, the digitalisation of books, music and videos, the increasing use of the internet as well as cheaper and better sensors that allow us to measure and track everything. Just think about it for a minute:

• When you were reading a book in the past, no external data was generated. But if you're using a Kindle or similar device, they track what you are reading, when you are reading it, how often you read it, how quickly you read it, and so on.
• When you listened to CDs in the past, no data was generated. Now we listen to music on our iPhone or digital music player and these devices are recording data on what we are listening to, when and how often, in what order etc.
• Today, most of us carry smartphones and they are constantly collecting and generating data by logging our location, tracking our speed, monitoring what apps we are using as well as who we are calling or texting.
• Sensors are increasingly used to monitor and capture everything from temperature to power consumption, from ocean movements to traffic flows, from dustbin collections to your heart rate. Cars are full of sensors and so are smart TVs, smart watches, smart fridges, etc.
• Finally, combine all this with the billions of internet searches performed daily, the billions of status updates, wall posts, comments and likes generated on Facebook each day, the 400+ million tweets sent on Twitter per day and the 72 hours of video uploaded to YouTube every minute.

A growing flood

I am sure you are starting to get the point. The volume of data is growing at a frightening rate. Google's executive chairman Eric Schmidt brings it to a point: "From the dawn of civilisation until 2003, humankind generated five Exabytes of data. Now we produce five Exabytes every two days...and the pace is accelerating." For those who like seeing many zeros one Exabyte is one quintillion bytes OR 1 EB = 10006bytes = 1018bytes = 1000000000000000000B = 1000 petabytes = 1 billion gigabytes.

Not only do we have a lot of data, we also have a lot of different and new types of data: text, video, web search logs, sensor data, financial transactions and credit card payments etc. In the world of 'big data' there are 4 V's that characterise it:

• Volume - the vast amounts of data generated every second
• Velocity - the speed at which new data is generated and moves around (credit card fraud detection is a good example where millions of transactions are checked for unusual patterns in almost real time)
• Variety - the increasingly different types of data (from financial data to social media feeds, from photos to sensor data, from video capture to voice recordings)
• Veracity - the messiness of the data (just think of Twitter posts with hash tags, abbreviations, typos and colloquial speech)

Sorting the wheat from the chaff

There is an enormous amount of data, in different formats, that is often fast moving and of varying quality - but why would that change the world? In the past we had traditional database and analytics tools that couldn't deal with extremely large, messy, unstructured and fast-moving data. Without going into too much detail, there's now software which enables us to analyse large, messy and fast-moving volumes of structured and unstructured data. It does so by breaking the task up between many different computers (which is a bit like Google breaking up the computation of its search function).

As a consequence of this, companies can now bring together these different and previously inaccessible data sources to generate impressive results. Let's look at some real examples of how big data is already being used today to make a difference:

• In the States the FBI is combining data from social media, CCTV cameras, phone calls and texts to track down criminals and predict [prevent] the next terrorist attack.
• Facebook is using face recognition tools to compare the photos you have uploaded with those of others to find potential friends of yours (we are all fully aware of how Facebook is exploiting our private information using big data tools).
• Politicians are using social media analytics to determine where they have to campaign the hardest to win the next election.
• Video analytics and sensor data of rugby or cricket games and all major sports around the globe are used to improve performance of players and teams. In the recent Cricket World T20 event the advancement in batsman and bowler data monitoring is leaving no stone unturned, honing in on finding weaknesses and strengths in every player so that strategies could be put in place to potentially negate the opposition.
• Artists like Lady Gaga are using data of our listening preferences and sequences to determine the most popular playlist for her live gigs.

Just the beginning

These examples are just the beginning. Companies overseas are barely starting to get to grips with the new world of big data.

In conclusion then, big data will change the world. In terms of language it seems that they prefer to talk about the 'datafication of the world'. The time is fast approaching that companies and marketers understand the massive potential as well as big threats of big data. This door will open up new opportunities.

I have to sign out now as I am analysing my personal drone which has been patrolling the Durban beachfront this weekend, sending images to my iPhone so that I can specifically pin point the demographics of our population using the new facilities...