AN OVERVIEW
Not so long ago, in fact when I started working in our profession in 1989, the demand was for more and more information until … as the years rolled by and the internet arrived and evolved…we all began to get rather swamped. It became nearly impossible to keep up because information had grown exponentially. Then gradually the problem of Too Much Information came to the fore. Indeed the BIALL conference that was held in Sheffield in 2007 even took this issue as its theme. Meanwhile, in the US, Clay Shirky, who is a writer, consultant and teacher on the social and economic effects of internet technologies, saw things differently and reframed the issue as:
“It's not information overload, it's filter failure”
So, in a nutshell, could Big Data really provide the magic bullet that could give us the all important filter?
THE ORIGINS OF BIG DATA
McKinsey, the management consultants, are credited with inventing the term ‘Big Data’. The general consensus seems to be that the rise of Big Data started over a decade ago. The growth of the internet, mobiles, smart data, social data, and so on, led to a massive increase in data, in the size and volume of data sets, and an increase in the speed of data processing. More data was being produced automatically.
This exponential growth led to a need for cloud computing where data can be stored and analysed on a Pay As You Go (PAYG) basis making it both scalable and affordable to smaller companies. This democratisation of IT made it more accessible than previously, allowing companies with only 8 or so employees or even charities to work with Big Data. Cloud computing underpins the rise of big data and means organisations can break out of geographical boundaries and in terms of scale.
DEFINITIONS OF BIG DATA
Commentators tend to define Big Data as “data sets so large that they're impossible or really hard to process with existing systems” or “any data that you can't process in your own organisation. Data which is too large to be stored, managed and analysed in your own organisation”. According to Manny Cohen, CEO of RM online producers of ARMadillo, the first company information app launched in June 2011, “Big Data = Direct Data”. He says there will be a demand for business information ‘on the go’ and ‘at your fingertips’. He says future access is going to be direct and that all screens will soon have apps access.
Matt Benati, Vice President, Global Marketing at Attunity, specialises in the transfer of Big Data and describes Big Data somewhat differently as the “analysis and outputs that are possible when you combine massive computing power and gargantuan datasets”.
BIG DATA COULD PROVE TO BE A BIG WEAPON IN THE FIGHT AGAINST CANCER
What sort of scenarios are we talking about? They range from vast international science projects like the Hadron Collider to the more mundane commercial uses such as Sainsbury's checking over 14,000 product prices at the till against other stores and giving you a voucher for the difference back if you could have got them cheaper elsewhere! And potentially ground-breaking and life-changing ones such as targeting drug trials more effectively and for developing so called “personalised medicine” when Big Data is used to find out quickly which of the thousands of available treatments for cancer will work best with a particular patient.
So where does Google slot into all this since it is a filter for the internet of sorts. Well Google is also Big Data and can feel a bit like Big Brother looking over one's shoulder.
The areas where Big Data comes into its own are many and various:
UNSTRUCTURED DATA
There is an enormous amount of data on the web that is unstructured and hard to pull into an analytical system. Companies like Connotate monitor this unstructured data and deliver it in a very structured format, which can be plugged into an analytics platform. The result is that this unstructured Big Data can be turned into actionable information. The key here is to ensure that the results inform decisions that affect the organisation.
Basically Connotate automate the process of web data extraction and monitoring. They simplify web data for businesses. These days companies want to know how data can improve their businesses. Most organisations say they don't know enough about what they do. Analysis of internal data is therefore important.
According to Keith Cooper, CEO of Connate, amid the talk of volume, velocity, variety and voracity in Big Data discussions, it is more important to ask the question “what's the problem you're really trying to solve.” What is needed is to transform huge datasets into an outcome that presents “action critical” findings or “anticipatory intelligence for frontline workers”. It is better to have rough answers to the right questions than accurate answers to the wrong ones.
If we take competitive pricing as an example. If a customer wants to get every price on every product from every competitor to inform their own pricing decisions…. That is a lot of data! It is better to try and boil it down “What is it you really want?” It might be a subset of products that you view as most competitive, or ones where the difference of a nickel will make a real difference to the outcome, say 10% of the client's profits. So Connotate start there.
They then narrow it down further. Not every competitor is so sharp in changing those prices regularly to alter the demand and supply patterns; that is not 1,000 competitors, it is more like 10 competitors. This produces a manageable set of this big data – and it is the data that matters.
This sharpening of requirements is a good idea even if you do have the computing power to work with more data. There is a better chance of getting an answer that way.
Three components of big data that are vital to understanding what a big data project and solution require:
1. The importance of rapid and pristine transfer of data.
2. Making sense of volumes of structured and unstructured data elements and somehow creating analysable graphics and tools.
3. Semantic-driven text mining.
DATA MOVEMENT
Data movement is crucial. It is often a hidden variable in big data projects. It is important because big data relies on data of all kinds streaming continuously into the system, e.g. HR data, financial data, inventory data. It must get there efficiently and in a timely manner before it can be analysed.
Analytics relies on 3 steps:
1. Build a model
2. Train the model
3. Score the data
Fresh data needs to be coming in all the time, moving through an iterative process to train the model. And when the model is trained you have to keep bringing in that data to get the benefit.
According to Matt Benati, VP Global Marketing at Attunity, most big data today is moved by Fed Ex. Data is transferred onto a set of CDs because there is too much for a single disk. The CDs are shipped to a data warehouse centre to upload the data to the cloud. And then you are ready to train the model, score the data, do the analysis and so on. Even if you do this as quickly as possible the data is already a day old. This is known as the “big data bottleneck”. Annuity has streamlined the process of moving data. Traditionally data transfer systems are “all or nothing” propositions; you transfer the whole dataset or none of the dataset.
Attunity focuses on only a small fraction of the data that might have changed. After the initial data load, if you transfer just the changed data that has a huge impact on the speed and efficiency of the transfer. A comparison would be between traditional transfer that behaves like a commuter train making several stops along the way and a bullet express train going from point A to B rapidly, without stops.
Or in the words of Dr Phil Hastings, SVP of Sales and Marketing at Linguamatics, who deploy innovative, natural language processing-based text mining for high-value knowledge discovery and decision support, “we're providing the correct filters on big data to get you to that key information you are seeking”, even if it is on Twitter which he describes as “the world thinking aloud”. As well as structuring the unstructured data world, Linguamatics also specialise in knowledge synthesis' which in practice means extracting and combining key knowledge buried in documents. It extracts and synthesises structured and unstructured data presenting it in a value added way.
Hopefully, the above explanation gives you a clearer idea of Big Data ‘in action’.
THE IP AND DATA PROTECTION ISSUES WILL BE CHALLENGING
The ethics of collecting data is an issue with Big Data storing information on people in granular detail. Data footprint and data compliance are concerns – the rule of thumb being that data should be kept only for as long as reasonable.
In terms of data archives for academic research, stored items need to archived securely and stably according to Monique Ritchie, Research Information and Copyright Officer at Brunel University. Open access and open data are also important topics. Information professionals in that sector need to educate users about issues. Researchers should make sure what they store is anonymous. Big Data can look like Big Brother unless it keeps to principles. To take one example, Sainsbury's do not push alcohol even though they know the spending habits of customers. Social media accounts may be targeted to profile/customise marketing using the mantra of right place, right time, right message.
There are a plethora of concerns around the use of data as an asset; among them the analysis of data, holding of data and sale of data to someone else, whether data should re-used and/or “re-purposed” to answer new questions, etc. Fundamentally it boils down to the question of who owns whose data and whether it can be traced back to the individual.
More personalised marketing can be achieved. A student may be asked by a charity to send a £2 donation by text. Or in my case I was contacted via e-mail (and post!) by the University of Birmingham, my alma mater as one of 13 alumni whom the University hoped would agree to give £120 to help with an early stage prostrate cancer trial for which no other funding was available (allegedly). I am happy to give the money – it's a great cause, I love my old university and they are sending me a pin badge for my trouble but what if the marketing was for something unwelcome, intrusive or threatening? Or even all three. I would feel very cornered.
JOB OPPORTUNITIES
Do you fancy becoming a data wrangler, metadata librarian, data archivist, research data manager? “Data is information – this is our domain!” says Monique Ritchie, Research Information and Copyright Officer at Brunel University. Sadly, according to Dom Pollard, Editor at the Big Data Insight Group, “not many people have the right mix of talent for data sciences” i.e.
1. Statistical analysis
2. Data mining
3. Business hat – understanding of business processes
4. Communication skills
5. Technology/Software
Dundee University is running an MSc course which covers the perceived skills required. It describes a data scientist as follows; “a person who excels at manipulating and analysing data, particularly large data sets that don't fit easily into tabular structures (so-called “Big Data”).”
THE FUTURE
There may seem to be a lot of hype about Big Data but it has not yet matured as a technology . It is reaching the top of its hype i.e. the so called peak of inflated expectation. This article is probably just a taster of what is to come.
