Small Data is the Future

02 Jun 2016 12:00 am by Leela Bozonelis

LexisNexis held The Small Data Forum in London's Hospital Club on Tuesday 17th May, where I had the pleasure and honor to moderate an engaged discussion with Neville Hobson of IBM Social Consulting and Sam Knowles, MD of Insight Agents, a brand storytelling agency.

A diverse audience of big and small data users listened and engaged in order to understand more about big and small data. Everyone agreed that from the point of view of commercial data processing, it is much more interesting to look at what percentage of "big data" can actually be harvested for the purpose of analysis. In 2013, that percentage was just over 20% and only a quarter of that was actually analysed and leveraged for generating insights. These values are set to increase significantly over the coming years. The world of Big Data is full of clever algorithms and data scientists are the new rock stars. Leaders in data science, such as Blue Yonder, are leveraging those algorithms and rock stars to build data driven 'Predictive Analytics'.

But just because we are capable of generating hundreds of Terabytes of data per second, and feeding complex data models for business models, does not mean that we can automatically predict the future, especially human behaviour, based on previous observations.

The desire to predict the future is a basic human need that predates the Oracle of Delphi some 3000 years ago. It is therefore not surprising that Big Data promises are quickly exaggerated to the clairvoyant (like in this January faz-net article).

The problem is not the data, but the way in which it is used. Nate Silver, one of the Big Data rock stars (his book The Signal and The Noise is an international bestseller), is also one of predictive analytics' biggest critics. According to Nate, when humans give meaning to the numbers they are not often objective in doing so. One need only be reminded of the many cognitive psychology studies proving confirmation bias; our tendency to shape reality in accordance with our ideas through the selective absorption  of information and facts. Even the most cunning of data scientists are not immune to this.

We base our models on cycles and relationships which we believe to be true. It is however, according to Silver, remarkable how frequently in reality we are confronted with 'unique events'; surprising events that, in spite of all the available data, may only become explicable after the fact.

Nassim Taleb, the author of The Black Swan, is in full agreement with Silver. Taleb has set his sights on disproving the disproportionately optimistic predictive models of the financial industry. His criticism is directed at the fact that the 'human element' in many Big Data models is neglected. Psychologist and Behavioural Economist Daniel Kahneman, whose insights into the human side of the 'homo economicus" were nevertheless worthy of a Nobel Prize, joins this point of view.

Even Big Data cannot get around the 'human problem'. We need data and models that are based on understanding of context and the human ability to judge. Our best chance of getting there is through an interdisciplinary approach; bringing together data science, social sciences and management science. Together, we will achieve solutions considerably faster.

About the author:

Thomas Stoeckle is Global Head of Evaluation & Insights at LexisNexis BIS. His area of activity encompasses the observation and analysis of traditional and increasingly social media, the development of innovative research methods and approaches, as well as offering support and guidance regarding utilized communications research.

Related Blogs
p.s. 3 ways you can apply this information right now to better understand news monitoring and analytics
  1. Monitor industry conversations to keep track of what the press and internet are saying. Our innovative technology and premium content – including traditional and new media – will help you transform information into actionable intelligence. Find out more.
  2. Follow this blog series; subscribe to our blog to have the updates delivered to your inbox.
  3. Leave a comment below. Let's start a conversation