What’s Holding up Big Data?

The impact of big data and data analytics on payments hasn’t materialized yet. That doesn’t mean it isn’t going to be important once the kinks get worked out. But some of the kinks have proven to be quite difficult to resolve. Issues that are not yet solved run from purely technical problems to legal and ethical considerations.

Data Collection. Every year more and more data is created. It would seem that collecting the data would be easy. But collecting data in a meaningful way—in a way that it can be analyzed usefully—continues to vex those who know that value lies locked inside the data.

The first step to obtaining actionable insights into shopping, spending, and payments is to identify when the same individual is showing up making payments in multiple, unrelated data sets. This data-collection problem is mostly a problem of differences of definition and context. Exactly what data, and how much of it, is sufficient to identify a person?

The full set of data needed for identifying a specific individual with a high degree of surety is never going to be collected all at once in any one commercial interaction, other than perhaps at account opening. After that, it becomes a problem of taking partially identifiable data collected in various interactions with a customer and trying to determine if it is the same person.

The use of loyalty card numbers will do the job within any retailer. But with no central directory of such numbers, knowing if Jane at the grocery store is the same Jane at the restaurant is difficult to determine to a useful level of surety.

From Correlation to Insight. When an individual can be identified across various touch points, the next problem is making sense of the data. Jane may show a pattern of buying red wine when she shops for groceries. And Jane (whom we know to be the same Jane) shows a preference for a certain type of restaurant. But what does this possible correlation mean? And how can it be used to either sell her more wine or get her to try a new restaurant?

From Insight to Meaning. Correlating the days on which Jane buys wine with the days that she goes to a restaurant (or, for that matter, with the weather on each day) would not be hard. But again, what does it mean? A good statistician can tell us how likely it is that the seeming correlation is real or just random. But knowing that it isn’t random still doesn’t let us reach an actionable conclusion.

From Meaning to Action. When big-data analytics is able to tease out meaning, the question then becomes what to do with it. Knowing that patterns may reveal behavior in a limited way still leaves retailers and bankers with the task of figuring out what is the opportunity to take profit-producing action.

What is being found out is that the ability to analyze the data is far in advance of the data itself. It is not hard to figure out why. Probably well over 90% of the data that is available on people’s behavior has been collected without foreknowledge of the kind of data that would be most amenable to the analysis techniques we have available today.

This is not a problem that will go away with time. More time just means more unusable data being collected. What is needed is for the analysts to start specifying how the data is captured. Where this has been done, the promise of big data and data analytics is proving to be real. As this “backwards” approach—driving the data collection by the input needs of the data analysis—spreads, the promise of big-data analytics comes closer.

Privacy. Concern about privacy continues on the part of both consumers and those who would like to have more data about the consumer. The questions are how privacy is defined and how it can be protected. The European Union’s determination that people have a “right to disappear” from the Internet is just the most striking example of how the concern over privacy is a growing issue.

Where the problems lie is becoming clear as more and more banks, retailers, and others get involved with analyzing big data. Analyzing large data sets will prove to be a boon to understanding customer behavior. It will come to fruition. It’s just not here yet.

— George Warfel, george.warfel@edgardunn.com