Sci-fi technology will conquer big data

In the era of big data, we cannot hope to cope with all that information on our own. Processing power is just the beginning.

The promise of quantum computing is still distant by technology standards - but it does spark the imagination.

The phenomenon of Big Data – vast, fast and varied – has raised the challenge of how to store it and how to make sense of it.

That challenge may be overcome in the not too distant future. The use of quantum optics has just won two researchers the 2012 Nobel Prize in Physics, and this could pave the way for a new era of high-speed communication and computation.

The discoveries of Serge Haroche and David Wineland eclipsed the more well-known work on the Large Hadron Collider, the world’s largest particle accelerator, which was expected to take the award.

In locations across the world, teams of scientists are using desktop laboratories to design experiments to trap and manipulate individual atoms or light particles using cheap, high-precision lasers, atomic-scale transistors and other quantum optical technologies.

As Adam Frank, professor of physics at the University of Rochester states: “This could well usher in a radical new era of technology, one that makes today’s fastest computers look like hand-cranked adding machines.” (New York Times, 14 October 2012.)

Quantum physics has given rise to technologies such as the transistor and the laser, currently embedded in computers and smartphones. Now it is being used in quantum computing and information processing.

Today, this technology remains more in the realm of physicists than commercial IT practitioners, and the promise of quantum computing is still distant by technology standards.

However, it does spark the imagination. It leads one to consider the vast opportunities the continuous increase of processing power and storage capacity could open up. The challenges we face now in terms of speed and the amount of data we can store will be overcome. However, will the challenges of what we do with that information and computing power remain?

Intelligence can be defined as the ability to adapt and learn quickly. It means being able to draw conclusions and make decisions based on information.

That’s what good business intelligence allows us to do. BI can provide insight by summarising data and making connections we are not able to see on our own. As we move into the realm of predictive intelligence, BI can assist us to learn from past mistakes and work out better solutions in the future. Machine learning is used, for example, in the currency trading markets, where computers make decisions for dealers.

Companies in the US already use BI to increase operational efficiency by predicting seasonal staff requirements. This has changed the way many retail and services businesses, like restaurants, employ people. They are able tailor their workforce to meet demand and can notify staff automatically when they are needed.

We are getting to the point where computers can make operational decisions for us. In essence, we are following mathematical philosopher Mark Hogarth’s notion of machines using information to make decisions that affect people’s lives. We will rely on machines to process basic data inputs and make decisions on our behalf. This will allow us to focus on higher-level decisions and strategies – the areas where Hogarth argues our energy should be spent – so as to better make strategic decisions.

With Project Glass, Google has developed augmented reality using hi-tech goggles that provide situational awareness. The wearer can recognise places around him and obtain information about those places, people and objects. This has been the substance of sci-fi for years. However, once the technology is finally commoditised, it will provide another example of how we rely on machines to make decisions for us and to channel the right information our way.

In the era of big data, we cannot hope to cope with all that information on our own. Processing power is just the beginning. Then comes the process of distillation. In our lifetime, we will be able to rely on machines to distil the data we need – and transform relevant information into insight at the right time and in the right context.

The data deluge will not put us under water. We will learn to surf the wave. Technology will help us beat big data. It will help us to survive and thrive.

By Sean Paine, COO of information solutions specialist, EnterpriseWorx