The Next Big Data Debate Emerges

The ongoing discussion about big data, which continued last week in Waters' Big Data Webcast, appears to be turning away from a debate between using cloud computing or the Hadoop standard to a concern with rapidly increasing volume and velocity of data creating a need for greater use of big data systems.

An unspoken context underlying the webcast discussion, which had participants from Credit Suisse, BNY Mellon, Intel, IBM's Platform Computing and Sybase, is that the industry already seems to be leaning or moving away from Hadoop and toward cloud as being more effective for handling big data.

"The cost per gigabyte of storing that transaction over time is pushing us into cheaper, non-SQL, big data-type solutions," said Ed Dabagian-Paul, a vice president at Credit Suisse who works on setting strategy and direction for technology infrastructure at the firm. "The traditional big data solutions haven't mapped to our problems. We can answer most of our existing problems with existing data analytics or very large databases."

Daryan Dehghanpisheh, global director of the financial services segment at Intel, identified "volume, variety, value and velocity" as the four pillars of big data. He had already noted volume, and processing speed and time as key areas for big data when speaking with us in November.

Intel works with partners to produce solutions for operational issues such as big data. According to Dehghanpisheh, the company aims to achieve complex machine learning, statistical modeling and graphing of algorithms within big data, rather than the traditional business intelligence of query reporting and examining historical data trends. Orchestrating use of metadata and setting data usage policies are important parts of administering big data operations, he adds.

An extensible framework is needed to manage the volume and velocity at which big data now pours forth, as Dennis Smith, managing director of the advanced engineering group at BNY Mellon, sees it. "There are tremendous cost benefits to this from a scale standpoint and particularly looking at volume use cases," he said. Cloud computing inherently offers greater scale, of course, and analytics can be layered onto it or attached to it. As Smith also explains, Hadoop-related technologies, or standalone analytics infrastructures and traditional data warehouses as staging areas may all be ways to manage big data in tandem with cloud resources.

The question to ask, or the discussion to have, now, is how to marry big data, sourced from or processed through the cloud, with analytical systems that can derive actionable meaning from the data, for all its increased volume and velocity.