Cutting Big Data Down to Size
Editor's View
Last week, I wrote about a choice that could need to be made between cloud resources and the Hadoop tool for managing and working with big data. A better question, or a better way to frame the debate, could be as a decision about what is the best way to make use of cloud computing for data management, especially for "big data."
Tim Vogel, a veteran of data management projects for several major investment firms and service providers on Wall Street, has focused views on this subject. He advises that the cloud is best used for the most immediate real-time data and analytics. As an example, Vogel says the cloud would be an appropriate resource if one was concerned with the most recent five minutes of data under volume-weighted average pricing (VWAP).
"The cloud isn't cheap," he says. "Its best use is not for data on a security that hasn't traded in two weeks. Unless the objective is to cover the complete global universe, like [agency broker and trading technology provider] ITG does." Vogel points to intra-day pricing and intra-day analytics as tasks that could be enhanced, accelerated or otherwise improved upon through use of cloud computing resources. Data managers should think of securities data in two layers—a descriptive or identification layer and a pricing layer—both of which have to be processed and filtered to generate usable data that goes into cloud resources.
The active universe of securities as a whole, which includes fundamental data and analytics on securities, is really a super-set of what firms are trying to handle in terms of data on a daily basis, observes Vogel. With that in mind, the task for applying cloud computing to big data could actually be making big data smaller, or breaking it down into parts—cutting it down to size. That certainly will cut down on the bandwidth needed to send and retrieve data to and from the cloud, and consistently reconcile local data and cloud-stored data.
If nothing else, this is certainly a different way of looking at handling big data. It is worth considering whether going against the conventional or prevailing wisdom could lead data managers to a better way. Inside Reference Data would like to know what you think about this. We've reactivated our LinkedIn discussion group, where you can keep up with new stories being posted online, live tweets covering conference discussions, and provide feedback to questions and opinion pieces from IRD.
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
To access these options, along with all other subscription benefits, please contact info@waterstechnology.com or view our subscription options here: http://subscriptions.waterstechnology.com/subscribe
You are currently unable to print this content. Please contact info@waterstechnology.com to find out more.
You are currently unable to copy this content. Please contact info@waterstechnology.com to find out more.
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. Printing this content is for the sole use of the Authorised User (named subscriber), as outlined in our terms and conditions - https://www.infopro-insight.com/terms-conditions/insight-subscriptions/
If you would like to purchase additional rights please email info@waterstechnology.com
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. Copying this content is for the sole use of the Authorised User (named subscriber), as outlined in our terms and conditions - https://www.infopro-insight.com/terms-conditions/insight-subscriptions/
If you would like to purchase additional rights please email info@waterstechnology.com
More on Data Management
Nasdaq reshuffles tech divisions post-Adenza
Adenza is now fully integrated into the exchange operator’s ecosystem, bringing opportunities for new business and a fresh perspective on how fintech fits into its strategy.
This Week: HKEx's new derivatives platform; GoldenSource; Quonian-SimCorp, and more
A summary of some of the latest financial technology news.
Chris Edmonds takes the reins at ICE Fixed Income and Data Services
Edmonds is now leading ICE’s fixed income and data business as the rush to provide better data and analytics in fixed income builds.
Deutsche Börse democratizes data with Marketplace offering
Deutsche Börse Group is set to unveil its Marketplace, a one-stop data shop designed to simplify and streamline data acquisition and consumption for its clients, while also surfacing data from across the firm to its own users. Jan Stiebing and Sven…
DSB says industry is ready to meet UPI mandate ahead of deadline
The Unique Product Identifier will be required for certain OTC derivatives in the EU at the end of April, following US adoption in January.
Mapping a successful data journey: strategy, execution and sustainability
A well-planned data journey can positively impact an organization’s long-term trajectory. However, it is important to have clarity not only in the strategy but also in successful execution and sustainability for the long haul, argues data veteran Subbiah Subramanian.
The IMD Wrap: The growing data catalogue space
With their potential to manage costs and surface strategic datasets, it’s no wonder Max gets excited about data catalogs. This week, he takes a look at a new startup entering the space.
LSEG to sunset Redi EMS in favor of Tora
Sources say competitors will look to seize on the decision to win over Redi’s sizeable US client base.
Most read
- Chris Edmonds takes the reins at ICE Fixed Income and Data Services
- Deutsche Börse democratizes data with Marketplace offering
- Waters Wavelength Podcast: Broadridge’s Joseph Lo on GPTs