Anthony Malakian: The Swimmers in the Lake
The arms race for programmer talent has been a long-discussed topic in Waters, and there’s no reason to think it won’t continue to be a challenge for the foreseeable future—at least until our robot overlords finally wrest control of civilization from our hands.
While at the Buy-Side Technology North American Summit, held in mid-October in Manhattan, one panel discussed the merits of deploying data lakes versus data warehouses. The debate was interesting and led me to write a feature on the subject. It also got me thinking about the talent challenge, but more on that in a minute.
Two of the panelists—one from MetLife, the other representing JPMorgan Asset Management—were fairly evangelical on the subject of data lakes. The funny thing is, however, that their definition of a data lake would, I imagine, differ from some hardcore programmers. JPMorgan’s Scott Burleigh, in fact, said that his firm governs the data before it enters the lake, which I think many would argue makes the lake more of a warehouse than a pure lake.
But I also think that this is a natural progression of new technologies and strategies: The definition is morphed, stretched and molded to fit a particular industry. The capital markets space is likely to have a different view of data lakes than that adopted by music streaming services, in the same way that Wall Street had a different definition of the cloud compared with Silicon Valley where the technology first emerged.
Limited Skillset
I was speaking with Gartner’s Nick Heudecker on this topic, who has written a research paper on data lakes. He has also noticed that the term data lake is being thrown around to describe a lot of things. It’s even been used interchangeably with Hadoop. The key, he says, is to look at the underlying architecture if you really want to see what it is.
You’ll need innovative programmers sitting alongside the data scientists and quants to help extrapolate value from these lakes.
“If I’m building a very detailed semantics layer and add security in the way that I would do in my data warehouse, and I’m optimizing data for storage or reuse, is it really a data lake, or is it a different implementation of a data warehouse architecture? People confuse the physical implementation of the data warehouse with the architectural considerations,” he said. “And that’s fine; call it what you want. But I want to know about the architecture and from there we can figure out what characteristics you’re going to need.”
The real question is, who is going to build that architecture? As Dan Graham of Teradata told me, a data lake is “limited to what a programmer can do. It’s up to a programmer to do a lot of do-it-yourself work in the data lake.”
At some point, this raw, unstructured data is going to have to be cleaned and used, otherwise it’s just a data dumping ground without a purpose. And the cleaning part might not be as easy when it’s on such a massive scale.
“All of these data cleansing techniques are a foreign idea and a new area of exploration for the Hadoop data lake people,” Graham noted.
The problem for capital markets firms is they don’t want to have to hire separate cleansing specialists to go along with their Hadoop specialists.
Data Needs Programmers
I’m tired of the term “big data.” It’s too all-encompassing and vague. But as the volume of data grows at exponential rates—according to IBM, 2.5 quintillion bytes of data is created daily—a term needs to be created and accepted, so for the time being let’s just go with big data.
If firms want to find purpose and value in that sea of information, data lakes will prove to be a starting point. It’s like oil refining: you collect it, separate it, convert it, and treat it. You want to collect as much as possible, but the art is in the downstream stages.
You’ll need innovative programmers sitting alongside the data scientists and quants to help extrapolate value from these lakes. So, if you think the talent race was going to slow down anytime soon, you’re wrong.
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
To access these options, along with all other subscription benefits, please contact info@waterstechnology.com or view our subscription options here: http://subscriptions.waterstechnology.com/subscribe
You are currently unable to print this content. Please contact info@waterstechnology.com to find out more.
You are currently unable to copy this content. Please contact info@waterstechnology.com to find out more.
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. Printing this content is for the sole use of the Authorised User (named subscriber), as outlined in our terms and conditions - https://www.infopro-insight.com/terms-conditions/insight-subscriptions/
If you would like to purchase additional rights please email info@waterstechnology.com
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. Copying this content is for the sole use of the Authorised User (named subscriber), as outlined in our terms and conditions - https://www.infopro-insight.com/terms-conditions/insight-subscriptions/
If you would like to purchase additional rights please email info@waterstechnology.com
More on Emerging Technologies
Ice moves to meet demand for greater cloud, AI capabilities
The exchange also outlined competitive advantages behind managing its data and cloud strategy internally during its Q1 earnings call on Thursday.
FactSet looks to build on portfolio commentary with AI
Its new solution will allow users to write attribution summaries more quickly and adds to its goal of further accelerating discoverability, automation, and innovation.
How Ally found the key to GenAI at the bottom of a teacup
Risk-and-tech chemistry—plus Microsoft’s flexibility—has seen the US lender leap from experiments to execution.
The IMD Wrap: Beginning of the end for data audits?
This week, there’s exciting news for data bean-counters in the form of a partnership between two vendors that could change the way we view and track data usage and audits.
S&P debuts Spark Assist genAI copilot, draws up ‘Blueprints’ of combined datasets
S&P’s Kensho subsidiary has rolled out new emerging tech products leveraging AI to explore and combine the vendor’s wealth of datasets to solve common use cases.
Nasdaq reshuffles tech divisions post-Adenza
Adenza is now fully integrated into the exchange operator’s ecosystem, bringing opportunities for new business and a fresh perspective on how fintech fits into its strategy.
Liquidnet sees electronic future for gray bond trading
TP Icap’s gray market bond trading unit has more than doubled transactions in the first quarter of 2024.
Verafin launches genAI copilot for fincrime investigators
Features include document summarization and improved research tools.
Most read
- Chris Edmonds takes the reins at ICE Fixed Income and Data Services
- DTCC urges affirmation focus ahead of T+1 move
- Women in Technology & Data Awards 2024: All the winners and why they won