Anthony Malakian: The Swimmers in the Lake

The arms race for programmer talent has been a long-discussed topic in Waters, and there’s no reason to think it won’t continue to be a challenge for the foreseeable future—at least until our robot overlords finally wrest control of civilization from our hands.
While at the Buy-Side Technology North American Summit, held in mid-October in Manhattan, one panel discussed the merits of deploying data lakes versus data warehouses. The debate was interesting and led me to write a feature on the subject. It also got me thinking about the talent challenge, but more on that in a minute.
Two of the panelists—one from MetLife, the other representing JPMorgan Asset Management—were fairly evangelical on the subject of data lakes. The funny thing is, however, that their definition of a data lake would, I imagine, differ from some hardcore programmers. JPMorgan’s Scott Burleigh, in fact, said that his firm governs the data before it enters the lake, which I think many would argue makes the lake more of a warehouse than a pure lake.
But I also think that this is a natural progression of new technologies and strategies: The definition is morphed, stretched and molded to fit a particular industry. The capital markets space is likely to have a different view of data lakes than that adopted by music streaming services, in the same way that Wall Street had a different definition of the cloud compared with Silicon Valley where the technology first emerged.
Limited Skillset
I was speaking with Gartner’s Nick Heudecker on this topic, who has written a research paper on data lakes. He has also noticed that the term data lake is being thrown around to describe a lot of things. It’s even been used interchangeably with Hadoop. The key, he says, is to look at the underlying architecture if you really want to see what it is.
You’ll need innovative programmers sitting alongside the data scientists and quants to help extrapolate value from these lakes.
“If I’m building a very detailed semantics layer and add security in the way that I would do in my data warehouse, and I’m optimizing data for storage or reuse, is it really a data lake, or is it a different implementation of a data warehouse architecture? People confuse the physical implementation of the data warehouse with the architectural considerations,” he said. “And that’s fine; call it what you want. But I want to know about the architecture and from there we can figure out what characteristics you’re going to need.”
The real question is, who is going to build that architecture? As Dan Graham of Teradata told me, a data lake is “limited to what a programmer can do. It’s up to a programmer to do a lot of do-it-yourself work in the data lake.”
At some point, this raw, unstructured data is going to have to be cleaned and used, otherwise it’s just a data dumping ground without a purpose. And the cleaning part might not be as easy when it’s on such a massive scale.
“All of these data cleansing techniques are a foreign idea and a new area of exploration for the Hadoop data lake people,” Graham noted.
The problem for capital markets firms is they don’t want to have to hire separate cleansing specialists to go along with their Hadoop specialists.
Data Needs Programmers
I’m tired of the term “big data.” It’s too all-encompassing and vague. But as the volume of data grows at exponential rates—according to IBM, 2.5 quintillion bytes of data is created daily—a term needs to be created and accepted, so for the time being let’s just go with big data.
If firms want to find purpose and value in that sea of information, data lakes will prove to be a starting point. It’s like oil refining: you collect it, separate it, convert it, and treat it. You want to collect as much as possible, but the art is in the downstream stages.
You’ll need innovative programmers sitting alongside the data scientists and quants to help extrapolate value from these lakes. So, if you think the talent race was going to slow down anytime soon, you’re wrong.
Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.
To access these options, along with all other subscription benefits, please contact info@waterstechnology.com or view our subscription options here: http://subscriptions.waterstechnology.com/subscribe
You are currently unable to print this content. Please contact info@waterstechnology.com to find out more.
You are currently unable to copy this content. Please contact info@waterstechnology.com to find out more.
Copyright Infopro Digital Limited. All rights reserved.
As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (point 2.4), printing is limited to a single copy.
If you would like to purchase additional rights please email info@waterstechnology.com
Copyright Infopro Digital Limited. All rights reserved.
You may share this content using our article tools. As outlined in our terms and conditions, https://www.infopro-digital.com/terms-and-conditions/subscriptions/ (clause 2.4), an Authorised User may only make one copy of the materials for their own personal use. You must also comply with the restrictions in clause 2.5.
If you would like to purchase additional rights please email info@waterstechnology.com
More on Emerging Technologies
EU banks want the cloud closer to home amid tariff wars
Fears over US executive orders have prompted new approaches to critical third-party risk management.
Growing pains: Why good data and fortitude are crucial for banks’ tech projects
The IMD Wrap: Max examines recent WatersTechnology deep dives into long-term technology projects at several firms and the role data plays in those efforts.
Waters Wavelength Ep. 317: Bitdefender and Transilvania Quantum
This week, Bitdefender’s Adrian Coleșa and Transilvania Quantum’s Sorin Boloș join to discuss security vulnerabilities in quantum computing.
Investing in the invisible, ING plots a tech renaissance
Voice of the CTO: Less than a year in the job, Daniele Tonella delves into ING’s global data platform, gives his thoughts on the future of Agile development, and talks about the importance of “invisible controls” for tech development.
Evalueserve tames GenAI to boost client’s cyber underwriting
Firm’s insurance client adopts machine learning to interrogate risk posed by hackers
Waters Wavelength Ep. 316: Finbourne Technology’s Toby Glaysher
This week, Toby Glaysher, chairman at Finbourne Technology, joins the podcast to discuss the asset servicing industry.
State Street’s interop play for FX and easing technical debt
Waters Wrap: About six years ago, State Street partnered with Interop.io to tie together its GlobalLINK suite of platforms. Anthony explores how this plays into the “reuse” mantra.
As costs rise, buy-side CIOs urge caution on AI
Conference attendees encouraged asset managers to tread carefully when looking to deploy AI-driven solutions, citing high cost pressures.