Plumbing the Depths of Unstructured Data

Turning to organizing measures like collaboration, ‘microdata’ and application of standards can help manage a deep source of information.

Michael Shashoua, editor, Inside Reference Data

Can collaboration and sharing of data, within the context of adhering to a standard when devising data governance plans, help solve issues with handling unstructured data? This question arises when hearing how firms try to build data foundations to produce higher quality data analytics.

Data governance and analytics can be improved by building mechanisms to share data, says Leif Hanlen, a business development executive at Data61, a data requirements organization that is part of CSIRO, a digital development agency backed by the federal and state governments of Australia. 

Yet, however advanced a data sharing system is, the rules that data vendors and exchanges have about their data make it challenging to join such disparate sets together, says John Denheen of Tyler Capital in London. “One of the big issues with getting good reference data is that you need to be able to tack it on to other data sets,” he says.

Before compatible standards can be achieved, firms have to identify the business practices that differ, says Sydney Hassal, a director at Scotiabank. Along with this, standards can vary among jurisdictions, observes Allie Harris of Bank of Montreal. 

The US Office of Financial Research (OFR) points to “microdata,” which OFR chief data officer Cornelius Crowley defines as granular, targeted and specific information about entities, instruments, transactions or products. He advises that breaking down standards efforts to individual data elements like those he calls microdata can increase accuracy. In this context, one wonders whether unstructured data can be organized with a “microdata” approach—and what that can mean for following standards.

Harnessing unstructured data could be most useful to support consistent application of data standards.

Analytics Goldmine

Unstructured data is emerging as a “goldmine” for the application of data analytics, CSIRO’s Hanlen says, although it is nothing new—it is akin to “50 million Word documents saved over the last 10 years and put in a box.” 

Firms should now understand the two types of unstructured data, says Hanlen. “Unstructured data sitting inside the enterprise—in the customer relationship-management system, in fields called ‘other’—is like a hole in the ground that’s yet to become a goldmine,” he says. External unstructured data from sources such as social media and market surveillance is already being tapped for its golden value, he says.

Analytics can come into play for the former type of unstructured data. “The task for analytics of unstructured data is not to build a brand new goldmine, but to extract elements of information from that unstructured data,” he says.

CSIRO and Data61 have worked on analyzing payments, documents predicting market outcomes and price signals based on the likelihood of future events. The challenge Hanlen finds in working with such unstructured data of a predictive—rather than measuring—nature is placing that data in a credible spectrum, which can be used to anticipate the future or a likely outcome.

Unstructured data may be a goldmine, but it is one that industry experts are just figuring out how to tap, for a variety of purposes. In March, ISITC executive Jeff Zoller pointed to unstructured data about investment behavior and patterns, as well as industry analysts’ commentary, and social and economic behavior, potentially supporting the disruption of data-management technology. In May, MUFG Canada Branch chief information and operations officer Ron Lee, speaking about how his firm is consolidating systems, noted this effort would increase the capability to work with unstructured databases.

Before any of these more aspirational purposes, however, harnessing unstructured data could be most useful to support the consistent application of data standards. The legal entity identifier (LEI) is ripe for this, as Karla McKenna, head of standards at the Global Legal Entity Identifier Foundation, observed in June, saying entity legal form information—a data element in the LEI repository—remains in “unstructured form.” 

Only users who have a paid subscription or are part of a corporate subscription are able to print or copy content.

To access these options, along with all other subscription benefits, please contact or view our subscription options here:

You are currently unable to copy this content. Please contact to find out more.

Waters Wrap: Examining ASX’s CHESS do-over

The Australian exchange was the first exchange to be all-in on DLT—and the project failed. Anthony speaks with ASX’s Tim Whiteley to discuss the lessons learned and why he thinks the second attempt will succeed.

GenAI and HFT: A competitive edge?

Timing is critical in the capital markets. IBM researchers say genAI’s predictive abilities in combination with AI/ML models can translate into profitable HFT opportunities.

You need to sign in to use this feature. If you don’t have a WatersTechnology account, please register for a trial.

Sign in
You are currently on corporate access.

To use this feature you will need an individual account. If you have one already please sign in.

Sign in.

Alternatively you can request an individual account here