Opening Cross:‘Big Data’ Is Really About ‘Big Service’

Yet it isn’t new, per se: An entire class of traders exists who trade based on how the market reacts to news. Before electronic feeds, speculators and news barons used the telegraph, carrier pigeons and semaphore towers to communicate across distances as quickly as possible to take advantage of market-moving news.

Besides news about specific commodities or companies, traders realized that macroeconomic and geopolitical events could also impact markets in general and specific sub-sets of securities or derivatives. This became so competitive that in the US, announcements of market-moving government figures and reports are strictly controlled, governing the precise time that reporters can transmit stories from so-called “lockup” rooms.

Over time, the type of news that could impact a company’s price expanded, and hence so did the sources from which traders needed to capture data. The impact of weather on crop yields made weather data invaluable to commodities traders. A gaffe by a company exec in the society pages—or nowadays, a careless blog post or tweet—could lead to them getting fired or a public backlash, either of which could impact a company’s stock price. And so, not only did traders and aggregators need to capture and process unstructured data like news stories, but they also began to trawl the web for sources that others either hadn’t yet discovered or hadn’t learned how to decipher and understand meaningfully.

Other new types of data include analysis of public sentiment via Twitter as a leading indicator of price movement, and behavioral analysis of historical investor activity in response to price movements, such as that provided by vendors including Lucena Research, MarketPsych, and the now-defunct Titan Trading Analytics—itself a lesson that, as articulated by speakers at the Frankfurt Financial Information Summit, technical innovation must always target customer service.

Capturing, formatting, storing and retrieving—not to mention actually analyzing—this volume of data has led to the creation of a new cottage industry of high-performance database tools, while others still argue about the merits of investing in Big Data architectures, and whether they can deliver the results they promise for traders. And these naysayers may have a point: after all, the genesis of Big Data was geared towards better customer service.

For a fine example of an early use of Big Data, I suggest Setting the Table—part autobiography, part hospitality manual by restaurateur Danny Meyer, who describes how his staff constantly collects and records data about customers and their preferences, through feedback cards presented with the check, by engaging customers in conversation when they phone to make a reservation, or from their interactions with the wait staff. By knowing whether a diner is a repeat customer or a regular, what table they prefer, their taste in wine, and whether they had a good or bad experience on previous visits, the restaurant can tailor its service to provide the best experience for each diner—and make them a more loyal customer and maximize that relationship.

Data vendors do this for asset managers and advisors, collecting client information and linking it to relevant content, so users can keep track of their clients’ activity, as well as being able to quickly navigate to anything that would affect their portfolios.

Some would say that what we’ve come to call Big Data is really nothing more than big amounts of market data. But I see Big Data as the process of leveraging that data to its full potential and improving your business by understanding to whom existing inputs are most valuable.