The Storage Tsunami

The industry is pushing software and hardware engineering to some pretty amazing heights around pre-trade risk checking and low-latency execution, but the coming year will be all about the middle and back office.

Several people at the Waters USA 2010 conference and the American Financial Technology Awards this week said that buy-side clients will demand transparency—for example, why brokers selected to route order to specific venues, the state of the market when the broker routed the orders, as well as tick-by-tick coverage of how the order is doing in the market.

Thus firms are seeing an explosion in data storage demands and it’s getting to a critical point, as regulators have not been keeping up in advances in storage technology. Much of the critical performance data is moving out of databases stored on spinning disks to memory caches replicated and managed across a cloud environment. When it comes time to back up the data to tape or optical disks, which versions of the cached data will be considered the “true” data? Or will it be a situation where every cache will need to be backed up individually?

When financial services first adopted cloud computing, one of the major savings was seen in the decrease in server sprawl where servers running at 95 percent CPU utilization could easily replace racks full of standalone servers that historically operated at 5 to 8 percent utilization. This free space, however, is quickly being filled by storage systems and this growth is only going to continue.

I can easily see a day in the not-too-distant future where the most of a firm’s computing is done on the outside of its firewall and data storage takes up the majority of what remains inside the firewall.