Max Bowie: Do Crisis Closures Make Redundancy Redundant?

Aside from its tragic and destructive impact on the inhabitants of parts of New York and New Jersey, October’s Hurricane Sandy proved two key things for the financial markets: First, the business-continuity plans of datacenters where most marketplaces and trading firms host their trading engines and servers proved effective in keeping everyone’s systems running; and second, in spite of this, the markets still closed for two days because of old-fashioned physical issues that these centers were, in part, designed to eliminate.

The New York Stock Exchange initially planned to open for electronic trading as usual, anticipating that its physical floor would be closed following the storm—as indeed it was, re-opening after two days on generator power, though the floor was never flooded, contrary to some news reports—but changed its mind after discussions with other US exchanges and the Securities and Exchange Commission (SEC). Ultimately, all US equities markets remained closed for a full two days—an occurrence not experienced in decades. And while at time of writing, thousands of New Yorkers remain without power, the US financial markets could have largely continued unhindered.

According to Plan
This is testament to the integrity, security and policies of the datacenter facilities constructed by vendors such as Equinix, Telx and Savvis in recent years—veritable digital fortresses designed to withstand all manner of attacks, whether they be caused by humans or nature. Though new facilities tend to be built to take power from two power grids, most in the New York and New Jersey area experienced outages and had to run on generators during and after the storm. To ensure this could happen smoothly, datacenter operators tested generators beforehand, ensured fuel supplies, arranged hotel rooms for staff near facilities, even bringing cots and ready-to-eat meals into their datacenters, in some cases. This aspect of the markets worked fine, although it does raise the questions of whether so many critical markets should operate from the same datacenters—an outage at Equinix’s NY4 facility, which many markets have found offers the best connectivity to client firms and other venues, for example, could throw multiple markets into disarray—and what impact the changes in latency introduced by switching over to a backup location would have on firms’ trading strategies and routing decisions.

What failed, though, is that in spite of having increasingly electronic, automated and virtual markets, these still rely to a large extent on a legacy physical infrastructure overseeing them—an infrastructure of human administrators and operations staff, whose ability to do their job is limited by, say, flooding at their place of work, subways not running, or trees blocking roads. To a large extent, it seems that the market closures were a result of infrastructure supporting the people who manage the markets not being as resilient as the infrastructures that actually run them. Or perhaps, to look at this from another point of view, people didn’t trust the systems designed to run the markets to do their job without human oversight.

Limits
Some of these limitations cannot be overcome: Storms will always cause havoc. But other factors can be: Administrative and supervisory functions can be automated to the same extent as the markets they oversee, and can be performed as remotely and virtually as the infrastructures of the trading firms they monitor—even possibly by putting the logic to perform these functions on satellites to eliminate the risk of running these from a geographical location subject to disruption.

To a large extent, it seems that the market closures were a result of infrastructure supporting the people who manage the markets not being as resilient as the infrastructures that actually run the markets.

But the more systematic issue at stake has little to do with the systems that support the markets, and more to do with the system that governs them: Under what circumstances should all markets be forced to close, if the systems of some—or all—are able to keep running? After all, aren’t technology and reliability supposed to be differentiators? And what does it say about our confidence in these systems—or the system—if we feel the need to pull the plug when we can’t be in the room?