When securities markets went on hiatus following the terrorist attacks of September 11, attention turned to the New York Stock Exchange and the Nasdaq Stock Market as proxies for the U.S. economy and its resiliency. But the executives in charge of the exchanges’ technology and operations - the NYSE’s Robert Britz and Nasdaq’s Steven Randich - remained out of sight.
NYSE chairman Richard Grasso and his Nasdaq counterpart Hardwick Simmons were their organizations’ very public and confident faces, while Britz and Randich toiled tirelessly in the back rooms, restoring communications lines and priming computers to handle the pent-up rush of orders when markets reopened on September 17. They and their systems came through with flying colors; the Big Board broke a volume record that first day, and both exchanges easily handled trading activity that has remained at levels well above those before September 11.
For Britz and Randich, that frantic recovery week provided only a hint of the rebuilding and reengineering challenges to follow and of how their worlds may change. Both fill the chief technology officer role, and as competitive as their exchanges are, they stand united in confronting the now much bigger issues of operational reliability and disaster planning.
Britz and Randich have something else in common: promotions. Effective in January, Britz will become the NYSE’s co-chief operating officer, retaining his current CTO duties. In October Randich became Nasdaq’s chief information officer, adding market operations, information security and the program management office to his technology development and data center responsibilities.
“What was previously unthinkable is now, unfortunately, thinkable, and you have to think about what you want to defend against in a much more robust way,” says Britz, 50, a 29-year NYSE veteran who has been group executive vice president of market operations, technology and information products since 1995.
Adds Randich, 38, who joined Nasdaq in 2000 after four years at the Chicago Stock Exchange, where he was chief information officer, “If there is one lesson to learn from all this, it is that disaster-recovery plans need to be even better than was demonstrated on September 11.”
The two technologists looked back at, and forward from, that fateful day in interviews with Institutional Investor contributing writer Marianne Sullivan. They responded separately to a similar set of questions.
Institutional Investor: How directly did your infrastructures get hit?
Robert Britz: Things were always normal at the New York Stock Exchange per se. Our trading facilities and data sites were untouched. Our people were unharmed. Our heavy infrastructure - power and water and steam - was fine. Even our telecommunications, because we have very diverse routing, was overwhelmingly untouched.
Steven Randich: Our systems were unhurt because they are not located in Lower Manhattan. [Nasdaq’s primary data center is in Trumbull, Connecticut. Its backup site is in Rockville, Maryland.] Fortunately, we have 20 WorldCom telecommunications hubs around the U.S. through which our market-maker subscribers connect to the Nasdaq network. Two of those hubs are in the New York metro area, in Midtown and in New Jersey. The problem was that a number of firms in Lower Manhattan lost all connectivity to the outside world. That meant a number of our market makers lost their primary connections and had to work throughout the 11th and the following days to reestablish connectivity, not just to the Nasdaq but to everybody, from their primary or backup sites.
Britz: This is a hub-and-spoke industry; destination markets like the NYSE are the hubs, and the broker-dealers are the spokes. The essence of the problem post- September 11 was that the spokes in many cases had substantial impairment of their ability to connect to our networks.
How did you cope with what for many market participants was a complete connectivity breakdown?
Britz: They and we jointly engaged in a series of communications workarounds, many of which are still in place and stable. Would I describe those lines of communication as being as robust as they were prior to September 11? Probably not, but the workarounds have worked splendidly. There is no question that everybody who needs to get to the NYSE point of sale is more than capable of doing that. The proof is that on the first day when trading resumed, there was record volume.
Randich: We did pretty well because we require that any market maker anywhere in the country have two ways of connecting into the Nasdaq network. That is with separate local phone carriers, each directed at a separate WorldCom hub, so that there is no single point of failure. All market makers in Lower Manhattan would have connected to New Jersey as well as Midtown, and they would have used separate carriers. That all worked well. Unfortunately, the damage was so bad in and around the World Trade Center that both carrier connections were lost, which in practice almost never happens.
As you came back, did you experience any capacity problems?
Britz: None whatsoever. We have such excess throughput in our order-processing and switching systems and excess bandwidth in our networks that the systems barely broke a sweat. We averaged about 20 percent utilization of systems even with the record numbers that first week.
Randich: We had no problems - and we have had some big days since September 17. We spend a lot of time predicting demand and ensuring, every week on Sundays, that our systems are capable of meeting that demand with a margin of comfort. We have had a pretty good record of keeping up with the record peaks that we have experienced in the past six months.
Given that common carrier telecommunications are beyond your control, what can you do to shore up the markets’ connectivity?
Randich: The carrier connections have to be more dispersed, and then the firms need to have the ability to quickly move to another site and come back up operationally. A number of market makers have a hot disaster-recovery site backing up their primary operations as a part of normal business. If they lose one site, they don’t have to reestablish operations at the other location. They just need to expand it to accommodate the lost functions at the disabled site. A number of firms have done that, and those are the ones that fared the best.
Britz: It’s one thing to say the NYSE is sufficiently hardened, both in terms of its facilities and networks and switching and order-processing systems. But it is somewhat of a hollow victory if the broker-dealers are unable to send us orders because they have lost connectivity to our main sites. So we are working on an industry contingency plan, which recognizes the extraordinary interdependence in this industry. We are only as strong as the weakest link.
What will be in the plan, and how does that compare with previous levels of contingency planning?
Britz: We formed a task force to identify areas where we have learned lessons and can do better. There was a false sense of security, and when you actually get down to the wires-and-pliers level, the kind of redundancy that people thought they had really was not there.
Randich: We have a plan that is very well defined, and we do at least a half dozen tests a year to re-prove it. Every time a message comes into Nasdaq in Connecticut - and we have as many as 3,000 per second - it is immediately, within a subsecond time frame, replicated in Maryland. Essentially, we have a high-bandwidth network with synchronous communication between the locations.
Nasdaq focuses a lot of effort on disaster planning and is going to focus even more and conduct more frequent tests. In one type of test, we pump transactions through the backup site in an effort to see where its breaking point is. That’s something we do every week in our primary site. We want to ensure that the backup site performs and functions at the same level as the primary site.
What is the likelihood of more redundancy - a second trading floor for the NYSE or a third data center for Nasdaq?
Britz: We are certainly thinking about and debating it. One justification would be that we could bring an active second site up sooner than other backup alternatives. But there are other issues to consider. If you are trying to defend against the loss of people, having more than one trading site, at least in theory, gives you some level of protection against that. But if you carry this to an extreme, terrorists who really wanted to take out the NYSE would presumably make an attempt on both sites. And if there were six trading sites, they might make an attempt on all six.
So you cannot defend against all possibilities. But it used to be that you did not have to plan for, or defend against, the unthinkable. When you realize two planes have hit the World Trade Center and taken it down, you begin to realize that nothing is unthinkable anymore.
Randich: We are seriously thinking about a third site. But don’t get me wrong; it is conceptual. It’s not like we’ve got this thing budgeted.
Did that thinking begin after September 11?
Randich: No, it started quite a while ago. Another thing we are thinking about is to eventually have all of our international and domestic systems run on a common platform. Much like we saw happen at many banks on September 11, our European and Asian operations could take over in an emergency, creating a continentally diverse disaster-recovery capability.
After the attacks, could the NYSE have reopened faster by moving to a backup site?
Britz: No. We had made a decision that we were not going to open that day, so there was no need to go elsewhere. But lack of access to our trading space was never an issue. We had alternative venues at our disposal and had the capability of bringing them up. If anyone needs proof of that, look at how we brought the American Stock Exchange up [on NYSE communications lines] overnight. Given that we were untouched, there was no need to move the trading facility, and frankly, moving the trading facility would not have cured the connectivity problem.
Does this experience strengthen the argument for a more electronic market infrastructure - the Nasdaq model as opposed to the NYSE’s centralized trading floor?
Britz: The NYSE has a trading floor because it gives the customers what they want, and we are not exempt from that basic fundamental of business. We provide a product - order execution - and like any other product over any period of time, it takes on the characteristics that its customers want it to have, or else it is not going to succeed. We give them so many different choices in terms of connectivity and delivery systems and order types and automated versus human intervention at the point of sale. For example, they use Dot, our automated order routing system, and 90 percent of our orders come in that way, accounting for about 52 percent of our volume. Day in and day out, they also choose to send 48 percent of our volume to a broker to represent their interest at the point of sale.
Randich: We’ve always felt that our model, with geographically diverse data centers and a fault-tolerant network connecting decentralized market makers, is the optimal one. New York’s is very different, and there is a lot of tradition in it. I think that the NYSE’s focus will be more on backing up its trading floor than on changing its centralized auction approach. This debate has been going on for ten years and hasn’t changed. I can say that our model has attracted a lot of popular attention, particularly after the attacks, because that geographic diversity is very compelling. We are happy for that and are going to look to improve it further.
