When he’s not teaching and writing textbooks, Yale University finance professor Frank Fabozzi spends his working hours vetting academic papers. But the editor of the Journal of Portfolio Management also once had designs on becoming the next Robert Ludlum. Fabozzi dreamed of penning a thriller when he retires — about terrorists who plan to smash capitalism by simultaneously assassinating the risk managers of every major financial firm. "Now I’m not sure if that would be meaningful at all," he says in his raspy tenor. "I don’t know what they’d be destroying, given what we’ve seen happen to risk management systems."

What happened is that banks, asset managers and hedge funds screwed up royally in last fall’s crash. Fabozzi chastises them for letting star traders and portfolio managers run wild because those forceful types were making so much damn money. "I don’t care if you have a risk management system that was developed by Albert Einstein," he says. "You still need the ability to tell people who seek to violate the system, ‘No.’"

But even if they’re in play, Fabozzi says, some risk management systems fizzle when confronted with extreme events. Long before the worst financial crisis in 60 years pulverized global markets, he tried to sound the alarm. In 2005, Fabozzi co-authored a book about the dangers of fat tails — dramatic price changes that elude standard risk models. But nobody paid much attention, he says.

When disaster struck in 2008, senior executives tried to absolve themselves by labeling it a one-in-10,000-years occurrence, set off by an entirely unpredictable confluence of events. Yes, last year was unusually severe, but worldwide financial meltdowns aren’t some unicorn-like beast. In fact, they pop up every couple of years. That means the next unforeseen event — or "black swan," to use the term popularized by Nassim Taleb — could already be slouching its way toward the New York Stock Exchange.

If investment firms don’t want to get caught with their pants around their ankles yet again, they’d best rethink risk. Forcing them to take fewer chances with other people’s money is just one potential solution. Risk managers may also need better tools, especially if their mathematical models can’t tell when strategies are getting too crowded. In search of answers, some very smart people are designing new risk models that take human behavior — namely, fear and greed — into account. But to make these gizmos work, the eggheads want better data from banks and hedge funds — a wish that only the regulators can grant.

There’s no doubt that some commonly used risk-control methods are duds. Ask a risk professional what went wrong last year, and he’ll tell you that many CEOs and CFOs trusted a single number coughed out of a computer. Take value at risk — please. Also known as VaR, this ubiquitous metric aims to forecast how much a given portfolio might lose today, using recent history as a guide. We saw how well that worked out for American International Group and Lehman Brothers Holdings.

If human behavior is the problem, then the captains of risk may need special powers. It’s one thing to reject the Efficient Market Hypothesis — the cherished but increasingly kooky-sounding belief that investors are rational and prices tell you everything you need to know. Nor is it outrageous for lawmakers to gut the so-called bonus culture, which gives financial institutions a compelling reason to keep blowing themselves up. Still, it’s quite something else to build a risk apparatus that factors in the powerful emotions behind cataclysms like 2008.

But the latter is exactly what some cutting-edge academics and risk experts have set out to do. Universal physical laws may govern the movements of planets and protons, this group argues, but they aren’t much good at explaining the unpredictable ways of markets. Counterbalancing the powerful influence of physics on financial modeling, these risk pioneers have turned to such fields as cognitive science, complexity theory and evolutionary biology for answers.

Among them is Damian Handzy, chairman and CEO of Berkeley Heights, New Jersey–based risk consulting firm Investor Analytics. His beef with current risk management tools is that they explain how, but not why, markets tank. The former nuclear physicist wants to find root causes, the way geologists got the jump on earthquakes in the 1960s by mapping the planet’s tectonic plates. "We need a plate tectonics equivalent for the market," Handzy says.

Joining him on his quest is Massachusetts Institute of Technology economist Andrew Lo. In financial engineering circles, many roads lead back to Lo, a prolific writer and speaker who seems to be everywhere at once. The professor and hedge fund manager has long contended that markets are inefficient and ever evolving.

Out in the New Mexico desert, another Lo associate, J. Doyne Farmer, holds similar views. A physicist-turned-trader-turned-academic at the Santa Fe Institute, Farmer was an early adopter of chaos theory in the 1970s. Today he’s building simulated markets inhabited by synthetic buyers and sellers, in the hope that these agent-based models will show how systemic risk foments crises.

But even if such experiments yield practical results, science has its limits. At New York hedge fund firm D.E. Shaw & Co., chief risk officer Peter Bernard stresses the importance of judgment. Typically, significant losses result from numbers and exposures that never appear on a risk report, he warns: "In risk management — and we certainly do it here — one spends a lot of time imagining what could happen to the world and then thinking about what that would do to your portfolio."

In his tell-all 2006 book, Traders, Guns & Money: Knowns & Unknowns in the Dazzling World of Derivatives , Satyajit Das detailed the skullduggery that goes into making and selling financial products. The Sydney, Australia–based consultant is equally scathing about risk management. "If there’s one immutable law of the universe that holds, it’s that people have to take risks to make money, so why would you control risk?" he asks. "I always joke with people in banks that if they properly stress-tested and risk-managed their portfolios, they wouldn’t take their money out of Treasury bills."

Before becoming a derivatives and risk management specialist in 1994, Das worked for several banks, including Merrill Lynch & Co. He describes risk modeling as a fig leaf behind which senior managers hide. With sophisticated-looking systems in place to wow their clients, firms are free to roll the dice. When this gambling backfires, the CEO blames a rogue trader or an act of God.

"What I really dislike about this whole process is the underlying intellectual dishonesty," says Das. "I’d be very happy if people said, ‘We’re in this to rip off corporates, we’re in this to rip off regulators, we’re in this to rip off investors, and we’re going to do it with a lot of nice science.’"

Das says the profession of risk management didn’t exist when he began his career, in 1977. That’s because no one used to worry about risk — until two events conspired to make them care. The first was deregulation after the Bretton Woods fixed-exchange-rate system collapsed in 1971. As the ’70s wore on, companies became more exposed to financial ups and downs. Then in the early 1980s, the derivatives market found its stride. "The banks were trying to figure out how to use these instruments," Das says. "So it was almost like a strange coincidence of supply and demand."

Since then, a jungle of accounting and other infrastructure has grown up around the derivatives business, Das adds. Accounting standards are no match for modern financial instruments, so the rules get more and more esoteric.

Every seven to ten years, Das explains, a market meltdown sets off a risk crisis. In 1998 it was Long-Term Capital Management, the Greenwich, Connecticut–based hedge fund whose sudden demise threatened to topple the system. Because investors lose billions in these calamities, much hand-wringing ensues, and the regulators step in and change the rules. Then the cycle repeats itself.

The cause of the latest crisis is simple, according to Das. The banks needed higher returns because their investors demanded them, so they became adroit at using derivatives to create risk.

Financial institutions use an array of risk tools with a nomenclature that would make Dr. No blush. Besides VaR — hatched by New York–based J.P. Morgan & Co. in the late 1980s — there is the ironically named Monte Carlo simulation. Risk managers use it to gauge VaR by churning out thousands of possible scenarios. Then there’s expected tail loss, or ETL. Billed as an improvement on VaR, ETL purportedly accounts for fat-tail risk. Another popular risk technique is stress-testing, whereby institutions expose their portfolios to shocks both historical and imagined.

Not everyone agrees that risk management is about preventing loss. Gregg Berman is head of risk business at New York consulting firm RiskMetrics Group, which was spun off from J.P. Morgan in 1998. He says managing risk means understanding where you might lose so you can place a bet. "The more returns you want, the bigger those losses have to be," Berman argues. "Are you losing money when you think you were supposed to and under the circumstances that you were supposed to?"

RiskMetrics provides tools to help clients understand how their portfolios will behave in different market conditions. Its offerings include a library of stress tests. In regular stress-testing a manager picks a real-life nightmare like the 1994 Mexican peso crisis and sees what it does to his portfolio. Reverse stress-testing does the opposite, Berman says. A manager decides how much it can stand to lose and sees what might cause that to happen — then makes a judgment call about whether such events will transpire.

In the aftermath of 2008, more clients have been building their own stress tests, using the RiskMetrics library as a guide, Berman says. "Everybody’s portfolio is different, and what you really need to do is create stress tests that are specific to your investment strategy, your time horizon and your portfolio."

AIG seems to have taken Berman’s advice to heart — at least the part about placing a bet. The insurance giant finished 2008 with the biggest quarterly loss of all time — a staggering $61.7 billion — mostly thanks to bad investments in mortgage-backed securities. Brought in to help clean up this toxic mess was New York–based BlackRock, which will become the world’s largest money manager after it completes its acquisition of Barclays Global Investors this quarter. Last December the Federal Reserve Bank of New York tapped BlackRock to manage some $21 billion in mortgage assets from AIG.

Bennett Golub, BlackRock’s chief risk officer, doesn’t blame risk management systems for what happened in 2008. Instead, he says, people chose to ignore what those systems told them. "I also think a lot of the professional risk managers understood the limitations of their models quite well," Golub adds. "Where they may have fallen down is maybe some of the risk managers needed to advocate and communicate more effectively than they did."

By all accounts, the Santa Fe Institute’s Farmer is an eclectic genius. Born in Houston, he grew up in Silver City, New Mexico. In 1981 he received his Ph.D. in physics from the University of California, Santa Cruz. As a graduate student, Farmer was one of the first academics to explore chaos theory.

Chaos theory has many forebears, among them the late MIT meteorologist Edward Lorenz. While developing computer simulations of weather patterns in the 1960s, Lorenz came up with the butterfly effect — the idea that an insect beating its wings could cause or prevent a tornado halfway across the world. In other words, tiny variations in a nonlinear system like the Earth’s atmosphere can lead to unpredictable and wildly divergent outcomes.

Farmer, 57, spent the 1980s working as a theoretical physicist at the Los Alamos National Laboratory, where he founded the Complex Systems Group. Often described as existing at the edge of chaos, a complex system creates its own order and adapts to changing conditions — with no leader or central command. Two examples are bee swarms and cities.

Farmer’s interest in chaos and complexity theory drew him to time-series forecasting, the practice of using mathematical models to predict events based on past ones. In 1991 he and fellow physicists Norman Packard and James McGill launched Prediction Company.As chief scientist of the Santa Fe–based quantitative trading group, Farmer read widely in finance. He says many of the ideas and techniques he encountered didn’t do much to improve his firm’s trading performance. "The real story of what’s making markets behave the way they are has a lot more in common with things in biology and ecology and evolution, and that’s just not reflected in the finance literature."

Seeing an opportunity, Farmer returned to the lab a decade ago. Today, his big interest is what he calls quantitative theories of social evolution. Markets are the ideal subject — a complex environment where people leave a record of their decisions.

Farmer and his colleagues have built a simple agent-based model for assessing systemic risk when firms are leveraging. In the 2008 crisis, he says, nobody considered how the various pieces of the system fit together. Combined with leverage, that oversight triggered a cascading failure when so many investors in the same strategy tried to sell at once — the so-called crowded-trade phenomenon.

Farmer’s simulation includes value investors, who buy underpriced assets; and noise traders, who have only some idea of the value of those assets. There are also banks that lend money to the funds so they can leverage their investments. Normally, the funds are buying when prices fall and selling when prices rise, thereby stabilizing the market. But when they’re fully leveraged and prices drop, the banks make margin calls. So the funds are forced to sell, which induces fat tails because it amplifies the price fluctuations.

"We actually get more extreme events, crashes become more likely, and a lot of other behavior looks like what you see in real markets," Farmer says.

Farmer believes agent-based models like his could have two uses. The first is for competitive advantage. If market participants understand how the system works, they might be able to predict its behavior to the point of making directional bets.

The second potential use is for risk management. Farmer thinks his simulation could show banks that they’re making margin calls at exactly the wrong times — and provoking crashes by doing so. "It may be that the solution is for regulators to change the rules about how this kind of stuff is done, so that you don’t end up having everybody selling at the same time," he adds.

Building risk models that truly reflect market behavior is an arduous task, says Edgar Peters. Co-director of global macro at Pasadena, California–based investment firm First Quadrant, Peters has applied chaos and complexity to finance since the late 1980s. He says most risk management is tied to Newtonian physics. It assumes that markets run like a Swiss timepiece, when they actually resemble the weather. Peters wants people to think of the marketplace as having seasons of high and low volatility. It needs complexity and uncertainty to create opportunity, and every so often, things tip over into chaos.

According to Peters, U.S. market cycles last, on average, about four years. A model developed using chaos theory is better than VaR at estimating risk because it uses ten entire cycles, he says: "To do a statistically significant risk analysis, you basically need 40 years’ worth of data."

But Peters says such models are unpopular because the math is so difficult, and because they don’t give pat answers. Instead, modeling a stable market environment that doesn’t really exist is all the rage. Peters hopes that will change. "It could be that after the tech bubble burst, and after Long-Term Capital Management and the credit crisis, everyone’s finally waking up to the fact that this is the way things really are," he says. "It’s not nice, it’s not clean, and it’s a mess. And dealing with it is going to be messy always."

MIT’s Andrew Lo had his epiphany in 1999, on a family vacation to Washington. Lo had been struggling with the realization that many of the financial models he’d learned in grad school didn’t jibe with actual market events. In Washington he visited the Smithsonian National Air and Space Museum and the National Zoo. At the zoo, Lo watched some caged chimpanzees fight with each other for territory while a group of children battled for a better view. "It occurred to me at that moment that we’re animals like the animals on the other side of that cage," recalls Lo, who earned his Ph.D. in economics from Harvard University at age 24.

He was also struck by the contrast between the physicist’s and the biologist’s interpretation of the world. "We have to recognize that financial markets are quite a bit more complex than the static physicist’s view that we’ve been working under for the last several years," says Lo, 49, Harris & Harris Group professor of finance at the MIT Sloan School of Management. "Once we start thinking about markets as an organic entity that changes and responds to many kinds of stimulus, then we will have a much more sophisticated view of how to measure risks."

Lo disagreed with the prevailing theory that markets are efficient. But if he was right, he wondered, why is it so hard to make money? His answer is the Adaptive Markets Hypothesis. "Markets are not efficient, but they’re highly competitive and highly adaptive," explains Lo, who co-wrote a 1999 paper about evolution and efficient markets with Farmer. "Unless you are able to have a competitive edge at a point in time, you will not be able to survive."

Convinced that the market is an evolving ecosystem, Lo wants to decode the behavior that drives it. The director of MIT’s Laboratory for Financial Engineering is trying to measure liquidity risk, which he says is probably the biggest issue with all financial crises. "The breakthrough is to try to understand how human behavior can trigger these types of events," he says. "Liquidity risk and unwind risk are manifestations of a flight to quality or a flight to safety."

Lo points out that fear is much stronger than greed — an imbalance that reveals itself when investors bolt for the exits. His former colleague Bernard of D.E. Shaw is a veteran of crowded trades. Bernard spent 15 years at J.P. Morgan before co-founding fixed-income arbitrage shop New Bond Trading in 1994. From 2000 to 2004 he was CFO of RiskMetrics. After a brief term as president of AlphaSimplex Group, Lo’s Cambridge, Massachusetts–based hedge fund firm, Bernard joined the now $28 billion-in-assets D.E. Shaw in 2006.

Back in 1998, New Bond got caught in the LTCM downdraft, along with D.E. Shaw. For Bernard, the experience was a visceral lesson. Although financial crises are tricky to forecast, he says, it’s crucial to recognize when a market mechanism is under stress: "It doesn’t necessarily tell us how the crisis is going to occur or what is going to precipitate it, but it does allow us to refine how we are managing specific exposures."

Over the next few years, Lo expects to see more crises spawned by crowded trades. He says part of the solution is to develop models that adapt to market dynamics. "Until we are able to improve our infrastructure — financial models as well as computer systems to handle financial transactions — we’re going to see more potential for dislocation as investors swing from one favored asset class to another and as they react to fear and greed in human ways."

Then again, not everyone believes in models. The chief dissenter is Nassim Taleb, bestselling author of The Black Swan: The Impact of the Highly Improbable . Taleb says the latest crisis vindicated his idea that it’s impossible to predict the probability of events. He explains that financial models flop out-of-sample — when they are fed data from outside the period they were built on.

Taleb, who is distinguished professor of risk engineering at Polytechnic Institute of New York University, urges investors and regulators to learn to live with uncertainty by building a black swan–proof world. His prescriptions include limiting the size of financial institutions and banning complex derivatives.

As far as Taleb is concerned, traders were better off before the Black-Scholes option-pricing model came into vogue, in the 1970s. "Models have interrupted the flow of apprenticeship, where an old-timer teaches a new apprentice heuristics," he says. "By trying to formalize things, they killed the whole learning process, which is why we blow up."

Having spent 21 years working with nonlinear dynamics as a trader, Taleb is wary of agent-based simulations because the smallest change in variables can lead to dramatically different outcomes. "You cannot do predictive things with it because degrees of freedom swell out of proportion," he says.

At the very least, models should be driven carefully. Vineer Bhansali is a managing director and the head of analytics for portfolio management at Newport Beach, California–based Pacific Investment Management Co. A physicist by training, he says that applying the mathematics of physics to finance means making assumptions that only hold true under ideal market conditions. The creator of a model knows this — and warns users. "But by the time the trader trades with it, he doesn’t care about that warning," Bhansali says. "I think that’s the biggest danger."

For Bhansali risk management is a way to generate excess returns over time by identifying attractive buying opportunities. Pimco, which managed some $940 billion in assets as of September, uses a technique called tail-risk hedging in many of its products, including the $186 billion Total Return Fund. An insurance policy against catastrophic losses, the strategy consists of macro hedges across markets, in everything from equities to interest rates to currencies.

But Bhansali says firms are reluctant to buy tail insurance, partly because it’s an admission that they can’t see the future. "For some reason, in investments we always believe we have a high degree of certainty in forecasting. We don’t."
Difficult problems have always appealed to Investor Analytics’ Damian Handzy. That’s why the New Jersey native earned a Ph.D. in nuclear physics from Michigan State University. In the early 1990s, Handzy found himself pushing particles at his alma mater’s National Superconducting Cyclotron Laboratory. His specialty: understanding the relationships between protons emitted in certain nuclear reactions.

Chasing bigger intellectual challenges, Handzy switched to finance. He spent four years as a Wall Street technology, management and risk consultant with Deloitte & Touche before co-founding Investor Analytics in 1999. "You often hear, ‘This isn’t rocket science,’" says the 41-year-old Handzy. "And I agree — it’s not. It’s harder."

The more Handzy learned about commonly used risk tools, the more questions he had. He asked himself why people cling to particular beliefs, even when the evidence tells them otherwise. This prompted him to study cognitive science and behavioral economics. He discovered that humans are lousy at understanding probability because that skill serves no evolutionary purpose. "This leads to some very, very bad things from a financial perspective, in terms of risk management," he says.

The other focus of Handzy’s study of risk is complexity, which he says ties in neatly with volatility. The traditional belief is that volatility exists because markets are "looking for the right price." When they find that price in an efficient market where information flows quickly, volatility will drop to zero. But Handzy points to the currency market, which trades in every country and is entirely electronic — but is also highly volatile.

Complexity comes at volatility from the opposite direction, says Handzy, whose firm recently developed a risk analytics software suite of tools with MIT’s Lo. A complex process, evolution needs volatility so that it can perform natural selection. Handzy says the same goes for the marketplace, which he regards as a complex adaptive system.

"Volatility is how markets try out new things, and this notion of equilibrium price is a complete fallacy," he explains. "The market doesn’t know whether oil should be $30 a barrel or $170 a barrel — and it changes its mind rather rapidly."

Calling math and physics a sound basis for risk management, Handzy thinks the best results will come from folding in cognitive, behavioral and complexity studies. In the search for signals of riskier markets, the goal is to understand why such changes happen, he contends: "If you don’t have that causality, you don’t have a chance of doing anything predictive."

Depending on whom you ask, though, blending various sciences together into new risk concoctions has limited value. D.E. Shaw’s Bernard enjoys reading about agent-based models and complexity, and he believes that financial algorithms have predictive powers — as long as somebody oversees them. But, Bernard says, "mathematical models inevitably rely on past events. And reliance on the past is not the way to protect yourself from future problems."

Still, the Santa Fe Institute’s Farmer is pressing ahead with his agent simulations of systemic risk. But creating a picture of what’s really happening hinges on securing better data — quarterly derivatives positions, for instance. "Even private players would have a significant incentive because it would help them do their risk management better," Farmer says. "It might even help them make better profits."

Lo wants firms to give risk officers more authority and link their compensation to corporate longevity and stability. If he has his way, everyone from banks to hedge funds will surrender information about their financial transactions, through the Federal Reserve Board or a national systemic risk regulator. "They can do it on a confidential, no-names, aggregated basis so that hedge funds are not giving up their precious secrets," Lo adds.

For Richard Bookstaber, former head of risk management at New York–based Salomon Brothers, the key is to find the components of risk that are timeless and universal. He says the only way to understand recurring liquidity crises is by gathering firm level, position and leverage data.

To make his point, Bookstaber compares the market to a darkened theater. For all anyone sitting there knows, the joint could be empty or packed to the gills. And looking at how people entered and exited it over the past year tells nothing about what they’ll do if a fire breaks out. "If that happens, it’s a different world," Bookstaber says. "And right now, we don’t model that world."

Asked how far away he and his fellow researchers are from explaining why markets do what they do, Handzy doesn’t risk naming a date. But he says it won’t happen unless they take a chance. "If it turns out that it doesn’t work, well, that’s reality," he adds. "But we know that the current idealizations do not work, so we need to look at something that holds more promise."