I FIRST PRESENTED THE IDEAS CONTAINED IN THIS article at an Institutional Investor conference in January. Before taking the podium I struggled to come up with something that would convey the importance of what was to follow. I decided to inform this audience of professional pension fund and endowment managers that their luck had run out:  They were going to learn about investing in risk, either from me or from someone else, but — not unlike having dinner with the in-laws — it was going to happen, so they would be well served to get it over with. The doors were locked, the audience lost hope, and we pushed forward.

For most of the audience, 2008 had been a shock, characterized by a serious loss of self-esteem and devastatingly poor performance that had been exacerbated by a fundamental lack of risk diversification. But such an outcome was not inevitable. In what follows I describe and substantiate how, through the use of a novel, rational allocation process, investors can achieve significant improvements in portfolio diversification, thereby potentially increasing the return and decreasing the risk of their portfolios.

Post-2008 most investment portfolios have remained one-trick ponies, effectively possessing a single source of variance, a typical proxy for risk. In the case of private investors, with a “standard” 60-40 stock-bond portfolio, roughly 97 percent of the portfolio’s variance can be explained by the equity allocation. Perhaps surprisingly, the typical U.S. pension fund doesn’t fare much better, despite a more broadly diversified asset allocation, full-time investment professionals and, in many cases, externally hired experts — variously referred to as “guys from out of town” or “guys with shiny shoes.” Accordingly, the typical U.S. pension fund has in excess of 90 percent of its variance explained by a single risk factor.

....