Advances in Risk Management: Glass Boxes, Black Boxes, CDOs and Grocery Lists

On November 22, 2008, the New York Times used this famous quote to describe how Citigroup got in trouble: “Chuck Prince going down to the corporate investment bank in late 2002 was the start of that process,” a former Citigroup executive said of the bank’s big C.D.O. push. “Chuck was totally new to the job. He didn’t know a C.D.O. from a grocery list, so he looked for someone for advice and support. That person was Rubin. And Rubin had always been an advocate of being more aggressive in the capital markets arena. He would say, ‘You have to take more risk if you want to earn more.’ ”

For the full text of the story, use this link:
http://dealbook.blogs.nytimes.com/2008/11/24/citi-saw-little-risk-as-it-made-bolder-bets/?scp=1&sq=chuck%20prince%20grocery%20list&st=cse

For those unfamiliar with grocery lists (such as Mr. Prince and a former President of the United States), they generally look something like this:

A collateralized debt obligation, by contrast, looks completely different.  Here’s an example from a 2003 deal underwritten by Merrill Lynch:

The funny quote from the New York Times might be a little harsh, because it’s literally impossible for the chief executive officer of a huge organization like Citigroup to understand every single aspect of the firm’s business in great detail.  Instead, Mr. Prince (pictured below), Mr. Rubin, and the other directors of Citigroup have to rely on both the line business experts and the risk teams watching them to insure that the institution, at least, knows the difference between a grocery list and a CDO, even if the CEO doesn’t understand the distinction.

Once we get to the front lines of risk management and new business generation, however, there should never be the case where the firm’s experts don’t recognize the difference between a black box like this

and a glass box whose appearance is as different from a black box as a CDO is from a grocery list:

Unfortunately there is a dizzying array of examples in the financial services and risk management business where business experts, risk managers, or both are relying on black box models or methods for their business decisions.  For more than 20 years, there has been strong regulatory pressure for financial institutions to move instead to a “glass box” modeling approach.  The passion of regulators in this regard has been building.  Even in February 1999 at the Annual Capital Markets Seminar of the Office of the Comptroller of the Currency, there was an amazing exchange at the St. Louis conference venue.  After a particular interest rate risk vendor presented to the assembled OCC risk experts, Nick Kiritz, one of the most talented people at the OCC at the time, stood up and said something like this: “Sir, I just want to put you on warning that the black box approach your firm takes is unacceptable to us.  If we go into a bank on an audit and fail to get complete answers to our questions about how your risk system works, that’s going to have an impact on our assessment of the bank.”  The vendor’s representative stammered for a second, and then only semi-facetiously gave that legendary challenge often heard on primary school playgrounds, “I want to meet you outside.”  That would have been an interesting follow-up because it turned out that both Nick and the vendor were black belts in karate.

Nick Kiritz’s quote illustrates an incredibly simple but powerful message: if you don’t know how a model works and you have no way of finding out, how can you as a risk manager and the institution itself certify its own safety and soundness?  Similarly, how can the CEO certify to the board that the institution has achieved best practice in risk management?  And how can the board make the same representation to shareholders?  And how can regulators then make a similar assessment on behalf of the taxpayers who suffer the consequences when things go wrong?

The inability to be sure of the institution’s safety and soundness is obvious, and yet look at this long list of circumstances where many of the world’s leading institutions have relied on black boxes for risk management:

• Ratings: The process by which ratings are awarded is both completely opaque and makes it impossible to understand how movements in macro-economic factors drive default probabilities
• Effective maturity of ratings: The rating agencies have never been able to articulate a simple fact: what is the maturity or term of the rating?  Only a fool would buy a bond without knowing the maturity of the bond, and yet ratings are in wide use with an unknown maturity
• Default probabilities associated with ratings: The default probabilities associated with ratings are unknown for two reasons.  First, even for a given rating grade like CCC, the annual default rate has varied from 0 to 44% if you accept the rating agencies’ self assessments of corporate ratings performance as correct.  Second, those self assessments are not correct and need an outside audit.  We noted that the ISDA-defined failures of FNMA and FHLMC were omitted from the default rates calculated by both Moody’s and Standard & Poor’s, and one of the world’s largest banks has suggested that Lehman and Bear Stearns’ collapse were also omitted
• “Merton” default probability formulas in wide commercial use have never been disclosed or subjected to third party accuracy verifications.  Merton default probabilities in wide commercial application have been notorious for the unwillingness of the vendor to disclose the formula by which they are generated.  Similarly, the vendor has not provided any auditable tests of the model’s accuracy.  The former head of North American sales for the vendor once joked that the former owners of the vendor didn’t disclose the models “because they didn’t want to” and that the current owners of the vendor didn’t disclose the models “because they didn’t understand them.”  One of the ironies of the use of the model is that use has persisted after Robert Jarrow and I published a study in 2004 that showed that Merton theoretical default probabilities, no matter how they are monotonically mapped to actual default rates, are less accurate than using the absolute level of the stock price as a “naïve” credit model.  One can get this naïve model free with a copy of the Wall Street Journal, Nikkei, or Financial Times. For a copy of that study, please contact us at info@kamakuraco.com.
• Risk vendors: The vendor that Nick Kiritz challenged in 1999, along with many others, routinely fails to allow clients sufficient access to model formulas and coding to do the normal model audits that good corporate governance demands, as we noted in our June 12, 2009 blog post “Risk Management Model Validation: Checklist and Procedures.”

In spite of Nick Kiritz’s spirited attempts to change this while he was at the OCC, regulators around the world have, until recently, tended to sweep these serious breaches of best practice in corporate governance and risk management under the rug.

How have risk managers justified their use of black boxes for so long? A partial list of excuses and the obvious retorts to them are listed below:

“Everybody else is using the same black box.”  Retort: Everyone with Jim Jones in Guyana drank the cool-aid. Almost everyone in the CDO market thought the copula model would work.  Fads in finance have been documented for hundreds of years by Charles MacKay and many others!

“The model must work—it has an amazing intuitive appeal.”  The graveyards of Wall Street are littered with the remains of those who believed in models that didn’t work.  This September 12, 2005 article in the Wall Street Journal documents one of them:

“Senior management is used to ratings and they just don’t understand the alternatives.”  One of the refreshing things about the current credit crisis is that ranks of chief executive officers are being purged of those who didn’t understand fundamental aspects of their business.  Boards are being upgraded, and so are the risk management efforts of firms world-wide.  This Darwinian process will take a while.

“Our staff in risk management is so small that we couldn’t audit the model even if the vendor were willing to make disclosure.”  The obvious retort is that the institution should then restrict itself only to buying three month Treasury bills.  A more thoughtful suggestion is that the firm should outsource the model audit or merge with a larger firm that has the resources to meet normal corporate governance standards.

The excuses that one hears are almost all variations on these themes.  How can institutions move to a glass box, fully transparent and auditable risk management infrastructure?  The steps are simple:

Documentation.  With respect to all internal and external models where disclosure is properly made, one needs to carefully document the model performance audit on a regular basis and make this detailed documentation available “on demand” to those who need to review it to insure best practice in risk management and corporate governance.

Replace black box vendors.  If a vendor refuses to make sufficient disclosure (all formulas, coefficients, and model test results), replace them as soon as possible with a vendor that does make such disclosure.

Change vendor behavior.  Refuse to purchase “black box” risk technology and make it clear to the vendor that lack of disclosure is the reason.

Abandon techniques, like ratings, that are opaque by their very nature.  We often hear “If it isn’t broke, don’t fix it.”  A corollary is that “If it is broke and can’t be fixed, throw it away.”  See our blog post on May 12, 2009 entitled “A ‘Rating Neutral’ Investment Policy” on how to do this.

Educate management.  Educating a management team is essential to deserving the title of “Risk Manager of the Year” as Bennett Golub, chief risk officer of BlackRock, argued in an exchange in our blogs.  Sometimes, that’s impossible to do successfully.  In that unfortunate case, Ben and I agree that resigning is the best course of action in the long run.  Joseph Tibman’s “The Murder of Lehman Brothers” relates how the chief risk officer of Lehman was named “Risk Manager of the Year” in 2006.  That honor now lies in tatters, however, after the CRO didn’t resign after being marginalized by Dick Fuld.

We believe the “glass box” era of risk management is here now as a correct and logical outcome of what we have all lived through the last two years.

Donald R. van Deventer
Kamakura Corporation
Honolulu, October 15, 2009