On March 9, 2009, my partner Prof. Robert Jarrow and I wrote a blog entitled “The Rating Chernobyl,” predicting that rating agency errors during the credit crisis were so serious that even their own self assessments would show once and for all that legacy ratings are a hopelessly flawed measure of default risk. That 2009 prediction hasn’t come true for a very simple reason: the rating agencies have left their mistakes out of their self-assessments. This blog shows the dangers of relying on rating agency default rates in credit risk management, because the numbers are simply not credible.
My original post with Prof. Robert Jarrow entitled “The Ratings Chernobyl” was redistributed on www.riskcenter.com and by the Global Association of Risk Professionals. For a copy of that blog, please use this link:
http://www.kamakuraco.com/Blog/tabid/231/EntryId/11/The-Ratings-Chernobyl.aspx
In that blog, we predicted that the ratings errors during the credit crisis were so egregious that even the rating agencies’ own annual self-assessments of ratings performance would show that an analyst of credit risk should not place any serious reliance on legacy ratings. For an example of such a recent self-assessment, please see this example from Standard & Poor’s released in 2011:
http://www.standardandpoors.com/ratings/articles/en/us/?assetID=1245302234237
The other two major rating agencies release similar self-assessments. In today’s blog, in the interests of brevity, we use the S&P self-assessment. An analysis of the other two rating agencies would yield similar results.
In the wake of the financial crisis, rating agencies were forced to make available their ratings for public firms for the years 2007-2009. We use those disclosed ratings from Standard & Poor’s in what follows. Like the rating agency self-assessments, we analyze firms’ subsequent default rates based on their legacy ratings on January 1 of each year. If we use all of the data released by Standard & Poor’s, we have 6,664 observations for the three year period, using the January 1 rating in each case for each firm:
How many firms failed during this three year period among firms with legacy ratings? To answer that question, we first use the Kamakura Risk Information Services data base and then compare with results from Standard & Poor’s. The KRIS definition of failure includes the following cases:
- A D or uncured SD rating from a legacy rating agency
- An ISDA event of default
- A delisting for obvious financial reasons, such as failure to file financial statements, failure to pay exchange fees, or failure to maintain the minimum required stock price level
The KRIS data base very carefully distinguishes failure from rescues. Bear Stearns, for example, failed. After the failure, Bear Stearns was rescued by J.P. Morgan in return for massive U.S. government assistance that is documented in this blog:
van Deventer, Donald R. “Case Studies in Liquidity Risk: JPMorgan Chase, Bear Stearns and Washington Mutual,“ Kamakura blog, www.kamakuraco.com, July 8, 2011.
The Kamakura failure count includes all firms which would have failed without government assistance. In many of the government rescues, common shareholders and preferred shareholders would be amused to hear that the rescued firm “did not default” just because the senior debt holders were bailed out. As the Kamakura “Case Studies in Liquidity Risk” series makes clear, the government’s definition of “too big to fail” changes within 2 or 3 days of the failure of Lehman Brothers, so the prediction of a probability of rescue is a much more difficult task than the prediction of failure. Using the KRIS data base of failures in 2007-2009, we have 86 failed firms during this period. This count of failed firms is being revised upward by Kamakura in KRIS version 6 in light of recent government documents:
We can calculate the failure rate in each year and the weighted average 3 year failure rate by simply dividing the number of failures by the number of observations. We do that and display the results in graphical form from AAA to B- rating:
The results are consistent with Jarrow and van Deventer’s “The Ratings Chernobyl.” The 3 year weighted average failure rate for AAA rated firms was 4.55%, much higher than any other failure rate except B- ratings in the B to AAA range. By contract, BBB firms failed at only a 0.12% rate and BB+ firms failed at only a 0.25% rate. In tabular form the results can be summarized as follows:
By any statistical test over the B to AAA range, legacy ratings were nearly worthless in distinguishing strong firms from weak firms.
How does the Standard & Poor’s self-assessment present these results? It doesn’t. Here is what it shows in Table 24:
Note that the first year default rate for AAA rated companies is zero. What happened to FNMA and FHLMC? And what about the 2007-2009 failure rate of 1.19% for A+ weighted companies? Why is the A rated first year default rate so small in light of that? The reason is simple. A large number of important failed companies have been omitted from the self-assessment, avoiding a very embarrassing self-assessment.
Which firms did Kamakura have listed as failures in the ratings grades from A to AAA?
Kamakura included these 11 firms:
|
AAA
|
FANNIE MAE
|
|
AAA
|
FEDERAL HOME LOAN MORTG CORP
|
|
AA
|
AMERICAN INTERNATIONAL GROUP
|
|
AA-
|
WACHOVIA CORP
|
|
A+
|
NORTHERN ROCK PLC
|
|
A+
|
MERRILL LYNCH & CO INC
|
|
A+
|
LEHMAN BROTHERS HOLDINGS INC
|
|
A+
|
AGEAS SA/NV
|
|
A
|
BEAR STEARNS COMPANIES INC
|
|
A-
|
WASHINGTON MUTUAL INC
|
|
A-
|
ANGLO IRISH BANK CORP
|
Remember also that American International Group was rated AAA in January 2005. Which firms were omitted from the Standard & Poor’s list? Nine of the 11 failures tallied by Kamakura were omitted by S&P from their self-assessment.
FANNIE MAE (FNMA)
FEDERAL HOME LOAN MORTG CORP
AMERICAN INTERNATIONAL GROUP
WACHOVIA CORP
NORTHERN ROCK PLC
MERRILL LYNCH & CO INC
AGEAS SA/NV
BEAR STEARNS COMPANIES INC
ANGLO IRISH BANK CORP
For those who wish to debate whether or not FNMA and FHLMC were in fact failures, here are the auction protocols announced by ISDA:
http://www.dechert.com/library/FS_FRE_10-08_ISDA_Credit_Default_Swap_Auction_Protocols.pdf
When Standard & Poor’s was asked by a regional bank in the United States why FNMA and FHLMC had been omitted from the corporate ratings self-assessment, the response from Standard & Poor’s did not discuss the nuances of a rescue of a failed firm. Instead, the e-mail reply stated only that FNMA and FHLMC were no longer listed private firms and therefore they were no longer an object of this study.
Conclusions
A firm selling credit assessment technology, including Kamakura Corporation, has a vested interest in presenting their accuracy in the best possible light. To avoid the obvious conflict of interest in preparing a self-assessment, Kamakura has offered its clients and regulatory bodies the right of audit using Kamakura default data on the Kamakura site subject only to a confidentiality agreement. These audits have been performed and confirmed the accuracy of Kamakura’s self-assessments.
Rating agency historical default rates are biased as shown above because 9 of 11 firms rated A or above listed as failures by Kamakura were omitted from the results reported by Standard & Poor’s. A complete third-party audit of the entire default history is mandatory before historical rating agency default rates can relied upon with confidence.
Donald R. van Deventer
Kamakura Corporation
Honolulu, January 31, 2012