The Federal Reserve released the results of the Comprehensive Capital Analysis and Review stress tests on the afternoon of Thursday, March 5. In this note, we follow the examples of two very well-known papers. Andrew Haldane of the Bank of England, writing with Vasileios Madouros, asked whether the Basel capital ratios could outperform a benchmark statistical model in predicting default during the credit crisis in “ The Dog and the Frisbee.” The authors found that the Basel capital rules failed this model validation test.
Sreedhar T. Bharath and Tyler Shumway (“Forecasting Default with the Merton Distance to Default Model”) subjected the Merton model to a similar model validation exercise and found that it provided no incremental explanatory power to a well-specified reduced form.
In this note, we use trade-weighted average credit spreads as the basis for a model validation of the Fed’s CCAR 2015 results for 31 financial institutions. We use credit spreads for two reasons. First, a sample size of 31 firms with no defaults in recent years means that a benchmark to actual defaults is impossible. Second, credit spreads represent a market-based valuation of an institution’s promise to pay, the ultimate measure of credit risk. Focusing on the ending and minimum projected tier 1 capital ratios from the CCAR process, we examine these questions:
- Among the traded bonds of the CCAR banks, are the ending and minimum tier 1 capital ratios statistically significant as single variable models in predicting credit spreads?
- Compared to other single variable benchmark models of credit spreads, how do the ending and minimum tier 1 capital ratios rank?
- As incremental variables in a well-specified multi-variable benchmark model, how much incremental explanatory power, if any, do the ending and minimum tier 1 capital ratios provide?
These are the same types of questions posed by Fed staff to the 31 financial institutions subject to the CCAR stress-testing process.
The Data
March 9, 2015 was the second full business day after the announcement of the CCAR results. Fridays are typically a light trading day, so we used trade-weighted average bond prices from TRACE, Market Axess and Kamakura Risk Information Services on Monday, March 9. The National Association of Securities Dealers launched the TRACE ( Trade Reporting and Compliance Engine) system in July 2002 in order to increase price transparency in the U.S. corporate debt market. The system captures information on secondary market transactions in publicly traded securities (investment grade, high yield and convertible corporate debt) representing all over-the-counter market activity in these bonds. We identified all senior fixed-rate non-call bond issues by any legal entity affiliated with the 31 CCAR institutions. We eliminated all bonds with a survivor option from the sample. We were left with the following data. In all, there were 436 bond issues of legal entities associated with 27 of the 31 CCAR firms. There were 3,403 trades representing $1.2 billion of underlying principal on the bonds, as shown in this table.
For each bond, we calculated the matched-maturity yield to maturity on U.S. Treasury bonds (TLT) (TBT), interpolated from the Federal Reserve H15 statistical release for that day. The credit spread was calculated by subtracting the matched maturity Treasury yield from the trade-weighted average yield on each bond.
We assembled a minimalist set of attributes of each bond and its associated ultimate parent company, including the following, for a benchmark model construction:
- Bond maturity date
- Bond coupon
- Bond years to maturity
- Bond years to maturity squared
- Legacy ratings index, set to 1 for AAA, 2 for AA+, and so on
- “KDP” for Kamakura default probability, the matched maturity Jarrow-Chava reduced form model version 5.0, explained in appendix A
- The matched maturity credit spread for the U.S. Dollar Cost of Funds Index on March 9, provided by Kamakura Corporation, and explained in appendix B.
- Tier 1 capital ratio at September 30, 2014
- The ending tier 1 capital ratio in the CCAR projections
- The minimum tier 1 capital ratio in the CCAR projections
Preliminary Results
We first regressed credit spreads for all bonds on the ending and minimum tier 1 capital ratios from the CCAR process, one variable at a time. Both variables were modestly statistically significant, with adjusted r-squareds of 1.44% and 1.20% respectively. Relative to other single variable naïve models, performance of the CCAR ratios was very poor. Even the years to maturity variable was a better predictor of credit spreads, with an adjusted r-squared of 13.96%. When the two CCAR ratios were added to a benchmark model, neither model was statistically significant using stepwise-type logic on variable inclusion at the 5% significance level.
We hypothesized that the data was “noisy” because of small trades and the extremely high CCAR ratios for Deutsche Bank, which exceeded 30%. We therefore dropped all bond issues with less than $1 million in trading volume on March 9, and we dropped Deutsche Bank AG and Deutsche Bank AG (London Branch) bond issues from the data set.
Revised Data Set
After the revisions to the data set, we were left with the following 23 institutions:
In total, there were 176 bond issues and 2,345 trades representing $1.1 billion in underlying principal.
The Analysis
The first model validation question on the revised sample is easy to state.
1. On a stand-alone basis as a single variable model, are the ending CCAR tier 1 capital ratio and the minimum tier 1 capital ratio statistically significant predictors of credit spreads?
Yes.
The results for the ending tier 1 capital ratio are shown here:
All other things being equal, a 1 percentage point increase in the ending tier 1 capital ratio lowered credit spreads by 0.051%. The adjusted r-squared of this single variable model is 6.68%.
The results for the minimum tier 1 capital ratio are shown here:
The minimum tier 1 capital ratio causes credit spreads to decline by 5.8 basis points for each percentage increase in the capital ratio. The adjusted r-squared is 4.96%.
We now pose the second question in the spirit of Bharath and Shumway’s model validation exercise on the Merton model of risky debt and in Haldane’s model validation of the Basel capital ratios:
2. Are the CCAR capital ratios “sufficient” models of the credit spread, such that no other incremental explanatory variables have statistical significance?
No.
This benchmark model, developed using a simple stepwise process, is vastly more accurate in predicting credit spreads than either of the two CCAR capital ratios on a stand-alone basis:
The benchmark model explains 81.56% of the cross-sectional variation in credit spreads among the 176 bond issues of the 23 CCAR-related firms. The explanatory variables are the actual tier 1 capital ratio as of September 30, 2014, the bond coupon, the years to maturity, the years to maturity squared, the matched maturity U.S. Treasury yield, the Kamakura matched-maturity default probability, and the rating index. Note that the coefficient on the Kamakura default probability implies that the credit spread increases by 68 basis points for each percentage point increase in the default probability. This in turn implies a 32% loss given default, on average.
We now turn to the third question, one also posed by Bharath and Shumway:
3. Given that the tier 1 capital ratios are not “sufficient,” do they at least bring some incremental explanatory power to the prediction of credit spreads?
Yes, they bring some modest incremental explanatory power.
Because of correlation among the actual, ending, and minimum capital ratios, only two of the ratios are retained in the “best model.” That model is summarized here:
The tier 1 minimum capital ratio is statistically significant and causes a 4.6 basis point decline in credit spreads for each 1 percentage point increase. The model which includes the tier 1 minimum capital ratio has an adjusted r-squared of 84.45%, which is 2.89% higher than the model without the tier 1 minimum capital ratio.
Serious analysts may well debate whether or not the ending tier 1 capital ratio should be included, even if it does not meet the normal tests of significance, because it is highly correlated with the other two capital ratios. If this were done, the adjusted r-squared would be 84.42%, slightly lower. That version of the model is shown here:
Conclusions
On Monday, March 9, there were observable credit spreads on 27 of the 31 CCAR institutions if one looks only at the U.S. fixed rate market for senior debt. These credit spreads, because they reflect all scenarios and are based on hard dollar trades, are more accurate indicators of credit quality than any formula.
We tested the incremental contribution to a model of credit spreads that stems from the existence of the CCAR tier 1 minimum and ending capital ratios. The addition of those two ratios improves the adjusted r-squared on a benchmark credit spread model by less than 3 percentage points. The benefits of this accuracy improvement have a value that is far less than the cost to the 31 CCAR institutions and their regulators of the CCAR process. Does that mean that CCAR should be discontinued? In the humble opinion of this author, the answer is no. The CCAR program has pushed many risk managers to improve the sophistication and accuracy of their analysis. Instead of cancelling CCAR, the CCAR program should be generalized to a full Monte Carlo simulation of the probability of default of each of the CCAR institutions over the CCAR standard 13 quarter time horizon or even longer. These probabilities of default should be measured for accuracy versus statistical estimates of default using public information only. One would expect that a high quality internal model using non-public information should be able to outcompete a model based solely on public information. Until that has been proven, there are many incremental improvements that one can make to the CCAR process.
Appendix A
Background on the Default Probability Models Used
The Kamakura Risk Information Services version 5.0 Jarrow-Chava reduced form default probability model (abbreviated KDP-jc5) makes default predictions using a sophisticated combination of financial ratios, stock price history, and macro-economic factors. The version 5.0 model was estimated over the period from 1990 to 2008, and includes the insights of the worst part of the recent credit crisis. Kamakura default probabilities are based on 1.76 million observations and more than 2000 defaults. The term structure of default is constructed by using a related series of econometric relationships estimated on this data base. KRIS covers 35,000 firms in 61 countries, updated daily. Free trials are available at Info@Kamakuraco.com. An overview of the full suite of Kamakura default probability models is available here.
General Background on Reduced Form Models
For a general introduction to reduced form credit models, Hilscher, Jarrow and van Deventer (2008) is a good place to begin. Hilscher and Wilson (2013) have shown that reduced form default probabilities are more accurate than legacy credit ratings by a substantial amount. Van Deventer (2012) explains the benefits and the process for replacing legacy credit ratings with reduced form default probabilities in the credit risk management process. The theoretical basis for reduced form credit models was established by Jarrow and Turnbull (1995) and extended by Jarrow (2001). Shumway (2001) was one of the first researchers to employ logistic regression to estimate reduced form default probabilities. Chava and Jarrow (2004) applied logistic regression to a monthly database of public firms. Campbell, Hilscher and Szilagyi (2008) demonstrated that the reduced form approach to default modeling was substantially more accurate than the Merton model of risky debt. Bharath and Shumway (2008), working completely independently, reached the same conclusions. A follow-on paper by Campbell, Hilscher and Szilagyi (2011) confirmed their earlier conclusions in a paper that was awarded the Markowitz Prize for best paper in the Journal of Investment Management by a judging panel that included Prof. Robert Merton.
Appendix B
The U.S. Dollar Cost of Funds Index
The U.S. Dollar Cost of Funds Index TM measures the trade-weighted cost of funds for the largest deposit-taking U.S. bank holding companies. The index is a credit spread, measured in percent and updated daily, over the matched maturity U.S. Treasury yield on the same day. The current bank holding companies used in determining the index are Bank of America Corporation (BAC), Citigroup Inc. (C), JPMorgan Chase & Co. (JPM), and Wells Fargo & Company (WFC). The index is an independent market-based alternative to the Libor-swap curve that has traditionally been used by many banks as an estimate of their marginal cost of funds. Kamakura Corporation is the calculation agent, and the underlying bond price data is provided by TRACE and the U.S. Department of the Treasury. The following graph shows the construction of the index on March 9:
The on-the-run values of the index on March 9 are as follows:
Copyright ©2015 Donald van Deventer