A common approach to risk measurement can be described by this example:

• We have three factors driving the yield curve: level, shift and bend 
• We chose “no arbitrage values” for each of these factors and choose 7 values for each factor, the mean and plus or minus 1, 2 and 3 standard deviations from the mean 
• We then calculate portfolio values for all 7 x 7 x 7 = 343 combinations of these risk factors 
• We calculate our nth percentile value at risk from the smoothed surface of values that results from these 343 portfolio values, using 1 million draws from this smoothed surface to get the 99.97th percentile of our risk.

What’s wrong with this approach?  It certainly seems reasonable at first glance and could potentially save a lot of computing time versus a large monte carlo simulation.  Let’s look at it out of the finance context.  We are posed this problem:

Our golf ball is placed in the location marked by the white square in the lower left-hand corner of this aerial photo of a golf course in Augusta, Georgia. However, we are blindfolded and cannot see. We are told that the ball is either on a green at Augusta National or that it is not, and our task is to determine which case is the true one.

We calculate the standard deviation of the length of the first put made by every golfer on every hole at last year’s Masters golf tournament.  From the location of our golf ball, we then get the latitude and longitude of 49 points: our current golf ball location plus all other locations that fall on a grid defined by 1, 2, or 3 standard deviations (of putting length) North-South or East-West with our golf ball in the middle.  The grid looks like this:

At each point where the North-South and East-West lines intersect, we are given very precise global positioning satellite estimates of the elevation of the ground at that point. Upon reviewing the 49 values, we smooth the surface connecting them and we find that the 99.97th percentile difference in elevation on this smoothed surface is only 2 inches.  Moreover, we find that the greatest difference in elevation among any of the points on the grid is only slightly more than two inches.  We know by the rules of golf that a golf hole must be at least 4 inches in depth.  We reject the hypothesis that we are on a green.  But we are completely wrong!

This silly example shows the single biggest pitfall of interpolating between Monte Carlo values using some subset of a true Monte Carlo simulation.  We have assumed that the surface connecting the grid points is smooth, but the very nature of the problem contradicts this assumption.  The golf green is dramatically “unsmooth” where there is a hole in the green.  Our design of the Monte Carlo guaranteed a near certainty that our analysis of “on a green”/”not on a green” would be no better than flipping a coin.

Similarly, we could have been placed blindfolded on a spot somewhere in the United States and told either we are within 100 miles of the Grand Canyon or we are not.  We again form a grid of 9 points all of which are plus or minus 100 miles North-South or East-West of where we were placed blind folded, as shown below:

Again, we take the GPS elevation at these 9 grid points, smooth the surface, and take the 99.97th percentile elevation difference on the smoothed surface.  We find that our elevation difference is only 100 meters at the 99.97th percentile.  We reject the hypothesis that we are within 100 miles of the Grand Canyon since we know the average depth of the Grand Canyon is about 1,600 meters.  Again, we are grossly wrong in another silly example, because our assumption that intervening values between our grid points are connected by a smooth surface is wrong.

Summarizing the Problems with Interpolated Monte Carlo Simulation for Risk Analysis

For relatively simple, low dimensional systems that arise in physics and engineering, where correlations are well-controlled even in the tails, a reliable and believable description of the asymptotic behavior of the global probability density function may indeed be possible.  In high dimensional cases, such as in the risk management of financial institutions, things are much more complex with respect to the global probability density function of values, cash flows and net incomes.  The simple examples above illustrate two problems that can be extremely serious problems if risk is measured by interpolating between Monte Carlo valuations on a grid or small subset of possible outcomes:

• Smoothness is an assumption, not a fact, as our simple examples show. Interpolating yield curves using a smoothness criterion is reasonable for good economic reasons, but as we show below, it is highly likely that portfolio values and cash flows in financial institutions have sharp discontinuities.
• The Nth percentile “worst case” derived from the simulation is not relative to the true “worst case,” it is relative to the worst case on the grid defined by the analyst.  In our Grand Canyon example, the “worst case” elevation differential was 100 meters, but the true “worst case” elevation difference was the true depth of the Grand Canyon, 1600 meters.

Almost the entire balance sheet of financial institutions is filled with caps, floors, prepayment risk and defaults.  One cannot assume that the law of large numbers will make these 0/1 changes in values and cash flows change in a smooth way as the number of transactions rises.  We can show that with a simple example.  Assume it’s February 2000.  3 month US dollar Libor is 6.00%.  The five year monthly standard deviation of 3 month US dollar Libor from March 1995 to February 2000 was 32 basis points.  We have only 3 transactions on our balance sheet:

•  The first digital option pays us $1 if 3 month Libor is 6.25% or higher in 18 months
• The second digital option pays us $1 if 3 month Libor is 6.05% or less in 18 months
• The third digital option requires us to pay $100 if 3 month Libor is below 4.00% in 18 months

We set up our grid of 7 values for 3 month Libor in 18 months centered around the current level of Libor (6.00%) and plus and minus 1, 2, and 3 standard deviations.  We do valuations at these Libor levels in 18 months:

• Scenario 1: 6.96% Libor
• Scenario 2: 6.64% Libor
• Scenario 3: 6.32% Libor
• Scenario 4: 6.00% Libor
• Scenario 5: 5.68% Libor
• Scenario 6: 5.36% Libor
• Scenario 7: 5.04% Libor

Our valuations of our portfolio are $1.00 in every scenario:

• Scenario 1: 6.96% Libor means we earn $1 on digital option 1
• Scenario 2: 6.64% Libor means we earn $1 on digital option 1
• Scenario 3: 6.32% Libor means we earn $1 on digital option 1
• Scenario 4: 6.00% Libor means we earn $1 on digital option 2
• Scenario 5: 5.68% Libor means we earn $1 on digital option 2
• Scenario 6: 5.36% Libor means we earn $1 on digital option 2
• Scenario 7: 5.04% Libor means we earn $1 on digital option 2

Typical interpolation of these Monte Carlo results would smooth these calculated values.  Ardent interpolation users then draw from this smoothed surface (we’ve heard up to 1 million draws from this interpolated surface).  In this case, all values on the grid are 1, the smoothed surface is 1 everywhere and we announce proudly “With 99.97% confidence we have no risk because we earn $1 in every scenario.”

WRONG!  We earn nothing in the range between 6.05% and 6.25%.  We completely missed this because we didn’t sample in that range and assumed cash flows were smooth, but they weren’t! Even worse, the actual value of 3 month Libor in August 2001 was 3.47%, and we had to pay out $100 on digital option 3!

This is a simple example of a very real problem with valuing from a small number of grid points instead of doing a true Monte Carlo simulation.  One of the authors remembers a scary period in the mid-1980s when typical interest rate risk simulations never simulated interest rates high enough to detect the huge balance of adjustable rate mortgages in bank portfolios with caps at 12.75% and 13.00%.  The young analysts, with no knowledge of history and no examination of the portfolio at a detailed level, did not detect this risk, just as our simple example above missed medium sized risk within the grid and huge risk outside the grid.

This graph of the 10 year and 5 year moving average standard deviations of month-end 3 month Libor shows how easy it is to design “grid widths” that end up being horribly wrong.

Take the 32 basis point standard deviation of 3 month Libor over the five years ended in February 2000.  The standard deviation of 3 month Libor for the 5 years ended in September 1981 was 450 basis points!

What’s the solution?

Going back to our August National example, if we don’t know the true width of the hole on a golf course, we at least know that 49 grid points is not enough. We do 1,000 or 10,000 or 100,000 random north-south/east-west coordinates looking for an elevation change of at least 4 inches (the depth of a golf hole). The degree of confidence we seek determines the number of random coordinates we draw. If we know by the rules of golf that the hole is exactly 4.25 inches wide, we can be much more refined in our search techniques.  The trouble in risk management, however, is that we don’t know if we’re looking for a golf hole (a few mortgage defaults in 2005) or the Grand Canyon (massive mortgage defaults from the huge drop in home prices in 2007-2009).

Given the current state of risk systems technology, there is no reason not to do a true Monte Carlo simulation that defines the amount of risk embedded in the portfolio with any reasonable desired level of precision.  The short cut of interpolating between a small subsample of scenarios is extremely dangerous.

Predrag Miocinovic, Alexandre Telnov and Donald R. van Deventer
Kamakura Corporation
Honolulu, October 7, 2009