Value-at-risk estimates are just that: estimates. Errors occur because the distributional assumptions may not correspond to the actual distribution of changes in the market factors (and will never correspond exactly to the actual distribution of changes in the market factors), because the delta-normal, delta-gamma-normal and grid Monte Carlo methods are based on approximations to the value of the portfolio and because the estimates are based on past data that need not reflect current market conditions. Further, even users who choose the same framework and avoid implementation errors will make different implementation choices, leading to different value-at-risk estimates. For example, there can be differences in the choice of basic market factors, the mappings of various instruments, the methods for interpolating term structures, the formulas or algorithms used to value various instruments (e.g., options) and the number of past data used to estimate the distributions of changes in the market factors. In addition, some value-at-risk systems will embody logical or computer coding errors of varying degrees of severity and it is conceivable that in some situations users will have incentives to introduce biases or errors into value-at-risk estimates.
As a result, back testing is crucial to verify model accuracy and identify areas in which improvement is needed. While some of the impetus for back testing has come from banking regulators, who need to ensure VaR models used to determine capital requirements are not systematically biased, verifying VaR models is important for anyone who uses them for decision making.
Underlying the simplest back testing framework is the idea that, for a 1 — α confidence VaR model, one expects to observe exceptions on α% of the days.
For example, if α = 0.05 and the model is back tested using the last 250 daily returns, the expected number of exceptions is 0.05 x 250 = 12.5. Of course, the actual number of exceptions depends on the random outcomes of the underlying market factors in addition to the quality of the VaR model; even if the VaR model is correct, the actual number typically will differ from the expected number. This leads to a rule based on a range, determined by the willingness to reject a correct VaR model. For example, above figure shows the distribution of the number of exceptions out of 250 daily returns for a correct model with α = 0.05. Even though the expected number of exceptions is 12.5, the probability that the number of exceptions e is outside the range 7 <= e <= 19 is 5.85%. If 250 daily returns are used and a probability of rejecting a correct VaR model of 5.85% is tolerable, then the model should be rejected if the number of exceptions is outside this range and not rejected otherwise.
This simple approach is widely used and is enshrined in Basle framework, allowing banks to use internal risk models to determine capital requirements. However, a crucial limitation should be clear—many incorrect VaR models will generate between 7 and 19 exceptions out of 250 returns.