Regulators and financial institutions rely on statistical models to assess market risk. Alas, a new Federal Reserve paper shows that risk models are prone to creating confusion when they are needed most: in financial crises. Acceptable performance and convergence of risk models in normal times can lull the financial system into a false sense of reliability that transforms into model divergence and disarray when troubles arise.

“Model Risk of Risk Models”, Jon Danielsson, Kevin James, Marcela Valenzuela, and Ilknur Zer.

Federal Reserve staff working papers in the Finance and Economics Discussion Series, 2014-34.

http://www.federalreserve.gov/pubs/feds/2014/201434/201434pap.pdf

*The below are excerpts from the paper. Cursive text and underscores have been added.*

### What are statistical risk models?”

[*Statistical risk models are*] models used for forecasting systemic and market risk…statistical risk measures are set to play a much more fundamental role in policymaking and decision making within financial institutions.

**Market risk regulations**have been based on daily 99% Value-at-Risk [*VaR, a widely used statistic defined as a threshold value of the probability that the mark-to-market loss on the portfolio over a time horizon exceeds this value*] ever since the 1996 amendment to Basel I. After the crisis started in 2007, the extant market risk regulatory models became to be seen as lacking in robustness, especially when it comes to tail risk and risk measure manipulation. In response, the Basel Committee proposed three major changes to the existing regulatory regime in 2013, to be incorporated into Basel III: The replacement of 99% VaR with 97.5% expected shortfall [*an alternative to Value-at-Risk that is more sensitive to loss distribution in the tail of the distribution*], the use of overlapping estimation windows, and the calibration of a risk forecast to the historically worst outcome.- Parallel to these developments…
**systemic risk identification**and forecast methods has now emerged as a key priority…A wide variety of systemic risk measures have been proposed…Perhaps the most common way to construct a**systemic risk model**(SRM) is to adopt existing market risk regulation methodologies to the systemic risk problem, an approach we term market data based methods.”

“While intended for different purposes, both the market data based systemic risk methods and the market risk regulation techniques are closely related, sharing a methodological common root, [namely] VaR.”

### What is model risk?

“Broadly speaking, model risk relates to the uncertainty created by not knowing the data generating process… Within the finance literature…authors have defined model risk as… inaccuracy in risk forecasting that arises from estimation error and the use of an incorrect model.”

“It has been known, from the very first days of financial risk forecasting, that different models can produce vastly different outcomes, where it can be difficult or impossible to identify the best model… The fundamental problem of model risk in any risk model such as VaR arises because risk cannot be measured, but has to be estimated by the means of a statistical model. Many different candidate statistical models have been proposed where one cannot robustly discriminate between them.”

“In spite of this, very little formal model risk analysis [*on the divergent outcome of different underlying conventional methodologies*] has been done on VaR…In order to assess the model risk…we propose a new method we term risk ratios. This entails applying a range of common risk forecast methodologies to a particular asset on a given day, and then calculating the ratio of the maximum to the minimum risk forecasts.”

“While there is a large number of candidate methods for forecasting risk, the following six techniques in our experience are by far the most common in practical use: historical simulation, moving average, exponentially weighted moving average, normal GARCH [*models that assume variance to be a linear function of previous time periods’ fluctuations*], student-t GARCH [*assuming greater outlier risk than classic normal distribution models*], and extreme value theory. For that reason, we focus our attention on those six.”

### How serious is model risk?

“The data set [*of the empirical analysis*] consists of large financial institutions traded on the NYSE, AMEX, and NASDAQ exchanges from the banking, insurance, real estate, and trading sectors over a sample period spanning January 1970 to December 2012… We calculate the highest to the lowest VaR estimates for the random portfolios employing the six VaR approaches ”

“The mean model risk across all stocks and observations is 2.26 [*maximum risk forecast relative to minimum risk forecast*]… In the most extreme case it is 71.4…By further segmenting the sample into calm and turmoil periods, we find that model risk is much higher when market risk is high, especially during financial crises…model risk is positively and significantly correlated with market volatility, where the VIX…causes model risk…In other words, market volatility provides statistically significant information about future values of the risk readings’ disagreement.”

“When we apply the risk ratio analysis to the overlapping 97.5% expected shortfall approach proposed by the Basel Committee, instead of the current regulations with non-overlapping 99% VaR, we find that model risk increases by a factor of three, on average, with the bulk of the deteriorating performance of the risk models due to the use of overlapping estimation windows [*N.B. overlapping windows means that the same observations enter different samples and hence outlier events have repeated impact and can greatly bias statistical inference*].”

“In the case of the systemic risk measures considered, we find quite similar results as for VaR; the systemic risk forecasts…highly depend on the chosen model, especially during crisis periods. This supports our contention that any VaR based systemic risk measure is subject to the same fundamental model risk as VaR.”

### What are the causes of model risk?

“We suspect the problem of poor risk model performance arises for two reasons.

- The first is the low frequency of actual financial crises. Developing a model to capture risk during crises is quite challenging, since the actual events of interest has never, or almost never, happened during the observation period…
- Second, each and every statistical model in common use is founded on risk being exogenous, in other words, the assumption that extreme events arrive to the markets from the outside, like an asteroid would, where the behavior of market participants has nothing to do with the crisis event. However…risk is really endogenous, created by the interaction between market participants, and their desire to bypass risk control systems.”

### What are the consequences of model risk?

“The empirical results are a cause for concern, considering that the output of the risk forecast models is used as an input into expensive decisions, be they portfolio allocations or the amount of capital… The absence of model risk during calm times might provide a false of confidence in the risk forecasts. From a macro prudential point of view, this is worrying, since the models are most needed during crisis, but that this when they perform the worst.”