Executive summary
- Validation is independent challenge, not replication of the developer's testing.
- Conceptual soundness is examined first, because a structurally inappropriate model cannot be rescued by good performance statistics.
- Data quality, lineage and representativeness are assessed as a distinct area, not as a preliminary step.
- Validation intensity should be proportionate to model materiality, defined through a documented tiering approach.
- The output is a conditional opinion with use restrictions, findings, owners and deadlines.
Six areas of a validation
- Conceptual soundness: is the theory appropriate, are the design choices justified, and are the assumptions reasonable for the intended use?
- Data: is the data complete, accurate, representative of the current portfolio and traceable from source to model input?
- Implementation: does the code do what the documentation says, and does the production system match the development version?
- Performance: how does the model behave against realised outcomes, out of sample and out of time?
- Sensitivity, stability and benchmarking: how does it respond to input perturbation and re-estimation, and how does it compare with alternatives?
- Governance and use: is the model used within its approved scope, monitored appropriately and owned by someone accountable?
The ordering is deliberate. A model with an inappropriate structure will sometimes produce good backtest results on a sample that resembles its estimation period. Beginning with statistics can therefore validate a model that should have been rejected on conceptual grounds.
Proportionality and tiering
Applying identical validation depth to every model is neither feasible nor sensible. Tiering allocates effort by materiality, using criteria such as the financial exposure influenced by the model, its regulatory relevance, its complexity, the availability of alternatives and the consequence of error for customers.
- Tier assignment should be documented, reviewed periodically and challengeable.
- Higher tiers warrant full independent replication or a challenger model; lower tiers may warrant targeted review.
- Validation frequency should also scale with tier, with event-driven triggers regardless of tier.
- Aggregate coverage — the proportion of the inventory validated within policy — should be reported to the risk committee.
What constitutes evidence
A validation opinion should rest on tests the validator performed or independently reproduced. Reviewing the developer's documentation and confirming it appears reasonable is a review, not a validation, and the distinction should be stated in the report.
- Independent replication of key results on the same data, and where possible on an independently assembled dataset.
- Construction of a challenger model, even a deliberately simple one, to establish what performance the added complexity delivers.
- Perturbation testing across the plausible input range, including at boundaries where behaviour is often poorly defined.
- Re-estimation on alternative samples to assess parameter stability.
- Review of actual use: which decisions rely on the output, and are they within the scope the model was built for?
The validation opinion and findings
An opinion should be conditional and specific: the model is fit for the stated purpose within the stated data range and market conditions, subject to the listed restrictions and findings. A binary approval without conditions communicates less than the validator actually knows.
- Each finding needs a severity rating, a named owner, a remediation deadline and a stated interim risk mitigant.
- Use restrictions should be enforceable through system controls where possible, not only recorded in a document.
- Overdue findings should be reported to the risk committee with the exposure they leave open.
- Re-validation triggers should be defined: material market change, portfolio change, performance breach or elapsed time.
Practical example
A behavioural deposit model shows strong statistical fit across its estimation window and passes every performance test the developer applied. Conceptual review notes that the estimation period contains no episode of rapidly rising rates, and that the model's functional form constrains the repricing response to be linear in the rate change.
No performance statistic would reveal this, because the data contains no observation of the condition in question. The validation opinion approves the model with a restriction limiting its use to rate movements within the observed range, requires a stress overlay beyond that range, and sets a re-validation trigger on any rate move exceeding it. That is a more useful outcome than either approval or rejection.
Limitations and caveats
- Validation cannot establish that a model will perform under conditions absent from the available data.
- Independence is structural: it depends on reporting lines, resourcing and access, not on the validator's intentions.
- Vendor models limit inspection of internals, and the resulting evidence gap should be documented rather than assumed away.
- Validation resource is finite, so tiering decisions themselves carry model risk.
Conclusion
A validation function exists to make model limitations visible before they become losses.
Its value is measured by whether conditional opinions, use restrictions and findings actually constrain how models are used — not by the number of validations completed.
References
- SR 11-7: Guidance on Model Risk Management — Board of Governors of the Federal Reserve System, 2011
- Model risk management principles for stress testing (SS3/18) — Bank of England — Prudential Regulation Authority, 2018
- Solvency II Directive 2009/138/EC — European Commission, 2009
Author bio
Jonas Mohamed Osman Abdelghafour, known as Yonas Osman is an actuary, FRM and financial risk professional specialising in banking, insurance, model risk, capital modelling and quantitative risk management.