Model Risk and Validation

Definition

Model risk is the potential for adverse consequences from decisions based on incorrect or misused model output. It arises from fundamental errors, from correct models applied outside their intended scope, and from organisational over-reliance on results whose limitations are not understood.

Validation is the independent, evidence-based challenge that establishes whether a model is fit for its declared purpose, and under what conditions that conclusion ceases to hold.

Why it matters

• A model in production is a standing decision rule. Its errors compound quietly across every decision it touches.
• Machine-learning and generative components have expanded the model inventory faster than most validation functions have expanded capacity.
• Supervisors increasingly assess the validation function itself, not only the models within it.

Professional focus of Jonas Osman

• Validation scaled to model materiality rather than applied uniformly regardless of impact.
• Conceptual soundness as the first and most valuable test — before any performance statistic is computed.
• Explainability requirements set by the decision context, not by the technique chosen.
• Ongoing monitoring designed at build time, with thresholds and owners defined before deployment.

Key methodologies

• Conceptual soundness review. Assessment of theory, design choices, assumptions and their appropriateness for the intended use.
• Data quality and lineage assessment. Completeness, accuracy, representativeness, timeliness and traceability from source to model input.
• Back-testing and out-of-time testing. Performance against realised outcomes across periods the model did not see during estimation.
• Benchmarking. Comparison against challenger models, simpler baselines and external references.
• Sensitivity and stability analysis. Response to input perturbation, parameter change and re-estimation on alternative samples.
• Explainability testing. Global and local attribution methods with assessment of whether explanations are faithful and actionable.

Practical management applications

• Model inventory with tiering, ownership, validation status and known limitations.
• Approval conditions and use restrictions recorded and enforced through change control.
• Monitoring dashboards with pre-agreed action thresholds.
• Board-level reporting of aggregate model risk rather than a list of individual findings.

Governance and limitations

• Validation independence requires reporting lines, resourcing and access that do not depend on the model owner's cooperation.
• A validated model remains valid only within the data range, product scope and market conditions tested.
• Findings without remediation deadlines and owners do not reduce model risk.
• Vendor and third-party models require the same evidence standard; a supplier's documentation is not a validation.