Explainability, Fairness & Model Risk Management

In finance a model is not finished when it reaches a good AUC. It must be explainable, fair, documented, validated and monitored. This lesson covers SHAP and reason codes, fairness testing, and the model risk management lifecycle that validators and regulators expect.

Global vs local explanations

• Global: which features drive the model overall (mean |SHAP|, permutation importance, partial dependence).
• Local: why this customer got this score (SHAP values per prediction). These become reason codes.

import shap
explainer = shap.TreeExplainer(model)
sv = explainer.shap_values(X_valid)            # per-feature contribution to each prediction (log-odds)
shap.summary_plot(sv, X_valid)                 # global view
row = 42
top = sorted(zip(X_valid.columns, sv[row]), key=lambda t: -t[1])[:3]
reasons = [f"{f} increased risk" for f, v in top if v > 0]

• Map features to human reason codes, e.g. util_ratio → "High credit utilisation", dpd_6m → "Recent missed payments".
• Be careful with correlated features: SHAP splits credit between them, so group related features into reasons.

Fairness testing

• Do not use protected attributes as inputs, and check for proxies (e.g. some PIN codes or names).
• Compare outcomes across groups where lawful and appropriate: approval-rate ratios, false-positive and false-negative rates, calibration by group.
• Common metrics are demographic parity difference, equal opportunity (TPR gap) and the disparate-impact ratio (the "80% rule" heuristic).
• Mitigate with feature review, reweighting, threshold policies or constraints, and document the trade-offs.

Model risk management (MRM) lifecycle

1. Inventory: every model is registered with its owner, purpose, tier (materiality) and status.
2. Development documentation: data, assumptions, methodology, alternatives considered, limitations.
3. Independent validation: conceptual soundness, data quality, outcome analysis (back-testing), sensitivity and stress tests, benchmarks or challenger models.
4. Approval and controls: usage limits, overrides policy, fallback procedures.
5. Ongoing monitoring: performance, PSI/drift, overrides, incidents, with thresholds that trigger review.
6. Periodic revalidation, and retirement when the model is no longer fit for purpose.

US guidance such as SR 11-7 formalised these ideas, and banks worldwide, including in India, run similar frameworks.

GenAI under model risk

LLM apps (chatbots, document extraction, summarisation) are increasingly brought into MRM:

• Define the intended use and out-of-scope uses.
• Evaluation sets: accuracy, faithfulness and hallucination rate, toxicity, prompt-injection robustness.
• Controls: human-in-the-loop for customer-impacting outputs, citations, logging, vendor-model change management (pin versions, re-test on upgrade).

Check yourself