AI models may struggle to cope with market downturn
By: Agus Sudjianto, Jacob Kosoff and Aaron Bridgers
There is a lot of hype surrounding the benefits of using artificial intelligence and machine learning models for credit underwriting.
However, these models introduce fair loan risk, reputational risk, and significant credit risk.
Many fintechs and newly formed non-banks that enter the credit market at the end of the cycle favor the use of AI and ML underwriting methods that focus on thin, near-prime borrowers. In addition, established banks are under pressure to expand. And some are competing for the same types of high-risk clients, perhaps with misplaced reliance on credit decisions based on machine learning models.
This new enthusiasm around AI / ML underwriting and the current expansion of credit at the end of the cycle should lead to caution. Many AI / ML underwriting models are built using data assembled from various sources that is potentially biased in favor of a benign credit cycle. These models are likely to underestimate actual credit defaults and institutions using such models will experience significantly higher losses than their counterparts during the current market downturn.
One of the most difficult lessons the financial sector learns from the market downturn of March 2020 – and the financial crisis of 2008 – is that credit models can deteriorate quickly, and borrowers with identical credit scores can. have radically different performance depending on the time of the credit cycle. the loan is initiated.
These results by credit score strip depend heavily on the economic conditions found in the development data set. This means that vintage analysis, which assesses credit quality by loan date, provides the most reliable tool for understanding whether actual results are as expected.
Vintage analysis also creates an early warning indicator if patterns start to fail during a downturn in a credit cycle.
As seen in 2008, defaults became highly concentrated in the vintages closest to recession, and economic deterioration spilled over from the housing sector to all parts of the economy, invalidating assumptions of diversification. By the time most banks detected these modeling flaws, it was too late.
Model risk experts already recognize that many of the new models for mortgage or AI / ML credit card underwriting do not take into account certain risk factors, such as negative vehicle equity. Borrowers with negative auto stocks may find it difficult to manage multiple credit obligations, as noted.
The credit bureau attributes that most banks use for non-auto mounts via AI / ML models do not provide negative auto stocks, which leads to underestimating default rates in stress scenarios.
Model risk practitioners also point out that many machine learning models are based on additional factors with fragile correlations calculated during good economic times. As a result, a model’s ability to rank customers can quickly wane or even collapse during an economic downturn.
There is a modeling concept borrowed from biostatistics called “heterogeneity,” which seeks to identify higher risk groups within the same credit band based on subtle factors.
If these factors are not modeled correctly, it will be difficult for risk managers to identify the riskiest segments among credit applicants.
People who exhibit these factors may be considered “fragile” or more sensitive to environmental risks and will behave differently during a recession. In credit modeling, these include indirect factors such as wealth accumulation, financial lifestyle, access to credit, and other aspects of financial health that tend to be masked in the models. machine learning.
There are unproven claims from AI / ML providers who use alternative data and the sophistication of their new models that they can find factors explaining heterogeneity. Note that the heterogeneity is considerably amplified during a credit slowdown, as evidenced by the 2008 experience of rapid deterioration in vintage quality.
Credit modeling performance based on past experiences indicates that predictive fragility factors are elusive and should be humbly recognized as a risk not taken into account in underwriting credit models.
Anticipating unobserved factors of fragility, risk management teams must prepare for the next recession by creating an end-to-end approach. Such an approach must capture and effectively integrate early warning signals from underwriting, portfolio management, watchlist and collections.
Risk managers must continue to exercise healthy skepticism about the newest and most successful machine learning credit models. Models that claim to avoid fragility by effectively discerning credit quality within credit tranches should be approached with further consideration. This includes demanding interpretable models to question the correctness of the factors determining the predictions.
All origination decisions made by credit models should be monitored so that vintage deterioration caused by model assumptions can be quickly detected. Banks need to create a playbook that quickly adapts underwriting standards to mitigate faulty model assumptions and ensure that this playbook can effectively curb shoddy edits once detected.
In the end, it is impossible to observe all the variables that determine the ability of people to repay their loans. As such, not all models fail to capture some risk.
The claims of AI / ML models using alternative data to select better clients in near-premium segments are exaggerated and untested during a recession. During good times, loans are easier to repay because people have more stable incomes and can refinance themselves with the almost limitless opportunities offered by a fully functioning credit market.
It also means that it is difficult to determine which models are good at making predictions and which models are bad in a growing economy. It’s a very different situation during this current market downturn, when causation is important. As Warren Buffett said, “It is only at low tide that we find out who swam naked.”
Model risk professionals during this market downturn of March 2020 need to look at the risk of credit underwriting models more holistically; not only the performance of the traditional model. They must also understand and manage the risks inherent, and sometimes unpredictable, in the current market fallout.
Agus Sudjianto is Executive Vice President and Head of Business Model Risk for Wells Fargo. Jacob Kosoff is responsible for model risk management and validation at Regions Bank. Aaron Bridgers is Senior Vice President and Head of Risk Test Optimization at Regions Bank.
The opinions expressed in the presentation are statements of the opinion of the speaker, are intended for informational purposes only and are not formal opinions of, nor binding on, Regions Bank, its parent company, Regions Financial Corporation and their subsidiaries, and any representation to the contrary is expressly excluded.