How Model Validation Helps Credit Scoring Evolve With The Times

Fixed versus dynamic is the push-me-pull-you of payments.

The move to digital is forcing the payments and financial services ecosystem to adopt a more dynamic way of thinking and delivering services. Fixed infrastructure can sometimes make that a challenge.

Take credit scores.

Over the years, credit scores have increasingly been used to inform and influence decisions for lenders and consumers.

But one glaring misconception has remained — that a credit score is a fixed measure of repayment risk.

The truth is more nuanced: A credit score evaluates a consumer’s risk of defaulting on a loan (i.e., going 90 days or more past-due on a payment) at the moment in time that the score is calculated. The default risk associated with that score value (any credit score value) is not a fixed quantity: It will shift continually as economic conditions fluctuate, as lenders develop new loan products and as consumer behaviors change accordingly. That, said Sarah Davies, senior vice president of product development and analytics at VantageScore, is why it’s so critical that credit models have sensitivity to changing data and are refreshed over time.

Since its founding 10 years ago, VantageScore has published the results of its annual exercise to measure its credit scoring models so that financial institutions can develop more insight around credit scoring models and how they are performing.

“Transparency is a critical part of our mission,” Davies said, “and we felt it was appropriate to start a dialogue and conversation on a consistent basis with the industry about how these scores are performing and what [financial institutions] should do about it when they’re not performing well.”

There’s been a great deal of misunderstanding that credit scores predict absolute risk, but Davies noted that what they really do is rank-order consumers according to their relative risk of payment default — assigning lower scores to higher-risk consumers and higher scores to those with lower predicted risk. This can make it difficult for lenders to truly understand model performance and how to actually test for it.

“What we’ve done over these 10 years is provide lots of different ways to think about that, all within the context of understanding what the credit score does,” she added.

The outside-looking-in approach involves randomly taking credit files for 15 million consumers from the databases of the national credit reporting companies (CRCs — Equifax, Experian and TransUnion) in order to test how its own models stack up, performance-wise.

“Each model is validated by comparing predictive performance on both originations and existing-account management applications against the CRCs’ in-house models for the bankcard, auto and mortgage industries,” VantageScore wrote in a whitepaper, highlighting its findings, while reflecting on trends that have emerged over the years.

VantageScore’s process takes an ongoing measurement of performance based on the factors that matter most to lenders — predictiveness, universe expansion and score consistency across the three CRCs.

Learning From The Data

With a decade of credit model validations under its belt, Davies said one of the biggest takeaways from its validations year after year is that modern credit scoring models are extremely robust.

“The insights in the paper specifically show how you should be thinking about what credit score model is relevant for your industry and how you should be thinking about working with a contemporary timeframe,” Davies explained.

To put it simply, one-size-fits-all just doesn’t work anymore.

“You simply can’t afford to live in an environment where you’re thinking this one model applies across all products for all time,” she added. “You really do need to develop much more of an infrastructure that can handle different tools by product and different tools perhaps even by business line, whether it’s origination or existing accounts.”

VantageScore’s validation results assess various characteristic trends, risk and population volumes in order to determine true performance measures. And from there, VantageScore has discovered the following assessments:

• Over the past 10 years, the VantageScore models have consistently outperformed all CRC in-house models and scores for all major industries and business applications.
• VantageScore 3.0 continues to score 30 million–35 million more consumers than conventional credit scoring models.
• Due in part to the highly conservative post-recession lending and risk environment, all VantageScore credit score models continue to perform strongly.

“While all scores are performing at extremely high levels, VantageScore 3.0 outperforms the CRC models by an average of 1.7 percent to 3.4 percent in key industries,” the whitepaper stated. “In addition, VantageScore 3.0’s predictive performance is highly consistent across all three CRCs, varying by an average of only 0.34 Gini points between each CRC.”

To clarify, the Gini coefficient of a credit score compares the distribution of defaulting consumers with the distribution of non-defaulting consumers across the credit score range (the coefficient has a value of 0–100). The credit score fails to assign more defaulting consumers to lower credit scores.

Most importantly, at least from its most recent report, is what the company says about how VantageScore 3.0’s ability to reach the sizable “unscorable” population that is often overlooked by more traditional scoring models. By being able to score the conventionally unscorable, millions of people who would have otherwise been turned away due to “thin” credit files are presented with an expanded credit opportunity.

Davies noted that the credit score alone isn’t the most relevant piece of data anymore, but that it’s important to also dig into the data and get a much more granular understanding of what’s going on with the consumer.

The Next Frontier Of Credit Modeling

“At this point, we as an industry have a better sensitivity to the strengths and weaknesses of these tools. What we have to do now is move beyond the Petri dish we are working within,” Davies said.

While alternative data in credit modeling, such as paying rent and utility bills on time, have some value and an opportunity, Davies noted that it will take time for these sources to play out.

“You’ve got to now think about other types of modeling techniques,” she added. “We need to think about how to take the existing data, which is remarkably powerful, and see what more we can stretch out of it if we would shift out of these standardized techniques.”

It’s also important in the coming years for the ongoing fluidity and flexibility of operations infrastructures, meaning the capacity to switch out different credit models or use multiple models, to be considered. However, Davies advised that this should be done in a way that doesn’t cause any risk exposure.

“For any lender that goes down that path, they’re going to be lightyears ahead of anybody else,” she noted.