Validate Before You Speculate In Credit Scoring

The credit scoring world is coming out of a difficult environment in which lenders were compelled by the recession to operate their businesses in a very conservative fashion. Consumers were dealing with credit in a similar way…yet, as we’ve moved out of the recession, they have adjusted while many lenders have not.

In a recent conversation, MPD CEO Karen Webster and Sarah Davies, SVP of Analytics, Product Management & Research at VantageScore Solutions, discussed the vital importance for lenders to validate their credit scoring models both to account for consumer behavior during the recession as well as reflect the current paradigm. It’s likely that there is more to a consumer’s creditworthiness than traditional scoring models bear out — and if lenders don’t adjust, they’ll continue to miss out on opportunities.

KW:  Let’s talk today about something I know you regard as quite important, but not everyone in the credit scoring business necessarily pays attention to: validating your model.

You recently concluded the annual process by which you do that. Why is it so important?

SD:  It’s an interesting situation. I think the world of credit scores has been relatively sleepy for a long time. In many ways, score developers and users haven’t felt it very necessary to validate scores in terms of how predictive they are, and looking at where they are and are not performing well.

What we saw, however, through the last 5 to 6 years and all the volatility of the recession, was that score performance really did change very significantly — especially in the mortgage space. When validations weren’t being done on an annual basis, we really didn’t have a sense of how much exposure was being created by poorer performing models.

In our process, we poll 5 million consumers from each of the credit bureaus (for a total of 15 million consumers and 45 million credit files) and validate on that depth of information, looking for the ways in which the model is performing well and — more importantly for the industry — where it is not performing as well.

It’s critical information for a lender, and it really does help with risk management and risk strategy design.

KW:  What are you validating the scores against?

SD:  We’re validating against the consumer performance.

We score a consumer out at the beginning of a particular time frame, usually a two-year window, at the end of which we determine how they performed: Did they continue do pay their debts on time, or did they default?

We use statistics that are pretty familiar within the industry, perhaps not so much outside of it: GINI statistics or Kolmogorov-Smirnov (K-S) statistics. These are designed to identify how many consumers are going to default versus how many are going to pay on time, and we utilize them to measure how well the models are rank-ordering consumers.

KW:  What did you learn?

SD:  We learned that, during the recession, the models began to not predict as well. The volatility of behavior meant that models were having a harder time distinguishing good performance from bad performance.

To be very clear, all the models continued to do a pretty good job — it just wasn’t as good as when they were originally developed.

Post-recession, now, we see that the models are actually performing very well again, because the economy is much more stable.

KW:  In what ways, specifically, were the models not performing as well?

SD:  The performance was affected by a variety of underlying consumer behaviors in managing their credit during the period in question.

One example was the strategic defaulting of home loans. Consumers were defaulting on their mortgages before they were defaulting on credit cards and auto loans. That runs counter to the way that consumers typically respond to their mortgage; the mortgage is the largest and most important loan — it’s essentially the consumer’s primary asset.

Historically, people had always made sure they paid their home loan. But, during the recession, a small percentage of the population began to default on their home loans while they kept their credit card and their vehicle loans in good standing.

These shifts in traditional consumer behavior can contribute to uncertainty in the way the scoring model is trying to predict it.

KW:  Have you made adjustments, as a result of those changes in behavior, to the way you’re scoring risk?

SD:  Yes. As part of the validation, we work through myriad data and statistics, looking for instances like those behavioral changes and others. At some point, that triggers a global redevelopment: We update the model to reflect fresher data and a more recent time frame that more accurately represent how consumers are handling their debt.

After VantageScore 1.0 was launched back in 2006, by the time we got to the middle of the recession, we saw very different behaviors going on, so we built a new model. Now VantageScore 3.0 — which was launched three years ago — captures the post-recessionary environment, where consumers are handling their debts in a more conservative fashion. We wanted to make sure that the model picked up those new behaviors.

KW:  It sounds like it did. We’ve talked before about how your research and validation results support 36 million more consumers being scored using your method than can be using conventional ones.

How did what you learned influence your scoring so that it could include so many more consumers?

SD:  What we found, as we were looking at consumer behavior and data, was that a large number of consumers were not scored by conventional models — ones that had been used in the industry for several decades — simply because those models required consumers to have been using credit frequently within the prior 6 months, and to have at least 6 months of history on their credit file.

Although those parameters capture a large amount of the U.S. population, we found that our statistics, methodology and data were still capable of accurately scoring the consumers who failed to meet that criteria.

When we built specific score cards to look at those consumers — people who didn’t use credit quite as frequently, or didn’t have credit that was at least 6 months old — we discovered that there were about 36 million additional consumers that could be scored by looking at their data in a very unique fashion.

KW:  An absence of credit didn’t necessarily mean than someone was a bad credit risk; rather, it could mean that that they simply weren’t comfortable using credit but were nonetheless creditworthy individuals. Is that the conclusion?

SD:  That’s exactly right. Because of the recession, people were becoming more conservative in their credit management and credit usage practices, and were therefore failing the traditional scoring criteria at an increasing rate. Even if a consumer had used credit just within 30 days outside of that 6-month window I mentioned, they were deemed not creditworthy. We saw that as somewhat of an arbitrary line in the sand to draw.

If they’re looked at in a very specific fashion, those consumers can actually be scored very effectively. They should be on the same radar screen as everybody else.

KW:  Should lenders take on board the idea to validate their scoring models based on portfolios in addition to their own? Would you suggest that they cast a wider net?

SD:  Yes; I would recommend that.

One of the things we have done in the last several years of our validations is work with the Office of the Comptroller of the Currency (OCC) guidelines, which we find very helpful in terms of how to validate models.

We’ve refined and enhanced our analytics to take advantage of their guidance, and now present a much broader set of insights and statistics that help our own understanding of how the model performs. The OCC guidelines provide some useful transparency into model-validation routines, and we would encourage issuers to adopt them.

Let me be very clear: I think that an issuer needs to first and foremost always validate a credit score on their own portfolio and their own business. It captures exactly the unique traits of they consumers.

We hope to provide, in a sense, an educational platform for how to think about validation beyond the standard routines.

As we begin to open up the credit box, so to speak, and look for new consumers and new opportunities, we can learn from what happened in the past — and reflect as much within new validation routines — so that we won’t repeat those mistakes.

To download the full version of the podcast, click here.