In a prior post, I described the payment improvement that BackOnTrack generates for delinquent customers. It’s a powerful benefit, and one that we discuss in detail with financial institutions. A common question I get is this: “Is that a case of self-selection?” What the person is asking is whether the customers who elect to complete BackOnTrack were going to be better payers anyway. That the improvement is attributable to self-selection.

In a word: NO.

Let me explain.

Volunteer Bias

Self-selection or volunteer bias is a known risk in running interventions, experiments, and surveys. The qualities of those who volunteer to participate can differ from general population characteristics. This dynamic can undermine the findings of a trial. As an Oxford University-affiliated organization, the Catalogue of Bias, notes:

Volunteer bias can occur at all stages of the trial from recruitment, retention through to follow-up. Differences between volunteers and the target population are not restricted to socio-demographic factors but can include attitudes towards the trial and institutions involved.

In selecting people for experiments and surveys, researchers seek to minimize the risk of volunteer bias. When a financial institution offers BackOnTrack to delinquent customers, a percentage of people will elect to complete it. These completers are by definition volunteers, as there is not requirement that they go through BackOnTrack (BOT).

Does that mean volunteer bias explains their better performance? Let’s use analytics to answer that.

Test vs. Control

The baseline assessment of whether a treatment is effective is conducting a randomized test of it. This was done in a pilot of BackOnTrack. A population of more than 300,000 1-59 days past due collections customers were separated into two groups. Each group was similar in characteristics.

The test group was deemed the BOT Invite Group. All delinquent customers in this group were emailed an offer to complete BackOnTrack. The Existing Process Group were subject to the credit card issuer’s standard collection process. The objective of the test was to evaluate whether BackOnTrack improved collections customers’ subsequent payment performance.

Approximately 6 months after the two groups were in collections, their cure rates were examined:

The acid test: BOT Invite group, inclusive of those who completed BackOnTrack (4.5% of the group) and those who did not (95.5% of the group), outperformed the bank’s existing collections process by 53 basis points. The source of the improvement is the performance of the BackOnTrack completers, as you can see in the table below.

Those who completed BackOnTrack were significantly better performers, with a much higher cure rate at 64.46%. Those who declined the invitation to complete BackOnTrack cured at a lower rate than the Existing Process Group, by 37 basis points.

These results allow us to determine what percentage of the BackOnTrack completers’ better payment performance was based on self-selection.

BackOnTrack: Separating Treatment Effect from Self-Selection Bias

Given that people who declined the invitation to complete BackOnTrack were worse payers than the Existing Process Group, it’s fair to say there is some level of self-selection bias. In other words, after removing the BOT Completers, what was left – the non-completers – were statistically significant worse payers (p-value 0.0409). So there was some element of self-selection there.

Since BOT Invite Group outperformed Existing Process Group in a statistically significant manner, we know that BackOnTrack changed payment behavior. We now determine what percentage of the BackOnTrack completers’ better payment performance was true treatment effect vs. self-selection by people who were going to pay anyway.

In the graphic below, you’ll see the calculation of the self-selection bias and the treatment effect.

The key calculation is the hypothetical BOT completer difference vs. the Existing Process Group cure rate (second box, in red) that makes the BOT Invite Group cure rate equal to that of the Existing Process Group. That 7.81% higher cure rate represents the self-selection bias, the better payment performance these customers would have done anyway.

Subtracting this self-selection bias from the BOT Completers’ better cure rate of leaves us with the treatment effect of BackOnTrack: +11.86% better cure rates. That represents 60% of the better performance seen for BOT Completers. So to answer the question: 60% of BOT Completers’ better payment performance was caused by BackOnTrack influencing their behavior. 40% of the better payment performance is through self-selection by better payers.

To find out how BackOnTrack will make your collections customers into better payers, click here to learn more.

Signup to be notified when we’ve posted a new article to the Scorenomics Blog!