Churn Prediction: Which Accounts Need a Call?
A customer success team opens its Monday queue and sees fifty accounts marked “likely to churn.” One account has already submitted a cancellation. Another has stopped using the product but still pays. A third has an upcoming renewal and no recent support request. If all three receive the same urgent call, the prediction has done little to guide the work. The team needs to know what outcome the alert refers to, whether there is still time to act, and what the proposed action could change.
Churn prediction estimates which current customers may leave within a defined future period. IBM describes customer churn as the loss of existing customers over a period of time; an operating prediction needs a narrower definition than that general description (IBM). The useful question is not simply “Who has a high score?” It is “Which eligible customer might reach a specific loss outcome soon enough for a specific response to matter?” That wording forces the model and the retention team to work on the same decision.
I would begin with one outcome, one decision date, and one action the organization can actually deliver. Only then would I ask whether a model improves the queue. This approach costs some apparent breadth: a first model for subscription cancellation will not explain every form of disengagement. It gives the score a meaning that a customer success manager can check and use.
Define the loss before predicting it
“Churn” can mean a recorded cancellation, a relationship ending at renewal, or an account becoming inactive. Cancellation and inactivity are different outcomes in the supplied definition of the problem; silently combining them produces a label with competing meanings (IBM). An account can be inactive while still contracted, or active shortly before it cancels. The response to each case may be different. A cancellation prediction may prompt a conversation about renewal terms or an unresolved issue. An inactivity prediction may point to onboarding or product use. Combining both into one positive label makes it harder to tell what a high score asks anyone to do.
For a first pass, imagine a subscription business with account managers who review active accounts every Monday. The following numbers are illustrative, not results from a company: use activity from the previous 30 days to predict a recorded cancellation in the next 30 days. The account must still be active at the Monday decision point. If a cancellation request was already recorded before Monday, the account belongs in a cancellation workflow, not in a prediction queue. If the organization can only contact accounts once a week, a Monday score is a natural unit of work. If it can act continuously, a different scoring schedule may fit.
The unit matters as much as the event. A person who logs in, a billing account, and a contract are not necessarily interchangeable records. If one contract contains several users, user inactivity should not quietly become “customer churn” while the outcome is measured at contract level. Pick the record that can both receive an action and produce the stated outcome. Say what happens when an account has several contracts, changes ownership, pauses service, or later returns. These are definition choices, not model tuning. A team that cannot agree on them cannot interpret its own alert count.
The time horizon should match the action. A next-day cancellation may be too late for a relationship repair that requires a meeting, while a six-month warning may be too uncertain to justify an urgent call. Ask how long it takes to notice an issue, reach the customer, and offer a credible remedy. Then set the prediction window so the team can act before the event. Keep the outcome window fixed when comparing model versions; otherwise a change in reported performance may reflect a different question.
The loss measure also changes the ranking. A binary cancellation label treats the loss of a small account and a large account as the same event. That is appropriate if the purpose is to estimate cancellation probability. It is incomplete if the purpose is to allocate scarce retention time. Account value and the cost of an intervention belong in the later decision, with their own assumptions. A model that predicts cancellation well has not automatically learned the financial value of saving each account.
Build each historical record as the team would have seen it
Once the outcome is clear, create a series of historical decision points. At each point, record only information available to the team then, and attach the outcome from the declared future window after that window has ended. This creates a fair version of Monday’s question for earlier Mondays. It also makes the model’s predictions inspectable: for any account, a reviewer can identify what was known, what was predicted, and what later happened.
Availability is stricter than a database timestamp. Suppose a support case was entered on Tuesday but later backdated to Monday. The Monday prediction could not have used it. Suppose a cancellation reason is added when the customer leaves. That reason may be valuable for learning about departures, but it cannot be an input to an earlier alert. Scikit-learn calls the use of information unavailable at prediction time data leakage and warns that it makes offline performance look too optimistic (scikit-learn). For each proposed input, ask the practical question: could the person making the Monday decision have seen this value on Monday?
Candidate inputs should be chosen for that decision, not because a column happens to exist. A team might consider recent product activity, open service requests, billing status, time since signup, or the date of a contract decision. These are possible fields for the illustrative subscription account, not universal churn indicators. Their meaning depends on the label. A count of recent logins could be relevant to inactivity; a renewal date could matter to cancellation timing. A field created only after cancellation is unusable for either prediction. Missing data needs the same care. “No support case” and “support cases were not recorded for this account” are different situations.
Historical records also need separation by time. A model fitted on later outcomes should not be judged on earlier periods whose patterns it has already absorbed. Hold back a later block of decision points, let their outcome windows finish, and compare predictions with what subsequently occurred. If the same account appears on several Mondays, keep that structure visible. Several alerts for one account do not represent several independent customers, and a retention team may need one current case rather than repeated calls. The exact split depends on how often scoring runs, but the test should resemble the future operation it is meant to represent.
I would inspect a sample of historical records before fitting anything complex. For each record, can a colleague reconstruct the decision date, eligibility rule, input values, eventual outcome, and any action the account received? If not, a sophisticated model will only hide the ambiguity. Start with a simple ranking or rule as a comparison point. A new model earns its place by improving the actual queue under the same label, dates, and capacity. A better looking score on a differently constructed dataset is not a useful comparison.
Existing retention work complicates the history. An account that looked likely to leave may have stayed because someone intervened. Its observed “did not churn” label does not reveal what would have happened without that action. Conversely, accounts ignored in the past may differ from those contacted. Record past contacts and offers where possible, and avoid describing historical outcomes as an untouched picture of customer behavior. This is one reason a prediction score and a retention decision must remain separate.
Read a score as a forecast, not a reason or an order
A model may rank accounts from more to less likely to reach the chosen outcome. It may also output a probability. These are related but distinct uses. If a model assigns 0.40 to a group of accounts, a calibrated interpretation means roughly 40% of comparable accounts in that group reach the specified outcome over the specified period. Scikit-learn describes calibration as the agreement between predicted probabilities and observed event frequencies (scikit-learn). A score can rank accounts usefully without being reliable as a literal 40% forecast; check which claim the model can support before putting percentages in front of staff.
Calibration must be checked on accounts whose outcomes are known, using the same label and horizon as the live score. Group predictions into ranges, compare each group’s average predicted probability with its observed cancellation rate, and look for material gaps. Small groups can give unstable comparisons, so the apparent precision of a two-decimal score should not dictate how finely the team acts. After the account population or model changes, the earlier calibration may no longer hold (scikit-learn). A score shown as “probability of cancellation” needs periodic checks against newly matured outcomes.
A probability says nothing by itself about why one account may leave. A low activity count may be useful for prediction while remaining ambiguous in a conversation: perhaps the product is no longer needed, perhaps the work is seasonal, or perhaps the recorded activity misses the real user. Those are possibilities to investigate, not model findings. Staff should see the relevant, available account facts and a sensible next step, while treating the score as a prompt to ask rather than a script that claims to know the customer’s motive.
Turning a score into a yes-or-no alert requires a threshold. Google’s classification guidance explains that a threshold determines which scores receive a positive prediction; changing it changes who is included and the balance between missed cases and false alerts (Google for Developers). There is no special virtue in choosing 0.50 for every churn queue. The right cutoff depends on the number of accounts a team can serve, the cost of contacting customers who would stay, the cost of missing customers who leave, and the response available. Those are operating choices, not properties the model settles on its own.
A threshold is also different from a probability. If a customer’s estimated cancellation probability is 0.40 and the alert cutoff is 0.30, the 0.40 is still the estimate and 0.30 is the rule for placing that customer in the queue. Lowering the cutoff adds accounts to the queue. It does not make any individual customer more likely to cancel, nor does it explain the cause of cancellation. A team should be able to change the cutoff without retraining the model when its weekly capacity changes, while retaining the model version and cutoff used for each historical decision.
Judge the queue with the errors the team will actually face
Precision and recall answer different questions. Precision is the share of alerted accounts that actually reach the defined churn outcome; recall is the share of all accounts that reach that outcome that the alerts caught (Google for Developers). Precision describes how much work in the alert queue points to eventual churn. Recall describes how much eventual churn the queue covers. Neither tells us whether a call would prevent the outcome.
Consider an entirely illustrative completed month with 1,000 eligible accounts. Suppose 100 cancel. If a cutoff alerts 200 accounts and 60 of those cancel, precision is 60 out of 200, or 30%. Recall is 60 out of the 100 cancellations, or 60%. The queue also contains 140 accounts that do not cancel, and it misses 40 that do. These are the concrete consequences of the threshold. A report that says only “60% recall” conceals the 140 conversations the team may have to sort through.
Now suppose the team can serve only the 100 highest ranked accounts, and 45 of those later cancel. Precision becomes 45 out of 100, or 45%, while recall becomes 45 out of 100, also 45%. The narrower queue makes better use of each contact if the sole criterion is finding future cancellations, but it leaves more cancellations outside the queue. The extra 100 alerts from the wider queue would contain 15 additional future cancellations. Whether those 15 are worth another 100 contacts depends on the effort, possible customer annoyance, and what the team can offer. These illustrative numbers describe prediction outcomes, not saved customers.
Report counts alongside rates. A team of ten account managers needs the number of accounts per week, not just a percentage. Report how many eligible accounts were scored, how many were flagged, how many flagged accounts later churned, how many churners were missed, and how long the team had before the outcome. Check the same figures for the account groups the team handles differently. If a cutoff fills the queue with accounts whose decisions were already effectively final, a respectable precision figure will still produce poor work.
Do not score a model on accounts whose outcome period has not finished. A customer that has not canceled after ten days is not yet a confirmed non-churner for a 30-day prediction. Mixing open outcomes with completed ones can make recent performance seem better or worse for reasons unrelated to the model. It is more useful to accept a reporting delay and compare complete windows. A current queue can still be monitored for operational facts such as contacts made; its prediction quality must wait until the promised horizon passes.
Choose a response separately from the risk ranking
The largest mistake in using churn prediction is to treat the highest risk customers as automatically the best targets for a retention offer. Risk asks what may happen under the circumstances represented in the data. An intervention decision asks what would change if the business made a call, fixed a problem, or offered a concession. Research on churn retention distinguishes an intervention’s incremental effect from predicted churn risk (Information Sciences study). That distinction matters even when the model’s probability is well calibrated.
Imagine two illustrative accounts with the same predicted cancellation risk. One has an unresolved service issue that the team can fix. The other has decided to close its business. The same risk score would not justify the same retention spend. The first may have a remedy; the second may not. These invented cases are not claims about a real customer base. They show why a score alone cannot say who is persuadable or which action is worthwhile. The account’s value, the expected effect of a particular action, the cost of that action, and the chance of an unwelcome contact all matter to the decision.
I would make the initial alert lead to a low-cost, information-seeking response: review the account, confirm whether the customer has an open issue, and contact the customer only when the contact has a plausible purpose. I would not automatically attach a discount to every high score. A concession offered to someone who would have stayed anyway is a cost without a saved cancellation. A concession to someone who cannot or will not stay is also a cost. A helpful service repair and a price offer are different actions; each needs its own case.
For larger or more intrusive interventions, compare outcomes for eligible customers who receive the action with outcomes for comparable eligible customers who do not. If feasible, assign the action at random within a clearly defined eligible group, retain a comparison group, and use the same outcome window for both. For an illustration, if 20 of 100 untreated eligible accounts cancel and 14 of 100 treated eligible accounts cancel, the observed difference is six cancellations per 100 accounts in that test. It is not proof that each treated account was saved, nor a guarantee of the same result elsewhere. The team would still need to examine uncertainty, costs, customer experience, and whether the groups and treatment match the future rollout.
The test should answer the operating question the business has. Testing a courtesy call does not establish the value of a discount. Testing on large accounts alone does not establish the effect for new small accounts. Changing the offer, eligibility rule, or outcome period changes the question. When a good comparison is unavailable, say that the intervention effect is unknown. Continue using risk to organize investigation if it helps, but do not convert predicted churn probability into a promised number of retained customers.
Put the prediction into a workflow that can learn
A workable queue needs more than a score column. Each alert should carry the account identifier, the exact predicted outcome and horizon, the date and model version, the threshold or capacity rule, and the account facts that a person may use in a conversation. The workflow should show whether the account is already in a cancellation process and whether another colleague is handling it. It should let staff record what they did, when, and why. Without those details, a later review cannot tell whether the model missed the customer, the team lacked time, or the chosen response failed.
Assign ownership at the same level as the prediction. If the model scores contracts, a contract owner should receive the alert. If one customer has several contracts, the queue should make the relationship visible so that repeated alerts do not trigger several uncoordinated contacts. An account manager should be able to dismiss an alert for a recorded reason, such as an already resolved issue, while the underlying score remains available for later analysis. Human judgment can improve the action without erasing the history of the model’s recommendation.
Start with a modest weekly capacity and a defined action. If the team can review 100 accounts, rank eligible accounts and inspect the top 100 under the current rules. Keep an uncontacted comparison where appropriate for learning about the action, and avoid letting one person’s ad hoc preference silently change eligibility. After the outcome window closes, compare the queue with eventual cancellations and the action groups with their comparison groups. Prediction quality and action effect are separate results. Both matter, but one cannot substitute for the other.
As the queue runs, look for two kinds of change. First, have the scores become less consistent with subsequent outcomes? Calibration can shift as a model or population changes (scikit-learn). Second, is the queue still operationally useful? A model may retain its ranking while the team loses the ability to contact customers in time, or while a new policy makes a former remedy unavailable. Review alert volume, maturity of outcomes, time before cancellation, contact completion, and the actual response. Update a model only after identifying what changed; changing a threshold may solve a capacity problem without changing the forecast.
The best first deployment may be a small, disciplined alert list rather than an elaborate prediction platform. Define the loss, reconstruct the decision date, compare a simple baseline with a model, choose a queue size the team can handle, and keep the response measurable. Expand to other outcomes, such as inactivity, only when each has its own label, timing, and possible action. A churn prediction is useful when it helps a person reach the right customer with a credible reason while there is still time to help—and when the business can tell whether that effort made a difference.