Churn Modelling on Synthetic Data
The useful question isn't whether a model can predict churn, but how many false alarms you'll pay for to catch a churner.
Synthetic dataset. These results describe a generated dataset, not real customers.
Problem
Problem
A retention team can only contact so many customers. A churn model is only useful if its operating point matches that capacity: how many genuine churners it catches against how many loyal customers it flags by mistake.
Constraints
Constraints
- Synthetic data
- 100,000 generated telecom customers. Any pattern found was put there by whoever generated the data.
- Imbalanced classes
- 33% of rows are churners.
- Interpretability
- The point was to understand the trade-off, not to win a leaderboard.
Architecture
Architecture
Synthetic data100,000 rows
Input A generated dataset: its relationships come from whoever generated it, not from real customers.
Encodeone-hot, 11 cols
Deterministic step Categorical fields one-hot encoded; the ID dropped.
Split80/20 stratified
Deterministic step A stratified split keeps the 33% churn rate in both sets.
Modelweighted logit
Deterministic step Class-weighted logistic regression, chosen for interpretability.
Threshold0.5 vs 0.4
Stored data Compares the cost of missed churners against false alarms.
- Input
- Deterministic step
- Stored data
The modelling pipeline as built.
Key decisions
Key decisions
Logistic regression with class weights
Class-balanced logistic regression on one-hot encoded features, with a stratified 80/20 split.
Show reasoning
- Context
- One in three rows is a churner, and the drivers need to be readable.
- Options considered
- Tree ensembles · plain logistic regression · class-balanced logistic regression
- Trade-off accepted
- Likely leaves accuracy on the table compared with more flexible models, which weren't tried.
Treat the threshold as a business setting
Comparing 0.5 and 0.4: recall rises from 70% to 82% while precision falls from 52% to 48%.
Show reasoning
- Context
- The default 0.5 cut-off is a convention, not a decision.
- Options considered
- Report 0.5 only · compare operating points
- Trade-off accepted
- The threshold was compared on the test set itself; a separate validation split would be cleaner.
Implementation
Implementation
- pandas encoding, a scikit-learn stratified split, and class-weighted logistic regression.
- Accuracy, precision, recall, F1 and ROC AUC at two thresholds, plus coefficient inspection.
Validation
Validation
- Hold-out
- 20% stratified test split: 20,000 rows.
- Disclosed
- The dataset is synthetic, and that's stated wherever this project appears.
Results
Results
- 70% → 82%recall when the decision threshold drops from 0.5 to 0.4Dataset finding
- 0.77ROC AUC at the default threshold (accuracy 68.8%)Dataset finding
- Contracttype dominates the model; month-to-month carries the highest riskDataset finding
- Dataset finding At 0.5: accuracy 68.8%, precision 52.2%, recall 70.1%, ROC AUC 0.77.
- Dataset finding At 0.4: recall 81.6% and precision 48.0%; accuracy drops to 64.6%.
- Dataset finding Contract type dominates: one- and two-year contracts carry the largest negative coefficients, so month-to-month is the highest-risk group.
- threshold 0.5
- threshold 0.4
Source: notebook.ipynb, cells 6–7.
Show the data as a table
| Metric | threshold 0.5 | threshold 0.4 |
|---|---|---|
| Recall | 70.1% | 81.6% |
| Precision | 52.2% | 48% |
| Accuracy | 68.8% | 64.6% |
Limitations
Limitations
- Synthetic data: the “drivers” are properties of the generator.
- Features weren't scaled, so coefficient sizes aren't comparable across features, and the solver raised numerical warnings.
- No baseline or alternative model was compared.
Lessons
Lessons
- Model metrics only matter at an operating point someone has chosen.
- Know where your data came from before you believe what it says.
What I'd do differently
- Scale features, pick the threshold on a validation split, and compare against a simple baseline.
Where the numbers come from
- 70% → 82% recall when the decision threshold drops from 0.5 to 0.4. 20% stratified hold-out (20,000 rows); precision falls from 52% to 48%. Source: customer-churn-prediction/notebook.ipynb (cells 6–7)
- 0.77 ROC AUC at the default threshold (accuracy 68.8%). Same hold-out; 33% of rows are churners. Source: notebook.ipynb (cells 2, 6)
- Contract type dominates the model; month-to-month carries the highest risk. Largest coefficients: one-year −1.72, two-year −1.71 vs month-to-month. Source: notebook.ipynb (cell 8)