Predicting User Churn for a Digital Health App

Nadaa Taiyab

More Related Content

1.Predicting User Churn for a Digital Health AppGalvanize Data Science ImmersiveCapstone ProjectPresentation to Ayogo, Oct 2017Nadaa Taiyab
2.BACKGROUND• Investment Banking• International Development• Health Tech Startups (Finance & Operations)• Health Coaching
3.DIABESTIES APP• IPhone app built by Ayogo in partnershipwith the College Diabetes Network
4.P• Track glucose, insulin,carbs, and mood• Share data with your“diabestie”• Export data
5.PROJECT:PREDICT USER CHURNUse machine learningalgorithms to predict whichusers would stop using theapp
6.WHY CHURN MODELING?• Increase persistence by intervening with users atrisk of churn (eg. phone call, email, reminder,etc.)• Reduce cost and improve effectiveness ofinterventions• Instructive for future app design and predictivemodeling
7.TECHNOLOGIESseaborn
8.DATA• MySQL Database• 3 years of data (2012-2015)
9.3080Users51,125Logs387,188ClicksUser Data(Demographics)BehavioralData
10.User DataWhat can we learn about the users?
11.30% were college age (18-25), but the average age was37.
12.51% had Type 1 Diabetes and 42% had Type IIDiabetes.
13.58% of participants werefemale
14.50% would not specify ethnicity, so hard to draw firmconclusions,but likely that majority were caucasian. and living in US.
15.DEMOGRAPHICS• Many of actual participants differed fromintended target of college students with Type 1Diabetes• Huge age range, including a lot of people in latethirties• Ethnicity and location inconclusive, but likelymajority caucasian in the US
16.BehavioralDataWhat can we learn about how users interactedwith the app?
17.Used mainly as a glucose tracker.Insulin and mood tracked 30% of thetime.97% of entries includedglucose log
18.Many users tried the app a few times and quit.A few users logged hundreds of times over 2-3 years.85% entered data 0-10times15% entered more than 10times
19.23% of users were active more than 30 days, though only 15%logged more than 10 times, indicating stop-start usage pattern.
20.DIABESTIE EFFECT• Only 3% of users had a diabestie• Of users with a diabestie, 44% used the appfor more than a week• Diabestie didn’t seem to affect churn
21.BEHAVIORAL DATA• Mainly used a glucose tracker• Small core of 10-15% of users that are persistent• Diabestie no effect
22.Churn ModelCan we use machine learningalgorithms to predict churn?
23.FEATURE ENGINEERING• Behavioral Data• # page views• # total entries• # moods,• # notes• # A1C updates• User Data• age, gender,ethnicity,education• diabetes type• diabestie: yes / no
24.CHURN DEFINITIONUsers who completed less than tenadditional entries after the first week ofuse
25.CLASSIFIER MODELS• Logistic Regression• Random Forest• Gradient Boosted Trees• AdaBoost
26.Gradient Boosted Trees had highest Area Under the Curve(AUC), indicating that it had the most predictive powerTrue Positive Rate (Sensitivity) = True Positives / (True Positives + False Negatives)False Positive Rate = 1 - (True Negatives / (True Negatives + False Positives))
27.MODEL SCORESLogisticRegressionRandomForestGradientBoost AdaBoostAccuracy 92% 92% 94% 93%Precision 93% 95% 95% 95%Recall 98% 96% 98% 97%AUC = Area Under theCurveAccuracy = (True Positives + True Negatives) / Total SampleSizeRecall or Sensitivity = True Positive / (True Positive + FalseNegative)Precision = True Positives / (True Positive + FalsePositive)
28.CONFUSION MATRIXn= 770 PredictedChurnPredictedNot ChurnActualChurnTruePositive692FalseNegative16708ActualNot ChurnFalsePositive34TrueNegative2862726 44Results from Gradient BoostedTrees
29.Behavioral features had greater predictive power thandemographicsexcept for age.Feature importance scores calculatedusingRandom Forest algorithm
30.TAKE AWAYS• How an app is used can evolve in unexpected ways• Behavioral data has more predictive power than demographics• Improve Data Quality• Structured data for location (maybe for a1c and glucose too)• Safeguards against data entry errors• Timestamp data important• Better app usage data• Possibly improve model by only including users that made at least one entry
31.Nadaa Taiyabnadaa.taiyab@gmail.comgithub.com/nadaataiyab/diabesties