Here’s a quick run down of my consulting project at Insight Data Science with the startup, OneSignal.
If you own a mobile app, you know that user engagement is key and push notifications are an important tool to improving engagement. What OneSignal offers is a platform that manages, sends, and tracks all of your app’s push notifications for all your users. They have an extensive SDK that allows apps to have greater control over their notifications. With OneSignal, apps can segment their user base and personalize their notifications so the right people see the right notifications minimizing the number of users getting unwanted messages. They also include features such as A/B testing of messages and real time tracking.
My project for OneSignal was to find the optimal times to send out these notifications for the purpose of converting the most users.
Conversion – When a user clicks on the notification to get into the app
Reengagement Notification – A notification specifically targeting an inactive user
Conversions Over Time
There is simply not enough space for One Signal to store all the details of every individual notification sent. However, I do have a majority of the information of each conversion made. I first visualized these conversions over the 24 hour day cycle by the genre of the app in use.
From here, we can see that different genres of apps can have wildly varying conversion patterns. Gaming apps have the most conversions at 8pm while news apps show two peaks of conversions, one in the morning (10am-12pm) and another in the afternoon at 5pm. If apps time their notifications at or preceding these times, they can better target and engage their users. Here are the peaks in number of conversions across the top 10 genres that OneSignal deals with:
- Games: 8pm
- Social: 10pm
- Education: 11am-1pm, 8pm
- Entertainment: 8pm-10pm
- Lifestyle: 12pm, 5pm, 9pm
- News: 10am-12pm, 5pm
- Business: 10am, 3pm, 7pm
- Travel: 9pm
- Health and Fitness: 12pm,
- Media and Video : 3pm, 7pm
Some things to keep in mind are that these peaks don’t correct for the pure number of notifications sent. For example, if the most notifications were sent at 8pm to gaming apps, you would naturally expect more conversions during that following hour. OneSignal doesn’t keep track of the results of all the notifications sent in general, so I wasn’t able to correct for this.
Tracking Re-engagement Chances
Re-engagement notifications are only a subset of the notifications I looked at in the previous section. They are especially important to track though as they are targeted at users going inactive in order to reduce churn and extend the lifetime value of each user. One big benefit too for focusing on re-engagement notifications, is OneSignal has more information on these notifications so I can create a more powerful model.
The main question OneSignal wanted me to answer was: “How does the timing of the notification influence conversion chances?” I smoothly modeled this relationship using kernel density estimation. Below, you can see the results of my model for gaming apps alone.
Re-engagement notifications sent around noon have the best conversion chance. However, in reality gaming apps send most of their notifications in the evening around 8pm. If OneSignal were to push these notifications earlier in the day, these gaming apps could retain an additional 10,000 users per week. Extending this to the other genres with a large enough pool of re-engagement notifications gave the following optimal push times:
- Games: 11am-1pm
- Social: 10am, 11pm
- Education: 4pm
- Entertainment: 10pm
- Travel: 3-7pm
- Health and Fitness: 9am
For these genres, I then tested my model performance based on how well I could predict if a conversion is successful or not. I measured this using an ROC curve.
The area under the curve (AUC) represents a model’s ability to distinguish true positives while minimizing false positives. My model has an AUC of 0.76 which is a lot better than the 0.5 that you would obtain from a random classifier.
I wanted to push my AUC metric higher and one topic I was particularly excited to tackle was user personalization. OneSignal has a lot of information of how each user interacts with their notifications, so I leveraged this. To accomplish this, I used Bayesian learning. When a user downloads a new social app, I have no information about their activity. I start with the prior assuming they act like the average social app user does.
Here you can see that most social users have the highest conversion chances in the morning and evening, which makes sense for a typical working adult. However, as a specific user receives more notifications, I slowly shift away from this prior towards a more personalized model. Take this one social user:
You can see that unlike the average user, this specific user was most likely to interact with their social app around 3pm. This user most likely was not a working adult, but instead a student getting out from school. With this user personalization, my model performance (AUC) was pushed from 0.76 to 0.86. OneSignal is excited about this and currently working on including this model in as a notification delivery option for their apps.
Model Details and Scalability
This Bayesian learning model essentially works as a weighted sum. I select how many observations to weight the prior (genre only based model). Then, as I accrue more and more real user observations, the model updates. If the prior is worth 10 observations and I have 10 user observations, then the final model is just an even average between the user observations and priors.
I chose this model structure particularly as it lends itself easily to scalability of including future observations. Once a single model is created for a user, I only need to continue tracking the “total number of observations” (number of user observations + weight of prior) that go into the model (and of course the model itself). For any new observations, this is the only number required to update the model.