Skip to main content
A fixed share of every app’s users, 10% by default, is in the holdout. They receive no notifications from PushNotifAI, from flows or broadcasts. That is permanent, and it’s on purpose.

​
Why

The usual push dashboard says “notifications drove 40,000 opens.” Most of those people would have opened the app anyway. The notification just got the credit. Some programs even make things worse: notification opens go up while people open the app on their own less, and the attribution numbers look great the whole time. Comparing the users who got notifications with the users who didn’t is the only way to see whether your notifications cause more use, or just claim it.

​
How users are assigned

Each user is placed by a stable hash of the user id you pass to identify. The same user is always in the same group: reinstalling, switching devices, or changing your flows never moves anyone in or out.

​
What it costs, and what to expect

The holdout gives up notifications to 10% of users. In return you get the only trustworthy answer to “is this working?”. With modest traffic, small improvements take a long time to show up. Early on, the holdout’s main job is catching harm: a program that quietly reduces how often people come back. The dashboard and the Composer both say “not yet significant” rather than draw a line that isn’t there.