We finally connected our testing framework to Heap

  • 24 February 2022
  • 0 replies
  • 49 views

Userlevel 5
Badge +4

We’re 100% mobile. We make it easy and affordable for migrants, typically members of various African and Asian diasporas, to send money to their loved ones back in their home countries.  

As much as possible, we try to keep our app simple. This means all the juicy bits related to onboarding flow and transaction flow are generally controlled server side to address compliance issues. We have a pretty lightweight (homegrown I’ll add) server side a/b testing framework. This framework allows us to add users into experimental groups and then, based on their group memberships, have the apps treat them differently. Till now we’ve always analyzed the outcomes of the tests by using our standard BI tool (Sisense Periscope) and just looking at test outcomes (i.e. final number of conversions by cohort).

This past month we finally connected our testing framework to Heap so we could better understand what was happening in-app during the test. Not just what happened, but also how it happened.  

How did we do it? When our testing framework assigns a user into an experimental group, it now sends a [subscribed to experiment] event to Heap for that user. The event contains all the custom properties we need to understand what test it is as well as which cohort the user was made a part of. We then defined segments in heap based on [users who have done] [subscribe to experiment X group Y] where the latter is a custom event defined using the properties of the initial [subscribed to experiment] event. And voila, we now have the experimental segments mapped in Heap and can do all of our usual conversion, retention etc. type analysis with them. If there’s better way to accomplish this I’d love to learn, but for now this works for us


0 replies

Be the first to reply!

Reply