How do others use Heap for A/B Testing?

  • 9 March 2022
  • 2 replies
  • 107 views

Userlevel 3
Badge +1

We’ve integrated Heap and VWO, our A/B testing platform, in order to get a more comprehensive and detailed look into A/B testing data. We’d sometimes lose cross-device data or multiple session tracking in VWO, maybe due to the length of our conversion funnel. So Heap fills in those gaps and is actually where we do most of our performance analysis and statistical significance checks.

 

What’s been most beneficial is to be able to dive into A/B test results based on all the customer attributes we have in Heap. We've had A/B tests where we found that they perform really well for organic traffic, but actually hurt paid traffic, or other tests where desktop and mobile are performing differently, or we can look at the customer source, or if they’re visiting different pages pre-conversion funnel, etc. etc. - a lot of deeper cuts. It's been super useful for us! Something that I struggled with in the past was that we could only group by two attributes, but I know that was just changed, so I think we can now group by up to four attributes, which is very exciting!

 

Curious to hear how others use Heap and VWO or other platforms to explore A/B testing data!


2 replies

Userlevel 3
Badge +2

Our main use case for Heap is A/B test analysis and so we use Google optimize to set up our tasks, the website and then Firebase for the app, and then we also trigger an event to tell us on the Heap side what the experiment is and then what the variant is. Very similar to how we send the data to Periscope data visualization tool. This is done currently via Redshift but we're slowly moving over to snowflake. And, there we are able to connect the data to internal data, so we've got our user segmentation that we can layer on top. And it helps us look at retention metrics, and we've also been able to set up dashboards so product managers can self-serve analysis. They can see how experiments are performing in terms of encouraging different feature engagement and browsing metrics as well.

Userlevel 2
Badge +1

We use Optimizely as our A/B testing tool and Heap is always a very critical part of our analysis, even though we do use back-end data as our source of truth to read experiment results. I find Heap so useful for quick, real-time answers. When we first launch a test, everyone wants to know how it’s going right away. Heap's a really easy way to share out results with people in all different parts of the organization. I'll usually create a dashboard that's very visual, in which it’s very easy to see the core conversion rates that we're tracking. I share that out and people really love that. It also helps me triangulate data from our back-end as well.

Reply