Defining the Responsibilities of Our Product Analysis Function

  • 28 February 2022
  • 3 replies
  • 98 views

Userlevel 1
Badge +1

My company has grown very rapidly in the last couple years and as the demands on the product team have grown significantly, the value around adding the Product Analysis function also grew.  As more and more new team members joined, we needed to change the structure of our team to be more data-driven.  Out of this need, we defined four guiding principles for this new Product Analysis function:

Own Instrumentation Suite - Setup and Maintenance

Heap, Optimizely, Google Analytics, A/B testing, and other measurement tools all fall under this new function. Previously, these applications were managed by different teams. These teams were unaware if we launched new pages or changed URLs, and that led to data quickly becoming outdated. It made sense for these to all be centrally owned by someone with deep product knowledge, so that we could ensure the data was current. 

Provide Product-Level Insights on Usage

This function is all about knowing the customer. Where are they falling out of the funnel? What are the bounce rates? How can we improve engagement with our products? The Product Analysis function helps us see what the customers see. This includes identifying how we can improve the loan application to provide the best and fastest experiance for our customers. 

Guide, Monitor, and Design Experiments

The drop off points identified in our Heap funnels, are turned into opportunities to work with Product Managers and Designers to create experiments that solve the root cause of that dropoff. Running A/B tests is a big way we validate these hypotheses, and although we want every A/B test to “win” we know that many will not, but can still provide great insights. 

Reporting

As we’ve grown, our metrics have evolved from being project-based to product-driven based on our customers’ needs. The Product Analyst ensures our OKRs are on track, guiding us at biweekly meetings on how we’re progressing on those goals and also sharing what we have learned from our experiments.

This is what worked for us, but I always love learning from others. Please add your thoughts in the comments below on what you’ve done to build this function on your product team.


3 replies

Userlevel 4
Badge +3

I love this post @PhillipD ! Would you be willing to share a little bit more about how you convinced your organization to invest in a distinct Product Analysis function? It’s so much easier to make analytics work when you have a dedicated owner and it would be great to learn more about how your org made the case to senior leaders/executives to make the investment. 

Userlevel 1
Badge +1

Great structure - I really appreciate the detailed information. Thank you. 

Badge +1

Thank you @PhillipD for sharing. I would like to understand the structure of your “Setup and Maintenance” team. Is this one person or a dedicated function of multiple folks? What is the ideal profile of these folks, and how are you holding them accountable to using data governance best practices? I’d recommend checking out these product experimentation templates as well to see how these compare to your existing templates and processes: https://heap.io/resources/ebooks-whitepapers/3-templates-for-uncomplicating-product-management

I would say additively to what you have described here that we recommend to our customers is digging in more on how you are defining and updating your core metrics. Every product launch and experiment then can be influenced and tied to how those are impacting the bigger picture - your core metrics of success. Here is a great whitepaper we published about how to define those core metrics: https://heap.io/resources/ebooks-whitepapers/5-core-metrics-framework

Great work overall, I love the framing and way you have structured your team to ensure data driven decision making at all levels of the organization!

Reply