Impact evaluation allows you to monitor how effectively a segment is converting in comparison to all other users. It also provides an estimate of potential revenue and the impact that a change could have on your earnings.
This is the first insight that you can see regarding your segment performance!
Note that when the results are not statically significant, the difference in revenue is still displayed, but the tooltip will explain: "We are less than 99% certain the difference in conversion rates is not due to either segment being too small or the conversion rates being too close".
Correlation does not mean causation. To avoid false positives, Contentsquare assesses the reliability of the impacts that are calculated. This is called “Statistical significance”.
“Statistical significant” means that we are 99% confident that the observed difference is not due to coincidence. Otherwise, the calculation is “‘not significant”.
To determine if Opportunities results are statistically significant, we check the population size: If at least one of the populations compared is too small, (i.e. less than 30 sessions), the calculation is "not significant".
How to calculate an impact
1. Set up your Analysis context
Open the Analysis Context and choose your first segment and then click Compare to choose your second segment.
2. Define a goal
What's your analysis goal? Define it clearly to measure success and assess its impact on conversion.
Once the Analysis context and goal are defined, you'll start uncovering initial insights such as:
- The extent of the impact.
- The magnitude of the impact on conversion (measured in missed conversions).
- The potential increase in conversions if a certain percentage of one segment converted similarly to the other.
To delve deeper, you can quantify the impact on revenue and refine the improvement scenario by adjusting the impact calculation (refer to step 3)
3. Edit Impact Calculation
Decide on a value for each conversion to estimate revenue impact.
By assigning a value to each conversion related to your goal, you can calculate missed or additional revenue based on this value.
Whether your e-commerce tag is enabled or not, you can visualize and adjust the value per conversion to accurately measure the impact on revenue:
(For e-commerce goals) How to define the value per conversion
|
Select between median cart (default) and average cart For your chosen goal, you have the option to select:
Both values are automatically calculated based on the segment that performed better. Once you've made your selection, simply click Apply, and your missed or additional revenue will be calculated accordingly based on the defined value.
|
(For non-e-commerce goals) How to define the value per conversion
|
Select a custom value You have the option to define a custom value for each conversion related to your selected goal. This value will be saved with your goal and updated for all users. Once you've made your selection, simply click Apply, and your missed or additional revenue will be calculated based on the value you define.
|
Adjust the improvement scenario.
You can adjust the detected opportunity by using different improvement scenarios.
- Improvement scenarios
When using Opportunity in Comparison mode, you have the ability to define an improvement scenario. They allow you to predict how much money you would have earned if your worst performing segment
converted as well as your best-performing segment (the one you compare it to).
E.g. If X% of segment B sessions converted as well as segment A sessions, you would earn X amount of additional revenue.
Choosing an improvement scenario allows you to take into account different levels of impact.
E.g. it is feasible to completely fix an error but not feasible to fully eliminate all bounce users.
They enable you to get a more accurate estimate of the ROI opportunity
Improvement Scenario percentage quick guide
- 100% (Default)
Calculation: If SEGMENT A outperforms SEGMENT B, 100% of the traffic SEGMENT B X (CR of SEGMENT B - CR of SEGMENT A) X Median Cart of SEGMENT A
Use cases: Recommended for very impactful changes that could unblock user journeys
E.g., Fixing an API error on a payment page, fixing a wrong redirect (e.g. 404 page), fixing a JS error in a form
- 50% (High)
Calculation: If SEGMENT A outperforms SEGMENT B, 50% of the traffic SEGMENT B X (CR of SEGMENT B - CR of SEGMENT A) X Median Cart of SEGMENT A
Use cases: Recommended for high-impact changes like structural changes
E.g., Adding automatic suggestions for the address field, Adding a "Login as a guest" option or easy login with Google, Adding breadcrumbs to help users easily navigate between product categories
- 25% (Medium)
Calculation: If SEGMENT A outperforms SEGMENT B, 25% of the traffic SEGMENT B X (CR of SEGMENT B - CR of SEGMENT A) X Median Cart of SEGMENT A
Use cases: Recommended for medium impact changes like the change of the wording of the CTA
E.g., Adding a cart summary at each step of the funnel, Adding an error notification in form submission, Adding breadcrumbs in Checkout funnels for easy navigation back and forth, Moving main CTAs (Add to Basket, Add to Cart, Color, price, and size selectors) above the fold line, Making the search bar more visible
- 10% (Low)
Calculation: If SEGMENT A outperforms SEGMENT B, 10% of the traffic SEGMENT B X (CR of SEGMENT B - CR of SEGMENT A) X Median Cart of SEGMENT A
Use cases: Recommended for low-impact changes like changing a color/ design of a CTA
E.g., Changing color to help the user identify the ‘Add to cart' CTA, Allowing users to apply all filter options at the same time.
Use cases
- Monitor for sudden dips after implementing changes to your app, add widgets to your Dashboard to track performance at a glance.