You've just launched a new release on my website and you want to check its performance without using an A/B Test. To do so, follow the 3 steps below:

Before starting your analysis, make sure you answer these questions and proceed to step 2:

• What changed? Is this...
• New feature (Chatbot? Quickview for PDP? ...)
• New design (CTA color? Size? Position? Hover effect? ...)
• New path or navigation (Checkout in one-page rather than 3? New menu? ...)
• New content (More information in PDP? New images? ...)
• ...
• Which device is impacted? (Desktop, Mobile, Tablet, App)
• When did your release occur? (Time period)
• Where can you find the new releases? (On a specific page? On all pages?)
• Who is targeted? (All? Returning? New visitors? ...)

Objective(s) of the Release

To run a thoughtful analysis, you need to clarify the objective of your release.

To do so, think about:

Purpose of change

Which insights lead to this change? Is it data, bug, intuition, market trend, ...?

Results expected

Which hypotheses were made? Example: “By adding a quick view CTA on list page, I expect my add to cart to increase”

Expected impacted metrics

• Main metrics (Example: “Exit rate”)
• Secondary metrics (Example: “Click rate, hesitation, number of page views, add to cart, time spent, performance…”)

Now that your objective is clear, you are now ready to start your analysis.

Impact & Performance of the Release

Note: To measure the impact of your new release, you must have a sufficient volume of visitors (ideally 10K). In addition, you should keep in mind that any release implies an adaptation phase.

⇨ The table below provides some tips to run your analysis based on a before/after comparison using different metrics.

Comparison Before* / After the Release

* For an optimal analysis of the release performance, it is best use to have a version of your page(s) before the release so you can compare better its impact.

Conclusion & Iteration

Compare good and bad behavior

Is there a huge gap between two KPIs? Are your hypothesis confirmed?

Are you satisfied with the results?

If you’re not, try to understand which KPI is underperforming

Once you’ve identified it, create 2 segments based on:

• the good behavior (e.g. 'clicked on X', 'viewed X page')
• the bad behavior (e.g. 'didn’t click on X', 'didn’t view X page')

Once you have them, go through the different modules comparing your good and bad segments to identify the pain points.

Note: you can use the Session Replay with the bad behavior segment to visualize the anomaly in action.

Going further

To dive deeper in your analysis, you can use:

• Performance: set Alerts to follow the main KPIs of your release (bounce rate, click rate…) and a dashboard to monitor these over time.
• CS Live: quickly check how your release is performing in time through the different metrics.
• A/B Test Analysis: check which version performs better and make a data-driven choice on your final version.

Check Contentsquare Methodology which helps guide you through starting your analysis and provides the tools to establish an analytic scope which is easily applicable and repeatable.

One change at a time

It’s hard to evaluate the performance of one specific feature when it’s mixed with many changes. To have an understanding of the impact of one specific release, you might consider launching the releases one by one.