Features used: Goals, Segments, Impact Quantification
Time: 15 minutes
The success of your testing program depends on defining not only the right things to test but also the right things to measure. Is it clicks? Is it transactions or revenue? How should you decide? And how can you align them with the overall business objectives?
Key principles when developing your KPIs
Ensure that your metrics are:
- Simple: Easily understood by all stakeholders
- Measurable: Even in the online environment, not all effects are easy to measure. Post-purchase satisfaction or generating cross-channel leads (e.g., online to brick-and-mortar stores' sales) can be hard to measure.
- Attributable: To measure the success of your test, you must be able to assign actual values to your testing segments. For example, if you want to analyze if the Treatment is causing higher app crashes or higher loading times, you must be able to attribute the issue to its variant.
- Sensitive and timely: Your metric should be able to capture changes that happen in a timely fashion. An example of an insensitive metric is trying to make a change to the stock price of your company by running a test with improved product placements.
Techniques for developing metrics
- Use hypotheses to generate ideas and validate them through a robust data analyses.
- Remember that metrics are proxies - always come back to the reason you created your test and link this back to the objective of your page. Is your page's objective to lead more people into the conversion funnel, add to cart or be used as a navigation tool in your sire? And what do you want to achieve with your test?
Use case: You want to improve the navigation capabilities of your Homepage and you are thinking of rearranging your menu. Success metric here could be click rate on the new element but also reach rate of the next step of the funnel.
- Think of edge/niche cases. Each metric has a corresponding set of failure cases.
Use case: As a travel site you are heavily reliant on your search engine. You want to increase the engagement and your success metric is click-through rate. By only looking at CTR you might miss usability issues your visitors are encountering (e.g., seeing irrelevant results). Hence, you must look into additional metrics to account for such edge cases (e.g., the rate of reaching a designated follow-up page).
- Sometimes is easier to measure what you don't want to achieve, such as frustration or disappointment, than what you really want to accomplish. A decrease of a negative metric (e.g.,bounce rate) can be used as a measure of user satisfaction.
Setting up testing KPIs in Contentsquare
The goals are either a click on a Zone or viewing a certain page/objective. To assess their performance:
- Create and favor the relevant page goals (e.g., Reached Product Page, clicked 'Add to Bag' CTA, launched a search).
-
Apply them in the different CS modules (e.g.,Impact quantification, Zoning, Page comparator ) to see the percentage of visitors achieving this goal and identify potential factors that are driving these numbers up or down.
Create segments and checking data
- Create one segment for each variant with dynamic variables.
- Check that visitors number match in Impact Quantification.
Go further
Learn the statistics behind the online testing and how to set up a hypothesis.