How Not to be Wrong*

*(or, at least discover that you are sooner)

Everyone agrees that testing stuff is important. Unfortunately just because someone in marketing can string together the phrase “A/B testing” on a slide does not mean it will be done well. And all too frequently in practice it is poorly run, poorly documented and poorly managed by people who did not have the skills required to design the experiment to address their needs or even understand what the data generated actually represents.

A while ago I quickly drafted a fairly informal document to send around the office to lay out a very basic framework for running tests within the web side of the business.

Testing needs purpose

Articulate the need for testing and define what is to be trialed to address this. Identify who the stakeholders are and inform them of the test where appropriate. Consider whose activity will be affected by this.

Testing needs focus

Identify the metrics that the proposed change should affect. Identify other factors that could possibly influence these metrics. Assess the quality of the data available and identify any issues with tracking and reporting that may compromise the validity of the test.

Testing should not needlessly duplicate previous work

Establish if similar, overlapping, tests have already been done on any of the brands. If there is existing material review previous testing across all brands that may be relevant to the proposed trial. Identify where the information addresses similar areas of interest, and how the new test will differ and reveal new information.

Testing needs to be linked to business outcomes

Outline the value of the information generated by the trial in terms of business outcomes. Address information generated from all possible outcomes. Define how the trial will relate to the business’ general strategy.

Testing works when what Success or Failure look like is known

Identify what the change being tested is supposed to improve. Define how much this metric would need to change for it to be worth the cost of implementing to create value for the business. Project how much data would be required to prove a change of this magnitude. Identify what sources of data will be relevant for this

Testing requires design

Determine the scope of the test by defining how many resources to devote to the trial and the control groups. Establish how long you require to produce a result based on the data available at the time. Identity confounding factors and establish a plan for how to control or adjust for these. Consider how to limit impact of the trial on other areas of the business.

Testing needs to be implemented

Begin running the trial and collecting information as it is generated and assess periodically. Establish a framework for assessing in-progress results, taking into account what would be considered an egregious trend, requiring an early termination of the trial. Conversely monitor for especially conclusive results allowing for the test to be terminated early.

Testing must finish

On conclusion of the test, document and share the data generated during the trial. Should the data support the implementation of the change, do so. If the data does not show an improvement or not one large enough to be worth implementing, do not. Revise the change in light of the new information, and plan another trial.

Testing must be reassessed

Plan to reassess the elements tested at a later date based on real data gathered. Assess where the information generated differs or otherwise from that acquired during the test. Determine if any differences indicate that the testing methodology used needs to be adjusted.

And the point is?

It is insanely easy to run tests on a website these days. There is a range of tools available to support execution and management, and it is practically expected in most businesses. However just because someone has access to Visual Web Optimiser or can find ‘Experiments’ in Google Analytics it does not mean they actually know what they are doing. Being able to run a testing tool is not enough. Understanding what the project is supposed to accomplish, actually doing due diligence on experiment design, setting aims, determining how much data is required, and executing this to serve an actual business outcome may require something beyond just knowing the names of the tools.