In my last post, “Too soon for decisions”, I discussed applying a consistent set of rules to campaigns to assess the performance of new ads and targeting. However, in practice, assessment and tracking an AdWords or Facebook campaign can be an interesting exercise.

The data generated by a campaign is not a true representation of the population. The data is a snapshot limited by the markets targeted and the visibility available for the budget spent. Any single campaign can be exposed to direct competition over the whole market or specific subgroups. For example, just because “Campaign A” does not get traffic from Victoria does not mean that no-one in that state is searching for “Keyword B”.

A competitor could simply be focused on that market and value the traffic more. Other factors to consider are the effectiveness of the competition’s creatives and offers, the appeal of their product, efficiency of their site in turning clicks into sales and how much they return per conversion. All of these factors will influence their budget, and how much they are willing to spend per click or impression. Tools provided by the advertising networks that increase the efficiency of campaigns like Remarketing are also worth considering.

According to Wikipedia, a confidence interval is defined as:

…a particular kind of interval estimate of a population parameter. Instead of estimating the parameter by a single value, an interval likely to include the parameter is given. Thus, confidence intervals are used to indicate the reliability of an estimate. How likely the interval is to contain the parameter is determined by the confidence level or confidence coefficient. Increasing the desired confidence level will widen the confidence interval.

In use here, it is assumed that between similar competitors, the average Cost Per Acquisition (CPA) within the group is likely to be within a 95% confidence interval of the known CPA.

Confidence interval can give you an estimate of what other bidders may be paying for a conversion, assuming they are operating as efficiently as you are. In the graph included above, confidence interval of the CPA is used to estimate the most likely highest possible CPA a campaign can still compete on. In conjunction with Cost Per Click data, it is fair to assume that the competitors in the query space are willing to spend over the highest likely observed CPA. Reasons for their bidding strategy can vary from shutting out competitors by absorbing a short term loss, to a higher sustainable CPA. In a query space where a number of different verticals are competing for the same traffic, this metric is considerably less useful and your mileage may vary. For comparing CPA campaigns, creating a model for understanding the market, or simply to assess which ads are potentially performing a lot better or worse than your target in the face of direct competition, it is a useful tool.

Confidence interval can be a guide to how much your competitors expect to spend per conversion, assuming a lot of similarity in product and business practices. Arbitrage and industries with heavily commodified products are prime candidates for this, as well as campaigns with a very aggressive high cost bidding strategy, such as those competing directly with another member of your industry.