How To Test Bids In PPC Advertising Without Applying Them

A/B tests of bids have several obstacles:

In general, even with an infinite amount of data, we will be able to determine which option is better only if the options have a difference in the efficiency of more than 3-5%. Let's call this theoretical sensitivity.

The following factors influence theoretical sensitivity:

In practice, since we don't have an infinite amount of data, even for large sites, the sensitivity threshold of the A/B test is about 10-20%. If the difference in effectiveness between the two options is less than this threshold, then doing A/B is not much better than flipping a coin. Let's call this practical sensitivity.

As a result, A/B testing:

• is not accurate;
• takes much time;
• requires a lot of work and specific knowledge;
• cannot be fully automated.

And after all, there is always a possibility that new rates will work worse.

Therefore, there may be a metric that is more sensitive (even theoretical) than the A/B test. Moreover, this metric can be simply calculated using historical data without taking into account bids. If we automate this, then it can be done in a couple of clicks.

However, since we don't know the dependence of the number of clicks and CPC on the rate, then this metric will produce quantitative results with a rather large margin of error, but possibly quite accurate qualitative results.

In other words, this metric will answer the question "Which option is better?" Rather well, but rather poorly on the question "How much better?".
Intermediate calculations may seem voluminous to you, but the final formula will be short.

For the sake of simplicity, let's assume that each keyword has:
CPC and number of clicks are proportional to the bid;
conversion probability is independent of the bid.

Let's set the bids in proportion to the estimate:

Where:
f - forecast the probability of conversion
L is a constant to fit our constraints (budget, target CPA)

Let p be the probability of conversion, then the expected number of conversions is:

The expenditure for a keyword is:

Let our goal be the maximum conversions with a fixed CPA = TargetCPA.
The expected CPA is:

To find L, let's equate the target CPA with the expected one:

From here you can calculate L:

We introduce the weighted mean operator:

From which it follows that:

You can also easily see that the constant can be taken out of the weighted mean:

Let's get back to L:

For simplicity, we will assume that c and f are not correlated.

Let's calculate the expected number of conversions:

Let's calculate what will happen at equal rates. Let our forecast for all keys be the same and equal to some constant m. Without loss of generality, we can say that m is equal to the weighted average conversion:

Now let's divide the number of conversions for arbitrary bids by the number of conversions for equal bids. Let's get how many times bids bring more conversions than equal bids for the same CPA. Let's call this metric the Equal Bid Improvement Factor (IBIF).

With equal rates, IBIF should be equal to 1. For example, all rates are equal to the constant S:

W [p] is just a multiplier that doesn't depend on the estimate.
W [fp] - grows with an increasing correlation between the estimate and the probability of conversion. It also grows with bid variance (spread).
W [ff] - grows with increasing variance of bids.

IBIF can be expressed in terms of rate variance and the correlation between bids and estimates.

Where:

C is Pearson's weighted correlation coefficient between bids and predictions. The measure of the adequacy of the bids doesn't exceed 1. Equal to 1 for an ideal estimate, equal to 0 for random rates.
V [] - weighted relative variance, the measure of the spread. Greater than or equal to zero. For example, if V [f] = 0, it means that all bids are the same. V [p] <1 is almost certainly true. Usually V [p] is about 0.5. V [f] is usually not very different from V [p].

If R[p] = 0.5:

It can be seen that, in general, the larger V [f], the stronger the IBIF depends on the correlation. It is also easy to see that at high correlations (more than 0.95), the IBIF is practically independent of V [f].

At ideal bids, f=p, C=1:

The more different the conversions for keywords, the higher the IBIF can be obtained with an ideal estimate.

From the fact that it is almost certain that V [p] <1, we get that the IBIF is almost always less than 2. The opposite indicates, most likely, that a significant number of keywords have almost zero conversion (much less than the average for the site).

This can happen, for example, when you buy most of the clicks on products you don't have in stock, on 404 pages, or irrelevant pages; or some of the pages don't have a conversion tracking code.

In other words, IBIF > 2 may indicate that rates solve some specific problem.

You can calculate IBIF from historical data using the formula:

Where:

r - keyword conversion rates
n - number of clicks on keywords

Note that f should not directly depend on r. That is, f and r must be calculated at different periods.

In other words, you need to run the test like this:
For example, on the 1st day we withdraw bids or calculate the forecast. Remember f.
After a couple of weeks, for example on the 16th, we take data from the 2nd to the 15th. And we get n and r.
We calculate the IBIF, the variant with the higher IBIF is considered the winner.
Since we don't apply bids, we just look at the statistics, we will call this test passive.

If we multiply f by some constant factor S that is the same for all keywords, then the IBIF will not change:

Therefore, instead of f, you can take not only the forecast but also a set of rates that are proportional to the original estimates. That makes it possible to compare not only forecasts but also sets of bids with each other or a set of bids with a forecast.

You can give the following assessment of the maximum IBIF (with an ideal forecast):

Where:

r - conversion rate for the first period (regular rates are considered in this period)
n - number of clicks in the first period
R - conversion rate for the second period (periods should not overlap)

If you don't have a second period, then you can divide the first period into 2 parts (equal) then:

r - conversion rate in the first part
R - conversion rate in the second part
n - number of clicks for the entire period

This formula doesn't depend on uniform seasonality when the conversion of all keywords decreases or increases by the same number of times.

The accuracy of calculating the limiting IBIF for small data is several times less than that of a conventional IBIF.

To compare 2 sets of bids or forecasts, you need to calculate this parameter:

Where:

r - keyword conversion rates or revenue-to-click ratio
n - number of clicks on keywords
f - bids or forecast for conversion or revenue per click. When calculating them, data for the period for which r was taken should not be used.
W [] average weight operator

Equal bids give IBIF = 1. The maximum IBIF can be calculated as follows:

Where R is the conversion rate of keywords or the ratio of revenue to clicks for another period that does not overlap with the first.

To calculate this indicator in Excel, first you need to get rid of the weighted mean.

Where:

n (clicks) are in column A
r (conversion) are in column B
f (forecast/bids) are in column C

Then the formula will look like this:

However, the formula can be simplified if column D contains conversions and revenue:

Let R be in column G. Then the formula is: