The Small (but Great) SpyFu Experiment |

The Small (but Great) SpyFu Experiment

By Richard Fergie / September 19, 2008

For those of you not in the loop, is a web service that provides information on who is bidding on what keywords as well as further information on competitors’ daily ad spend as well as their average CPC.

No Mr Bond, I Expect you to Disagree.

Everyone seems to have an opinion on how useful spyfu is, ranging from the standard “it works/doesn’t work for me” to the commonly held belief that it is only really useful for large accounts. People disagreeing on the Internet? It’s time to try science!

The Experiment

I selected 20 of SEOptimise’s clients, discarded two of them for using foreign currencies (spyfu has a UK and US version but I didn’t want the hassle of fiddling with exchange rates) then ignored a further two when spyfu provided no data on them. I then compared the percentage accuracy for SpyFu’s estimates for daily ad spend and CPC with the actual daily spend and average CPC as well as the number of impressions the ad generates. I even drew graphs.

The Perils of Percentages

I decided to work with percentages rather than actual pounds and pence for two reasons; client confidentiality and because there is such a massive variation in budgets and CPC’s across the selected accounts. Considering percentage variations means that a £10 error on an account spending £1000/day counts less than for one on a spend of £10/day. But you knew that already, didn’t you?

One side effect of this is that, particularly on a graph, it looks like the lower estimates are a lot more accurate then the upper estimates. However, this is because the lower estimate is never less than zero so it can never be more than 100% different from the actual value.

Enough writing. Time for some pictures.

Bigger is Better?

With a bigger account, spyfu has more data so its estimates will be more accurate. Right?

This first graph is of the error in spyfu’s ad spend estimates against total ad spend. I expected that the error would be less for big spending accounts.

As you can see, it looks like my hypothesis is correct (at least for the upper estimate, which tends to dominate these charts) but then the error increases again for the account with the largest ad spend. There are three possibilities here:

1. Accuracy is unrelated to ad spend.

2. Accuracy increases with ad spend and the last result is just a freak

3. Spyfu is actually most accurate somewhere in the middle range of ad spends.

To me, option 3 seems the most unrealistic and option 2 the most likely. Let’s check by looking at CPC. . .

O dear, this graph of CPC error against total ad spend shows no correlation of any sort whatsoever.


You’re Using the Wrong Big

Ok, so it might be unfair to say an account is big just because it spends a lot of money. Spyfu uses web scraping so it should be more accurate for account with a large number of impressions.

The following graph shows the percentage error ad spend against the number of impressions the account has generated:

This graph looks very similar to the Ad Spend Error vs. Ad Spend graph, probably because daily ad spend is quite closely linked to the number of impressions an account gets.

The graph of CPC error against number of impressions, below, also doesn’t show any correlations.

The graphs so far, particularly for CPC show that there is not a strong relationship between the size of an account and the accuracy of spyfu.

Assuming our largest account is just an anomalous result I think there is a case for saying that estimates of daily ad spend are more accurate for the larger accounts but that the error in CPC does not improve (or get worse) as account size increases.


Spyfu’s algorithm is a bit of a mystery, perhaps it calculates CPC entirely separately from daily ad spend. Perhaps comparing CPC with the size of the account is like comparing apples and pears.


Comparing Apples and Apples

Now let’s look at graphs for the CPC error against actual CPC:

My goodness, there might even be a trend there. It looks as though the upper estimate gets more accurate as CPC increases. The account with the highest CPC is an anomaly since spyfu has given both a lower and upper estimate of 0; any PPC marketer with common sense would ignore this anyway.

This next graph shows the same information, but with the upper estimate plot removed so that it is easier to spot trends in the lower estimate:

So as CPC increases the lower estimate actually gets worse. This is all very confusing, what does it all mean?

What’s Going On?

So far the results seem to indicate the following:

1. Spyfu’s estimate of ad spend is more accurate for larger accounts whether size is measured by ad spend or number of impressions.

2. The accuracy of the CPC estimate does not depend on the account size.

3. The upper estimate for CPC improves as CPC increase, but…

4. The lower CPC estimate gets worse as CPC increases.


What Does This Tell Me?

Nothing at all. The sample size is small and the correlations are weak; you would be a fool to use this data to decide how accurate spyfu is. Besides, it’s not like the four conclusions written above are even quantitative. It would’ve been much more useful if I’d been able to say “The lower estimate of ad spend for an account is within 50% of the actual value 90% of the time” but as you have seen the results do not support anything like that. The best I can hope for is that someone will be intrigued by what they’ve seen here and decide to test clients from their own MCC. If they have more data, or if lots of people with not much data arrive at the same conclusions then perhaps I can say I was onto something.


So SpyFu is Useless?

Far from it. When it provided information, true ad spend was within spyfu’s estimates 71% of the time (ignoring the accounts for which it returned 0) with CPC being slightly better at 79%. I will continue to advise and act as if true ad spend and CPC lie within spyfu’s estimates because the information it provides is better than no information at all.

More posts from this author

blog comments powered by Disqus
01865 703000