• About Us
  • Services
  • Case Studies
  • Blog
  • Contact Us
  • The Small (but Great) SpyFu Experiment

    For those of you not in the loop, SpyFu.com is a web service that provides information on who is bidding on what keywords as well as further information on competitors’ daily ad spend as well as their average CPC.

    No Mr Bond, I Expect you to Disagree.

    Everyone seems to have an opinion on how useful spyfu is, ranging from the standard “it works/doesn’t work for me” to the commonly held belief that it is only really useful for large accounts. People disagreeing on the Internet? It’s time to try science!

    The Experiment

    I selected 20 of SEOptimise’s clients, discarded two of them for using foreign currencies (spyfu has a UK and US version but I didn’t want the hassle of fiddling with exchange rates) then ignored a further two when spyfu provided no data on them. I then compared the percentage accuracy for SpyFu’s estimates for daily ad spend and CPC with the actual daily spend and average CPC as well as the number of impressions the ad generates. I even drew graphs.

    The Perils of Percentages

    I decided to work with percentages rather than actual pounds and pence for two reasons; client confidentiality and because there is such a massive variation in budgets and CPC’s across the selected accounts. Considering percentage variations means that a £10 error on an account spending £1000/day counts less than for one on a spend of £10/day. But you knew that already, didn’t you?

    One side effect of this is that, particularly on a graph, it looks like the lower estimates are a lot more accurate then the upper estimates. However, this is because the lower estimate is never less than zero so it can never be more than 100% different from the actual value.

    Enough writing. Time for some pictures.

    Bigger is Better?

    With a bigger account, spyfu has more data so its estimates will be more accurate. Right?

    This first graph is of the error in spyfu’s ad spend estimates against total ad spend. I expected that the error would be less for big spending accounts.

    As you can see, it looks like my hypothesis is correct (at least for the upper estimate, which tends to dominate these charts) but then the error increases again for the account with the largest ad spend. There are three possibilities here:

    1. Accuracy is unrelated to ad spend.

    2. Accuracy increases with ad spend and the last result is just a freak

    3. Spyfu is actually most accurate somewhere in the middle range of ad spends.

    To me, option 3 seems the most unrealistic and option 2 the most likely. Let’s check by looking at CPC. . .

    O dear, this graph of CPC error against total ad spend shows no correlation of any sort whatsoever.

     

    You’re Using the Wrong Big

    Ok, so it might be unfair to say an account is big just because it spends a lot of money. Spyfu uses web scraping so it should be more accurate for account with a large number of impressions.

    The following graph shows the percentage error ad spend against the number of impressions the account has generated:

    This graph looks very similar to the Ad Spend Error vs. Ad Spend graph, probably because daily ad spend is quite closely linked to the number of impressions an account gets.

    The graph of CPC error against number of impressions, below, also doesn’t show any correlations.

    The graphs so far, particularly for CPC show that there is not a strong relationship between the size of an account and the accuracy of spyfu.

    Assuming our largest account is just an anomalous result I think there is a case for saying that estimates of daily ad spend are more accurate for the larger accounts but that the error in CPC does not improve (or get worse) as account size increases.

     

    Spyfu’s algorithm is a bit of a mystery, perhaps it calculates CPC entirely separately from daily ad spend. Perhaps comparing CPC with the size of the account is like comparing apples and pears.

     

    Comparing Apples and Apples

    Now let’s look at graphs for the CPC error against actual CPC:

    My goodness, there might even be a trend there. It looks as though the upper estimate gets more accurate as CPC increases. The account with the highest CPC is an anomaly since spyfu has given both a lower and upper estimate of 0; any PPC marketer with common sense would ignore this anyway.

    This next graph shows the same information, but with the upper estimate plot removed so that it is easier to spot trends in the lower estimate:

    So as CPC increases the lower estimate actually gets worse. This is all very confusing, what does it all mean?

    What’s Going On?

    So far the results seem to indicate the following:

    1. Spyfu’s estimate of ad spend is more accurate for larger accounts whether size is measured by ad spend or number of impressions.

    2. The accuracy of the CPC estimate does not depend on the account size.

    3. The upper estimate for CPC improves as CPC increase, but…

    4. The lower CPC estimate gets worse as CPC increases.

     

    What Does This Tell Me?

    Nothing at all. The sample size is small and the correlations are weak; you would be a fool to use this data to decide how accurate spyfu is. Besides, it’s not like the four conclusions written above are even quantitative. It would’ve been much more useful if I’d been able to say “The lower estimate of ad spend for an account is within 50% of the actual value 90% of the time” but as you have seen the results do not support anything like that. The best I can hope for is that someone will be intrigued by what they’ve seen here and decide to test clients from their own MCC. If they have more data, or if lots of people with not much data arrive at the same conclusions then perhaps I can say I was onto something.

     

    So SpyFu is Useless?

    Far from it. When it provided information, true ad spend was within spyfu’s estimates 71% of the time (ignoring the accounts for which it returned 0) with CPC being slightly better at 79%. I will continue to advise and act as if true ad spend and CPC lie within spyfu’s estimates because the information it provides is better than no information at all.

    ABOUT THE AUTHOR

    14 Responses to “The Small (but Great) SpyFu Experiment”

    1. Chelsea Blacker says:

      Richard, this is great! I’ve always wondered about SpyFu accuracy and your lab writeup certainly begins to identify SpyFu’s strengths and faults. I’ve learna a lot, thanx.

    2. Thanks Chelsea,
      Hopefuly I’m going to post more on SpyFu later in the week.

    3. Vishwa says:

      Nice analysis….

    4. Sean Blanks says:

      Good post! I wonder whether the results are affected by the age of the accounts? For instance, we have some pretty big accounts, but for those which have recently grown a lot, spyfu is still reporting that they are small (or barely there at all!)

    5. Hi Sean,
      SpyFu don’t update their database very often, I’ve seen a variety of figures, some say it can take as long as 90 days.
      I might cover this a bit in my next post which, hopefully, will be coming next week.

    6. [...] SpyFu say about their own algorithm and techniques and then I’ve combined this knowledge with what I observed last week to come up with these ideas about how to turn SpyFu into a double [...]

    7. Frank Grasso says:

      has anyone seen any real benefit from using SpyFU? its nice to know what the competitors are spending and what their cpc is but how does this help you to build a better campaign?

    8. SEO Stuart says:

      Richard, some great information you have gathered here, I have been mulling over taking out the paid SpyFu service for a while now. I think I will give it a go, and maybe try a few tests myself.
      You make a very good point by saying: “because the information it provides is better than no information at all.” I’ve kinda thought that myself when hovering over the join now button. :)

    9. Hi Frank,
      You’re wondering how SpyFu helps you build a better campaign. I’d say that it doesn’t, but it does help you tell if a campaign is good or not.

      If you were managing a campaign in a vertical you knew nothing about then you could use SpyFu to compare you daily ad spend, avg CPC and ad position with competitors. This would give you some indication of whether or not it was a good campaign.

    10. [...] blogged on here for a while and I’ve been asked to step up to the plate without mentioning SpyFu! I should’ve been thinking of what to post for a few days now but I’m afraid I’ve [...]

    11. Best of The Web - Blog Posts and Articles Of Year 2008 says:

      [...] The Small (but Great) SpyFu Experiment on SEOptimise. An excellent post analysing the accuracy and usefulness of PPC competitor review tool Spyfu. [...]

    12. Best of The Web - Blog Posts 2008 says:

      [...] The Small (but Great) SpyFu Experiment on SEOptimise. An excellent post analysing the accuracy and usefulness of PPC competitor review tool Spyfu. [...]

    13. Thanks I use spyfu and have been wondering about this for a while. The data might not be completely useful,m but it’s a start and it is interesting.

    14. Sajan Kota says:

      I was thinking of buying spyfu or a similar software however looks like its not worth the money and effort. Looks like the hunt for the software which meets my requirement goes on.

    Leave a Reply