Search Engine Ranking Software Comparison

As I mentioned in a previous post about the numerous reasons for organic search engine ranking discrepancies, there are a wide variety of tools that can help simplify the process of checking a site's rankings for multiple keywords across the primary search engines. In this post I will explore four methods for tracking rankings and compare the features, costs, and results of each.

Methodology

I tested the same set of 78 keywords on Google, Yahoo!, and MSN using each of the four methods listed below. The destination domain was AmericanAqua.com, a Michigan-based small business that provides a variety of services including bottled water delivery, salt delivery, water treatment, and water analysis. The keywords selected were a mix of branded terms, product/service terms, and geographic terms. As a control, several broad, generic keywords were chosen where the company has no supporting content on their site and thus would not expect to be ranking. Only the first listing for the destination domain was counted, so if the site was listed in positions 4 and 12 for the same keyword on the same engine, only the position #4 ranking was counted in the results.

In order to control for as many variables as possible, the data was collected in a 4-hour window from the same computer, using the same browser. I was not logged in or cookied by any of the engines at the beginning of the data collection period. The automated rank checkers were set to query no faster than once every 5 seconds to avoid triggering the engines' anti-scraping mechanisms. Only the first 30 organic results (3 pages) were scanned because let's face it, if your site doesn't appear in the first 3 pages you might as well be invisible.

The Contenders

• Web Position 4 is a feature-rich software package that was launched by WebTrends in 2005. Judging by the age of the content and "latest" press releases on their site, there hasn't been much activity or improvement of the software since then. The software is available in "Standard" and "Professional" packages. Only results from the Professional version are included below.
• SEOBook's Rank Checker Firefox Extension is a free product offered by Aaron Wall. Once downloaded, the tool itself lives within your Firefox browser as an extension and does not require any additional software.
• SEOMoz's Rank Checker is another web-based tool offered by the SEOMoz.org team. Unregistered users can run a limited number of queries per day for free, but premium members get unlimited usage for a monthly fee.
• Manual Rank Checking is done the old fashioned way...with a keyboard, mouse, and a spreadsheet. No bells and whistles, just you and a mild case of Carpal Tunnel Syndrome.

¹Web Position 4 Standard edition is $189 and performs all of the same organic rank checking functions. The Professional edition has a few useless add-ons that are not useful for experienced SEO's or no longer considered useful, i.e. search engine submissions.

²SEOMoz allows registered, non-Premium members to check up to 5 keywords a day for free. Unlimited usage of their Rank Checker tool requires a $49 per month Premium membership, although discounts are given for 6 and 12-month subscriptions.

³For some reason there is an arbitrary 45 minute delay in running Web Position reports on Yahoo after about 50 keywords. Total running time was 84 minutes, but 45 of those were just waiting around.

⁴This probably would have taken much longer, but I didn't bother checking the ranks of about 150 keywords (50 KWs * 3 engines) that did not show up in any of the other tools.

The Results

Despite the various data collection methods and reporting tools, all 4 data collection methods returned the exact same rankings nearly 72% of the time. Where they were off, the average standard deviation was only 0.84, meaning the difference between collection methods was off by less than 1 position on average. That indicates that the tools are consistently pulling very similar rankings.

However, it should be noted that of the 78 potential rankings on each engine, at most we saw 29 keywords ranking at one time. When the non-ranking keywords are removed from the previous calculations, we only see the same results from each tool 35% of the time with a much higher standard deviation of 1.95.

Interesting Findings

The following tidbits stuck out while collecting and analyzing the data:

• The four rank checkers agreed most often on MSN rankings. Of the keywords that ranked, the standard deviation on MSN was only 1.08, compared to 1.99 on Google and 2.77 on Yahoo.
• Yahoo's results were the most inconsistently reported by the four rank checkers.
• SEOBook's Firefox Rank Checker was the fastest, but missed a few rankings than the other tools found.
• SEOMoz's tool only allows one query at a time, which slows down the data collection considerably for larger keyword sets.
• SEOBook's Firefox Rank Checker managed to delete one word from the file export, which caused a few minutes of working backwards to determine what was dropped. I ran that one again and it was fine.
• SEOMoz's tool reports results as "Found on page X, position #Y", which slows down the process even more to manually convert to an exact position out of 30.
• Many of Web Position's additional features are no longer selling points and are practically worthless, such as their search engine submission tool. The only compelling features are the ability to run multiple keywords across many engines and the data trending and export functions.

Conclusions

Given the similarities in results between all four methods, I recommend SEOBook's Rank Checker Firefox Extension because it is free, fast, can handle multiple keywords across the top engines, and has sufficient data export capabilities. It won't trend your results over time, but a simple Excel spreadsheet or database can make some great charts and graphs from the raw data. Each rank checking method has their own pros and cons, but Aaron's tool has the right combination of cost (free) and features (just enough to get the job done, without too many bells and whistles). Other tools have more robust feature sets but are too expensive, or are free but too time-consuming and limited in functionality.

None of the organic search ranking tracking methods proved to be dramatically more or less accurate than the others. There are many external variables that could influence rankings by a few positions or more, and these are likely the main cause for the discrepancies between the various tools.

These data only represent rankings at one point in time. The best way to determine the effectiveness of your SEO campaigns is to measure rankings across multiple engines at regular intervals so you can examine the trends. Do not put too much emphasis on any one data point, since they are very likely to fluctuate over the short term. Instead, pay attention to the longer-term trends to look for signs of improvement in your rankings.

As with any software decision, you must draw your own conclusions based on what is important to you. What features do you think are lacking from today's rank checkers? How should they be extended to be more useful? Which do you use, and which are your favorites?