A Call for Scientific SEO Testing

“The trouble with the world is that the stupid are so confident while the intelligent are full of doubt.”
Bertrand Russell

I’m not sure about the world at large, but that quote by Bertrand Russell pretty much nails the SEO industry.

In an industry governed by an algorithm that reportedly incorporates over 200 factors and has countless PHDs working to improve it, we’re constantly bombarded with people claiming to have discovered the key to the topic du jour. To make matters  worse, there always seems to be a massive number of people willing to believe the claims, no matter how bombastic!

Wild A** Guesses

In the aftermath of Google’s Penguin updates, the blogosphere has been flooded with articles discussing what Google changed, what sites it impacted, and most importantly, how to recover if your site was spanked… all within a week of the update rolling out!

We literally had dozens of articles “educating” readers on how to recover from an update before even a SINGLE site recovered!

Readers, one can only assume, were supposed to make business altering decisions based on the theories, suppositions & W.A.G.s of these SEO experts without a single data point or piece of supporting evidence.

Strategic Wild A** Guesses

“Education has failed in a very serious way to convey the most important lesson science can teach: skepticism”
David Suzuki

Once Penguin finally WAS refreshed (albeit for a very small portion of the impacted sites), SEO sites hungry for content began featuring case studies of the lucky few sites that actually did recover.

While case studies can be interesting, when they aren’t thinly veiled self sales pitches, they are anecdotal at best. Maybe the steps the author took allowed the site in question to recover, or maybe the site was able to generate enough sympathetic press that Google felt pressured into taking action on their behalf.

Or, maybe the author of the case study simply wants to elevate their profile within the industry and they manufactured a few images to support their W.A.G.s and called it a case study. Did I ever tell you about my really hot girlfriend from 6th grade? Yeah she lived really far away and you probably wouldn’t know her but she TOTALLY existed.

Correlation is NOT Causation

Correlation studies seem to be the new chic within the SEO and Social Media communities and we’re constantly warned that correlation is NOT causation. Of course, the publishers of these correlation studies hype their findings in an understandable effort to get as much attention as possible. All too often that hype results in readers (and occasionally publishers themselves) blowing right past any causation disclaimers and putting all sorts of unrealistic expectations on the data.

Now don’t get me wrong, correlation studies have their place and at least they’re generally based on data, a LOT of data and math (occasionally faulty math, but if the data is shared, those kinds of issues can be spotted and corrected). But in the end, these studies still leave readers with a high degree of uncertainty.

Facebook shares are closely correlated with high rankings, awesome! Now is that because sites that rank well get more traffic, resulting in more Facebook shares? Or do the Facebook shares help boost your rankings?  Hell, I don’t know but I DO know I NEED Facebook shares and lots of ’em!

“Who do I trust? Me!”
Tony Montana

I will admit, I’m a skeptic by nature. If you want to convince me of something, I need to see some evidence convincingly laid out in a logical manner, and even then I’ll probably want to test that evidence for myself. When a waiter or waitress at a restaurant tells me “be careful, the plate is extremely hot” the first thing I do is touch it to see exactly how hot it really is.

Thankfully, there’s an entire process designed to help satisfy stubborn skeptics like myself.

The Scientific Process

Look, I’m not calling for all of us to don lab coats and become molecular biologist SEOs. We’ve already got one of those and I’m not sure the world can handle more.

But the (slightly dumbed down) purpose of the scientific method is to prove or disprove theories based on actual evidence and to document the entire process in such a way that others can review, repeat, and build upon your work.

Instead of being left with the “chicken or the egg” conundrum presented by correlation studies, or a internal evaluation of an SEO bloggers credibility, readers are presented evidence and are allowed to draw their own conclusions or perform tests of their own.

The old standbys of of the scientific process:

  • Observation
  • Hypothesis
  • Prediction
  • Experiment
  • Analysis
  • Publication
  • and Retesting (yes, I totally had to ask someone much smarter than me for half of these)

would be welcome additions to many SEO posts.

Our industry is governed by PHDs constantly working on an algorithm. It’s time for our analysis of that algorithm, and investigations into ways of influencing it to our advantage, to evolve beyond the anecdotes, correlations and strategic wild ass guesses we’ve grown accustomed to reading.

About Ben Cook

Ben Cook is the owner of Direct Match Media, an internet marketing firm specializing in responsive WordPress design, SEO consulting and viral content creation.

Filed under: Featured, Strategy


Stuart Draper

Someone needed to say it Ben. Thanks! I have always asked myself why you see so many more posts about SEO compared to PPC. I think that the reason is that you can’t get away with as much opinion and guesswork with a post about PPC. PPC posts better have some hard data behind them and some screenshots of proof before I give them much credibility. With SEO, you see so many posts where people can get away with opinion as fact because they spoke once at XYZ conference.

So often though, even when you do test, and go through the entire process, there are so many variables in an SEO test that people will always criticize your work; the accuracy and surety.

Ben Cook

Stuart, even though SEO tests are extremely difficult to isolate, I think they are absolutely worthwhile. And, as long as you document your test carefully, and don’t get too far ahead of yourself on the claims you make based on it, I think people will generally be accepting.

Where people get into trouble is claiming that their single test proves X Y or Z when in fact there still needs to be a lot more testing done before anything is firmly established.

Eider Vasconcelos

Absolutely on board with this idea. I just moved to Mexico, where most people still look at overall visits to measure traffic or number of followers as a goal.
I have loads of data to experiment with from some of the biggest sites in Latin America, but I lack experience in scientific experimenting.
You have any suggestions of sites where one could start in this process? if related to SEO, even better.

Jake Webster

“The trouble with the world is that the stupid are so confident while the intelligent are full of doubt.”
– Bertrand Russell

I find this to be true with just about anything I do. Love this quote!

Comments are closed.