We've had a lot of discussions recently about SEO as a Science. Unfortunately, these discussions sometimes devolve into arguments over semantics or which approach is the "best" in all situations. I'd like to step back for a few moments today and talk about the wider world of SEO evidence. While not all of these types of evidence are "science" in the technical sense, they are all important to our overall understanding. We need to use the best pieces of all of them if we ever hope to develop a mature science of SEO.

The Fundamental Assumption

All science rests on a fundamental assumption, long before any hypothesis is proposed or tested. The fundamental assumption is that the universe is orderly and follows rules, and that through observation and experimentation we can determine those rules. Without an orderly universe, science would be impossible (as would existence, most likely). A related assumption is that these rules are relatively static – if they change, they change very slowly. Our view of the universe may change dramatically, resulting in paradigm shifts, but the underlying rules remain roughly the same.

The advantage we have as SEOs is that we know, for an absolute fact, that our universe is orderly. Like Neo, we have seen The Matrix. The Algorithm consists of lines of code written by humans and running on servers.

The disadvantage for SEO science is that the rules governing our universe are NOT static. The algorithm changes constantly – as often as 400 times per year. This means that any observation, any data, and even any controlled experiment could turn out to be irrelevant. The facts we built our SEO practices on 5 or 10 years ago are not always valid today.

(1) Anecdotal Evidence

All science begins with observation. In SEO, we make changes to sites every day and measure what happens. When rankings rise and fall, we naturally try to figure out why and to tie those changes to something we did in the past. Although it isn't "science" in the technical sense, the evidence of our own experience is very important. Without observing the universe and creating stories to explain it, we would never learn anything from those experiences.

PROS – Anecdotal evidence is easy to collect and it's the most abundant form of evidence any of us have. It's the building block for just about any form of scientific inquiry.

CONS – Our own experiences are easily affected by our own biases. Also, no single experience can ever tell the whole story. Anecdotal evidence is just a starting point.

(2) Prophetic Evidence

SEOs have a unique type of available evidence. Every once in a while, a prophet will descend from the Mountain Top (or Mountain View), shave his head, and speak the words of the Google Gods. Whether or not we choose to believe these prophets, the fact remains that there are people who have seen and written the Algorithm, and those people have access to facts that the rest of us don't. Their statements (and our ability to critically reconcile those statements) are an important part of the overall puzzle.

PROS – The prophets are as close to objective reality as we're ever going to get. They have direct insight into the algorithm.

CONS – The prophets don't have a vested interest in telling us the whole truth. Their messages can be cryptic and even misleading.

(3) Secondhand Evidence

When you hear "secondhand" evidence, you may naturally think of the extreme examples, like hearsay and urban legends:

My cousin's neighbor's stylist said that she once changed all of her META tags to "sex poker sex poker sex" and her site immediately jumped to #1 on Google!

To be fair, though, secondhand evidence also includes the legitimate science that came before us and the experiences of our peers. If we were forced to confirm and replicate every single conclusion for ourselves, we would never make any progress. Ultimately, we build on the reliable conclusions of other experts, past and present.

PROS – Secondhand evidence is the foundation for scientific progress.

CONS – Sometimes, experts are wrong, and you have to learn how to tell the difference, especially in a field as young as SEO.

(4) Experimental – "The Wild"

Experimentation is the heart of Capital-S Science. The most basic experiments happen something like this:

• You form a hypothesis ("Adding keywords to my title tag will improve rankings").
• You make a change to test that hypothesis.
• You measure the outcome and find out if you were right.

Most SEO experimentation, by its nature, occurs in the "wild". We have to put our sites out in the world, and we often have to use existing sites that are already complicated and changing.

PROS – By directly forming and testing a hypothesis, we can start to determine causality. We can also repeat the process, helping to validate what we've learned.

CONS – Using existing sites in the wild introduces a lot of extra noise. Often, our sites have to keep changing (even during the experiment), and Google is always changing. There's also a fair amount of risk – if we change our bread-and-butter sites to test SEO theories, mistakes can be costly.

(5) Experimental – Controlled

This is the classic SEO experiment, where we register one or more new domain names and build sites from the ground up. We can even introduce a control group, by building both sites up to Step X and then only changing one of the sites after that point. Even then, it might be best to call these experiments "semi-controlled," since the Google algorithm can still change and we can't always control outside influences (like someone accidentally linking to one of the sites).

PROS – This approach is about the best we can do, in terms of control, and it separates out a lot of confounding factors.

CONS – The artificial sites we set up in these experiments (often using nonsense words) aren't always representative of real, complex sites. In addition, these experiments are usually conducted on a sample of just one or very few sites, to save time and money. Statistical significance can be very difficult to achieve.

(6) Correlational Evidence

Sometimes, either we can't separate out the variables involved in a complex situation (like the 200+ factors Google uses in its ranking model) or direct experimentation would be impossible or unethical. For example, let's say you want to understand how smoking affects mortality. You can't take 1000 5-year-olds, force them to smoke for 70 years, and compare them to 1000 non-smoking 5-year-olds. In these cases, you take a very large data set and look at the correlations. In other words, if I look at 1000 smokers and 1000 non-smokers, how likely is each group to die at a certain age? Correlation can help you understood how changes in X (smoking, in this case) co-occur with changes in Y (mortality).

PROS – Correlation can help us mathematically find relationships when direct experimentation is impossible or impractical. These techniques can also help model complex situations where multiple variables are affecting the same outcome.

CONS – Correlation does not imply causation. We don't know if changes in X cause changes in Y or if they just happen to co-occur (maybe even due to a Factor Z affecting them both).

(7) Large-scale Simulation

If we can collect enough data, we can build a model of the universe and test hypotheses against that model. Now that large-scale indexes are being built to mimic Google (including our own Linkscape and indexes like Majestic), it only stands to reason that we'll eventually be able to run experiments directly against these models. Although the conclusions we draw from these simulations are only as good as the models themselves, simulation data can help us both improve models and conduct something closer to a laboratory test than is usually possible in SEO.

PROS – Simulations can be controlled. Unlike Google, we know whether we've changed the model or not. Experiments can also be run very quickly and on a very large-scale.

CONS – The result of any simulation is only as good as the model it's built on, and our models are still in their infancy.

Which One Is The Best?

Any type of evidence, including controlled experimentation, has limits. In a field like SEO, where the Google algorithm is constantly changing, relying too much on any one type of evidence can either stall progress or lead us to bad conclusions (or, in some cases, both). Understanding every available source of evidence not only helps us paint a broader, more comprehensive picture, but it also helps us cross-test our hypotheses and prevent mistakes. SEO science is a young and constantly changing field, and, at least for now, SEO scientists will need to adapt quickly.