How to Run Digital Marketing Experiments

Digital marketing is awesome. It’s one of the only industries where you can trace a legitimate change in revenue to something as simple as changing the color of a button or the location of a form. The only problem is that there are so incredibly many variables on a website that it can be hard to tell the effect of one or two particular changes.

Fortunately, there are plenty of tools to help you make and keep track of changes. Unfortunately, none of those help if you don’t understand where, how, or why you’re testing a change.

This gets even more complicated when—like many digital marketers tend to do—the Scientific Method is entirely ignored. This can be frustrating, as the Scientific Method isn’t exactly a new concept. But truth be told though, it can be hard to apply it to digital marketing.

While you undoubtably understand the importance of testing and running experiments (“I want to know something, so I tested it”) and probably where to use it (landing pages, conversion forms, ad text, page titles, etc.), this blog post will formalize a process for running and approaching digital marketing experiments, as well as clear up any misconceptions surrounding how experiments work.

What is the traditional Scientific Method?

Let’s start off first with a refresher. First formally designed by Francis Bacon, the Scientific Method is a process designed to clearly define and answer a question. It is broken into six steps, outlined below.

1. 1. Define the question: Okay, so this isn’t technically the first step. You always want to start with an observation of something weird that you want to know more about. After finding something that doesn’t add up, begin to formulate your question. Keep in mind that digital and otherwise, you don’t have the time or the ability to test everything.

1. 1. Gather information & resources: Before getting started down a new route, it’s always helpful to take a look around and see what has already been learned. There’s a good chance that someone has done a similar experiment to what you’re testing, and that’s a good thing. You’ll want all of that information, as it can help you draw or reinforce conclusions at the end of your experiment. Knowing what has been done before can also help narrow your hypothesis to something very specific.As you take in information from a number of different resources, be wary of non-standardized terminology, misinterpretation of results, and questions on the writer's behalf, and most importantly; be wary of social media as a whole. Try to stick to resources from well-respected individuals or organizations, ‘cause remember; no one lies on the internet.

1. 1. Establish your hypothesis: Your hypothesis is the actual experiment. This is where you clearly define the answer you think you’ll find. It can be pretty tricky to narrow your hypothesis down, but through research and breaking a big topic into digestible bits it can be easier. For example, start off with the topic “Does the amount of sunlight a tomato plant receives affect the size of the tomatoes?” and then narrow it down to a more specific question. “If a tomato plant receives more sunlight, then the tomatoes will grow larger.” It can be helpful to frame your hypothesis in a “if, then” format, but it’s not 100% necessary. It may be more helpful to pose your hypothesis in this format though, if for no other reason than that it formalizes what you expect to see and how you expect to get this result.

1. 1. Perform experiment, collect data: AND DOCUMENT ABSOLUTELY EVERYTHING. If it seems important, take note. With thorough notes you’ll have a clearer idea of the circumstances that surrounded your particular experiment. Taking thorough notes of what happens during a testing window can also help provide insights into preventing the number one problem of all experiments: failure to replicate.

1. Interpret data & draw conclusions: After your experiment runs its course, it’s time to compile all of the data obtained and begin narrowing down the results into a digestible format. A few things to think about as you complete this stage:
   1. Does the conclusion agree with expectations?
   2. Does the conclusion have an alternative explanation?
   3. Does the conclusion agree with other existing data? If not, what was different?
   As you complete your experiment, remember that there is no such thing as a definite conclusion—only increasing levels of certainty that your conclusion rejects or fails to reject your hypothesis.
2. Publish & retest results: Now that your experiment is complete, you get the chance to tell everyone what you’ve learned and how to recreate this. You also now get the chance to test your results and to make sure that your experiment can be replicated again and again. See why the notes help, now?

How is the Scientific Method used for Digital Marketing?

Okay, fine. It’s largely the same, except with a lot more legwork involved for digital marketers. Digital testing is more involved than tradition for two reasons. First, there are so many different ways to introduce noise into an experiment that it can be hard to control for just one or two variables. Second, it’s also exceedingly difficult (read: damn near impossible) to create a true control group—doubly so in SEO.

So, if you were going to test landing page copy, you won’t actually be able to establish a true control group. Instead, use the previous month's data, assuming that you’re examining traditionally similar months. Try to look for time periods that are fairly stable, rather than those that experience massive seasonality, like eCommerce websites tend to see in November/December. Then, as you make your experiment, avoid making any other changes (adding extra variables), as this could disrupt your experiment and invalidate your conclusion.

Anyways, let’s examine the differences a little more in-depth.

A Modern Scientific Framework

1. 1. Evaluate all existing data sources: Start with an audit of your data sources. You want to be sure that you’re tracking absolutely everything that can actually be tracked, and that your data is coming in as clean as possible. This means checking things like popup windows, iframes, links taking users off sites, tabbed content, any third party websites or plugins (both of which can break at any time, and can be difficult to discover).

1. 1. Review website basics: This is when you start checking things like site-wide performance, how your conversion funnel performs, technical errors on the site, what customer segments are clearly identified and how they interact with your site, and the baselines of how each marketing channels performs. Essentially, after you audit your data sources, this is how we determine our baselines and when the best time is to run our experiment.
      

1. 1. Generate your hypothesis: This part doesn’t really change from traditional to digital.You still want to be sure that your question is data-backed, and will generate improvements that will outperform your current setup. Conversion Sciences, one of the top CRO companies out there, recommends approaching a hypothesis with this simple flow chart. To be honest, we’ve put a lot of thought into this and still have yet to determine a better approach.

1. 1. Develop your approach to testing: How are you going to actually determine how things are changed? Write this all down. Ideally, your experiment can be replicated by others on your team, and even other marketers (should you choose to brag about your incredible 10,000x results) due to your meticulous note taking. Be sure to mark all of the following:
      • Experiment duration: How long will it last?
      • Goal: What are you trying to see?
      • Percentage: What percentage of your site’s total traffic will be seeing the test?
      • Targeting: How will you determine who will be entered into the experiment?
      • Treatment design: The creative change for the test treatments.
      • Test code: How are you implementing or moving things around the page for each experiment?
      • Approval: Do you even have internal approval of the test and approach? Is this documented with intent and expected results?

1. 1. Run & monitor the test: Hit ‘go’ on your experiment for your stated period of time, and check in to make sure all is running well. If things break, you’ll want to know right away as this may cause your experiment data to be invalidated, so be very active in checking to make sure that everything runs appropriately. Think of this as running QA throughout the testing period to make sure all is well.

1. Evaluate test results: At the end of the day, regardless if you’re looking at a traditional experiment or a digital marketing experiment, you’ll still need to determine the outcome. We’re going to borrow some terminology from statistics to make this clear. There are only two possible outcomes of your experiment. Let’s take a look:
   1. The test was inconclusive: Sometimes these inconclusive experiments yield the best insights, so don’t be discouraged if none of the alternatives beat the control, indicating that the hypothesis was not disproven, or that we can reject the hypothesis. If this is the case, then we want to examine each segment and channel to determine if there is any difference in how users are engaging with the control (what we didn’t change) and the variant (what was changed).
   2. One or more of the treatments beats the control: Of course the key here is that a statistically significant amount of users exposed to the treatment managed to surpass the outcome of the control. To determine this, analyze the quality of the new versions and determine if you're looking at a legitimate statistical increase in the conversion rate. Then, examine to make sure that the new conversions are just as valuable as those in your control group. Ultimately, you’re using this step to determine “Why did one variation win and what does this tell us?”

Go Forth And Test

So we may have been a bit over dramatic. The Scientific Method doesn’t dramatically change as you go from testing in a lab to testing on a site. You still need to be sure you’re asking the right, testable, question and examining it with the cleanest data possible in a statistically significant sample size. It’s still best to do so with a somewhat rigid structure to allow for replicability, and you still want to explicitly document everything. Problems only arise for digital marketers when it comes time to reduce all that noise in your data set, as well sa control for only specific changes. As a result, most people choose to ignore what 700 years of science has taught us just because it doesn’t seem like it intuitively applies.

That’s what we want to change. Science has given us a rigorous procedure designed to conduct and interpret experiments.

Use it.

To make get you started, download this Experiment Outline, created to contain your notes, experiment design, and results.

Happy testing!

Do you have any questions, comments, or concerns? Reach out to speak with one of our expert analysts!