Measuring Success: An Introduction to Testing

Do you remember your first science fair project?  Did you build a fully functional erupting volcano?  Or did you recreate a diorama of the solar system?  Science projects are a common educational activity for kids all over the world and they function as a means to teach the scientific method; how to ask a question and find an answer.  Testing and recording the results of any variable on your website works the same way.

In continuing our Measuring Success series, we want to express the importance of testing and recording.  Too often companies make changes to their websites without tracking or testing.  Imagine your fully functional volcano letting out an anticlimactic puff of smoke instead of a steady flow of lava!  Clearly something went wrong with your science experiment, but without tracking your changes, how would you know what to fix?  If the scientific method has taught us anything, it’s this:  Not tracking the cause and effect is a sure path to failure.

Constructing the Hypothesis:  Why Should I Track My Changes?

By tracking and recording variables and by adding quantifiable values to the data, you can begin to calculate conversion rates and determine what impact the changes have on your overall success.  In addition, you can track visitor web page trends and correlate them to the factors of your website, economic state and even seasonality.  The variables are limitless.

Just like a science fair project, one of the first things you have to do is choose your experiment.  Perhaps your website is an e-commerce site and the most important measurement is the conversion of site visits to sales.  You might measure conversion like this.

# of visitors who purchase an item / # total number of visitors   = conversion rate

If one hundred people visit your website but forty people visiting your site make a purchase, your conversion rate is 40%.  The more statistical information you can generate from your testing and the changes you make, the greater the chance to enhance your current and future strategies for success.

Testing the Hypothesis:  How Should I Measure the Success of My Changes?

A/B Split Testing is the simplest and fastest approach to testing changes made to the user interface of your website.  A/B Testing lets you compare several versions of the same web page at the same time in order to see which web page produces the greatest results.  You can do this by navigating half of your website visitors to one version of a web page and the other half to another.  A/B Testing simply takes one variable on a page and alters it, while the other page remains the same.  Like a science experiment, one page is your “control group” which allows you to determine exactly which variables have an impact on the way people interact with your web page.  Maybe you see a higher conversion rate on a page with a blue background versus your control page with a yellow background.  The variables to test are virtually endless.

Taken from Paras Chopra’s case study. Click to view

Another tool for testing data is Multivariate Testing.  Multivariate Testing uses the same concept of testing variables, but rather than testing one item at a time, it changes multiple variables and tests the best combination simultaneously for the best optimization.  Maybe you see a higher conversion rate on a page with a blue background and an interactive interface and flashy action buttons; multiple variables.  This optimization testing helps quantify the best changes and offers an ending recommendation for the final website design, layout and content.

When constructing and testing a hypothesis in a scientific experiment, you always want to keep in mind that you are making an educated guess about how things will work:  “If ____ [I do this], then ____[this]___ will happen.”  Using A/B Split Testing and Multivariate Testing correspondingly is the best way to increase effect with the greatest success.  Maybe if you change a layout or design interface, your conversions will increase by 40%…  you just have to test in order to optimize success.

Best Practices For Tracking and Recording


You’re the scientist conducting an experiment, so recording your data as thoroughly as possible is necessary to later communicate your results and prove your “hypothesis.”  The metrics you track become a data log of information that can help define past performances and influence future variable changes for your company’s success.  Like a science experiment, testing can only have three outcomes: no result, a positive result or a negative result.  In order to optimize conversion rates, you have to track the results in order to get to the positive outcome.


A/B Testing is not intended for use on your frequent visitors or repeat customers.  Testing is temporary in nature, so frequently changing your website may cause discomfort and irritation, drastically reducing the success of your efforts.  The variations you change may not ultimately be implemented if you test on regular site visitors.  It is better to test A/B Splits on new or infrequent visitors in a way that the testing cannot be detected.  The most meaningful and relevant testing results will be objective.


A/B Split Testing is a great funnel for initial results.  It can eliminate testing variables by determining poor results quickly and allow for additional testing soon after.  For this reason alone, A/B Testing should be your primary test strategy for determining optimal site changes.  Once those variables have been determined through A/B Testing, you can use Multivariate Testing to then determine appropriate optimization efforts.  We use this type of testing to determine which combinations of changes have the greatest optimization and conversion.


By employing the first three Best Practices, you have a good start to continue building on the success of your website through testing, and we recommend you do multiple tests in order to optimize conversion rates.  It probably took you quite a few tries to get lava flowing from your homemade volcano, right?  There are limitless items to test and endless combinations to test so it makes sense to do many tests.  Data is collected daily and testing will continue refining your customers’ interactions as well as optimize your efforts for conversion.

The ultimate goal of testing is to create a better user experience, which in turn helps increase conversion rates and your success.  Testing turns browsers into buyers – homemade volcanoes into science fair winners.

We would love to hear from you as we continue this ongoing series on Measuring Success.  What are some of the combinations of tests you have performed on your own web site?  Have you seen a change in conversions as a result?  Coming up next in the Measuring Success series, we will look at A/B testing more in depth and how the theory relates to reality.

Tagged with: , , ,


Handcrafting technology and marketing solutions since 2001.
Subscribe to Our Blog

Get valuable tips and ideas via email.
Sign Up Now