How to Understand & Calculate Statistical Significance [Example]

Have you ever presented results from a marketing campaign and been asked “But are these results statistically significant?” If you were feeling snarky, you might have responded “Well, the results are different from what we saw before. Isn’t that significant?”

All kidding aside, as data-driven marketers, we’re not only asked to measure the results of our marketing campaigns, but also to demonstrate the validity of the data.

Just recently, I had a call with a customer asking that exact thing. The two marketers had each created a version of a landing page and used HubSpot’s A/B testing functionality to collect the results. They had a friendly wager over which one would win.

After a few days, they had the results; one had a slightly higher conversion rate, but they were left wondering if the results were statistically significant. (I’m going to guess that it was the person with the lower conversion rate that asked this question.) I love a little friendly competition -- my family still tells the story about how I challenged my brother to an apple-peeling contest just to “make things interesting” one Thanksgiving. Needless to say, I was happy to help settle this bet.

While there are a number of free tools out there to calculate statistical significance for you (HubSpot even has one here), in order to truly understand what these tools are telling you, it’s helpful to understand what they’re calculating and what it means. We'll geek out on the numbers using a specific example below to help you understand statistical significance. 

1. Determine what you'd like to test.

First, decide what you’d like to test. This could be comparing conversion rates on two landing pages with different images, click-through rates on emails with different subject lines, or conversion rates on different call-to-action buttons at the end of a blog post. The number of choices are endless.

My advice would be to keep it simple; pick a piece of content that you want to create two different variations of and decide what your goal is -- a better conversion rate or more views are good places to start.

You can certainly test additional variations or even create a multivariate test, but for the purpose of this example, we’ll stick to two variations of a landing page with the goal being increasing conversion rates. If you’d like to learn more about A/B testing and multivariate tests, check out "The Critical Difference Between A/B and Multivariate Tests."

2. Determine your hypothesis.

Before I start collecting data, I find it helpful to state my hypothesis at the beginning of the test and determine the degree of confidence I want to test. Since I’m testing out a landing page and want to see if one performs better, my hypothesis is that there is a relationship between the landing page the visitors receive and their conversion rate.

3. Start collecting your data.

Now that you’ve determined what you’d like to test, it’s time to start collecting your data. Since you’re likely running this test to determine what piece of content is best to use in the future, you’ll want to pull a sample size. For a landing page, that might mean picking a set amount of time to run your test (e.g. make your page live for 3 days).

For something like an email, you might pick a random sample of your list to randomly send variations of your emails to. Determining the right sample size can be tricky, and the right sample size will vary between each test. As a general rule of thumb, you want the expected value for each variation to be greater than 5. (We’ll cover expected values further down.)

4. Calculate Chi-Squared results

There are a number of different statistical tests that you can run to measure significance based on your data. Determining which is the best one to use depends on what you’re trying to test and what type of data you’re collecting. In most cases, you’ll use a Chi-Squared test since the data is discrete.

Discrete is a fancy way of saying that there are a finite number of results that can be produced. For example, a visitor will either convert or not convert; there aren’t varying degrees of conversion for a single visitor.

You can test based on varying degrees of confidence (sometimes referred to as the alpha of the test). If you want the requirement for reaching statistical significance to be high, the lower your alpha will be. You may have seen statistical significance reported in terms of confidence.

For example, "The results are statistically significant with 95% confidence." In this scenario, the alpha was .05 (confidence is calculated as 1 minus the alpha), meaning that there's a one in 20 chance of making an error in the stated relationship.

After I’ve collected the data, I put it in a chart to make it easy to organize. Since I’m testing out 2 different variations (A and B) and there are 2 possible outcomes (converted, did not convert), I’ll have a 2x2 chart. I’ll total each column and row so I can easily see the results in aggregate.

statistical-significance-chi-squared-observed-values-1

5. Calculate your expected values.

Now, I’ll calculate what the expected values are. In the example above, if there was no relationship between what landing page visitors saw and their conversion rate, we would expect to see the same conversion rates with both version A and version B. From the totals, we can see that 1,945 people converted out of the 4,935 total visitors, or roughly 39% of visitors.

To calculate the expected frequencies for each version of the landing page assuming there’s no difference, we can multiply the row total for that cell by the column total for that cell, and divide it by the total number of visitors. In this example, to find the expected value of conversion on version A, I would use the following equation: (1945*2401)/4935 = 946

statistical-significance-chi-squared-expected-values

6. See how your results differ from what you expected.

To calculate Chi-Square, I compare the observed frequencies to the expected frequencies. This comparison is done by subtracting the observed from the expected, squaring the result, and then dividing it by the value of the expected frequency.

Essentially, I’m trying to see how different my actual results are from what we might expect. Squaring the difference amplifies the effects of the difference, and dividing by what’s expected normalizes the results. The equation looks like this: (expected - observed)^2)/expected

statistical-significance-chi-squared-values

7. Find your sum.

I then sum the four results to get my Chi-Square number. In this case, it’s .95. To see whether or not the conversion rates for my landing pages are different with statistical significance, I compare this with the value from a Chi-Squared distribution table based on my alpha (in this case, .05) and the degrees of freedom.

Degrees of freedom is based on how many variables you have. With a 2x2 table like in this example, the degrees of freedom is 1. 

In this case, the Chi-Square value would need to be equal or exceed 3.84 for the results to be statistically significant. Since .95 is less than 3.84, my results are not statistically different. This means that there is not a relationship between what version of landing page a visitor receives and conversion rate with statistical significance.

Why Statistical Significance Is Significant

You may be asking yourself why this is important if you can just use a free tool to run the calculation. Understanding how statistical significance is calculated can help you determine how to best test results from your own experiments.

Many tools use a 95% confidence rate, but for your experiments, it might make sense to use a lower confidence rate if you don’t need the test to be as stringent.

Understanding the underlying calculations also helps you explain why your results might be significant to people who aren't already familiar with statistics.

If you’d like to download the spreadsheet I used in this example so you can see the calculations on your own, click here.

Editor's Note: This blog post was originally published in April 2013, but was updated in January 2020 for freshness and comprehensiveness.



from Hubspot

Comments

  1. Wonderful illustrated information. I thank you about that. I am truly glad to read this web site posts which consists of tons of helpful data, thanks for providing these kinds of data.

    Thanks for sharing this nice and informative article. This is a good change. I am really impressed! All the information is dealt with clarity.

    Want to recover from an unexpected financial emergency? Get Emergency Payday Loans Online Now.

    Online Payday Loans
    Fast Payday Loan
    Cash advance loans
    Bad Credit payday Loans

    ReplyDelete
  2. Need an Emotional Support Animal Letter Fast? We help people with anxiety, depression, or chronic stress connect with a mental health professional that will provide an ESA letter quickly. An emotional support animal letter can help you bring your pet into the cabin of an airplane and/or into apartment complexes without paying any fees.

    ReplyDelete
  3. Looking for a reliable digital marketing firm that you can trust? Look no further! Let our team help you find your way on top of your game. Talk with our experts today!

    ReplyDelete
    Replies
    1. Sell My House Fast Arkansas & Nationwide USA
      Sell My House Fast for Cash Nationwide USA. We Buy Houses. Fair Cash Offers. We Buy Houses Arkansas. Any Location, Houses & Land: Residential, Commercial, Industrial, Agricultural. Sell My House for Cash Arkansas.

      Delete
  4. Thanks for sharing. I really enjoyed reading this article.

    International Clubbing Experience Unlike anything in Dubai.

    ReplyDelete
    Replies
    1. This is really helpful.

      Jin Kim is a tax attorney in Sacramento, California. Her tax resolution practice helps individuals and businesses get out of tax debt owed to the Internal Revenue Service (IRS), Franchise Tax Board (FTB), California Department of Tax and Fee Administration (CDTFA), and other tax agencies.

      irs tax attorney | about jin kim tax attorney in sacramento california | tax defense attorney

      Delete
  5. This is an interesting topic. Thank you for sharing.

    ReplyDelete
  6. We Buy Phones Near Me Miami South Florida | Cash for Electronics Miami South Florida | Need to Sell My iPhone, Sell My Laptop Near Me, Tablets or Sell Any Other Electronics? | We pay cash for electronics laptops smartphones in Miami Florida & Nationwide USA

    ReplyDelete
  7. Looking For Mobile Notary Near Me Florida, South Florida? Need a Florida Notary that Makes House Calls? 24 Hour Mobile Notary South Florida; Florida Mobile Notary Services. Traveling Notary in Florida that Comes to You. (305) 874-0269

    ReplyDelete
  8. If you don't know how to book your own fixed place in your client's newsfeed, if you don't know how to make your clients thirsty for your news don't worry welkin house of Digital Marketing Agency in Abu Dhabi مستشارك القانونى اون لاين will give you the perfect opportunity to do so.

    ReplyDelete
  9. If you don't know how to book your own fixed place in your client's newsfeed, if you don't know how to make your clients thirsty for your news don't worry welkin house of Digital Marketing Agency in Abu Dhabi will give you the perfect opportunity to do so.

    ReplyDelete
  10. Choosing a Social Media Marketing Agency in Dubai, and in the United Arab Emirates, in general, is a tiring process, as there is a plethora of advertising and digital marketing agencies out there and it is somehow difficult to choose just one agency.

    ReplyDelete
  11. منافسيك على بعد نقرة واحدة منك ، حركة المرور العالية ليست فقط على الطريق ، بل يمكنك الحصول عليها على موقع الويب الخاص بك أيضًا. مركز وكالة event management companies in dubai يعرف كيف يجلب لك هذه الحركة ويحسن عائد استثمار عملك.

    ReplyDelete
  12. منافسيك على بعد نقرة واحدة منك ، حركة المرور العالية ليست فقط على الطريق ، بل يمكنك الحصول عليها على موقع الويب الخاص بك أيضًا. مركز وكالة smart board in Egypt يعرف كيف يجلب لك هذه الحركة ويحسن عائد استثمار عملك.

    ReplyDelete
  13. No Credit Check Title Loan in Arizona Fast AZ & Nationwide USA. Get Qualified for Title Loans Near Me Today. Fair Cash Offers. Any Location: Online Title Loans Arizona, Get a Loan With Car Title in Arizona Nationwide USA! 1 (833) 461-0143

    ReplyDelete
  14. No Credit Check Title Loan in Oregon Fast & Nationwide USA. Get Qualified for Title Loans Near Me Today. Fair Cash Offers. Any Location: Online Title Loans Oregon, Get a Loan With Car Title in Oregon Nationwide USA! 1 (833) 461-0143

    ReplyDelete
  15. No Credit Check Title Loan in Indiana Fast & Nationwide USA. Get Qualified for Title Loans Near Me Today. Fair Cash Offers. Any Location: Online Title Loans Indiana, Get a Loan With Car Title in Indiana Nationwide USA! 1 (833) 461-0143| https://www.titleloanindiana.com

    ReplyDelete

Post a Comment

Popular posts from this blog

February Social Media News: Weather on Facebook, SNL on Snapchat & More

How to Process Negative Feedback Without Triggering a Stress Response [Infographic]