Statistical Significance: FAQ

Say, you have collected humidity of 5 days for city A and city B- that are adjacent.

City A = {40,45,42,50,42}
City B = {38,45,40,48,52}

The average humidity of City A is 43.8 and City B is 44.6.

Most of the times, we come to a decision that City B is more humid than City A by just taking a look at the average humidity. This is inappropriate. To tell this, we need to further investigate whether or not they are "statistically significant". Average is just a tool to assume not to claim.

To say that two classes significantly differ from each other, we need to test their "statistical significance". There are numbers of tools to test it. I am not discussing them here because one can google them and read more about it. What I am answering here can be seen as FAQs.

What is parametric and non-parametric tests?

If you know that your data of two classes follow normal distribution, then you can choose several significance tests that are parametric. If they don't then choose a non-parametric test.

Link to non-parametric test list

How do I know that my data of two classes follow normal distribution?

A novice approach can be to have class intervals and frequency of occurrence, and then a plot. The plot should contain the class intervals in x-axis and frequency in y-axis. If they form a Bell shaped curve, then your data is following normal distribution.

For in depth precise analysis, click here

Click here if you don't know what a Bell curve is

And to find several normality tests by which you can be confirmed that your data are normally distributed, click here

How do I determine whether I need a parametric test or non-parametric test?

1. If you know that your data follow normal distribution, use parametric test; non-parametric test otherwise.

2. Some values are extremely lower or higher and can even follow normal distribution. Use non-parametric test in this case.

3. If you are confused about the distribution of sample, try to look at the whole dataset rather than the sample.

4. Try to find out the sources that cause the data to scatter. If you have numbers of sources, then it is most probably following normal distribution.

5. If you have large dataset, you can try any one of this- from experiment, it is proved that both of the tests perform well on large dataset. In contrast, they are poor on small dataset.

Last but not the least, many people choose parametric tests as they are not confirmed if the data has lost following normal distribution and many people consider non-parametric tests as they are not sure if the data met the requirements to be normally distributed.

I have seen paired and unpaired tests- which is appropriate?

If you feel that the values of your dataset match with each other, you have to experiment with unpaired tests, paired tests otherwise.

Good, I have seen one-sided and two-sided p value also- can you tell me about them

First, tell me if you know what a null hypothesis is.

No, what is a null hypothesis?

A null hypothesis tells that there is no statistical significance between the two datasets. If you see their average is differing, they are differing by chance only.

Oh, okay, then tell me now about the one-sided and two-sided p value.

If the null hypothesis is true, the one-sided P value is the probability that two averages would differ as much as was observed or further (see the example, they differ, don't they?) in the direction specified by the hypothesis just by chance, even though the means of the overall populations are actually equal. The two-sided P value also includes the probability that the sample means would differ that much in the opposite direction (i.e., the other group has the larger mean). The two-sided P value is twice the one-sided P value.

So, when should I use them?

When you can state with certainty (and before collecting any data) that there either will be no difference between the means or that the difference will go in a direction you can specify in advance (i.e., you have specified which group will have the larger mean), you should use a one-sided p value during your test, otherwise select a two-sided P value.

1. If you select a one-sided test, you should do so before collecting any data
2. You need to state the direction of your experimental hypothesis.
3. If the data go in the "wrong" direction, then you should use a two-sided P value.

It is recommend that you always calculate a two-sided P value.