Statistical Significance: FAQ

Say, you have collected humidity of 5 days for city A and city B- that are adjacent.

City A = {40,45,42,50,42}
City B = {38,45,40,48,52}

The average humidity of City A is 43.8 and City B is 44.6.

Most of the times, we come to a decision that City B is more humid than City A by just taking a look at the average humidity. This is inappropriate. To tell this, we need to further investigate whether or not they are "statistically significant". Average is just a tool to assume not to claim.

To say that two classes significantly differ from each other, we need to test their "statistical significance". There are numbers of tools to test it. I am not discussing them here because one can google them and read more about it. What I am answering here can be seen as FAQs.

What is parametric and non-parametric tests?

If you know that your data of two classes follow normal distribution, then you can choose several significance tests that are parametric. If they don't then choose a non-parametric test.

Link to non-parametric test list

How do I know that my data of two classes follow normal distribution?

A novice approach can be to have class intervals and frequency of occurrence, and then a plot. The plot should contain the class intervals in x-axis and frequency in y-axis. If they form a Bell shaped curve, then your data is following normal distribution.

For in depth precise analysis, click here

Click here if you don't know what a Bell curve is

And to find several normality tests by which you can be confirmed that your data are normally distributed, click here

How do I determine whether I need a parametric test or non-parametric test?

1. If you know that your data follow normal distribution, use parametric test; non-parametric test otherwise.

2. Some values are extremely lower or higher and can even follow normal distribution. Use non-parametric test in this case.

3. If you are confused about the distribution of sample, try to look at the whole dataset rather than the sample.

4. Try to find out the sources that cause the data to scatter. If you have numbers of sources, then it is most probably following normal distribution.

5. If you have large dataset, you can try any one of this- from experiment, it is proved that both of the tests perform well on large dataset. In contrast, they are poor on small dataset.

Last but not the least, many people choose parametric tests as they are not confirmed if the data has lost following normal distribution and many people consider non-parametric tests as they are not sure if the data met the requirements to be normally distributed.

I have seen paired and unpaired tests- which is appropriate?

If you feel that the values of your dataset match with each other, you have to experiment with unpaired tests, paired tests otherwise.

Good, I have seen one-sided and two-sided p value also- can you tell me about them

First, tell me if you know what a null hypothesis is.

No, what is a null hypothesis?

A null hypothesis tells that there is no statistical significance between the two datasets. If you see their average is differing, they are differing by chance only.

Oh, okay, then tell me now about the one-sided and two-sided p value.

If the null hypothesis is true, the one-sided P value is the probability that two averages would differ as much as was observed or further (see the example, they differ, don't they?) in the direction specified by the hypothesis just by chance, even though the means of the overall populations are actually equal. The two-sided P value also includes the probability that the sample means would differ that much in the opposite direction (i.e., the other group has the larger mean). The two-sided P value is twice the one-sided P value.

So, when should I use them?

When you can state with certainty (and before collecting any data) that there either will be no difference between the means or that the difference will go in a direction you can specify in advance (i.e., you have specified which group will have the larger mean), you should use a one-sided p value during your test, otherwise select a two-sided P value.

1. If you select a one-sided test, you should do so before collecting any data
2. You need to state the direction of your experimental hypothesis.
3. If the data go in the "wrong" direction, then you should use a two-sided P value.

It is recommend that you always calculate a two-sided P value.

Micro- and Macro-average of Precision, Recall and F-Score

I posted several articles explaining how precision and recall can be calculated, where F-Score is the equally weighted harmonic mean of them. I was wondering- how to calculate the average precision, recall and harmonic mean of them of a system if the system is applied to several sets of data.

Tricky, but I found this very interesting. There are two methods by which you can get such average statistic of information retrieval and classification.

1. Micro-average Method

In Micro-average method, you sum up the individual true positives, false positives, and false negatives of the system for different sets and the apply them to get the statistics. For example, for a set of data, the system's

True positive (TP1)= 12
False positive (FP1)=9
False negative (FN1)=3

Then precision (P1) and recall (R1) will be 57.14 and 80

and for a different set of data, the system's


True positive (TP2)= 50
False positive (FP2)=23
False negative (FN2)=9

Then precision (P2) and recall (R2) will be 68.49 and 84.75

Now, the average precision and recall of the system using the Micro-average method is

Micro-average of precision = (TP1+TP2)/(TP1+TP2+FP1+FP2) = (12+50)/(12+50+9+23) = 65.96
Micro-average of recall = (TP1+TP2)/(TP1+TP2+FN1+FN2) = (12+50)/(12+50+3+9) = 83.78

The Micro-average F-Score will be simply the harmonic mean of these two figures.

2. Macro-average Method

The method is straight forward. Just take the average of the precision and recall of the system on different sets. For example, the macro-average precision and recall of the system for the given example is

Macro-average precision = (P1+P2)/2 = (57.14+68.49)/2 = 62.82
Macro-average recall = (R1+R2)/2 = (80+84.75)/2 = 82.25

The Macro-average F-Score will be simply the harmonic mean of these two figures.

Suitability
Macro-average method can be used when you want to know how the system performs overall across the sets of data. You should not come up with any specific decision with this average.

On the other hand, micro-average can be a useful measure when your dataset varies in size.

Research Writing: That or Which?

Very simple explanation on "that" and "which". But I would say- this one is the best clarification I found on the web so far. I am confident enough now to use either of the two as the papers I review mostly mix up them.

[Originally from Mignon Fogarty]

"
Restrictive Clause--That

A restrictive clause is just part of a sentence that you can't get rid of because it specifically restricts some other part of the sentence. Here's an example:

Gems that sparkle often elicit forgiveness.

The words that sparkle restrict the kind of gems you're talking about. Without them, the meaning of the sentence would change. Without them, you'd be saying that all gems elicit forgiveness, not just the gems that sparkle. (And note that you don't need commas around the words that sparkle).

Nonrestrictive Clause--Which

A nonrestrictive clause is something that can be left off without changing the meaning of the sentence. You can think of a nonrestrictive clause as simply additional information. Here's an example:

Diamonds, which are expensive, often elicit forgiveness.

Leaving out the words which are expensive doesn't change the meaning of the sentence. (Also note that the phrase is surrounded by commas. Nonrestrictive clauses are usually surrounded by, or preceded by, commas.
"

He as well as Me or He as well as I?

Found a very useful article (don't know the name of the poster in a language forum, but I am acknowledging him/ her with the deepest).

"as well as" functions as a conjunction in #1, not a preposition:

#1. She was into drama and took part in many youth theater productions as well as [took part in] singing in choirs.

"as well as" has two functions:

conjunction: courageous as well as strong.
preposition: The rhetoric, as well as the reasoning, is appreciated.

Notice the commas on each side of the prepositional phrase. They set off or bar the grammar from counting it as part of the subject. That's why the verb is singular "is", and not plural "are". Take the commas away and the prepositional phrase changes identity. It becomes a conjunction + noun phrase that's counted as part of the subject:

conjunction: The rhetoric as well as the reasoning are appreciated.

Below in #2a, there aren't any commas setting off "as well as" from the grammar, so it's counted as part of the subject. "He as well as I" is a compound subject so the verb should be plural "are" (#2b), not singular "is":

#2a. He as well as I is satisfied with the result.
#2b. He as well as I are satisfied with the result.

Subject verb agreement is also a problem for #3a. "He as well as me" is a compound subject; the verb should be plural:

#3a. He as well as me is satisfied with the result.
#3b. He as well as me are satisfied with the result.

Now, add in the commas and "as well as" functions as a preposition,

#2c. He, as well as I, is satisfied with the result.
#3c. He, as well as me, is satisfied with the result.

As a conjunction, "as well as" joins two like forms;i.e., courageous as well as strong; you as well as Sam, but in #3b, below, "as well as" joins two unlike forms, the subject pronoun "He" and the object pronoun "me".

#3b. He as well as me are satisfied with the result.
#3d. He as well as I are satisfied with the result.

Now, "as well as me" is non-standard English, but nevertheless speakers will use "me" as well as "myself" as a way of placing the other person above them. It's a way of humbling oneself.

Excel Graph to EPS (MS Office 2007)

Here is how I convert an excel (MS Office 2007) graph to EPS so that I can use that graph in a TEX file.

1. Open MS Excel. Copy the graph to a new sheet and be careful so that it fits in one page (you can double check if the graph fits in a page from print preview option).

2. Go to File-> Print-> Properties-> Advanced-> Postscript Option and select EPS.

3. The file will be saved as EPS so give the file an extension .eps

4. Open the GSview and open the .eps file you just saved. Go to File and select ps to eps. The file should be given an extension of .eps. This is the final eps file that you can insert in your TEX code.

From Tex to PDF

There are several ways to generate a PDF from a TEX file. I am stating the most popular 4 ways here.

Method 1

If you do not have bibliography file

% latex myfile (to generate myfile.dvi from myfile.tex)
% dvips myfile (to generate myfile.ps from myfile.dvi)
% ps2pdf myfile.ps (to generate the file myfile.pdf)

Method 2

If you have bibliography file

% latex myfile (to generate myfile.dvi from myfile.tex)
% bibtex myfile (uses the .aux file to extract cited publications from the database in the .bib file, formats them according to the indicated style, and puts the results into in a .bbl file)
% dvips myfile (to generate myfile.ps from myfile.dvi)
% ps2pdf myfile.ps (to generate the file myfile.pdf)

Method 3

If you want to convert a TEX file directly to PDF and do not have a bibliography file


% pdflatex myfile

N.B. If you have images in EPS format, you need to convert it into PDF format with the following command-

% epstopdf image.eps

Method 4

If you want to convert a TEX file directly to PDF and have a bibliography file


% pdflatex myfile
% bibtex myfile
% pdflatex myfile
% pdflatex myfile


N.B. If you have images in EPS format, you need to convert it into PDF format with the following command-

% epstopdf image.eps

Two Figures Side-by-Side in Latex

Many times you can come across a situation where you need to put two figures side-by-side in a paper written in latex format. Well, of course, they are two different figures: one is say Figure 1 and the other is say Figure 2. How are you going to achieve this? Well, simply the following latex code helps.

\begin{figure}[b]
\begin{minipage}[t]{0.48\linewidth}
\centering
\includegraphics[scale=0.4]{\string"Path and file name without extension\string".pdf}
\caption{Number of new connections in five chunks for six papers on Ischemia
and Glutamate}
\label{fig:figure1}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[t]{0.48\linewidth}
\centering
\includegraphics[scale=0.4]{\string"Path and file name without extension\string".pdf}
\caption{Number of dropped connections in five chunks for six papers on Ischemia
and Glutamate}
\label{fig:figure2}
\end{minipage}
\end{figure}

The thing to remember is to create minipages. One minipage serves the purpose of putting one figure. All you need to adjust here is the position of the figure (here, I chose the bottom of the page), position of the linewidth of the minipage (mine here is 0.48, yours definitely depend on the page size), and the graphics size.

Qualitative and Quantitative Analysis

Often in research papers, we find that "quantitative analysis on A and B could not be performed... so, we proposed a qualitative analysis on A and B"

What do they mean by quantitative and qualitative analysis and which fits in what cases?

Let me give an example. Sachin Tendulkar and Inzamam ul Huq are two great players of cricket. They have qualities- they are master batsmen, they have strokes, they have techniques and strategies. They, however, have quantities- number of hundreds, number of fifties, number of games played, games as captains.

Now, if someone tells you to make a quantitative analysis on them, what you do is to chart their statistics and make a compare-contrast between them. If someone asks you to provide a qualitative analysis, you analyze their quality as a cricketer and its impact on their game and their teams.

Quantitative analysis is not suitable here. Because, Sachin is opening batsman and has plenty of chances to score hundreds and fifties than Inzamam- who is a middle order batsman. This is just one example- you can find many others. As they are differing in batting positions, quantitative analysis as batsman is not a good choice. However, you can quantify their records as captains. Again, in some cases you cannot do that. Like- maybe Sachin took over captaincy when his team lacked quality players but Inzamam, for example, had the best men in his team. So, it depends.

Quantitative analysis should be done more carefully in compare to qualitative analysis. But again, sometimes it depends. You cannot make a qualitative analysis on Waqar Younis and Sachin Tendulkar as bowlers- it simply is wrong.

I hope you are now able to map these examples in your research and find a way to analyze either quantitatively or qualitatively or even both!

True Negatives and Accuracy

Hello folks! Welcome back! We have talked about Precision and Recall and eventually three terminologies appeared: True positives, false positives and false negatives. Today, we will talk about accuracy and hence, we need to know another term: True negative.

Let's go back to our old example.

You were given 10 balls in a box, 6 of which are white and 4 are red. You were asked to pick only the red balls from the box and you picked 7 balls- 2 of them are really red but 5 white balls fooled you.

Now, our true positives are 2, false positives are 5 and false negatives are 2.

A true negative is what you thought negative and really was negative: in our case, which is- you thought a ball as white in the box, and that ball is really white. So, number of true negatives for our example is:

True negatives = Total Balls - (True positives + False Positives + False Negatives)
= 10 - (2 + 5 + 2) = 1.

Why are we eager to know the number of true negatives? Well, because if we want to measure how accurate you were in picking up red balls, then we need to know your true negatives as well.

The formula for accuracy is

Accuracy = (True positives + True negatives) / (True positives + True negatives + False positives + False negatives)

In your case, your accuracy is equal o (2 + 1) / 10 = 30%.

Efficiency and Effectiveness

Do you use these terms as synonyms? You are wrong!

Efficiency is scientifically output / input. Efficiency is measured to maximize output with minimum resources. It refers to doing things in right way. Say, you produce 10 kgs of potatoes with 1 kg of fertilizer. To be efficient, you need to produce more potatoes with fertilizer measuring 1 kg or less.

Effectiveness, on the other hand, means doing the right thing. Say, you produce 10 kgs of potatoes with 1 kg of fertilizer. To be effective as a farmer, you need to produce more good potatoes than bad potatoes; you don't have to bother about your fertilizer.

Proactive and Reactive Research

Sometimes you hear from your supervisor- "You have to be proactive in research" or sometimes they ask you to be reactive. What does he mean?

Proactive activity means you are cautious about your future, you are planned to face problems in future, or you are prepared for something "bad". When you are saving money for facing troubles in future, it is said to be a proactive activity. So, proactive research means if you know a "situation" may occur in future, prepare yourself. To understand "situations" that may occur, you need to dig your problem and find all possible "situations" that may occur in future.

So, you are a proactive researcher when you analyze your problem and formulate solutions prior to some situations in future.

Reactive activity means a little bit carelessness: I will provide solutions when "situation" occurs or I will react when time comes. So, reactive researchers do not bother about future; they just do what they are meant to do and when they face problems, they try to find a solution.

Mostly, in my opinion, you need to have a blending characteristic to succeed in research: at times you need to proactive and sometimes being reactive will bring you success.

Precision and Recall

These are very confusing terms- precision and recall. You have to understand these terms completely before you are moving forward.

Say, you have 10 balls (6 white and 4 red balls) in a box. I know you are not colorblind but still somebody asked you to pick up the red balls from them. What you did is that you thought 7 balls as red, picked them from the box and put them in a tray. Among these 7 balls, you picked 2 red balls and 5 white balls (but you thought all of them are red).

Your precision in picking red ball is number of correct pick-ups/(number of correct pick-ups + number of wrong pick-ups) which is 2/(2+5) = 2/7 = 28% in this case. Now, look carefully that your denominator can also be like (total pick-ups).

Your recall in picking red ball is number of correct pick-ups/(number of correct pick-ups + number of red balls that you missed) which is 2/(2+2) = 2/4 = 50% in this case.

Now, what do they mean? Precision says how exact you were among your pick-ups. So, as you picked them up as red balls, you were 28% exact. Recall says, how complete you were among your pick-ups. So, as you picked them up as red balls, you were 50% complete in identifying all the red balls.

We learn at this point- Precision describes exactness and Recall describes completeness.

From the same example, we will now take a look how to combine various terminology with these simple examples.

Number of correct pick-ups can be said "true positives" as they were red ball that you picked up and you were asked to pick the red ones. The balls you picked as red but eventually are white can be called "false positives"- you thought they are positive but they are not.

So, if we modify this formula of precision with these terms, it turns into-

Precision = true positives / (true positives + false positives)

Again, the number of red balls you missed are thought as you missed them thinking them as white. So, they can be called "false negatives", which means you thought they are not red balls, but they are.

So, if we modify this formula of recall with this new terminology, it turns into-

Recall = true positives / (true positives + false negatives)

So, from this "re-written" version of recall formula can pop-up one thing: this is the "rate of true positives". In other words, from all the red balls, what percentage of red balls were grabbed. You had 4 red balls but you got 2 and missed 2: means you could took 50% of the red balls!

THIS IS WHAT YOU WILL FIND PRECISION AND RECALL IN THE REALM OF CLASSIFICATION PROBLEM.

Now, we will move to a different realm: INFORMATION RETRIEVAL. It will require sets, so we will change our scenario a little.

You have 10 files in a folder, 4 of which are about games and 6 of which are about weather. Now, somebody asked you to copy only the game files to another location. And what you did is you copied 7 files (thinking all of the 7 files you picked up is about games) and put them in a different location. But you picked only 2 game files and 5 weather files (but you thought they are all games files).

REMEMBER, THE ANALOGY HERE IS THE SAME AS THE RED BALL-WHITE BALL PROBLEM.

Now, your precision of copying games files will be:

Precision = Number of games file both in new location and in old location / number of files you copied.

In this case, which is 2/7 = 28% (the same as our previous example)

Now, your recall of copying games files will be:

Precision = Number of games file both in new location and in old location / total number of games files.

In this case, which is 2/4 = 50% (the same as our previous example)

Some more identical definitions or explanations of these two terms:

Precision
- A measure of the ability of a system to present only relevant items
- The fraction of correct instances among all instances that the algorithm believes to belong to the relevant set
- It is a measure of exactness or fidelity
- It tells how well a system weeds out what you don't want (Confused about this, but it is written in a document)
- Says nothing about the number of false negatives


Recall
- A measure of the ability of a system to present all relevant items
- The fraction of correct instances among all instances that actually belong to the relevant set
- It is a measure of completeness
- It tells how well a system performs to get what you want
- Says nothing about the number of false positives