The 7 Research Sins

Scenario: A class was given a pre-test just before the holiday period, and the mean of the results was obtained. The students went for their vacation and came back to school 6 weeks later. A similar test was given to the students and the results were collected. This time, it was observed that there is an improvement in the results despite no intervention on the teacher's part.

Why is there an increase in the marks over the holidays?

According to Dr Soh, this is due to the 7 Research Sins, which are due to:

1. Instrument: Topic tested could be easier or marker could probably be more lenient during the second test.
2. Testing Effect: Both topics (before and after) could probably be the same, so students have some recollection of the test.
3. Maturation: Pupils might mature during the holidays based on their experiences, which could be provided by parents.
4. History: In between pre-test and post-test, students might learn from other factors (eg: attending workshops/ tuition classes) despite teacher not teaching.
5. Mortality: A few students might transfer out of school during the holidays and they happened to be the ones that are very weak.
6. Regression to the Mean: This is the most interesting concept that I have learnt and I will explain it later...
7. Performance effect: Students are prepared to work harder because they suddenly developed the impression that the teachers care about them.

Why are they called the 7 Research Sins? According to Dr Soh, if they happened in a research study, then they may mislead the results. As researchers, we might interpret it wrongly too. Hence, this is the reason why there is a need for a control group. A control group will eliminate the '7 Research sins'...and this will help to make the research study more accurate and reliable.

As much as I understand the idea of the '7 Research sins' and the need for a 'control group', I also feel that it depends on the reality of the school context. Ideally, it is good to have one but in the case of a school whereby the whole curriculum is to affect the whole level, then it is almost impossible to have a control group. Of course, there could be alternatives such as comparing to another school of the same standard and by making their students our 'control group'. However, it will be a tedious process as different schools will have different cultures and different teachers. No two schools are alike.

So do we have a control group? Not yet, but I think we can try 'make' one...