An often heard criticism of null-hypothesis significance testing is that the null is always false. The idea is that average differences between two samples will never be exactly zero (there will practically always be a tiny difference, even if it is only 0.001). Furthermore, if the sample size is large enough, tiny differences can be statistically significant. Both these statements are correct, but they do not mean the null is never true.

[Now with update for STATA by my colleague +Chris Snijders] [Now with update about using the MBESS package for within-subject designs] [Now with an update on using ESCI] Confidence intervals are confusing intervals. I have nightmares where my students will ask me what they are, and then I try to define them, and I mumble something nonsensical, and they all point at me and laugh.

I used to be really happy with any p -value smaller than .05, and very disappointed when p -values turned out to be higher than .05. Looking back, I realize I was suffering from a bi-polar p -value disorder. Nowadays, I interpret p -values more evenly. Instead of a polar division between p -values above and below the .05 significance level, I use a gradual interpretation of p -values.

Recently, people have wondered why researchers seem to have a special interest in replicating studies that demonstrated unexpected or surprising results. In this blog post, I will explain why, statistically speaking, this makes sense. When we evaluate the likelihood that findings reflect real effects, we need to take the prior likelihood that the null-hypothesis is true into account.

I recently read a meta-analysis on precognition studies by Bem, Tressoldi, Rabeyron, and Duggan (available on SSRN). The authors conclude in the abstract: 'We can now report a metaanalysis of 90 experiments from 33 laboratories in 14 different countries which yielded an overall positive effect in excess of 6 sigma with an effect size (Hedges’ g) of 0.09, combined z = 6.33, p = 1.2 ×10^-10). A Bayesian analysis yielded a Bayes Factor of 1.24 ×

It’s important to communicate any remaining uncertainty in the results of an experiment.

‘The last couple of years it has been repeatedly pointed out that it is essential to perform replication studies.’ This is the first line of a Dutch article by Annie van Bergen, written in 1963 for the Nederlands Tijdschrift voor de Psychologie en Haar Grensgebieden (Dutch Journal for Psychology and her Border Areas). I first heard about it one month ago, when a group of researchers were giving short presentations about the value of replication