I’m often asked “What is the minimal sample size I need to do…” This is perhaps one of my favorite questions, as I certainly don’t perform sample size calculations off the cuff, yet there should be some notion as to what is an appropriate minimal N. Often I’m answering this question in the context of reproducibility/reliability analyses and I typically find myself saying “10, no less than 10”. As students we’re taught that there should be 10 subjects per independent variable (IV) in a regression model to prevent under fitting, yet why does this notion of appropriate number of subject s per IV propagate itself into a minimal sample size. As it turns out, there are some valid reasons to think that 10 (or thereabouts) is a magical number in sample size estimation.
Let’s assume we have a standard normal distribution (µ=0, sd=1). Using some simple code in Stata we can see how the 95% Confidence Interval span changes relative to the sample size.
set obs 20
*Calculate and Display Percentage Change in CI span as N increases
list n perci, noobs sep(1)
twoway (connected ci n, mcolor(black) msymbol(diamond) lcolor(gs8)), ///
ytitle(95% CI Span) xtitle(“”) xlabel(#20, labsize(small)) xmtick(, noticks) ///
title(Change in 95% CI Span as N Increases, size(medsmall)) legend(off) scheme(s1mono)
The results show that as N increases, the span of the CI decreases exponentially, with the shape of the curve becoming more asymptotic at around an N of 12. If we examine the table, we can see how the percentage change in the span of CI ( (Current-Previous)/Previous ) flattens after an N of 12.
N % Change
Similarly, when we examine how p values from a t-statistic are affected by changes in the sample size, we see a similar asymptotic relationship, such that increases in N past 10-12 do not offer much advantage in increasing statistical significance for a constant effect size for medium to large effects. Here, Cohen’s d standard effect sizes are presented as small (0.2), medium (0.5), and large (0.8) effect sizes.
While certainly one should not negate the importance of conducting true power analyses and sample size estimations, an N of 10 is by no means an arbitrary number and represents the balance of additional data collection efforts and the ability to detect significant effects (of course only within the confines of the normal distribution).