I gave an example in this post of an experiment with low power. I thought it would be interesting to do a calculation to actually show how large of a sample that one would expect to need to show a significant effect.
Suppose you have a magic pill that stops you from dying, you give it a group of undergraduates and give a placebo to a control group. You measure the effect on mortality after one year. How large of a sample do you need to find a statistically significant result at the 95% confidence level?
So let's get specific. Sometimes you'll run the experiment and find no effect. The best you can hope to do is have a large enough sample to find the effect a good proportion of the time. It's typical to hope a test is high enough powered to find an effect 80% of the time. We know the drug works (we're assuming that), what we hope to do is design a study large enough that the results will show that the drug works better than a placebo 80% of the time.
So how large should the study be. Well, first you need to know the baseline mortality for undergraduates. I'm getting the number from this study, which shows 22.404 per 100,000 undergrads die over one year. Thankfully, the number is small (.000224), 99.978% of undergraduates won't die over the course of this year.
Using this power calculator, we can see that we'd need 87,000 undergraduates in our study. And this is just to show any effect different than zero.
This calculation is with a drug that ends mortality. That's pretty unrealistic. Suppose the drug just decreases the mortality rate 10%. That'd be a wonderful thing. Now we expect a proportion of .000224 deaths in the control group and .0002016 in the treatment group. Now we need 13.5 million undergraduates for the study just to show a statistically significant effect 80% of the time. That's pretty remarkable.
In the real world, we do mortality studies over many years and in populations with higher mortality than your typical undergraduate. Those tests will have more power. But notice, the effect isn't small in importance. Ending all death among undergraduates or lowering their mortality rate 10% is, I think, really significant. I wonder how many drugs can claim that. The effect is a small number, though, and that makes it difficult to detect.
Suppose you have a magic pill that stops you from dying, you give it a group of undergraduates and give a placebo to a control group. You measure the effect on mortality after one year. How large of a sample do you need to find a statistically significant result at the 95% confidence level?
So let's get specific. Sometimes you'll run the experiment and find no effect. The best you can hope to do is have a large enough sample to find the effect a good proportion of the time. It's typical to hope a test is high enough powered to find an effect 80% of the time. We know the drug works (we're assuming that), what we hope to do is design a study large enough that the results will show that the drug works better than a placebo 80% of the time.
So how large should the study be. Well, first you need to know the baseline mortality for undergraduates. I'm getting the number from this study, which shows 22.404 per 100,000 undergrads die over one year. Thankfully, the number is small (.000224), 99.978% of undergraduates won't die over the course of this year.
Using this power calculator, we can see that we'd need 87,000 undergraduates in our study. And this is just to show any effect different than zero.
This calculation is with a drug that ends mortality. That's pretty unrealistic. Suppose the drug just decreases the mortality rate 10%. That'd be a wonderful thing. Now we expect a proportion of .000224 deaths in the control group and .0002016 in the treatment group. Now we need 13.5 million undergraduates for the study just to show a statistically significant effect 80% of the time. That's pretty remarkable.
In the real world, we do mortality studies over many years and in populations with higher mortality than your typical undergraduate. Those tests will have more power. But notice, the effect isn't small in importance. Ending all death among undergraduates or lowering their mortality rate 10% is, I think, really significant. I wonder how many drugs can claim that. The effect is a small number, though, and that makes it difficult to detect.
No comments:
Post a Comment