Saturday, October 26, 2013

What is Statistical Power?

by Charlie Clarke

More here and here.

Statistical power is the ability to reject false hypothesis.  Intuitively, it means you are measuring the effects of an experiment with small errors.

In a follow up to the post that started a long argument between Scott Sumner and I, Anon Ymous demonstrates exactly what you shouldn't conclude from the Oregon Health Study, "To put it simply, the Oregon study showed that Medicaid does a good job of protecting the poor from crushing medical expenses, but it doesn't make them healthier or save lives."

The correct statement is, "the Oregon study showed that Medicaid does a good job of protecting the poor from crushing medical expenses, but we don't know if it makes them healthier or saves lives."

If you were arguing with someone about whether Medicaid makes people less likely to face financial hardship or improves self-reported well-being, then that argument has been settled.  It does.  You can start arguing about how much, but the effect is not zero.

If you were arguing about whether people on Medicaid have better physical outcomes, then keep arguing, because that has not been settled.  The results in the Oregon Health Study are consistent with a wide range of positions, thus is unlikely to persuade you or your opponent.

Let's take an example of the studies finding on blood pressure.

1)  Medicaid decreased blood Systolic Blood Pressure on average by -.52 mmHg

2)  Medicaid decreased the percent of people with high blood pressure by 1.33%

Both of the effects are measured with very large errors.  The effect of Medicaid on blood pressure is likely somewhere between -.2.97 mmHg and 1.93 mmHg.  Medicaid either lowers the percent of people with high blood pressure by 7% or raises it 5%.

That is, there is a good chance Medicaid lowers blood pressure a lot or raises it a lot.  We don't know.  If you were arguing with someone, as long as there position wasn't Medicaid raises percent of people with high blood pressure by more than 5% or lowers it more that 7%, you won't be able to settle the argument.

The temptation is to do what Anon Ymous did and conclude that since the effects are statistically different from zero that the effect is zero.  But you can see from the confidence that the effect isn't significantly different from -5% either.  The error is so large that it probably didn't reject your prior.

What's the best guess?  The best guess with no other information is always the point estimates.  So with no priors, your best guess is that Medicaid lowers the percent of people with high blood pressure by 1.33%.  That's a far cry from what Scott Sumner and Anon Ymous have been saying.

No comments:

Post a Comment