It Is Not Significant When Your Low Power Test Is Not Significant

In Anon Ymous's  latest response, it seems he feels he's being misunderstood.  Yet, he keeps making statements that demonstrate exactly the view I'm arguing against.

"The Oregon study did not find evidence of statistically significant improvements in physical health. That is a pretty significant result."

No, this is exactly what I've criticized.  If you run a test that has low power, it is NOT significant if you find no effect.  Suppose you have a magic pill that stops you from dying, you give it to 10 undergrads and use 10 as a control group and measure the effect after one year.  You find no statistically significant effect.  Is that finding significant?  NO!!!  Even if the pill prevents 100% of all deaths, you'd have to give it to thousands of undergrads to get statistical significance, because undergrads just don't die very often.

"Obviously, it is not as bad as positive evidence that Medicaid does not improve physical health. Still, if we had the critics and supporters of Medicaid in its current form predict the results on physical health in advance, I think that it is fairly clear that the critics would have been closer."

That the results are far from what supporters of Medicaid would predict is far from clear.  In fact, it is exactly what we are arguing over.  The whole point of my post was to show that the study was consistent with "bold claims about Medicaid" as well as with the view that Medicaid doesn't improve health.  The whole argument is over whether the results were consistent with "bold claims about Medicaid."  Let's try to define "bold claims."  I don't know who particularly Anon Ymous is arguing with, but I can put bounds on what kinds of "bold claims" this is consistent with.

First, what are the bold claims such that you would still be closer to the study's estimate than someone whose prior is an effect on health of zero.  Let's take the percent of people with high blood pressure and the percent of people with high glycated hemoglobin which is a key diagnostic for diabetes.  I use these because there significance is pretty interpretable even for non-medical folks like myself.

For high blood pressure, there is a fall from 16.3% of people to 14.97% of people.  That's an 8% reduction in incidence of high blood pressure.  It's more consistent with a "bold claim" of up to a 16% reduction of high blood pressure than it is a zero percent reduction.

For diabetes, there is a fall from 5.1% to 4.17%.  That's an 18% reduction in the incidence of diabetes.  That's more consistent with a "bold claim" of up 36% reduction in the incidence of diabetes than no reduction in the incidence of diabetes.

I don't see any indication that the study was less consistent with bold claims than critical claims, and I can't imagine why Anon Ymous would say otherwise.

Yet, it goes much further.  If you are a liberal with "bold claims" about Medicaid, we can also ask how bold of a claim you can have before you can no longer say, "The Oregon Health Study does not provide evidence against my bold claim" (to use the phrasing Anon Ymous prefers--I'd put statistically significant in front of evidence, but apparently Anon Ymous is very into classical statistics).

"This study provides no evidence against my belief that Medicaid lowers the incidence of high blood pressure by 40%."

"This study provides no evidence against my belief that Medicaid lowers the incidence of diabetes by 87%."

Those are ridiculously bold claims, but they are also perfectly consistent with the results of this study.

So the real question is, why are the opponents of Medicaid so focused on a this portion of the study where we basically learn nothing about the effects of Medicaid on physical health?

Here is Anon Ymous:

"The defenders of Medicaid seem to think that the burden of proof is shared on this question. It is not. It is squarely on the shoulders of those making these bold claims about Medicaid, as they are the ones defending the status quo of a program that commands a lot of resources at least partially based on very bold claims about what good the program supposedly does. Claims that, thus far, are quite scant on evidence." 

If you think that Medicaid improves access to health care and access to health care is likely to improve health, there is still no evidence that your wrong.  But even though that is a pretty common-sense prior, Anon Ymous doesn't think you should be able to have it.  As best I can tell, he's really upset that Raj Chetty didn't report that his prior was not rejected by the evidence, even though virtually every other prior was also not rejected by the evidence.

Raj Chetty's piece is on the ability of economics to settle disputes.  Raj reported on all the ways the Oregon Health Study settled disputes.  He omitted the portion of the study that didn't settle any disputes.  Perhaps, he could have included a sentence such as, "The effects of Medicaid on physical health were anything from large positive effects to large negative effects."  I think omission is better than including the statement, "the study found no statistically significant effect on health," because as is clear from this discussion, people have a very hard time understanding the meaning of that phrase.