A "Bayesian" Analysis of the Oregon Health Study

by Charlie Clarke

(A follow up to this post, more here)

Ok, a real Bayesian analysis very carefully formulates priors (best guess hypotheses) and then updates those priors based on the results for a study.  I'm not going to do that, as it requires lots of field specific knowledge and is a fair bit of work after that.  But one of my points of contention with Scott Sumner and Anon Ymous is that as a whole the OHS was bad for people that didn't think Medicaid improved health.  It's not particularly bad the study had low power, but it's pretty clear, which way the sign points.  It's the people that went into the study thinking that Medicaid had no effects on measures of physical health that should be revising up their priors a little.

The reason, as I alluded to last post, is that with no other information, the point estimates of the study are the best guess for the effect of Medicaid on some measure.  In general, the point estimates point in the direction of Medicaid improving outcomes.  These improvements are quite small relative to the errors with which they are measured, but that does not mean they are small in absolute terms.  To make the criticism Scott and Anon Ymous want to make, they really need to argue the whole confidence interval around an estimate is economically insignificant.  If they can show that, then they can argue this study conclusively supports their view.

So how do the point estimates of the actually study look?  I'll try to classify them below, by there answer to this question, "Does Medicaid improve measures of physical health?"  If the point estimate has a sign in the direction of better health, I'll classify it yes.

Physical Health Measures:

Yes

Systolic blood pressure is lower
Diastolic blood pressure is lower
Percent of people with elevated blood pressure is lower

Lower percent of percent of people with high cholesterol
HDL is higher
Percent of people with low HDL is lower

Percent of people with high Glycated Hemoglobin (diabetes marker) is lower

Framingham measure for risk of heart attack is lower

No

Total Cholesterol is up

Glycated Hemoglobin is up

Frammingham measure for high risk individuals is higher


Those are the results.  To me, it looks like Scott and Anon Ymous should be epsilon less confident in there current positions, not asking us to join them.  Two of the No answers aren't necessarily bad.  Since the percent of people with low HDL is lowering, it might be good that total cholesterol is rising.  When its high, glycated hemoglobin is a marker for diabetes and Medicaid lowers that, is a change within the safe range a predictor of anything?

Realize, none of the results are statistically significant.  All of the effects are measured with large error.  But in a perfect world, we'd all carry around really rational and well-thought out prior beliefs and update those beliefs whenever we are confronted with new evidence.

As far as the politics of Medicaid, this certainly doesn't mean that supporters are immune from criticism.  In a more perfect world, Scott and Anon Ymous would be arguing the effects estimated by this study are too small to be worth the money or much smaller that liberal so and so would have thought going in.  If even the high end of the estimates are not worth the cost, then that is a big feather in the cap of Medicaid opponents.

Alas, the commentary we get is on the order of: Study proves Medicaid doesn't make people healthier.  And then we get attacks on Raj Chetty for not parroting that false interpretation.