Though not as clear-cut as you might want, and more gradual than your examples, pre and post WWii Okinawa provides a natural experiment, in particular when it comes to the food environment.
The socioeconomically relatively poor Okinawans had, up to WWii, access largely to vegetables, including a staple, sweet potatoes, and had e.g. exceptionally high longevity and number of centenarians per capita. After WWii their food access gradually Americanized, but also Japanized, and 2015 male Okinawans ranked 36th of 47 prefectures in life expectancy. For example, Okinawa has Japan's highest number of hamburger restaurants per capita. [1]
(All weight gain can be explained with increased supply.[1])
[1] For references to the claims, see the linked draft, section 1.5.2, first paragraph:
Another important aspect of RCTs and applying to the care of individual patients is answering the question, “Were patients like the one in front of me in the RCT?” Often study participants have more social capital and higher levels of functioning than the population at large. The examples you cite help address this concern as well as the “Study Effect” which has a favorable influence regardless of the group to which participants are randomized.
This is a great point. It's why I think natural experiments are useful to understand what treatment effects look like in the real world. Also, the ability to detect heterogenous treatment effects is much greater in large scale real-world data than in RCTs.
I suspect the "doctor roulette" findings are probably real, but also probably not as powerful as you think.
I am going to be the cynic here again, but I don't know how you can calculate how many games people play, both patients and doctors on these issues. I know for a fact that in my small town patients call up to the urgent care to find out if the provider there is one "that they like" or not. There are people who just won't come in if they know it is someone who won't give them abx for their allergies.
You would like to think that ER docs wouldn't cherry pick who they see, but I also think there is a real possiblity that the "no narcs!" doctors might see a 34 year old boarded for "chronic back pain, out of meds" and just pass that guy over for a chest pain knowing their less stringent partner will pick him up in 10 minutes and do what needs to be done to get him out of the ED.
In any case I think there are more variables, deeply hidden, at play here than just day of the week. There is also an arithmetic where some of the people who want antibiotics are going to go back to the ER day after day until they get them. So some fraction that sees the "it's just a virus" doc on Monday is destined to end up at a more lenient doc eventually and it doesn't work the opposite way. The folks who see the Levaquin for all doc get seen once and are happy.
Clearly we somehow/somewhere DO in fact create the monster of "I always need antibiotics for this", but once created it is self-perpetuating.
This is a good point. The "identifying assumption" in these kinds of doctor-variation studies is that within some setting (a given hospital or a given ED) the assignment of a patient to a provider is as good as random. Now clearly, some EDs or urgent care centers are different than others so the key is to looking *within* the same facility. That said, concerns about selection may still remain if certain doctors in a facility choose certain patients or vice versa. The standard tool we use to try and assess if that's happening is look at the observable characteristics of the patients. Do they vary by provider, on average? If so, that's concerning for selection. But typically we don't see that, i.e., we see that patient characteristics are "balanced" across the high- and low-prescribing providers. Now, it's still possible that patients may differ on unobservables when they are balanced on 15-20 observables, but i tend to think that's less likely. Either way though, the concern about non-random assignment is one to take seriously!
I always wonder a lot about the value of "placebo-controlled" although we all sort of want to see that as the gold standard. I don't think we talk enough about placebo and how IT works.
I was looking at a placebo-controlled study recently for esketamine for depression. Is that something that CAN be placebo-controlled? I am sure one group knows they are getting active drug and the other group knows they are not. The group that got dissociated did better than the group that did not. Depression measures are pretty subjective. You see this with all the psych meds. The placebo arm in the SSRI trials all improve and then the SSRI arm does slightly better. But probably the study arm knows they are getting the real deal. They know they can't orgasm.
Although it is a terribly fraught area to even look at I wonder if there is a "natural experiment" way to shed light on some of the transgender medicine questions society is asking. In the same way as with the antidepressants it is hard to really blind someone as to if they are getting puberty blockers or testosterone. But maybe you could compare some sort of outcomes in groups where one had access and the other did not.
Don't forget that people also experiment on themselves (see: Quantified Self) with a wide variety of drugs and natural supplements to see if they can get positive benefits. I do this regularly.
I am currently testing Lion's Mane supplement, which purportedly has neuropathy benefits and Longjack, which purportedly increases testosterone and might help in workouts. I also consume a wide variety of other supplements. Most of the things I try don't seem to bring any clear benefits.
One that has failed me is Hyaluronic Acid. I wanted to take this for potential skin and joint benefits but it cause an allergic reaction resulting in itchiness and raised welts on my body.
OTOH, at age 71, I am in VG health and don't have need of any prescription drugs, which I am confident annoys the pharmaceutical industry to no end.
Though not as clear-cut as you might want, and more gradual than your examples, pre and post WWii Okinawa provides a natural experiment, in particular when it comes to the food environment.
The socioeconomically relatively poor Okinawans had, up to WWii, access largely to vegetables, including a staple, sweet potatoes, and had e.g. exceptionally high longevity and number of centenarians per capita. After WWii their food access gradually Americanized, but also Japanized, and 2015 male Okinawans ranked 36th of 47 prefectures in life expectancy. For example, Okinawa has Japan's highest number of hamburger restaurants per capita. [1]
(All weight gain can be explained with increased supply.[1])
[1] For references to the claims, see the linked draft, section 1.5.2, first paragraph:
http://dx.doi.org/10.31219/osf.io/bq438
PS: Great NYT article!
This is so interesting. This reminds me of something that David Epstein wrote in his piece about our book last week! See here: https://davidepstein.substack.com/p/freakonomics-but-for-medicine.
Another important aspect of RCTs and applying to the care of individual patients is answering the question, “Were patients like the one in front of me in the RCT?” Often study participants have more social capital and higher levels of functioning than the population at large. The examples you cite help address this concern as well as the “Study Effect” which has a favorable influence regardless of the group to which participants are randomized.
This is a great point. It's why I think natural experiments are useful to understand what treatment effects look like in the real world. Also, the ability to detect heterogenous treatment effects is much greater in large scale real-world data than in RCTs.
I suspect the "doctor roulette" findings are probably real, but also probably not as powerful as you think.
I am going to be the cynic here again, but I don't know how you can calculate how many games people play, both patients and doctors on these issues. I know for a fact that in my small town patients call up to the urgent care to find out if the provider there is one "that they like" or not. There are people who just won't come in if they know it is someone who won't give them abx for their allergies.
You would like to think that ER docs wouldn't cherry pick who they see, but I also think there is a real possiblity that the "no narcs!" doctors might see a 34 year old boarded for "chronic back pain, out of meds" and just pass that guy over for a chest pain knowing their less stringent partner will pick him up in 10 minutes and do what needs to be done to get him out of the ED.
In any case I think there are more variables, deeply hidden, at play here than just day of the week. There is also an arithmetic where some of the people who want antibiotics are going to go back to the ER day after day until they get them. So some fraction that sees the "it's just a virus" doc on Monday is destined to end up at a more lenient doc eventually and it doesn't work the opposite way. The folks who see the Levaquin for all doc get seen once and are happy.
Clearly we somehow/somewhere DO in fact create the monster of "I always need antibiotics for this", but once created it is self-perpetuating.
This is a good point. The "identifying assumption" in these kinds of doctor-variation studies is that within some setting (a given hospital or a given ED) the assignment of a patient to a provider is as good as random. Now clearly, some EDs or urgent care centers are different than others so the key is to looking *within* the same facility. That said, concerns about selection may still remain if certain doctors in a facility choose certain patients or vice versa. The standard tool we use to try and assess if that's happening is look at the observable characteristics of the patients. Do they vary by provider, on average? If so, that's concerning for selection. But typically we don't see that, i.e., we see that patient characteristics are "balanced" across the high- and low-prescribing providers. Now, it's still possible that patients may differ on unobservables when they are balanced on 15-20 observables, but i tend to think that's less likely. Either way though, the concern about non-random assignment is one to take seriously!
I always wonder a lot about the value of "placebo-controlled" although we all sort of want to see that as the gold standard. I don't think we talk enough about placebo and how IT works.
I was looking at a placebo-controlled study recently for esketamine for depression. Is that something that CAN be placebo-controlled? I am sure one group knows they are getting active drug and the other group knows they are not. The group that got dissociated did better than the group that did not. Depression measures are pretty subjective. You see this with all the psych meds. The placebo arm in the SSRI trials all improve and then the SSRI arm does slightly better. But probably the study arm knows they are getting the real deal. They know they can't orgasm.
Although it is a terribly fraught area to even look at I wonder if there is a "natural experiment" way to shed light on some of the transgender medicine questions society is asking. In the same way as with the antidepressants it is hard to really blind someone as to if they are getting puberty blockers or testosterone. But maybe you could compare some sort of outcomes in groups where one had access and the other did not.
Don't forget that people also experiment on themselves (see: Quantified Self) with a wide variety of drugs and natural supplements to see if they can get positive benefits. I do this regularly.
I am currently testing Lion's Mane supplement, which purportedly has neuropathy benefits and Longjack, which purportedly increases testosterone and might help in workouts. I also consume a wide variety of other supplements. Most of the things I try don't seem to bring any clear benefits.
One that has failed me is Hyaluronic Acid. I wanted to take this for potential skin and joint benefits but it cause an allergic reaction resulting in itchiness and raised welts on my body.
OTOH, at age 71, I am in VG health and don't have need of any prescription drugs, which I am confident annoys the pharmaceutical industry to no end.