> It’s inevitable that some of our interventions are going to fail. (If they don’t, we should either be suspicious of the results, or we should be more ambitious with our programs.) What we really want to know is: when we try to improve the world through some policy or program, how often do we succeed?
Looking at policy-making it seems we routinely mistake having a good reason to want something to happen for having good evidence that our proposed intervention will actually make it happen.
Perhaps the crucial missing ingredient is a requirement that policies contain their own mechanisms for being proved wrong?
> Plain packaging — where you strip cigarette boxes of their attractive branding — seems like a no brainer, but in Australia, it looks like the policy caused smokers to switch to cheaper brands and to smoke more
This example provides an easily understood illustration of why so many social programs fail. Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control. Like generals ensconced in their offices located not even on the same continent as the front line they study maps and tables and hear the advice of experts with even less direct experience than themselves and then dispatch orders that arrive too late and attempt to command a reality that does not exist. The reports they receive back show nothing has changed or the situation has got worse and yet they never adapt or see their own fault. Why should they? They've suffered no consequence.
The irony here of course is that most comments on HN - including this one and the one it responds to - are arm-chair commentary on generals' actions, even less informed, even more disconnected.
Unintended consequences are inevitable. The danger isn't that they'll exist - it's that they'll be ignored entirely or ignored until they cannot be, that the desire for panacea produces a willful blindness to their existence or myopic fixation on their effects to the point where the value of the intended consequences are ignored.
Another relevant example here is vaping: an amazing win for harm reduction in existing smokers, but an attractive on-ramp to the addiction for some who would have avoided it otherwise. No panacea.
If you read the report instead of a blog post with an agenda you will find out that the plain packaging accelerated decline in smoking by about 25% (p<0.01)
Was about to comment the same thing, even if they smoked more, the program was meant to reduce the trend of new smokers, period. And it was successful about that
But this policy has been very successful ... Every policy has some unwanted side effects but smoking overall has seen large reductions since its introduction.
Maybe we should save the anger for the rich and powerful people selling addictive cancer rather than the ones who need to find some way of reducing their harm without getting sued or voted out.
Imagine if they had been able to take proper measures decades ago, bankrupting the tobacco companies and imprisoning their executives for covering up the damaging effects of their products and engaging in a massive campaign of false advertising. Maybe that would have moved the needle.
It is not a surprise that when people are given virtually no power to do anything, they have a hard time accomplishing very much.
> One of my favorite genres of psychology experiment is called something like: “the study is probably bullshit but the effect is probably real”. At the top of that heap is The Illusion of Control, a classic from 1975 and a favorite of Psych 101 classes everywhere. Usually, this study gets summarized as “people value their lottery tickets more when you let them pick the numbers vs. when you pick the numbers for them.” It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4
> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5
Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
The thing that is missing from most of these government initiatives (in the US at least) is a mechanism and willingness to revisit and iterate. That's how we do software these days, but social programs are developed and shipped in a full waterfall cycle, then politicians just spend decades arguing about whether it should be cancelled due to some flaws or kept exactly as-is. What's missing are real data-driven attempts to close the feedback loop and fix the issues to achieve the stated objective.
Measuring success is good, but we also need to exercise the ability to make small updates and continue measuring, instead of just using that data to argue on the side of "throw it all out!" or "we can't touch it, it's essential!"
Not having a metric set out in advance may, in fact, be more rigging resistant! unfortunately.
But the real deal is that none of it matters because the program itself is the goat sacrifice. It's not there to work, it's there to signal the proponents values and ingratiate themselves with the goat suppliers. It's actually better for theses purposes if it DOESNT work, because then you can perpetually campaign on solving the same problem.
In most of the examples given the aim is to be seen to be doing something. It is cheaper and/or avoids unwanted change than doing something that actually works.
For example, lots of organisations will have diversity officers and training. Very few do things such as taking names, languages spoken (obviously you would leave any that are relevant to the job) and other clues to sex and race from job applications before shortlisting people for interview.
In the security realm we have security theater. In the social realm, we have compassion theater. "We are spending money on problem X" shows that the supporters of the spending care about people with problem X. Any proposed cut to that spending shows that the opponents to the spending hate people with with problem X and don't want to help them. Meanwhile, the money actually supports those in charge of the spending and various suppliers of goods and services.
> There ought to be an entire liturgy of grotesqueries, in fact—pricked fingers, names written in blood, robes, candles, chanting, obsidian blades, still-beating hearts, the whole shebang.
The idea that such ceremony induces humility may be true in a limited individual sense, in that a person may come to believe that they are nothing but their deity is everything. But at the same time it elevates the status of whoever is the current puppet master of their deity. The end result is net uptick in hubris, on behalf of their lord, whose relayed word is sufficient without any need to resort to such crudities as science or debate.
WASHINGTON (AP) Recent reports indicate the federal goat sacrifice program has failed to produce a meaningful improvement in the success rate of public policy over the past five years.
The program, which costs taxpayers over $1 billion annually at a minimum of $1 million per goat, has also had little impact on out of control spending in unrelated government programs, a finding that reportedly surprised the bill's original cosponsors.
Advocacy groups have condemned the practice as inhumane. Proponents, meanwhile, blame rising costs on the $300 million spent each year on therapy for all individuals involved. Critics are now questioning how much of that funding actually goes toward mental health care, and whether more attention should be paid to the care of the goat prior to slaughter.
Make graffiti classes mandatory in school. Ensure they are taught by gym teachers with no discernible artistic interest or talent. Subsidize a Graffiti text book industry. Create schools of approved graffiti. Launch a "America's Got Graffiti" television show with celebrity randos.
Yes, I’m not disparaging the practice (though I personally think those who do it are deceived). I’m simply making the point that it’s not a real factor in societal change (which is along the lines of the point the article was trying to make).
> The point of this ritual is not to signal our obeisance to the gods (although if any of them want to stop by and help us out, that would be cool), but to remind ourselves that we know nothing. This is procedurally-induced humility, which is desperately necessary, because an alarming number of our best-intentioned interventions do not work.
> For example:
> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.
> Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.
> The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1
> Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2
> Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.
Something that needs addressing on all but the laptop one: there are parties who had a vested interest in seeing these example programs fail. In order:
* people who think public schools exist only as instruments of mass indoctrination for secular values and who would rather not have their money go to that, and/or would like to be the investors in "market-based" solutions to America's supposed education problems
* people who think that anyone and everyone who isn't white, male, middle-class-or-above, and Judeo-Christian gets into college programs because the government forced them in there, and find other ways to get those people to leave, because they see them as a threat to their freedom to do and say whatever they want in the spaces of higher education, and a threat to their sense of intelligence and superiority
* the Taliban. I hope I don't need to explain that one further.
* medical grifters who can't push you things like bleach enemas and colloidal silver if you aren't sick because you got vaccinated, among others
I can't speak to what happened in Peru, but these are all examples where you can easily point to people in American or Afghan society participating in the efforts in bad faith, and thus sabotaging them.
> It’s inevitable that some of our interventions are going to fail. (If they don’t, we should either be suspicious of the results, or we should be more ambitious with our programs.) What we really want to know is: when we try to improve the world through some policy or program, how often do we succeed?
Looking at policy-making it seems we routinely mistake having a good reason to want something to happen for having good evidence that our proposed intervention will actually make it happen.
Perhaps the crucial missing ingredient is a requirement that policies contain their own mechanisms for being proved wrong?
> Plain packaging — where you strip cigarette boxes of their attractive branding — seems like a no brainer, but in Australia, it looks like the policy caused smokers to switch to cheaper brands and to smoke more
This example provides an easily understood illustration of why so many social programs fail. Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control. Like generals ensconced in their offices located not even on the same continent as the front line they study maps and tables and hear the advice of experts with even less direct experience than themselves and then dispatch orders that arrive too late and attempt to command a reality that does not exist. The reports they receive back show nothing has changed or the situation has got worse and yet they never adapt or see their own fault. Why should they? They've suffered no consequence.
The irony here of course is that most comments on HN - including this one and the one it responds to - are arm-chair commentary on generals' actions, even less informed, even more disconnected.
Unintended consequences are inevitable. The danger isn't that they'll exist - it's that they'll be ignored entirely or ignored until they cannot be, that the desire for panacea produces a willful blindness to their existence or myopic fixation on their effects to the point where the value of the intended consequences are ignored.
Another relevant example here is vaping: an amazing win for harm reduction in existing smokers, but an attractive on-ramp to the addiction for some who would have avoided it otherwise. No panacea.
If you read the report instead of a blog post with an agenda you will find out that the plain packaging accelerated decline in smoking by about 25% (p<0.01)
https://www.health.gov.au/sites/default/files/study-of-the-i... (page 12)
They also tried quitting more often (p=0.013)
https://pmc.ncbi.nlm.nih.gov/articles/PMC3710988/
Was about to comment the same thing, even if they smoked more, the program was meant to reduce the trend of new smokers, period. And it was successful about that
What agenda? Like a hidden motive?
But this policy has been very successful ... Every policy has some unwanted side effects but smoking overall has seen large reductions since its introduction.
Maybe we should save the anger for the rich and powerful people selling addictive cancer rather than the ones who need to find some way of reducing their harm without getting sued or voted out.
Those are inherent to complex systems.
You can still try and revert back if it fails.
Imagine if they had been able to take proper measures decades ago, bankrupting the tobacco companies and imprisoning their executives for covering up the damaging effects of their products and engaging in a massive campaign of false advertising. Maybe that would have moved the needle.
It is not a surprise that when people are given virtually no power to do anything, they have a hard time accomplishing very much.
> One of my favorite genres of psychology experiment is called something like: “the study is probably bullshit but the effect is probably real”. At the top of that heap is The Illusion of Control, a classic from 1975 and a favorite of Psych 101 classes everywhere. Usually, this study gets summarized as “people value their lottery tickets more when you let them pick the numbers vs. when you pick the numbers for them.” It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4
> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5
Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
Seems like every social program should have a verifiable measure of success and a termination clause if the measure indicates that it isn't working.
But as software people we all know that turning a measure into a target yields problems.
The thing that is missing from most of these government initiatives (in the US at least) is a mechanism and willingness to revisit and iterate. That's how we do software these days, but social programs are developed and shipped in a full waterfall cycle, then politicians just spend decades arguing about whether it should be cancelled due to some flaws or kept exactly as-is. What's missing are real data-driven attempts to close the feedback loop and fix the issues to achieve the stated objective.
Measuring success is good, but we also need to exercise the ability to make small updates and continue measuring, instead of just using that data to argue on the side of "throw it all out!" or "we can't touch it, it's essential!"
whomever is profiting will just rig the metric.
Not having a metric set out in advance may, in fact, be more rigging resistant! unfortunately.
But the real deal is that none of it matters because the program itself is the goat sacrifice. It's not there to work, it's there to signal the proponents values and ingratiate themselves with the goat suppliers. It's actually better for theses purposes if it DOESNT work, because then you can perpetually campaign on solving the same problem.
These programs are pitched along with statistics, so maybe it’s on us to demand the methods of those statistics continue with the program.
In most of the examples given the aim is to be seen to be doing something. It is cheaper and/or avoids unwanted change than doing something that actually works.
For example, lots of organisations will have diversity officers and training. Very few do things such as taking names, languages spoken (obviously you would leave any that are relevant to the job) and other clues to sex and race from job applications before shortlisting people for interview.
"to be seen to be doing something"
In the security realm we have security theater. In the social realm, we have compassion theater. "We are spending money on problem X" shows that the supporters of the spending care about people with problem X. Any proposed cut to that spending shows that the opponents to the spending hate people with with problem X and don't want to help them. Meanwhile, the money actually supports those in charge of the spending and various suppliers of goods and services.
> aim is to be seen to be doing something.
A signal of an act. Not the act itself. Post l-modernisms ugly head rears itself once again.
I have no comment on the content, but man is it refreshing to read such a well-written article
> There ought to be an entire liturgy of grotesqueries, in fact—pricked fingers, names written in blood, robes, candles, chanting, obsidian blades, still-beating hearts, the whole shebang.
The idea that such ceremony induces humility may be true in a limited individual sense, in that a person may come to believe that they are nothing but their deity is everything. But at the same time it elevates the status of whoever is the current puppet master of their deity. The end result is net uptick in hubris, on behalf of their lord, whose relayed word is sufficient without any need to resort to such crudities as science or debate.
5 years later...
WASHINGTON (AP) Recent reports indicate the federal goat sacrifice program has failed to produce a meaningful improvement in the success rate of public policy over the past five years.
The program, which costs taxpayers over $1 billion annually at a minimum of $1 million per goat, has also had little impact on out of control spending in unrelated government programs, a finding that reportedly surprised the bill's original cosponsors.
Advocacy groups have condemned the practice as inhumane. Proponents, meanwhile, blame rising costs on the $300 million spent each year on therapy for all individuals involved. Critics are now questioning how much of that funding actually goes toward mental health care, and whether more attention should be paid to the care of the goat prior to slaughter.
Problem: Graffiti
Solution:
Make graffiti classes mandatory in school. Ensure they are taught by gym teachers with no discernible artistic interest or talent. Subsidize a Graffiti text book industry. Create schools of approved graffiti. Launch a "America's Got Graffiti" television show with celebrity randos.
Graffiti will be dead within ten years. :-)
I realize the title is clickbait. Still, my immediate reaction is, “Yes, because that worked out so well for Haiti.”
It works for the Gurkhas, who fought in both World Wars. They still “sacrifice” goats for Dashain.
Yes, I’m not disparaging the practice (though I personally think those who do it are deceived). I’m simply making the point that it’s not a real factor in societal change (which is along the lines of the point the article was trying to make).
My Hindu friend sacrificed a gourd instead of a goat for the last full moon.
Love the title, made me laugh :)
> The point of this ritual is not to signal our obeisance to the gods (although if any of them want to stop by and help us out, that would be cool), but to remind ourselves that we know nothing. This is procedurally-induced humility, which is desperately necessary, because an alarming number of our best-intentioned interventions do not work.
> For example:
> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.
> Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.
> The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1
> Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2
> Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.
Something that needs addressing on all but the laptop one: there are parties who had a vested interest in seeing these example programs fail. In order:
* people who think public schools exist only as instruments of mass indoctrination for secular values and who would rather not have their money go to that, and/or would like to be the investors in "market-based" solutions to America's supposed education problems
* people who think that anyone and everyone who isn't white, male, middle-class-or-above, and Judeo-Christian gets into college programs because the government forced them in there, and find other ways to get those people to leave, because they see them as a threat to their freedom to do and say whatever they want in the spaces of higher education, and a threat to their sense of intelligence and superiority
* the Taliban. I hope I don't need to explain that one further.
* medical grifters who can't push you things like bleach enemas and colloidal silver if you aren't sick because you got vaccinated, among others
I can't speak to what happened in Peru, but these are all examples where you can easily point to people in American or Afghan society participating in the efforts in bad faith, and thus sabotaging them.