Who should you donate your money to? Not all charities are equally good at helping people. If your choice is between “The Pershan Family Vacation Fund” and “Give Food to Starving Children” you should definitely give money to hungry children. Then again, how do you know if the kids really are starving? And maybe the charity is corrupt? Say what you will about that Pershan Fund, the money really does go where they say it will.
These examples are silly, but reality isn’t that much different and it gets complicated very quickly. It’s good to give to charity, but where should you give? This is why Givewell exists — “We search for the charities that save or improve lives the most per dollar” is their motto, which is reassuring in its frank dorkiness. They are optimizing a rate, which means they are institutionally committed to solving a calculus problem.
For years, I’ve used Givewell to guide my giving. I’ve followed their advice in donating to various anti-malaria groups. Their ethical advice in has seemed to me entirely sound, and I’ve been grateful for their guidance.
But over the past few months, something has been bothering me, which is that in 2011 their top recommended charity for the United States was … KIPP, the charter school network. In particular, KIPP Houston, which they thought was especially in need of additional funding. (Givewell no longer lists top US charities in the same way, so the recommendation isn’t current.) The more I think about it, the weaker this recommendation seems. But I trust and admire their work in other areas so much, I don’t know what to make of the KIPP recommendation. And I still don’t.
One of the many admirable things about Givewell is that they make their reasoning public, so the case for donating to KIPP is crystal clear. The case begins by citing a trio of studies that find KIPP schools effective at improving test scores. They note concerns about students leaving and retention, but the studies seem to cover these issues. Basically, there is evidence that KIPP increases test scores and that is the basis for their recommendation.
If I’m reading correctly, the Givewell evaluators seemed to have two major concerns with KIPP: teacher burnout as a proxy for KIPP’s sustainability and capacity for growth, and whether KIPP really could do more with more funding.
It seems to me that the really obvious concern is not raised in their evaluation, which is “how sure are you that it’s very important to raise children’s test scores?” I find it genuinely confusing that this question wasn’t raised in the report? They write, “Note that we have focused our analysis on KIPP’s impact on middle-school test scores and not longer-term impacts such as high school and college graduation or adult earnings because we have not seen this data.” That’s the whole point, though!
What’s especially surprising is that even KIPP doesn’t treat test scores as their measurable endpoint. Their self-professed endpoint was reducing poverty by helping more low-income students get through college and towards higher-paying jobs. While Givewell labeled KIPP the top US charity, KIPP went through an effort to increase their college graduation rate, a crucial step in the “do well on dumb tests, go to college, and reduce poverty” plan. How is that going?:
The college graduation rate for KIPP alumni is about 35 percent, above the national average for low-income students but not nearly as high as its founders had envisioned. After years of attempts to help KIPP alumni graduate, the network is proposing new solutions, which it hopes other schools will emulate.
I raise this not to criticize KIPP or the way that it’s handled this. I wish more schools thought hard about how their students do in college. But if you’re an organization that recommends charities and you’re evaluating a anti-poverty project by how well it increases test scores, something is off. It’s like if instead of measuring the impact of the Anti-Malaria Foundation by how many lives it saved you measured it by how many mosquito nets they purchased — precisely the sort of mistake that Givewell is typically so good at avoiding.
There are other concerns that you could have raised in 2011 about KIPP. Research over the past decade has made it clearer that charters might really impose costs on surrounding public schools. Matt Barnum writes:
Charter schools really do divert money from school districts. Those districts can make up for that by cutting costs over time. But the process of doing so is often fraught, especially because the most straightforward way to reduce costs is to close schools.
The systemic issues are important, but they don’t seem as basic as the most basic point: did Givewell evaluate the value of increased test scores for poor children? No I don’t think they did? In which case what is the basis of their recommendation of KIPP as a top charity for the US, when I could give money instead to a local food shelter?
I’m trying to think this through in a non-hysterical way for two reasons. First, because I’m a teacher with teacher readers and it’s very easy for us to play Whack-a-Charter. I don’t want to do that. Second, because I genuinely admire the people at Givewell and I am inclined to respect their recommendations in general. And yet I can’t help but think that some sort of large mistake was made with KIPP.
If there was a mistake, what was its nature? Here are some options, and I don’t know which is true:
- They didn’t put much effort into evaluating US charities because people in the US are so much wealthier and better off than those in areas their top charities target. In other words, the KIPP recommendation was lazy.
- Charters were quite popular in a bipartisan way in the late 2000s and there was a general consensus that increasing test scores would decrease poverty. The mistake of Givewell was not critically examining this consensus.
- Givewell’s recommendations are in general not as reliable as they seem. I noticed this one because I work in education, but people in international development would also easily recognize mistakes in their giving recommendations overseas. This would be a case for expertise in a field being important in addition to a general commitment to an open, transparent, evidence-based process.
- I am wrong, and they did seriously consider the tradeoffs with KIPP and I’m just missing something.
What makes most sense to me is that it’s some sort of combination of the above. Maybe I’m being a bit unfair using my 2021 brain to evaluate a 2011 claim about charters. That movie “Waiting for ‘Superman'” came out in 2010 and it was a pretty blatant piece of pro-charter propaganda that generated a lot of discussion — such were the times. And maybe Givewell didn’t give the evaluation the full Givewell analytic effort. I see that a volunteer graduate student in plant biology evaluated the KIPP recommendation favorably. Would someone working in education have been able to raise concerns? Probably. And it’s to their credit that eventually Givewell probably realized that recommending US charities that weren’t really their recommendation (since what they actually recommend is giving internationally) wasn’t their core competency, and they stopped doing this.
Still, this gnaws at me. Is there a lesson here? And is the lesson, be very careful when thinking outside your expertise?