Apples, Oranges, and Outcomes

Apples, Oranges, and Outcomes

Imagine you are planning to make a charitable donation and want your gift to make a real difference. You’ve done your research and found three very effective programs: one provides cash transfers to increase the incomes of very poor households; one provides treatment to correct clubfoot, a congenital condition that causes pain and mobility loss; and one provides children with medication to prevent illness and death from malaria. How do you decide?

Like GiveWell, you may aim to maximize the amount of “good” your donation does. But how can you compare giving opportunities when the good that programs do often looks very different?

Let’s go back to the three options you found. We very roughly estimate that a $10,000 donation might do one of the following:

  • Double the consumption of around 15 very poor people for one year1This is based on only the direct effects of providing unconditional cash transfers that allow people to purchase more goods and services; it does not account for spillover effects or health benefits.
  • Correct clubfoot for eight children
  • Prevent the death of two children from malaria

Estimating the impact of your donation in this way doesn’t resolve the dilemma of where to give—it just raises other important questions. Which of those outcomes has the greatest impact? Is preventing a child’s death more or less valuable than doubling a household’s income? How much more or less valuable? In order to make a decision about where to donate, you’ll need a way to compare very different outcomes.

GiveWell faces the same set of questions. Because our funding is limited, we have to make choices about which programs to support, but those programs don’t all have the same outcomes. We need a way to compare programs to each other.

Assigning moral weights

We do this by using what we call “moral weights,” units of value that we assign to different outcomes. In our cost-effectiveness models, we use moral weights to compare outcomes and help us understand how much a particular program is likely to help people per dollar donated. We have established values for three key outcomes:

  • Mortality: How valuable is preventing a person’s death?
  • Morbidity: How valuable is reducing someone’s suffering?
  • Consumption: How valuable is increasing someone’s ability to purchase what they need?

Over GiveWell’s history, we have done ongoing work to determine values for these various outcomes, and our approach has changed over time.

At one point, we invited all staff to come up with a set of moral weights, then combined their resulting impact estimates in our calculations. Then, in 2019, we funded IDinsight to survey approximately 2,000 people living in poverty in Ghana and Kenya. The survey asked them about their preferences regarding averting deaths at different ages and trading off averting a death with increasing income. The results suggested that, when compared to the moral weights developed by GiveWell staff, the respondents valued averting death more highly than increasing consumption.

Combining the results of that survey with the other work we had done, we assigned a value of 1 to doubling consumption for one person for one year and 100 to averting a death. This basic ratio continues to inform our moral weights. We also found that those participating in the survey valued the lives of children much more highly than the lives of adults. This same pattern showed up when we surveyed our donors. In order to account for this, we now place the greatest value of the life of a five- to nine-year old, with comparatively lower values for people who are younger or older (see here for the range of values and an explanation of how we came up with them).

When a program’s outcome is reducing illness or suffering, we base our moral weights on YLDs (“years lived with disability”), a widely used global health metric equivalent that represents the equivalent of one full year of healthy life lost due to disability or ill health. We assign a value of 2.3 to averting one YLD, then weight particular conditions based on their severity.2We calculated the moral weight of averting one YLD by dividing the value of averting a death at each particular age by the average remaining years of life at that age, then averaging the results across ages. See here for more details.

A simplified example

Let’s look at a highly simplified example to see how these values are used as we evaluate funding opportunities.

Imagine that GiveWell is evaluating a program that prevents people from dying of a disease. If the program saves the lives of 100 people and each life is hypothetically valued at 100 units, the program would generate 10,000 units of value (100 lives x 100 units each = 10,000 units).

Now imagine instead that GiveWell is evaluating a program that prevents mobility impairments for 100 people, and the benefits last for 30 years. To determine the value of that program, we’d start by assessing the severity of the impairment based on values (between 0 and 1) from the Institute for Health Metrics and Evaluation, a research institute that collects and analyzes global health data. We’d then multiply this by the number of people, the number of years during which those people benefit, and our moral weight for a YLD. Let’s say the program prevents “severe motor impairment,” to which the IHME assigns a value of around 0.4.3IHME defines severe motor impairment as “is unable to move around without help, and is not able to lift or hold objects, get dressed or sit upright.” The value of the program would be 0.4 x 100 people x 30 years x 2.3 per YLD, for an overall impact of 2,760 units.4Our actual calculation of a program’s value would also include a discount rate to benefits occuring in the future in order to account for intertemporal uncertainty, such as possible fundamental changes that would render the program ineffective.

Assuming these two programs have the same overall cost, the first program is substantially more cost-effective according to our current moral weights, as it produces more than three times as much impact per dollar. (Again, this is a highly simplified example; our actual analyses incorporate a wide array of other factors.5For example, to determine the value of a program that averts deaths, we need to know the mortality rate associated with the particular condition being addressed in the specific location where the program is being implemented, how much we estimate the program will reduce mortality, the percentage of people we expect will have access to the program because of our funding who wouldn’t otherwise, and much, much more. In addition, programs that avert mortality usually have other benefits, such as decreasing illness or increasing children’s long-term income, so we also calculate those benefit streams and add them to the overall value of the program. And if those benefits extend over time, we apply a discount rate to account for factors like changes in consumption and intertemporal uncertainty. Despite all those considerations, our models are only very rough approximations. While they are useful for comparing programs, we think the specific details are quite uncertain.)

We’re not satisfied

We have invested a lot of time working on these questions, but we’re not satisfied with our current approach. It isn’t fully grounded in a clear rationale, it only accounts for three primary outcomes, and we know that many people (including people on GiveWell’s staff!) value the outcomes much differently.

We’re working on it: we recently updated our moral weights page to more clearly spell out our current approach and open questions that might lead us to change our mind. In addition, we published a report describing our efforts to model the value of contraception, some of our open questions, and considerations for grantmaking.

We also think it’s important to get more input from people who are demographically similar to the people affected by the programs we fund, so we recently funded IDinsight to review recent scholarly literature on how people in low-income countries make trade-offs between health, income, subjective well-being, and contraception, and to pilot new approaches for eliciting information about these trade-offs.

We also know that more work won’t resolve what are ultimately ethical judgments. Historically, we’ve directed more funding to health than to livelihoods programs in part because our current moral weights—which consider an averted death around 100 times more valuable than doubling a person’s income—make it harder for income-focused programs to meet our cost-effectiveness threshold. But we don’t view those weights as the only reasonable standard, so we’re considering how to better incorporate moral frameworks that place greater importance on economic empowerment (for more on this effort, see our recent podcast episode on livelihoods programs).

To allow you to explore the impact of your own values, we have developed a moral weights tool where you can enter your own moral weights to see how that affects our Top Charity programs’ cost-effectiveness.

You might disagree

Translating the suffering and death of real people into units of value can seem callous. Unfortunately, we don’t have enough funding to avert all preventable deaths or address all preventable suffering. We need a way to decide where to allocate scarce resources, and we want to allocate them in a way that has the greatest impact.

Our moral weights help us in that effort, but there are no easy—or absolute—answers to the questions they help us navigate. You might assign more (or less) weight to increasing consumption. You might value outcomes, such as autonomy, that are not included in our moral weights. That’s why, in line with our core value of transparency, we publish the full details of our analysis, including our judgment calls and tough moral tradeoffs: so you can evaluate our work and decide for yourself.

Stay Informed

Get the best articles every day for FREE. Cancel anytime.