In Plain View: A Transparent Systems Approach for Enhancing Health Policy Decisions

Modern times bring modern complexities that call for strategic priority setting. Markets effect some prioritization through the willingness of people to buy and sell products at competitive prices. Other activities—such as public investments in defense, regulation, research, and health services—take place outside of the market and benefit from careful planning, especially as resource constraints loom large in every sector.

In health and health care, global forces such as emerging and reemerging diseases, aging and associated disease burdens highlight the critical need for better understanding of the links between human behavior, culture, and environment to improve outcomes. We describe a systems-based tool that accounts for these factors and their influences while at the same time promoting convergence and transparency in the decision-making process.

Cost-Effectiveness Analysis

In the best of circumstances, planning and prioritization are informed by data and evidence, and many approaches have evolved accordingly. Using accounting and budgeting techniques, early planning efforts focused on program costs, which are easier to measure than outcomes and benefits. Then cost-benefit analysis emerged, providing a mechanism to combine both costs and benefits in widely diverse areas of policymaking.

Many health policy analysts use a similar approach, called cost-effectiveness analysis. While cost-benefit assessments place an explicit value on a life saved or a life-year gained, cost-effectiveness analyses measure health benefits in natural units such as life-years saved or premature deaths averted irrespective of age. The priority-setting process then uses the ratio of incremental costs to incremental health gains—the incremental cost-effectiveness ratio—as a ranking metric.

Health policy analysts and decision makers are often reluctant to set a specific monetary value on lives saved, quality-adjusted life years (QALYs) created, or disability-adjusted life years (DALYs) reduced. For this and other reasons, the World Health Organization recommends the use of a generalized cost-effectiveness analysis to evaluate health programs, eschewing the potentially more complete cost-benefit framework (Tan-Torres Edejer et al. 2003).

Invariably, however, cost-benefit and cost-effectiveness analyses cannot meaningfully incorporate positive or negative externalities that affect individuals who do not receive the intervention in question. They also omit many important factors that determine real policy decisions, such as socioeconomic inequality, public perception of a disease, or faith-based practices. Cost-effectiveness analyses typically carry a caveat to that effect.

Actual policy decisions take these other factors into account, but in ways that are often hidden from public view and discussion. We urge that these factors be brought into plain view as part of the formal decision analysis.

Subjective Attributes

In common practice, decisions about priorities (and actions that follow) rely on a blend of hard analysis and subjective attributes. How these qualitative attributes are weighted in the decision, and how they are combined with data-driven cost-effectiveness and cost-benefit exercises, is not articulated and may not even be explicitly recognized. How much emphasis does each attribute receive in the final decision? In general, there is no way to know. Preferences differ among various stakeholders, and it may not be worthwhile to argue about them: De gustibus non est disputandum (Stigler and Becker 1977).

Consider the domain of vaccine development. Vaccines prevent disease from occurring, and some offer the promise of wholly eradicating the disease. Prevention means that something does not happen, and is therefore hard to measure. (The nonoccurrence of disease or disability is also hard to measure in comparison to dramatic therapeutic interventions such as organ transplants, synthetic joints, or vision-restoring surgery.)

But because vaccines often have their greatest impact in early years of life, their ability to maintain a child’s health can lead to better growth, education, and development, greater lifetime earnings, and thus the improved success of subsequent generations (Rappuoli et al. 2014; Whitney et al. 2014). They can even, ultimately, lead to changes in family fertility decisions, and thus population growth (Montgomery and Cohen 1998) and eventual infrastructure requirements. Because few health interventions offer such dramatic and far-reaching prospects, the application of standard analytical techniques such as cost-effectiveness and cost-benefit to vaccines and other global health interventions is simply inadequate.

Similarly difficult to capture in a conventional cost-effectiveness analysis are issues that arise when the relevant disease—Ebola is a prime example—generates significant anxiety and fear, elements that are typically omitted from cost-benefit or cost-effectiveness models. This challenge is amplified by differing and even conflicting views and values among different stakeholders in the vaccine enterprise.

In the face of such difficulties, alternative techniques that rely on multicriteria systems analysis have greater potential to successfully incorporate the other, nonobjective factors that often drive real-world decisions.

A Systems-Based Tool

Systems-based approaches seek to include all relevant aspects of a decision in a cohesive model that treats all factors comparably by explicitly defining them and assigning them a weighted importance.

SMART Vaccines

In a recent project funded by the US Department of Health and Human Services, the National Academies of Sciences, Engineering, and Medicine developed a systems-based software for vaccine prioritization called the Strategic Multi-Attribute Ranking Tool for Vaccines (SMART Vaccines) (Madhavan et al. 2012, 2013, 2015).¹

SMART Vaccines integrates diverse elements that influence important decisions in a single unifying framework. The model creates a value score for each possible vaccine candidate, based on how well each choice performs against a standardized scale pertinent to each attribute, and then applies weights determined by the user/stakeholder that specify the importance of each attribute to produce a final SMART score. The key feature is the ability to specify each attribute on a common 0 to 100 scale, where nominally zero represents the worst-case and 100 the best-case performance of the candidates.

For example, one attribute measures premature deaths prevented by a vaccine, calculated using data on disease burden, breadth of the vaccination program’s coverage, and the vaccine’s efficacy. A score of zero would denote no deaths prevented, and a score of 100 might represent, for example, the highest envisioned achievement: elimination of half of a population’s deaths from the most serious known vaccine-preventable disease. Other high-end achievements associated with vaccine use could include enhanced quality of life, workforce productivity, and educational attainment. The value of some attributes will be subjectively assessed depending on the particular context—whether the vaccine fits into an existing immunization schedule, avoids the use of cold-chain storage, or benefits target populations (e.g., low-income, military, or native groups).

Once each vaccine’s attributes are measured on the 0 to 100 scales, its overall performance is measured by adding up these attribute-scores, weighted by the importance the user/stakeholder places on each attribute (with the weights summing to 100 percent). The scores are unique to each user since the weights are user-specified.

Applications and Advantages

One direct application of this systems approach is to create a set of possible vaccine strategies (e.g., one, two, or three doses, or increased length of protective immunity) and compare the SMART scores in real time as certain attributes assigned to the vaccine candidates are varied.

As an example, consider pneumococcal vaccines: Would it be more desirable to add more serotypes to the vaccine to increase efficacy or to reduce the number of doses required to achieve a given level of protection, and at what cost? How do these recommendations change if some pneumococcal strains become increasingly resistant to antibiotics?

Perhaps the most important advantage of the systems-based approach is that it brings the decision-making process into full view: How each candidate vaccine performs on each dimension is measured comprehensively. If different people have different estimates of these performances, the differences are obvious and can be resolved through discussion and/or better data. If different people have different preferences (i.e., they weight attributes differently), those differences are clear, since the weights are exactly specified and can be clearly seen and compared across stakeholders. Groups or organizations can create their own official list(s) by agreeing on relevant attributes and associated weights.

Facilitated Discussion and Convergent Decisions: A Hypothetical Illustration

To illustrate how multicriteria systems analysis can facilitate discussion and lead to convergent decisions among possibly competing stakeholders, we created a scenario using SMART Vaccines involving negotiations between a hypothetical health minister and finance minister. In our demonstration, the ministers seek convergence on the prioritization of three hypothetical vaccine candidates for the South African population—rotavirus for infants, or pneumonia or tuberculosis for all ages.

They approach their discussion having used SMART Vaccines with their own predetermined set of attributes and associated weights.

• The health minister’s selection includes three attributes: the potential of the vaccine to benefit infants and children (as a specific demographic consideration, ranked first with 61 percent weight, but applied only to the rotavirus vaccine), premature deaths averted (to represent health benefits, ranked second with 28 percent weight), and net direct costs or savings associated with vaccine use (as an economic factor, ranked third with 11 percent weight).
• The finance minister’s selection involves two attributes: cost-effectiveness calculated as cost per disability-adjusted life years ($/DALY; ranked first with 75 percent weight) and DALYs averted (ranked second with 25 percent weight)—with the understanding that the DALYs averted measure will also capture much of the potential workforce productivity gains in which the minister has interest.

As figure 1 shows, the ministers arrive at discordant priorities. The health minister shows rotavirus vaccine as the top performer (a score of 67) compared to the finance minister’s leading candidate, the vaccine for tuberculosis (a score of 106, exceeding the envisioned best score of 100). The absolute value of each minister’s SMART scores has no meaning in comparison to others’ scores; rather, each individual’s weights create an independent and unique yardstick to gauge vaccine ranking for that individual.

Figure 1

In discussing their chosen attributes and weights, the two ministers come to agree that using the DALYs averted measure could capture much of the health minister’s concern expressed in the earlier combination of life years saved and special benefits to infants and children. They also come to agree that the finance minister’s cost-effectiveness metric ($/DALY) captures in large measure the fiscal concerns expressed by the health minister’s use of net direct costs or savings. But the health minister insists that the special benefit of rotavirus vaccine for infants and children is still important, independent of the technical value of DALYs.

Using SMART Vaccines as a facilitator, the ministers agree on a new set of three attributes for a reanalysis: DALYs averted, $/DALY, and benefits to infants and children. The health minister ranks DALYs averted first, benefits to infants and children second, and $/DALY third (the updated scores are in panel C of figure 1). The finance minister ranks in the order of $/DALY, DALYs averted, and benefits to infants and children (panel D). Despite the difference in ranking of the attributes they have chosen, they now converge on a shared decision for a vaccine candidate: tuberculosis ranks first.

Looking Ahead

In addition to the obvious uses of this software for vaccine prioritization, we envision an extended set of applications based on the core platform that underpins SMART Vaccines. Adaptations of the software could be applied to

• evaluating benefits of diagnostics, therapeutics, informatics, or interventions (say, SMART Health);
• prioritizing among existing or new technology options to reduce the burden of leading chronic diseases (SMART Prevention);
• assisting patients’ choices among alternative treatment options (SMART Choices);
• analyzing response and countermeasure efforts against infectious pathogens (SMART Preparedness);
• considering insurance benefit design (SMART Benefits);
• exploring pricing options for various prescription drug programs (SMART Pricing);
• ranking endpoints/outcomes in clinical trial design (SMART Trials), and
• supporting allocation decisions about investments for innovation in science, engineering, and health (SMART Innovation).

All of these, of course, would require further software development and relevant data, but it is clear that the potential applications of multicriteria systems analysis and decision support are extensive.

We believe it is time to make otherwise hidden and subjective elements of major policy decisions visible and transparent. Globally, the pressure for sound policy decisions is rising in areas not subject to pure market resolution—defense, regulation, research, population health, and others. And recent developments in online social media (e.g., Twitter, YouTube, Facebook, and Instagram) reveal a public now accustomed to transparency. Governmental policy and decision making must accommodate to this reality.

From our experience in developing SMART Vaccines, we believe that policy tools can be designed to incorporate a wide variety of stakeholder preferences. SMART Vaccines is a tool that can serve as a prototype to stimulate product development efforts that facilitate discussion and deliberation among stakeholders and thus promote transparency in policy decisions.

Acknowledgments

We thank the Department of Health and Human Services’ National Vaccine Program Office and the National Institutes of Health Fogarty International Center for supporting this research in part through a contract with the National Academy of Sciences.

References

Madhavan G, Sangha K, Phelps C, Fryback D, Lieu T, Martinez R, King L, eds. 2012. Ranking Vaccines: A Prioritization Framework. Washington: National Academies Press.

Madhavan G, Sangha K, Phelps C, Fryback D, Rappuoli R, Martinez R, King L, eds. 2013. Ranking Vaccines: A Prioritization Software Tool. Washington: National Academies Press.

Madhavan G, Phelps C, Rappuoli R, Martinez R, King L, eds. 2015. Ranking Vaccines: Applications of a Prioritization Software Tool. Washington: National Academies Press.

Montgomery M, Cohen B, eds. 1998. From Birth to Death: Mortality Decline and Reproductive Change. Washington: National Academy Press.

Rappuoli R, Pizza M, Giudice G, Gregorio E. 2014. Vaccines: New opportunities for a new society. Proceedings of the National Academy of Sciences 111(34):12288–12293.

Stigler G, Becker G. 1977. De gustibus non est disputandum. American Economic Review 67(2):76–90.

Tan-Torres Edejer T, Baltussen R, Adam T, Hutubessy R, Acharya A, Evans D, Murray CJL, eds. 2003. Making Choices in Health: WHO Guide to Cost-Effectiveness Analysis. Geneva: World Health Organization.

Whitney C, Zhou F, Singleton J, Schuchat A. 2014. Benefits from immunization during the Vaccines for Children Program Era—United States, 1994–2013. Morbidity and Mortality Weekly Report 63 (16):352–355.

FOOTNOTES