With the release this week of the 2016 Federal Invest in What Works Index, I had the opportunity to talk with John Bridgeland and Bruce Reed, Senior Fellows at Results for America on The Business of Government Hour. Here are some of their key insights on
The U.S. federal government spends tens of billions of dollars each year on social programs that have been shown to produce modest results or worse. In other cases, billions of dollars have been spent on programs and funding streams for many years, and yet little is known from rigorous evidence about whether the programs are producing good outcomes.
Given the challenges facing the nation and the budget constraints at all levels of government, public funds must increasingly be invested in “what works.” While much of the current public debate has focused on whether our nation should spend more or fewer resources, it is also critical to get better results from existing resources. Data and evidence of impact are rarely used as criteria in decisions about how to allocate public resources. By investing in what works, limited dollars can be directed towards programs that demonstrate proven success and generate greater impact.
The nation’s social programs are unlikely to be improved until we learn to enact programs supported by rigorous evidence, to improve existing programs based on evidence, and to shut down failing ones, again based on evidence from high-quality program evaluations.
To coincide with the release this week of the 2016 Federal Invest in What Works Index, I had the opportunity to explore the questions and more with John Bridgeland and Bruce Reed, Senior Fellows at Results for America on The Business of Government Hour. I’d like to share some of their key insights on from this year’s Federal Invest in What Works Index as well as the impetus to transform Washington D.C., from an evidence free zone.
On the Importance of Evidence and Program Effectiveness. Evidence is just proof. Washington, as many have said, tends to be an evidence-free zone and what is important to have with any program is the ability to have scientifically rigorous evaluation of whether something works. It is important to agree at the outset what a program is supposed to accomplish, measure what it has done, and agree on the standards of whether it is actually working.
Less than one out of every one hundred dollars in domestic discretionary spending is backed by even the most basic evidence. We don’t want an evidence-free zone. so Results for America and the coalitions committed to improving that are focused on highlighting the importance of investing in what works. To put a finer point on the impetus of our collective effort, there is a GAO study showing that 37% of program managers, only 37% said that their programs had been evaluated in the last five years. A majority didn’t even know whether their programs had been evaluated. The state of play was pretty grim before RFA got into this mix.
To access program effectiveness, you need to have data. You need to collect that data, evaluate it to see whether it is living up to the program’s mission. It is also important to have a common evidence framework because you want to make sure that you’re comparing apples to apples.
The government is awash in data, but the key is identifying the best data and accessing it through a common evidence framework, so that you can discern whether program interventions actually impact outcomes. It is essential that departments and agencies do a better job in marshaling the evidence, committing resources to evaluation, and then integrating findings into resource allocation decisions. The latter will no doubt be the trickiest part of this journey toward evidence -- the tough play will be having leaders actually shift resources from programs that don’t work to programs that do work based on the evidence.
On the 2016 Federal Invest in What Works Index. Over the last several years, Results for American has compiled and published the Federal Invest in What Works Index, which describes how key U.S. federal agencies and departments are increasingly using evidence, data and evaluation to help government make better decisions and get better results. The index is one way RFA seeks to foster and accelerate a commitment to data and evidence amongst federal agencies with the ultimate goal of having them invest in what works.
The 2016 edition highlights the extent to which the Administration for Children and Families (within HHS); Corporation for National and Community Service; Millennium Challenge Corporation; U.S. Agency for International Development; U.S. Department of Education; U.S. Department of Housing and Urban Development and U.S. Department of Labor are currently building the infrastructure necessary to be able to use data, evidence and evaluation in budget, policy, and management decisions.
It is important to note that:
Instead of being a punitive exercise where the index was designed to shame departments and agencies, it was just the opposite. It was a cooperative exercise where more than seventy-five former government officials and those in government cooperated to design an index that would actually achieve the goal that RFA was trying to achieve -- to get government to move strongly in a direction of investing in what works.In turn, it designed about ten indicators of what constitutes a commitment to investing in what works. Initially, only a couple of agencies participated, the Department of Education and Labor, but we continue to ramped up with seven departments and agencies in the 2016 index that are committed to continuously improving in this area. These department and agencies should receive serious accolades for having stepped forward, putting themselves out there, and taking up the evidence challenge. It is particularly difficult in public life where everyone is watching and not everyone is rooting for you to succeed.
As they say, the definition of insanity is doing the same thing over and over again and expecting a different result. Well in government, the real definition of insanity is doing the same thing over and over again and not even measuring the result.
For example, while I [John Bridgeland] was in the Bush administration there was the Even Start Family Literacy Program, which sought to enhance the literacy skills of parents and that will in turn translate to the kids. Initially, it may have sounded like a good theory of change, but three randomized control trials proved that those who were subject to the intervention did no better than the control group. This program did not meet its intended outcome. The theory of change wasn’t supported by the evidence.
The administration recommended that the Even Start program resources be shifted to reading recovery programs, which were proven impactful. Unfortunately, it took eight years, a billion dollars later before Congress finally shut down the Even Start program, redirecting funds to more effective programs. The reality is that for almost a decade due to this inaction young people were not getting access to high quality literacy programs and the opportunity cost of a billion dollars going into programs that weren’t effective. This example hits home for me [John Bridgeland], the power of Bruce’s point about insanity -- the insanity of investing in things that were clearly proven to be ineffective.
On the Importance of Leadership in Pursuing an Evidence Agenda. The dedicated commitment of department and/or agency leaders is crucial. You get out of government what you put into it. It’s important to have senior leadership buy-in and promotion of this effort. It is also important to have an office and resources dedicated to this effort because the front line folks will take their cue from signals given by top department leaders. If you are serious and committed then, leaders need to show that by making it part of their agenda and dedicate the people and resources to make an impact.
As highlighted in the 2016 Federal Invest What Works Index, the Labor Department has the Chief Evaluation Office. Labor is a real leader in this area. It sets aside 1% of funding to do research and evaluation. The Department of Education is also quite a leader. It has an evaluation planning group that has done much over the last few years to push evidence at the state and local level as well. The Corporation for National and Community Service gives twenty-seven out of one hundred points for the applicants to state and national grants that take evidence seriously. The corporation is basically embedding in its reviews and grant making a real commitment to evidence. The reauthorization of the No Child Left Behind Act is littered with evidence-based requirements. For example, the federal government stepped up and said that under Title I, up to 7% of the funds can be spent on plans to reform low performing schools, but you’ve to follow the evidence. That was a big deal and a big signal that government was going to be on the side of those who have proof.
The U.S. Department of Housing and Urban Development’s HUDStat program represents a move in this direction as well. Using this program, the department underscores its commitment to tracking performance and measure what works. For example, HUD and VA are committed to ending homelessness among veterans, so they have a performance review around: “what does the data tell us, what do the evaluations tell us?” Then you can go on their website and you can see the cities where they have ended chronic homelessness among veterans.
For those skeptics of who wonder if government can ever do good, evidence is building that government is doing extraordinary things in areas like reducing homelessness among veterans. It is in part because departments like HUD are committed to taking data and evaluation seriously and then translating it into improving their programs and their outcomes.
Executive branch leadership can also play an important role in focusing the legislative branch. We talk a lot about the executive branch and how often agencies are tasked with carrying out a program that wasn’t that well thought out from the beginning. The very same people who designed that program hold a hearing holding a department or agency leader accountable for the failure of a program that they didn’t design. Within that context, it is critically important to put some onus on congress to think about evaluation, think about results when they design these programs, make sure that a program has a clear mission, that there are clear outcomes, and that the agencies have the resources to measure and evaluate whether they work.
On Pushing the Evidence Agenda Forward. With each successive administration, we’ve seen a renewed enthusiasm for evidence based decision-making and with good reason. The government serves at the will of the people and for the last twenty-five years, there has been a real challenge for anyone working in government to deal with the common belief among average Americans that the federal government couldn’t run a two car funeral.
The movement has come a long way. Now we know how many cars are in the funeral and the President Obama has done a terrific job of signaling to everybody who works in this administration that evidence and results are to be valued and congress is coming along. It’s important for the next administration and the next congress to do the same.
George Orwell noted that our duty in politics is to recognize the truth that is in front of our nose. There are lots of reasons, self-preservation, re-election, pride, and so on that get in the way of that in public life. Ultimately the best thing for everybody concerned, both the tax payers and those who serve them, is to deliver the goods.
If you believe in an affirmative role for government, it’s important for you to have evidence that the program or theory of change you’re pushing actually works. If you’re skeptical of government, the same is true. Americans are an incredibly practical people but the common language that we speak and respect is results.
* * * * * * *
I invite you to listen to my entire conversation on The Business of Government Hour with John Bridgeland and Bruce Reed on the importance of investing in what works and the effort to turn Washington DC into a evidence focused zone.