Harnessing Evidence and Evaluation: Insights from Kathy Stack, Advisor, Evidence-Based Innovation, OMB

The federal government spends tens of billions annually on social programs with modest or poor results. In other cases, billions have been spent on programs and funding streams while little rigorous evidence exists about program outcomes. In a climate of fiscal austerity, it is far better to cut programs with minimal impact and improve existing programs, based on evidence from high-quality program evaluations. What is program evaluation? How can evidence and rigorous evaluation be best integrated into decision-making?

Kathy Stack, Advisor for Evidence-Based Innovation, Office of Management and Budget (OMB) shares her insights on these topics and more. The following is an edited excerpt of our discussion on The Business of Government Hour.

What are you doing now at OMB?

OMB is pursuing an aggressive management agenda that delivers a smarter, more innovative, and more accountable government for citizens. An important component of this is strengthening agencies’ abilities to continually improve program performance by applying existing evidence about what works, generating new knowledge, and using experimentation and innovation to test new approaches to program delivery.

I’m advisor for evidence-based innovation within OMB, a new role created in July 2013. I have a staff of three and our mission is to help federal agencies use evidence and data to inform decision-making. It’s all about creating partnerships and coalitions of the willing who can try to make things happen together.

What are your top challenges and how have you sought to address those challenges?

I am rediscovering how important it is to build trust with agencies. Many are not used to sharing information. Making progress on my agenda requires encouraging people to be candid about the challenges they face or their lack of expertise. The second challenge has to do with available resources. Much of what we do is statutorily mandated, but no statutory mandate specifically requires agencies to use evidence, evaluation, and data to make better decisions. I am doing my best to find allies and partners inside agencies and also external partners … think tanks and nonprofits. The third challenge is maintaining momentum in the face of leadership transitions. Fortunately I had a great transition from the Bush administration into the Obama administration.

Along with the challenges you encounter, most governmentwide efforts can be fraught with unanticipated or unexpected surprises. What surprised you most?

I have been struck by how similar decision making is for OMB leadership regardless of a Republican or Democratic administration. When OMB leaders are presented with very compelling data and evidence they’re going to reach similar if not identical conclusions. When you don’t have data and evidence, ideology tends to fill that gap. It’s [also] amazing, the ability that OMB or the White House has, when they can bring agencies in and help them become part of creating that vision … they get excited about it and then some great things can happen.

The federal fiscal situation necessitates improvements in efficiency and doing more with less. Programs can use a broad range of analytical and management tools, i.e., an “evidence infrastructure,” to learn what works and what does not work. Would you briefly describe performance measurement and program evaluation?

Performance measurement is the ongoing monitoring and reporting of program accomplishments and progress toward established goals. It looks at inputs, process measures, outputs, outcomes, in order to manage programs, set goals, and continually improve performance. It’s well suited to dashboards [which show] performance over time on different indicators. It can provide valuable information that enables you to view your performance and if necessary figure out how to course correct. Every organization needs to use performance measurement as a management tool.

Program evaluation answers different questions. Evaluations are typically systematic studies conducted periodically to assess how well a program is working. There are many grant programs that address various issues while using a wide range of different strategies. We need to identify those that have the greatest impact. For example, in the 90s there was a battle between phonics and whole language. Fortunately, we had a strong child development center at NIH that was able to do controlled experiments and discover that phonics based approaches result in better impacts. It just makes sense to drive dollars to where there is evidence of impact.

It is also essential to bridge these tools, but unfortunately there aren’t many places where this is happening. New York City established the Center for Economic Opportunity to design a portfolio of strategies to reduce poverty. They are using data all the time to see which providers are doing better, and th[ose not] doing well, they get let go. New York City is working with some strong research firms to perform rigorous analysis demonstrating whether the intervention is getting results before investing more money.

Federal dollars flow to states and localities through competitive and formula grants. Grant reforms can strengthen the use of evidence in government. Among the most exciting advancements are so-called tiered-evidence or innovation fund grant designs. Would you tell us more about these designs?

These grant designs focus resources on practices with the strongest evidence, but still allow for innovation. In a three-tiered grant model, for example, grantees can qualify for:

• The “scale up” tier and receive the most funding
• The “validation” tier and receive less funding but evaluation support
• The “proof of concept” tier and receive the least funding,

but also support for evaluation With a tiered-evidence approach, potential grantees know they must provide evidence behind their approach or be ready to subject their models to evaluation. The Education Department’s Investing in Innovation Fund (i3) is a favorite of mine. These grants are expected to expand the implementation of and investment in innovative and evidence-based practices, programs and strategies that significantly:

• Improve K-12 achievement and close achievement gaps
• Decrease dropout rates
• Increase high school graduation rates

Grants are awarded to support scale-up, validation, or development activities, depending on the level of evidence. Ultimately, it is about letting states and localities know what works, so they can replicate these strategies in their own funding streams.

Right now, five agencies have tiered program designs. The Labor Department runs two programs, the workforce innovation fund and a community college initiative, using this structure.

At HHS we have teen pregnancy prevention and home visiting. The Corporation for National Community Service has a social innovation fund. USAID has a development innovation ventures program. All have shown that the tiered structure can be an overlay to a fairly traditional grant program.

Would you tell us about the new authority for Performance Partnership Pilots for disconnected youth that was recently included in the 2014 omnibus appropriations bill? What was the impetus for this initiative, and how will it work?

This is one of the most exciting examples of bottom-up policy making. Its genesis comes from a February 2011 presidential memorandum to agencies calling for administrative flexibility for states, localities, and tribes. It charged federal agencies to work closely with state, local, and tribal governments to identify administrative, regulatory, and legislative barriers in federally funded programs that currently prevent them from efficiently using tax dollars to achieve the best results for their constituents.

One of the most compelling examples was a coalition of states focusing on disconnected youth … 14 to early 20s, school dropouts who don’t have jobs. We have dozens of federal youth programs, but the way they are structured, it’s incredibly difficult for a locality to make the[m] work together. This coalition identified challenges they faced trying to weave these programs together to support the needs of these high-risk kids. The paper they presented led to a meeting with senior officials from a number of agencies.

It convinced them that things had to change. As a result, the 2014 budget would authorize up to 13 state or local performance partnership pilots to improve outcomes for disconnected youth. Pilots would use blended funds from separate youth-serving programs in the Departments of Education, Labor, HHS, HUD, Justice and other agencies, and the strategies would be subjected to evaluations to determine which efforts work best so they could be expanded. Interestingly, this authority was given in the 2014 appropriations bill. Frankly, it was the states who presented this ineffective and onerous situation to the attention of key members in Congress.

So-called “Pay for Success” approaches are another way to strengthen the use of evidence in government. Would you tell us more about the PFS model (also know as social impact bonds)?

At a time when government resources are constrained, an innovative approach is the Pay for Success funding model. This is where investors provide upfront capital for social services with a strong evidence base that, when successful, achieve measurable outcomes that reduce the need for future services. Efforts underway in New York City and Massachusetts look at the cost of recidivism. Many of the projects have [gotten] working capital from the private sector to run these prevention services. Rigorous measurement and evaluation methodologies assure that these new projects achieve results. If [investors] get a return, it’s because the government has realized savings and they are sharing it while the individuals served are realizing improvements in their lives.

I invite you to listen or download my complete interview with Kathy Stack, Advisor for Evidence-based Innovation at OMB on The Business of Government Hour.