Challenges to evaluating energy innovation policies and how to address them


A new whitepaper released by MIT Sloan School of Management’s Prof. Jacquelyn Pless identifies five key challenges to evaluating energy innovation policies and programmes.

The paper also recommends solutions for overcoming these challenges.

While substantial innovation is needed to enable the transition to clean energy systems, the evidence is limited as to what policy mechanisms actually work to help drive private-sector innovation, according to the whitepaper.

Related articles:
€4.9bn in investment for climate-action projects approved
12 rituals to enable extensive governance of innovation

The primary challenge relates to quantifying how much innovation activity can be attributed directly to a policy or programme as opposed to other factors. “Without a control group, you don’t know what would have occurred with funding versus without funding,” says Pless.

The second challenge involves significant and uncertain time delays that occur in the energy sector between funding and technology development. “In some industries, like software or app development, technology can be developed in a few days or weeks. In the energy sector, there are long timelines – sometimes more than 10 years — between idea generation and outcome. That makes it difficult to attribute results directly to a policy.”

The third challenge is determining how to measure outcomes. “What is the right measure of innovation for the energy sector? Inputs like spending on energy R&D can be measured, however capturing information on whether innovation actually occurs and subsequently impacts economic and environmental performance is less straightforward. Researchers often use measures like publications and patents, but new metrics are needed for the energy context.”

Distinguishing the direction of innovation when assessing these outputs is the fourth challenge. Did the innovation advance “clean” versus “dirty” technologies? This leads to the fifth challenge of disentangling policies and programs to determine the effects of each one and their interactions. That is essential in order to optimally design policy, she says.

“It’s critical to ensure that the public resources spent on supporting innovation are not wasted. We need to design policies and support programmes based on evidence of what mechanisms actually work,” says Pless. “This requires identifying the barriers to clean energy innovation as well as the policies and interventions for overcoming them.”

To address these challenges, governments and agencies can track successful and non-successful applicants of grants and researchers can compare the innovative activity of firms that receive funds relative to similar firms that do not receive funds.

This can help assess outcomes over the short- and long-term. It can also help measure whether innovation outcomes are environmentally-friendly, which is critical for assessing the direction of innovation.

Embedding some randomisation into policy and programme design, which can enable more rigorous analyses can also help in addressing challenges to evaluating energy innovation policies.

For example, the requirements attached to funding could be randomised, such as collaboration with specific types of firms, national laboratories, or universities. Randomisation combined with improved outcome measures can help determine whether collaborations affect commercialization, she says.

Pless points out that there are under-studied research areas that are critical for developing a better understanding of how to drive clean energy innovation, such as long-run changes to strategies, capabilities, and management practices that continue to enhance innovation.

“All of these solutions fall under the umbrella of multiple stakeholders working together, as access to data is critical for researchers to adequately answer these questions. Overcoming these challenges could lead to a significant increase in the evidence base of policies that can successfully drive innovation in the energy sector,” she says.

Prof. Jacquelyn Pless compiled the whitepaper in partnership with Cameron Hepburn of the University of Oxford, and Niall Farrell of Queen’s University Belfast.

The whitepaper is available for download here.