Friday 30 April 2010

Contradictory results in Court of Auditors report on the impact assessments of the Commission

A report by the EU Court of Auditors (that was so far confidential but now released by the Council) is generally positive about the way the EU Commission assesses the impact of its programmes. At the same time, the report leaves serious doubts about this conclusion.

For time reasons, I just took a look at the executive summary. But in there one can clearly see the contradiction between different conclusions drawn.

It starts very positive:
"On balance, particularly in recent years, the audit has shown that impact assessment has been effective in supporting decision-making within the EU institutions. In particular, it was found that the Commission had put in place a comprehensive impact assessment system since 2002.

Impact assessment has become an integral element of the Commission's policy development and has been used by the Commission to design its initiatives better. The Commission's impact assessments are systematically transmitted to the European Parliament and Council to support legislative decision-making and users in both institutions find them helpful when considering the Commission’s proposals.
So while this sounds good, the summary continues like this:
"Impact assessment reports do not easily reveal their key messages to the reader and comparing the impacts of the various policy options presented in an IA report is often impossible.

Also, due to a lack of appropriate data, the quantification and monetisation of the impacts of the different options assessed were weak and the analysis often did not provide an adequate basis for a comparison of policy options. In addition, comparing different options was made difficult because of the way in which they were presented in the impact assessment reports.
In other words, the Impact Assessments are a nice innovation but they are largely worthless because they are unclear and not based on good data.

So Impact Assessments have been introduced but they cannot be an effective tool because their results are blurry as much EU language in decisions and reports is blurry and indefinite. Great!


martinned said...

OK, but what lesson would you have us draw from this? Normally, I'd imagine you'd be in favour of innovations such as this, which improve the accountability and transparency of EU decision making.

After all, these impact assessments are hardly for the benefit of the Member States, who would have their own national ministries do the workup on any significant proposal anyway. And they are also not for the benefit of the European Parliament, which wouldn't let something as insignificant as reality get in the way of voting for something that they want. Impact assessments are part of the Community's (and the Commission's) accountability towards the broader stakeholders of their policies, including the public at large.

However, given that it turns out that an assessment of the impact of proposed legislation is only possible to a point, the question is: now what?

Julien Frisch said...

It is necessary to be more clear about goals, how they will be measured and at the end to be open whether this was actually achieved.

Doing assessments that have no clear results are a waste of time, even worse than not doing it.

martinned said...

Now you're just wishing the problem away. If it was possible "to be more clear about goals [and] how they will be measured", the Commission would have done so already. Lord knows they're enthusiastic enough about such innovations. The problem signalled here is that a lot of policy cannot be analysed in this way, at least not with the level of precision that one would like.

When we're talking about food safetly legislation, you can calculate the costs for the industry to do the proposed testing and labelling, and you can measure the reduction in food poisoning. That still leaves the issue of post hoc ergo propter hoc on the benefits side, but OK.

But what if we're talking about something like the Arhus Convention on access to documents, citizen participation and access to justice in environmental matters? (One that I've worked on myself in the past...) On the one hand, the compliance costs depend on how much citizens will actually use the rights they are given. That is still manageable from an Impact Assessment point of view. But what about the benefits? More use by citizens is not necessarily better. In fact, a cleaner environment is not necessarily better. How on earth can you be specific about the goals of legislation like that? How on earth would the benefits be measured?

That said, would we really prefer it if the Commission stopped doing impact assessments for proposals of the latter category? Surely it is better for them to attempt to define and measure at least some aspects of the proposal's impact, rather than giving up altogether?

Julien Frisch said...

"The audit analysed whether impact assessments supported decision-making in the EU

I base my comments just on the executive summary of the Court of Auditors so they who have some responsibility for the supervising the work of the institutions thought that it was noteworthy and raised these issues. I might need to read more into the report to get more specific.

What I wanted to highlight is that if you cannot clearly formulate what kind of impact you expect (under which conditions) it seems rather useless to spend time writing blurry reports that don't support future decision-making (which was the topic of the Auditors' report).

My assumption is that a lot of the blurriness is not the result of the specific policy issues but is rather connected to internal struggles within and between DGs and keeping things unclear reduces criticism.

And assessments that don't present clear are ways to reduce conflict because everyone involved can read into them whatever it wants.

This is convenient politically but doesn't serve the goal of impact assessments. So if you just want to be diplomatic you don't really need these assessments, they only serve a purpose if used as targeted as possible.

martinned said...

I think you're probably right. That also tracks with the CoA's recommendation to strengthen the mandate of the IAB. (The Impact Assessment Board.) I talked to one of those guys last year, and I got the distinct impression that they meant business. (Although that could have been caused by the fact that the person in question was Dutch, a people not known for their patience with politicking.)

I do wonder, though, how the goal of tightening up the impact assessments fits with the Court's recommendations as to consultations at the draft stage. I'd imagine adding more consultation there would only increase the scope for politics and compromise.