Úvodní stránka » Archiv čísel » Články 02/2017 » Evaluation culture and evaluation capacities

Evaluation culture and evaluation capacities in Central and Eastern Europe: An introduction

Oto Potluka, Jiří Remr

1. Introduction: Why Central and Eastern European countries?

In the field of evaluation, the globalized world enables sharing experiences and relevant knowledge easily across different parts of the globe. However, Central and Eastern European countries have been largely underrepresented in evaluation research[1]. Many studies on factors influencing the development of evaluation capacities and evaluation culture have been published but none of the EU’s Central and Eastern European member states are described in these reviews. This is especially true of a comprehensive comparison of evaluation development study involving 21 countries across the world (Furubo, Rist & Sandahl, 2002) and an even more recent discussion on global trends and challenges in evaluations (Barbier, Hawkins, 2012; Stockmann, Meyer, 2016). International comparisons of evaluation capacities and evaluation cultures are led exclusively by local initiatives (see example Olejniczak, Strzęboszewski & Bienias, 2012). Does this mean that there is nothing interesting in these countries concerning evaluation?

In addition to substantial political changes, Central and Eastern European countries have witnessed dynamic socio-economic development over the last two decades. This shift has also concerned the development, institutionalization and professionalization of evaluation during that time. EU membership introduced the concept of multi-annual programming to these countries and so has the evaluation of operational programs. It has been a new experience and a challenge for civil servants, consultants, and academics.

Evaluation practice had nevertheless been developing in these countries as well. Evaluation experts there took part in official development assistance and EU pre-accession assistance. They accumulated experience in cooperation with the European Commission, international organizations and by taking part in international expert teams.

We see some parallels between building evaluation capacities in the EU during the 1990s and in Central and Eastern European countries (CEE) after their accession into the EU. The process of gaining experience was long, as public servants perceived evaluation specifically as a tool for supervision and as a compulsory bureaucratic requirement of the EU. During the last few years, impact evaluation has been applied in some cases in order to inform decision makers in public organizations about the effects of implemented programs. Some departments and public agencies are now using evaluations as management and accountability instruments (Bienias et al., 2009). About one decade ago, evaluation associations gathered professionals interested in evaluations in these countries.

What approach toward evaluation capacity building is taken in Central and Eastern Europe? How is evaluation culture developing in these countries? This special issue seeks to answer these questions.

2. Theory of evaluation culture development and evaluation capacities building

In an evaluation culture, program managers and evaluators look for information on programs’ performance in order to learn how to improve management and delivery of programs. Empirical evidence on outputs, outcomes, and impacts is used for better performance of such programs. Moreover, evaluation culture encourages experimentation and change (Mayne, 2008).

Such processes do not manifest independently as building evaluation capacities requires leadership, supportive organizational structures and learning focus (Mayne, 2008; Stevenson, Florina, Mills & Andrade, 2002). Experience from Central and Eastern European countries proves that evaluation culture and evaluation capacities are developing but they need to be studied further to share experience, skills and knowledge.

2.1   Evaluation culture and evaluation capacity

Monitoring and evaluation help to find out what works and what does not work in the policy process using self-reflective and self-examination tools. Monitoring and evaluation provide arguments for policy improvements and policy dialogue (Mayne, 2008). They help organizations to learn and improve their performance.

Among the key determinants of the process of evaluation capacity building, we distinguish three important factors: a) leadership, b) organizational support, and c) learning focus (Mayne, 2008). We discuss them in the following part of the editorial.

 

Leadership

External leadership or “external pressure” and supra-national organizations play an important role in evaluation culture development. They concern not only official development assistance but also OECD countries (for Spain as an example, see Arriazu Muñoz, 2009; or an Irish case in McNamara, O'Hara, Boyle & Sullivan, 2009). In Europe, this role has been assumed by the EU cohesion policy. It helps to introduce and develop evaluation skills and knowledge in countries that do not have such experience. On the other hand, its pressure might be more harmful than helpful in terms of improvement (Wojtowicz, Kupiec, 2016).

Internal leadership is reflected by the existence of evaluation societies, which are set up to develop evaluation culture. Experience with evaluation societies varies across Europe. The Polish Evaluation Society has the highest number of members and benefits from a large evaluation market. The Czech Evaluation Society has a low number of members (currently about 45) but is very active and works hard to take a leading position in pursuing evaluation standards and ethics. In Slovakia, there are two independent evaluation societies and Hungarian Evaluation Society appears rather passive in these days.

 

Organizational support

There are three supportive factors that we consider the most important for evaluation capacity building. These factors are a) evaluation trainings and education, b) data availability and accessibility, and c) the willingness of commissioners to apply evaluation findings.

The availability of evaluation training varies across Central and Eastern Europe. The only university program on evaluations in Central and Eastern European countries drew attention in Romania (Friedrich, 2016). There are, however, less formalized trainings in other countries.

Data availability and accessibility is always an issue in evaluation. Summa and Toulemonde (2002) point out that limited accessibility and low reliability of primary and secondary data causes ambiguity of evaluation conclusions and impractical application in strategic decision-making. Usually, organizations commissioning evaluations also have access to primary data. Providing information about structure and variable definitions would enable the improvement of evaluation design prepared by evaluators. Obtaining secondary data for rigorous evaluations makes such evaluation an expensive exercise.

Moreover, the willingness of commissioners to apply evaluation findings differs across Central and Eastern European countries. External pressure is not sufficient to persuade all politicians to apply evaluation findings to make better policies (McNamara et al., 2009; Wojtowicz, Kupiec, 2016).

 

Learning focus

Learning focus belongs among the main factors establishing the success or failure of evaluation capacities development (Mayne, 2008). Although the evaluation societies of Central and Eastern European countries do not have the capacity to set up evaluation programs at universities, they are able to share information about courses and workshops, which would help to increase the knowledge and skills of evaluators. For example, the Czech Evaluation Society organizes the so-called “EvalCafé” that helps to share knowledge and exchange experiences among evaluators (i.e. practitioners, commissioners and academics). Moreover, the Czech Evaluation Society and the Slovak Evaluation Society organize the annual European Development Evaluation Training (EPDET, the European version of International Development Evaluation Training).

There is a 190-hour evaluation course held at the Centre for European Regional and Local Studies (EUROREG) at Warsaw University (National Evaluation Unit, Skórska, 2012). This training course is designed specifically for civil servants.

Beyond this intra-national activity, the Polish Evaluation Society took the initiative to organize a workshop during the European Evaluation Society’s Conference in 2016 to share experiences with an international auditorium.

3. The goal of this issue

This issue presents studies on evaluation culture and evaluation capacities in Central and Eastern Europe; these represent experience from the Czech Republic (Pělucha, Květoň, 2017), Slovakia (Švihlová, Shipley, 2017) and Poland (Olejniczak, Kupiec, & Newcomer, 2017). Their perspectives vary in terms of objects of evaluation (programs and projects) and perspectives they reflect such as policies or programs (e.g. EU cohesion policy and other national policies). The EU cohesion policy is among the most evaluated policies in Europe (Ferry, 2009). Thus, it also helps to develop the evaluation capacity in other policies. This policy includes evaluation requirements that create positive pressure on both the evaluation demand and supply. This increasing demand has enabled countries in Central and Eastern Europe to improve their evaluation capacities and experiences during the last few years.

This special issue is focused on evaluation culture and evaluation capacities; it does so by involving two articles focused on the EU cohesion policy of two major recipients of the funding from this policy in the period 2007–2013: the Czech Republic (the highest per capita allocation) and Poland (the highest absolute allocation).

In the first contribution, Pělucha and Květoň (2017) reflect the developing evaluation culture of the EU cohesion policy in the Czech Republic. In their view, a gradual improvement of evaluation culture tackles a number of internal and external factors. They distinguish the willingness of politicians to listen and to accept positive and negative findings as a major obstacle for the application of evaluation recommendations into practice. Their finding relates to previous observations that evaluations had been perceived as an artificial requirement creating a tool for the European Commission to supervise (Bienias et al., 2009). Evaluations were used mainly for the following purposes: accountability and legitimacy, improving quality and performance and improving planning (Batterbury, 2006). However, the other purposes were neglected: capacity building, learning, developing a sense of ownership among program participants, and empowerment (Batterbury, 2006). Evaluations started to be used in order to improve absorption capacity in financial sense, i.e. to increase spending EU funds instead of achieving long-term goals and improving implementation of policies (Ferry, 2009). Furthermore, some managing authorities started using evaluations as a management and accountability instrument (Bienias et al., 2009).

Pělucha and Květoň (2017) also discuss the development of methodological approaches to evaluation. They confirm that rigorous evaluation methods are rarely used in the EU cohesion policy (Frondel, Schmidt, 2005). The first systematic attempts to evaluate rigorously this policy appeared during an ex-post evaluation in the programming period of 2000-2006 (AVAPP, 2012; Czarnitzki, Bento & Doherr, 2011; GEFRA, IAB, 2010). The approach to evaluations has moved to a rigorous and theory-driven approach, although the change has been a slow-moving process (Hoerner, Stephenson, 2012).

The second paper on the EU cohesion policy (Olejniczak et al., 2017) confirms that evaluations are not key factors of policy change in Poland. This country represents a case with the highest allocation of EU cohesion policy funds in the EU during 2007–2013. Authors posit that evaluations represent a limited source of knowledge for program managers in Poland. The most important source of knowledge is the day-to-day contact between program managers and beneficiaries during the implementation of individual projects. Such findings reflect the trade-off between the independence of evaluations and the in-depth knowledge of implemented programs (Antosz, Drożdżak & Felcis, 2012; Naccarella et al., 2007). However, the trade-off relates to another issue: the speed of evaluation conduct. Policy makers need evaluation findings quickly and of high quality, yet it is not realistic to achieve both. Thus, the trade-off also reflects the question of whether to conduct evaluations quickly but at lower quality or to let the evaluators work longer and enable them to apply rigorous (time-consuming) approaches. These factors might be obstacles for implementing evaluations’ recommendations.

Recent development underlines the close relationship between individual projects and their evaluations. Therefore, evaluations are conducted not only on program and policy levels, but also on project, organizational, and municipal levels (see example Kokeš, 2017). The third paper in this special issue reflects this development. It compares evaluation culture at the local level in Slovakia in comparison to Canada, a country with a long-term tradition in evaluations.

Švihlová and Shipley (2017) point out that in Slovakia there are still issues hindering evaluation culture’s development in public health and education. First, a lack of evaluation courses at universities causes insufficiency of educated and skilled evaluators. Second, budgets for evaluation are the first to be scaled back when restrictions take place. With limited funding, it is not possible to attract high-skilled evaluators. Third, this study confirms the findings of Pělucha and Květoň (2017), Antosz et al. (2012), and Olejniczak et al. (2017) concerning the political willingness to translate evaluation findings into practical policy.

4. Conclusions

This issue shows that there is an interesting evolution in evaluations in Central and Eastern European countries. All studies included in this issue on evaluation culture and evaluation capacities confirm that evaluation culture develops and evaluation capacities grow. On the other hand, all of them reveal some obstacles of seamless development.

The main issue that underlines all studies is the willingness of commissioners to implement evaluation recommendations. Current experiences show that the chance of evaluations of having an impact might be increased by applying the rigorous evaluation methods while also decreasing the significance of unrealistic policy recommendations.

Learning focus is a crucial aspect of developing evaluation culture (Mayne, 2008) but is not present in the surveyed countries. Until now, evaluators and evaluation associations in Central and Eastern European countries have succeeded in developing even without specific programs at universities (except for the program at the University of Warsaw dedicated to civil servants). The evaluation associations of particular countries have helped to develop the skills and knowledge of evaluators by sharing information and knowledge and by peer review.

Evaluators, commissioners and evaluation associations remain on the front lines of the new challenges. The requirements for rapid analyses and usage of big data (and solving the issues of its volume, velocity, veracity, variety and variability) are among the greatest of these new challenges. Evaluation associations in Central and Eastern Europe are in a similar position as the other evaluation associations in the world.

 

 

References

1. ANTOSZ, P., Z. DROŻDŻAK and W. FELCIS. Kultura ewaluacyjna w Polsce. Zarządzanie Publiczne, Vol. 2, No. 18, pp. 9–19. doi: 10.4467/20843968ZP.12.007.0531
2. ARRIAZU MUÑOZ, R. The Development of Evaluation Culture in Spain: A Historical Approach Based on the Biographical Method. Journal of MultiDisciplinary Evaluation, Vol. 6, No. 11, pp. 102-108.
3. Counterfactual impact of cohesion policy: impact and cost-effectiveness of investment subsidies in Italy. Final Report to DG Regional Policy, 2012.
4. BARBIER, J.-C. and P. HAWKINS (Eds.). Evaluation Cultures: Sense-making in Complex Times. New Brunswick: Transaction Publishers, 2012.
5. BATTERBURY, S. C. E.. Principles and purposes of European Union Cohesion policy evaluation. Regional Studies, Vol. 40, No. 2, pp. 179-188. doi: 10.1080/00343400600600504
6. BIENIAS, S., T. GAPSKI, J. JĄKALSKI, I. LEWANDOWSKA, M. MACKIEWICZ, E. OPAŁKA and P. STRZĘBOSZEWSKI. Evaluation in Poland: Brief overview of evaluation process of EU Cohesion Policy funds. In BIENIAS, S. and I. LEWANDOWSKA (Eds.). Evaluation systems in the Visegrad Member States. Warsaw: Ministry of Regional Development, 2009.
7. BIENIAS, S. and I. LEWANDOWSKA (Eds.). Evaluation Systems in the Visegrad Member States. Warsaw: Ministry of Regional Development, 2009.
8. CZARNITZKI, D., C. L. BENTO and T. DOHERR. Counterfactual impact evaluation of cohesion policy: Work package 2: Examples from Support to Innovation and Research. In: European Commission [online]. 2011. Retrieved from http://ec.europa.eu/regional_policy/
   sources/information/evaluations/pdf/impact/ciewp2_final.pdf.
9. FERRY, M. Cohesion Policy Evaluation Systems In The Visegrad States: An Overview. In BIENIAS, S. and I. LEWANDOWSKA (Eds.). Evaluation systems in the Visegrad Member States. Warsaw: Ministry of Regional Development. 2009.
10. FRIEDRICH, V. European University-Based Study Programs in Evaluation: Characteristics and Future Challenges. In R. STOCKMANN and W. MEYER (Eds.). The Future of Evaluation: Global Trends, New Challenges, Shared Perspectives. Basingstoke: Palgrave Macmillan. 2016.
11. FRONDEL, M. and C. M. SCHMIDT. Evaluating environmental programs: The perspective of modern evaluation research. Ecological Economics, Vol. 55, No. 4, pp. 515-526. doi: 10.1016/j.ecolecon.2004.12.013
12. FURUBO, J.-E., R. C. RIST and R. SANDAHL (Eds.). International Atlas of Evaluation. New Brunswick: Transaction Publishers, 2002.
13. GEFRA & IAB. Ex-post evaluation of Cohesion Policy programmes 2000-2006 financed by the European Regional Development Fund Work Package 6c: Enterprise Support - an exploratory study using counterfactual methods on available data from Germany. 2010.
14. HOERNER, J. and P. STEPHENSON. Theoretical Perspectives on Approaches to Policy Evaluation in the EU: The Case of Cohesion Policy. Public Administration, Vol. 90, No. 3, pp. 699-715. doi: 10.1111/j.1467-9299.2011.02013.x
15. KOKEŠ, R. Evaluace projektu Místa zblízka vedená teorií: akční učení, teorie změny a process tracing. Paper presented at the Konference 2017: Evaluace programů, evaluace projektů, evaluace všedního dne, Prague: 2017.
16. MAYNE, J. Building an Evaluative Culture for Effective Evaluation and Results Management. ILAC Working Paper 8. Rome: Institutional Learning and Change, 2008.
17. MCNAMARA, G., J. O'HARA, R. BOYLE and C. SULLIVAN. Developing a Culture of Evaluation in the Irish Public Sector: The Case of Education. Evaluation, Vol. 15, No. 1, pp. 101-112. doi: 10.1177/
    1356389008095492
18. NACCARELLA, L., J. PIRKIS, F. KOHN, B. MORLEY, P. BURGESS and G. BLASHKI. Building evaluation capacity: definitional and practical implications from an Australian case study. Evaluation and Program Planning, Vol. 30, No. 3, pp. 231-236. doi: 10.1016/j.evalprogplan.2007.
    001
19. National Evaluation Unit, Skórska, P. Poland. In: OLEJNICZAK, K., P. STRZĘBOSZEWSKI and S. BIENIAS (Eds.). Evaluation of Cohesion Policy: Overview of practices. Warsaw: Ministry of Regional Development Poland, 2012. pp. 197-208.
20. OLEJNICZAK, K., T. KUPIEC and K. NEWCOMER. Learning from evaluation – the knowledge users' perspective. Evaluation Theory and Practice, Vol. 5, No. 2, pp. 49-74.
21. OLEJNICZAK, K., P. STRZĘBOSZEWSKI and S. BIENIAS (Eds.). Evaluation of Cohesion Policy: Overview of practices. Warsaw: Ministry of Regional Development Poland, 2012.
22. PĚLUCHA, M. and V. KVĚTOŇ. Development of evaluation culture in the context of institutional and methodological aspects: case of the Czech Republic. Evaluation Theory and Practice, Vol. 5, No. 2, pp. 1-26.
23. STEVENSON, J. F., P. FLORINA, D. S. MILLS and M. ANDRADE. Building evaluation capacity in human service organizations: a case study. Evaluation and Program Planning, Vol. 25, No. 3, pp. 233–243. doi: https://doi.org/10.1016/S0149-7189(02)00018-6
24. STOCKMANN, R., W. MEYER (Eds.). The Future of Evaluation: Global Trends, New Challenges, Shared Perspectives. Basingstoke: Palgrave Macmillan, 2016.
25. SUMMA, H. and J. TOULEMONDE. Evaluation in the European Union: Addressing Complexity and Ambiguity. In FURUBO, J.-E., R. C. RIST and R. SANDAHL (Eds.). International Atlas of Evaluation. New Brunswick: Transaction Publishers, 2002. pp. 407-438.
26. ŠVIHLOVÁ, D. and R. SHIPLEY. Program Evaluation in Municipal Sector – Case study Slovakia and Canada. Evaluation Theory and Practice, Vol. 5, No. 2, pp. 27-48.
27. WOJTOWICZ, D. M. and T. KUPIEC. (2016). Reluctant to learn? The use of evaluation to improve EU cohesion policy implementation in Polish and Spanish regions. Evidence & Policy: A Journal of Research, Debate and Practice. Vol. 14, No. 1, pp. 103-120 doi: 10.1332/174426416x14788831
    396548

[1]  Hereinafter, we refer to countries that entered the EU during the period 2004–2007 (i.e. Estonia, Latvia, Lithuania, the Czech Republic, Slovakia, Poland, Hungary, Slovenia in 2004, and Romania and Bulgaria in 2007).