The Evaluation of Socio-Economic Development
THE EVALUATION OF
SOCIO-ECONOMIC DEVELOPMENT
The GUIDE
December 2003
Tavistock Institute in association with:
GHK
IRS
CONTENTS
INTRODUCTION 1
PART 1 THE CONTRIBUTION OF EVALUATION TO SOCIO-ECONOMIC DEVELOPMENT 5 1.1 THE BENEFITS OF EVALUATION 5 1.2 INTRODUCING EVALUATION: HISTORY AND PURPOSE 10 1.3 METHODS AND THEIR ROOTS 17 1.4 EVALUATION TO STRENGTHEN SOCIO-ECONOMIC DEVELOPMENT 24 1.5 GOLDEN RULES 35
PART 2 DESIGNING AND IMPLEMENTING EVALUATION FOR SOCIO-ECONOMIC DEVELOPMENT 38 2.1 DESIGNING AND PLANNING YOUR EVALUATION 38 2.2 IMPLEMENTING AND MANAGING EVALUATIONS 56 2.4 GOLDEN RULES 81
PART 3 DEVELOPING CAPACITY FOR SOCIO-ECONOMIC EVALUATIONS 84 3.1 DEVELOPING INSTITUTIONAL CAPACITY 84 3.4 GOLDEN RULES 100
PART 4 CHOOSING METHODS, TECHNIQUES AND INDICATORS AND USING EVIDENCE IN EVALUATION 103 4.1 FACTORS INFLUENCING THE CHOICE OF METHOD, TECHNIQUES, DATA AND EVIDENCE 103 4.2 METHODS AND TECHNIQUES FOR EVALUATING DIFFERENT TYPES OF SOCIO-ECONOMIC INTERVENTIONS 109 4.3 METHODS AND TECHNIQUES FOR DIFFERENT EVALUATION PURPOSES 115 4.4 METHODS AND TECHNIQUES APPLICABLE AT DIFFERENT PROGRAMMES/POLICY STAGES 117 4.5 METHODS AND TECHNIQUES APPLICABLE TO DIFFERENT STAGES IN THE EVALUATION PROCESS 120 4.6 ACQUIRING AND USING DATA IN EVALUATION 123 4.7 CREATING INDICATORS AND INDICATOR SYSTEMS 127 4.8 USING INDICATORS TO IMPROVE MANAGEMENT 140 4.9 GOLDEN RULES 146
Annexes:
Annex A The Main Stages of Evaluation
Annex B Changes in Structural Fund regulations
INTRODUCTION
This GUIDE, is intended for those involved in the evaluation of socio-economic development in Europe. It is a successor to the MEANS collection (methods for evaluating structural policies). Whilst the GUIDE has a specific focus on evaluation within European Structural Funds, it is not confined to the evaluation of these interventions. Socio economic development is, after all, strongly featured in many national
SOCIO-ECONOMIC DEVELOPMENT
The GUIDE
December 2003
Tavistock Institute in association with:
GHK
IRS
CONTENTS
INTRODUCTION 1
PART 1 THE CONTRIBUTION OF EVALUATION TO SOCIO-ECONOMIC DEVELOPMENT 5 1.1 THE BENEFITS OF EVALUATION 5 1.2 INTRODUCING EVALUATION: HISTORY AND PURPOSE 10 1.3 METHODS AND THEIR ROOTS 17 1.4 EVALUATION TO STRENGTHEN SOCIO-ECONOMIC DEVELOPMENT 24 1.5 GOLDEN RULES 35
PART 2 DESIGNING AND IMPLEMENTING EVALUATION FOR SOCIO-ECONOMIC DEVELOPMENT 38 2.1 DESIGNING AND PLANNING YOUR EVALUATION 38 2.2 IMPLEMENTING AND MANAGING EVALUATIONS 56 2.4 GOLDEN RULES 81
PART 3 DEVELOPING CAPACITY FOR SOCIO-ECONOMIC EVALUATIONS 84 3.1 DEVELOPING INSTITUTIONAL CAPACITY 84 3.4 GOLDEN RULES 100
PART 4 CHOOSING METHODS, TECHNIQUES AND INDICATORS AND USING EVIDENCE IN EVALUATION 103 4.1 FACTORS INFLUENCING THE CHOICE OF METHOD, TECHNIQUES, DATA AND EVIDENCE 103 4.2 METHODS AND TECHNIQUES FOR EVALUATING DIFFERENT TYPES OF SOCIO-ECONOMIC INTERVENTIONS 109 4.3 METHODS AND TECHNIQUES FOR DIFFERENT EVALUATION PURPOSES 115 4.4 METHODS AND TECHNIQUES APPLICABLE AT DIFFERENT PROGRAMMES/POLICY STAGES 117 4.5 METHODS AND TECHNIQUES APPLICABLE TO DIFFERENT STAGES IN THE EVALUATION PROCESS 120 4.6 ACQUIRING AND USING DATA IN EVALUATION 123 4.7 CREATING INDICATORS AND INDICATOR SYSTEMS 127 4.8 USING INDICATORS TO IMPROVE MANAGEMENT 140 4.9 GOLDEN RULES 146
Annexes:
Annex A The Main Stages of Evaluation
Annex B Changes in Structural Fund regulations
INTRODUCTION
This GUIDE, is intended for those involved in the evaluation of socio-economic development in Europe. It is a successor to the MEANS collection (methods for evaluating structural policies). Whilst the GUIDE has a specific focus on evaluation within European Structural Funds, it is not confined to the evaluation of these interventions. Socio economic development is, after all, strongly featured in many national
References: Boyd, R., Grasper, P., Trout, J.D. The Philosophy of Science, Cambridge, MA: MIT Press, 1991 Boyle, R., Lemaire, D., eds Fetterman, D. M., Kaftarian, S. J., Wandersman, A., Eds. Empowerment Evaluation: Knowledge and Tool for Self Assessment and Accountability, Thousand Oaks, London, New Delhi: Sage, 1996 Furubo, J.E., Rist, R., Sandhal, R Jacob, S., Varone, F. Evaluer l’action publique: état des lieux et perspective en Belgique, Gent, Accademia Press, 2003. Leeuw, F. L., Rist, R. C. Can Governments Learn? Transaction Publishers, 1994 Nutley, S Patton, M. Q. Qualitative Research and Evaluation Methods. Thousand Oaks, London, New Delhi: Sage, 2002 Pawson, R Pawson, R. and Tilley. N. Realistic Evaluation. London, Thousand Oaks, New Delhi: Sage, 1997 Pollitt, C., Justification by Works or by Faith? Evaluating the New Public Management: Evaluation, Vol Rogers, E., Diffusion of Innovation. New York: The Free Press, 1995 Shadish, W., Cook, T Van der Knaap, P. Policy Evaluation and Learning. Feedback, Enlightenment or Augmentation? Evaluation Vol. 1 (2): 189-216, 1995 Wholey, J ----------------------- [1] Financial Regulation (Council Regulation 1605/2002), Article 27 [2] These typologies are consistent with those in the document “Common Guidelines for Monitoring and Evaluation” (European Commission, 1995, Luxembourg: OPOCE). [3] Innes de Neufville (1994) [4] For example, the US “Government Performance and Results Act (GPRA)” of 1993 made the use of new indicators compulsory through the entire government administration.