Handbook on Impact Evaluation

Handbook on Impact
EvaluationHandbook on Impact
Evaluation
Quantitative Methods and Practices
Shahidur R. Khandker
Gayatri B. Koolwal
Hussain A. Samad© 2010 The International Bank for Reconstruction and Development / The World Bank
1818 H Street NW
Washington DC 20433
Telephone: 202-473-1000
Internet: www.worldbank.org
E-mail: feedback@w
All rights reserved
1 2 3 4 13 12 11 10
This volume is a product of the staff of the International Bank for Reconstruction and Development / The World
Bank. The fi ndings, interpretations, and conclusions expressed in this volume do not necessarily refl ect the views
of the Executive Directors of The World Bank or the governments they represent.
The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors,
denominations, and other information shown on any map in this work do not imply any judgement on the part of
The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries.
Rights and Permissions
The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without
permission may be a violation of applicable law. The International Bank for Reconstruction and Development /
The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions
of the work promptly.
For permission to photocopy or reprint any part of this work, please send a request with complete information
to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400;
fax: 978-750-4470; Internet: www.copyright.com.
All other queries on rights and licenses, including subsidiary rights, should be addressed to the Offi ce
of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail:
pubrights@worldbank.org.
ISBN: 978-0-8213-8028-4
eISBN: 978-0-8213-8029-1
DOI: 10.1596/978-0-8213-8028-4
Library of Congress Cataloging-in-Publication Data
Khandker, Shahidur R. Handbook on impact evaluation : quantitative methods and practices / Shahidur R. er, Gayatri B. Koolwal, Hussain A. Samad.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-8213-8028-4 — ISBN 978-0-8213-8029-1 (electronic)
1. Economic development projects—Evaluation. 2. Economic assistance—Evaluation.
I. Koolwal, Gayatri B. II. Samad, Hussain A., 1963- III. Title.
HD75.9.K52 2009
338.90072—dc22
2009020886
Cover design by Patricia Hord.Graphik Design.Contents
Foreword ................................................................................................................................xiii
Preface ..................................................................................................................................xv
About the Authors ......................................................................................................................... xvii
Abbreviations ................................................................................................................................ xix
Part 1 Methods and Practices ................................................................................... 1
1. Introduction..............................................................................................................3
References .......................................................................................................6
2. Basic Issues of Evaluation ....................................................................................7
Summary ........................................................................................................7
Learning Objectives .......................................................................................7
Introduction: Monitoring versus Evaluation ...............................................8
Monitoring .....................................................................................................8
Setting Up Indicators within an M&E Framework ......................................9
Operational Evaluation ...............................................................................16
Quantitative versus Qualitative Impact Assessments .................................18
ve Impact Assessment: Ex Post versus Ex Ante
Impact Evaluations ...................................................................................... 20
The Problem of the Counterfactual ............................................................22
Basic Theory of Impact Evaluation: The Problem of Selection Bias .........25
Different Evaluation Approaches to Ex Post Impact Evaluation ...............27
Overview: Designing and Implementing Impact Evaluations ..................28
Questions .....................................................................................................29
References 30
3. Randomization .......................................................................................................33
Summary ......................................................................................................33
Learning Objectives .....................................................................................33
vContents
Setting the Counterfactual ..........................................................................34
Statistical Design of Randomization ..........................................................34
Calculating Treatment Effects .....................................................................35
Randomization in Evaluation Design: Different Methods
of Randomization ........................................................................................38
Concerns with Randomization ...................................................................38
Randomized Impact Evaluation in Practice ...............................................39
Diffi culties w.................................................................47
Questions .....................................................................................................49
Notes .............................................................................................................51
References 51
4. Propensity Score Matching ................................................................................53
Summary ......................................................................................................53
Learning Objectives .....................................................................................53
PSM and Its Practical Uses ..........................................................................54
What Does PSM Do? ...................................................................................54
PSM Method in Theory ...............................................................................55
Application of the PSM Method .................................................................58
Critiquing the PSM Method .......................................................................63
PSM and Regression-Based Methods .........................................................64
Questions .....................................................................................................66
Notes .............................................................................................................67
References 68
5. Double Difference.................................................................................................71
Summary ......................................................................................................71
Learning Objectives .....................................................................................71
Addressing Selection Bias from a Different Perspective:
Using Differences as Counterfactual ...........................................................71
DD Method: Theory and Application ........................................................72
Advantages and Disadvantages of Using DD .............................................76
Alternative DD Models ................................................................................78
Questions .....................................................................................................82
Notes .............................................................................................................84
References 84
viContents
6. Instrumental Variable Estimation ......................................................................87
Summary ......................................................................................................87
Learning Objectives .....................................................................................87
Introduction .................................................................................................87
Two-Stage Least Squares Approach to IVs .................................................89
Concerns with IVs .......................................................................................91
Sources of IVs ..............................................................................................9