«1 and Nathanael Goldberg. 2 Comments were provided by Markus Goldstein and Emmanuel Skoufias. This note was task managed by Markus Goldstein and ...»
Impact Evaluation for Microfinance:
Review of Methodological Issues
This paper was written by Dean Karlan 1 and Nathanael Goldberg. 2 Comments
were provided by Markus Goldstein and Emmanuel Skoufias. This note was task
managed by Markus Goldstein and financed by the Trust Fund for Environmentally and
Socially Sustainable Development supported by Finland and Norway and by the Bank-
Netherlands Partnership Program.
Yale University and Innovations for Poverty Action 2 Innovations for Poverty Action. The opinions reflected in this paper are the opinions of the authors and not opinions of their institutions.
TABLE OF CONTENTSINTRODUCTION: WHY EVALUATE?
I. DEFINITION OF MICROFINANCE
A. KEY CHARACTERISTICS OF MICROFINANCE
B. LIABILITY STRUCTURE OF MICROFINANCE LOANS
C. “OTHER” MICROFINANCE SERVICES
II. TYPES OF POLICIES TO EVALUATE
A. PROGRAM IMPACT EVALUATIONS
B. PRODUCT OR PROCESS IMPACT EVALUATIONS
C. POLICY EVALUATIONS
III. METHODOLOGICAL APPROACHES
A. RANDOMIZED CONTROLLED TRIALS FOR PROGRAM EVALUATION
Experimental Credit Scoring
Randomized Program Placement
Ethical Considerations of Randomized Evaluations
B. QUASI-EXPERIMENTAL METHODOLOGIES FOR PROGRAM EVALUATION
C. RANDOMIZED CONTROLLED TRIALS FOR PRODUCT AND PROCESS INNOVATIONS
D. OTHER CONSIDERATIONS
Determining Sample Size
Intensity of Treatment
IV. IMPACT INDICATORS
A. ENTERPRISE INCOME
B. CONSUMPTION OR INCOME LEVELS (POVERTY)
C. CONSUMPTION SMOOTHING
D. WIDER IMPACTS
F. IMPACT ON THE MFI
G. TIMING OF MEASUREMENT
CONCLUSION: OUTSTANDING ISSUES FOR EVALUATION
Introduction: Why Evaluate?
Impact evaluations can be used either to estimate the impact of an entire program or to evaluate the effect of a new product or policy. In either case, the fundamental evaluation question is the same: “How are the lives of the participants different relative to how they would have been had the program, product, service, or policy not been implemented?” The first part of that question, how are the lives of the participants different, is the easy part. The second part, however, is not. It requires measuring the counterfactual, how their lives would have been had the policy not been implemented.
This is the evaluation challenge. One critical difference between a reliable and unreliable evaluation is how well the design allows the researcher to measure this counterfactual.
Policymakers typically conduct impact evaluations of programs to decide how best to allocate scarce resources. However, since most microfinance institutions (MFIs) aim to be for-profit institutions that rely on private investments to finance their activities, some argue that evaluation is unwarranted. At the same time, MFIs, like other businesses, have traditionally focused on quantifying program outcomes; in this view, as long as clients repay their loans and take new ones, the program is assumed to be meeting the clients’ needs. Even if this is so, we propose four reasons to evaluate.
First, an impact evaluation is akin to good market and client research. By learning more about the impact on clients, one can design better products and processes.
Hence, in some cases, an impact evaluation need not even be considered an activity outside the scope of best business practices. For-profit firms can and should invest in learning how best to have a positive impact on their clients. By improving client loyalty and wealth, the institution is likely to keep the clients longer and provide them the resources to use a wider range of services, thus improving profitability. In many cases there also are financial infrastructure investments (e.g., credit bureaus) that improve the market as a whole, not any one particular firm. Public entities may wish to subsidize the research to make sure the knowledge enters into the public domain, so that social welfare is maximized. 3 Note that this point is true both for impact evaluations of an entire program (i.e., testing the impact of expanding access to credit) and impact evaluations of program innovations (e.g., testing the impact of one loan product versus another loan product). We will discuss both types of evaluations in this paper.
Second, even financially self-sufficient financial institutions often receive indirect subsidies in the form of soft loans or free technical assistance from donor agencies.
Therefore it is reasonable to ask whether these subsidies are justified relative to the next best alternative use of these public funds. Donor agencies have helped create national credit bureaus and worked with governments to adopt sound regulatory policies for microfinance. What is the return on these investments? Impact evaluations allow program managers and policymakers to compare the cost of improving families’ income or health 3 Note that for-profit firms could have an interest in keeping evaluation results private if they provide a competitive advantage in profitability. However, for-profit firms can and have made excellent socially minded research partners. When public entities fund evaluations with private firms they should have an explicit agreement about the disclosure of the findings.
1 through microfinance to the cost of achieving the same impact through other interventions. The World Bank’s operational policy on financial intermediary lending supports this view, stating that subsidies of poverty reduction programs may be an appropriate use of public funds, providing that they “are economically justified, or can be shown to be the least-cost way of achieving poverty reduction objectives.” (World Bank 1998).
Third, impact evaluations are not simply about measuring whether a given program is having a positive effect on participants. Impact evaluations provide important information to practitioners and policymakers about the types of products and services that work best for particular types of clients. Exploring why top-performing programs have the impact they do can then help policymakers develop and disseminate best practice policies for MFIs to adopt. Furthermore, impact evaluations allow us to benchmark the performance of different MFIs. In an ideal setting, we would complement impact evaluations with monitoring data so that we could learn which monitoring outcomes, if any, potentially proxy for true impact.
Lastly, while many microfinance programs aim to be for-profit entities, not all are. Many are non-profit organizations, and some are government-owned. We need to learn how alternative governance structures influence the impact on clients. Impact may differ because of the programs’ designs and organizational efficiencies or because of different targeting and client composition. Regarding the former, many organizations have found they have been better able to grow and attract investment by converting to for-profits. The advantages of commercialization depend on the regulations in each country, and some critics accuse for-profit MFIs of mission drift—earning higher returns by serving better-off clients with larger loans. Some governments have run their own MFIs as government programs. Historically, government-owned programs have had difficulties with repayment (perhaps due to the political difficulty of enforcing loans in bad times), but there are cases where government-owned programs can do well (e.g., Crediamigo in Brazil and BRI in Indonesia).
If, however, the main difference is due to targeting and client composition, impact evaluation is not necessarily needed in the long term. Impact evaluation can begin by measuring the relative impact on the different client pools. However, once the relative impact is known, simpler client profile data and targeting analysis could suffice for making comparative statements across microfinance institutions.
In this paper we seek to provide an overview of impact evaluations of microfinance. We begin in Section I by defining microfinance. This discussion is not merely an exercise in terminology but has immediate implications for how to compare evaluations across different programs. Section II discusses the types of microfinance impacts and policies that can be evaluated, including program evaluation and policy evaluations. Section III reviews experimental and quasi-experimental evaluations methodologies in urban and rural environments and discusses some of the key results from past studies. In Section IV, we review common indicators of impact and sources of data. The final section concludes with a discussion of impact issues that have yet to be adequately addressed.
2I. Definition of Microfinance
The first step in conducting an evaluation of a microfinance program is, perhaps surprisingly, to ensure that you are conducting an evaluation of a microfinance program.
This seems obvious, but is not, since the definition of “microfinance” is less than clear.
Broadly speaking, microfinance for loans (i.e., microcredit) is the provision of smallscale financial services to people who lack access to traditional banking services. The term microfinance usually implies very small loans to low-income clients for selfemployment, often with the simultaneous collection of small amounts of savings. How we define “small” and “poor” affects what does and does not constitute microfinance.
“Microfinance” by its name clearly is about more than just credit, otherwise we should always call it microcredit. Many programs offer stand-alone savings products, and remittances and insurance are becoming popular innovations in the suite of services offered by financial institutions for the poor. In fact, it is no longer exclusively institutions for the poor that offer microfinance services. Commercial banks and insurance companies are beginning to go downscale to reach new markets, consumer durables companies are targeting the poor with microcredit schemes, and even Wal-Mart is offering remittances services.
Hence, not all programs labeled as “microfinance” will fit everybody’s perception of the term, depending on model, target group, and services offered. For example, one recent study collectively refers to programs as varied as rice lenders, buffalo lenders, savings groups, and women’s groups as microfinance institutions (Kaboski and Townsend 2005). Another study, Karlan and Zinman (2006b), examines the impact of consumer credit in South Africa that targets employed individuals, not microentrepreneurs. Surely these are all programs worthy of close examination, but by labeling them as microfinance programs, the researchers are making an implicit statement that they should be benchmarked against other microfinance programs with regard to outreach, impact, and financial self-sufficiency. If the programs do not offer sufficiently similar services to a sufficiently similar target group, it is difficult to infer why one program may work better than another. Despite their differences, these programs do typically compete for the same scarce resources from donors and/or investors. Hence, despite their differences and lack of similarities, comparisons are still fruitful since they help decide how to allocate these scarce resources. Note that this argument holds for comparing not only different financial service organizations to each other but also interventions from different sectors, such as education and health, to microfinance. At a macro level, allocations must be made across sectors, not just within sectors. Hence lack of comparability of two organizations’ operations and governance structure is not a sufficient argument for failing to compare their relative impacts.
A. Key Characteristics of Microfinance It may be helpful to enumerate some of the characteristics associated with what is perceived to be “microfinance.” There are at least nine traditional features of
1. small transactions and minimum balances (whether loans, savings, or insurance),
2. loans for entrepreneurial activity,
3. collateral-free loans,
4. group lending,
5. target poor clients,
6. target female clients,
7. simple application processes,
8. provision of services in underserved communities,
9. market-level interest rates.
It is debatable which of these characteristics, if any, are necessary conditions for a program to be considered microfinance. The first feature, small loans, is likely the most necessary, though lending itself is not essential; some microfinance programs focus on mobilizing savings (although few focus entirely on savings without engaging in any lending). Although MFIs often target microentrepreneurs, they differ as to whether they require this as a condition for a loan. Some MFIs visit borrowers’ places of business to verify that loans were used for entrepreneurial activities while other MFIs disburse loans with few questions asked—operating more like consumer credit lenders. In addition, some MFIs require collateral or “collateral substitutes” such as household assets that are valuable to the borrower but less than the value of the loan. Group lending, too, while common practice among MFIs, is certainly not the only method of providing micro-loans.
Many MFIs offer individual loans to their established clients and even to first-time borrowers. Grameen Bank, one of the pioneers of the microfinance movement and of the group lending model has since shifted to individual lending.
The focus on “poor” clients is almost universal, with varying definitions of the word “poor.” This issue has been made more important recently due to legislation from the United States Congress that requires USAID to restrict funding to programs that focus on the poor. Some argue that microfinance should focus on the “economically active poor,” or those just at or below the poverty level (Robinson 2001). Others, on the other hand, suggest that microfinance institutions should try to reach the indigent (Daley-Harris 2005).