«ABSTRACT In this reﬂective, analytic story I discuss what it is like to be a parent and an evaluator in a participatory, democratic evaluation of ...»
What’s It Like When the Participatory
Evaluator is a “Genuine” Stakeholder?
In this reﬂective, analytic story I discuss what it is like to be a parent and an evaluator in a
participatory, democratic evaluation of an afterschool program. The story tells of tensions
between these roles, and how these tensions can instruct participatory evaluators in pursuing
goals of democracy, community, and justice in evaluation. This discussion can be situated among a number of recent articles in AJE that have focused on analyses of participatory or empower- ment evaluation. (See Schnoes, Murphy–Berman, & Chambers, 2000; and the exchanges between Smith, 1999 and Cousins & Earl, 1999.)
INTRODUCTIONI approach the double aluminum doors of the school gymnasium, predisposed to be annoyed.
It is a cold New York winter day and with my gloves still on I bang on the door with the palm of my hand. To knock would be fruitless. Inside the gymnasium, I hear yelling, laughter, the sounds of children in the Afterschool Program. I wait for someone to open the door, but no one does. More annoyed now, I bang harder on the door. And wait, again. Eventually, the door is pushed open, slowly, as if in slow motion. It opens slowly because it is heavy for the little girl, maybe 5 or 6 years old, to push open. I thank her and, in my mind, I run through my reactions to this daily occurrence.
I am annoyed at the school for making it difﬁcult for parents to pick up their children at the end of the day. Yes, I have heard the discussion about school safety, but isn’t it just a ruse, because small children are the ones to open the door—it isn’t necessarily monitored by a responsible adult. I am annoyed with the Afterschool Program staff for complying with the school, for being too gutless to advocate for smoother operations of their program. I sign out my son on the sheet provided, few words are exchanged with the staff, and I convince Sandra Mathison ● Associate Professor, Department of Educational Theory and Practice, SUNY at Albany, 1400 Washington Avenue, Albany, NY 12222; Tel.: (518) 442-4986; E-mail: firstname.lastname@example.org.
American Journal of Evaluation, Vol. 22, No. 1, 2001, pp. 29 –35. All rights of reproduction in any form reserved.
ISSN: 1098-2140 Copyright © 2001 by American Evaluation Association.
29 30 AMERICAN JOURNAL OF EVALUATION, 22(1), 2001 him it is time to go home. He is reluctant to leave; he is having a good time playing with his friends.
As we leave, I think well, ﬁnally, the access issue will be resolved now that I and my doctoral students will be doing an evaluation of this Afterschool Program. During the spring semester, I and a group of doctoral students, enrolled in an advancedcourse on participatory evaluation, conducted an evaluation of this program. I, as the professor, negotiate pro bono evaluations for community agencies to provide students with “real” experiences doing evaluation. In this course, and in these evaluations, the focus is on participatory evaluation that involves as many stakeholders as possible in all aspects of the evaluation, including deciding what the evaluation ought to focus on, questions to be asked, kinds of data to be collected, and how to communicate the understandings that develop from the evaluation.
Guba and Lincoln’s (1989) Fourth Generation Evaluation is a main text, and we use the construction of claims, concerns, and issues to start, focus, and continue the evaluation. The intent of these participatory evaluations is to create dialogue and deliberation among stakeholders, with an eye to empowering those who may have been disenfranchised in the program context, and to create a sense of community responsibility for quality programming.
We use a form of participatory evaluation, sometimes referred to as transformative (see House & Howe, 1999, Mathison, 1996, 2000, and Whitmore, 1998 for more detail on the approach to participatory evaluation referred to here). We use deliberative forums extensively throughout these evaluations (see MacNeil, 2000 for a description of deliberative forums).
I hesitated about this particular evaluation. I approached the Child Care Director for the YMCA, which runs most of the city’s afterschool programs, about possible evaluations.
Although there are many options within the Y’s programming, the director was keen to have the students look at what she considered her best afterschool program. I resisted, suggesting alternatives. She insisted, feeling it would be a great experience for all. I relented, and we moved ahead with the evaluation.
It is my practice to supervise the evaluations in these courses, to work with the doctoral students in planning and conducting the evaluations, but to leave most of the ﬁeld work to the students. In the past, even when things have been rocky, when my personal intervention would perhaps have made things go more smoothly, I have resisted direct intervention (offering instead counsel and advice) and allowed students to do their own work. I was to discover that I had been able to maintain this distance, at least in part, because I was not what I would call a “genuine” stakeholder in previous programs. Not so for the Afterschool Program. My role in this evaluation deviated from those I have done as part of the class experience in the past. I took a substantial role in doing the evaluation, including collecting data, conducting deliberative forums with stakeholders, and writing reports. I acted as an evaluator, not just a supervisor of an evaluation class project. Various stakeholders gave either my evaluator or parent self more prominence—I believe other parents and school staff saw me as a parent, while the staff and director of the Afterschool Program saw me more often as an evaluator.
My son, 5 years old, goes to this Afterschool Program. I am a parent who has been actively involved in my son’s care and schooling; his caregivers and teachers can be assured of my involvement. When I enrolled my son in this program, I requested a meeting with the Child Care Director to inquire about the speciﬁcs of the program. At the time, she remarked how unusual my request was. In this program, I am a “genuine” stakeholder (in the sense of real, not necessarily good).
Participatory Evaluator as a ‘Genuine‘ Stakeholder 31 Guba and Lincoln (1989) quite rightly tell us that evaluators are stakeholders too. But, in the end, evaluators are “artifactual” stakeholders (in the sense that their stake is created by the existence of an evaluation, not necessarily that it is accidental or bad). Evaluators do not have a fundamental stake in the program—their stake ends when the evaluation ends. When we completed the evaluation, my students no longer had a stake in the program—they were ﬁnished with it and gave little thought to the operations and future of the Afterschool Program. I, however, continue to be a stakeholder because my son still goes to the Afterschool Program. In fact, my longevity as a stakeholder extends beyond that of most on-site staff of the program, many of whom work for a year and sometimes less in the program.
OVERCOMING THE NIMBY (NOT-IN-MY-BACK-YARD) PHENOMENON
One of the reasons I relented on this evaluation was some reﬂection on what the value of democratic, participatory forms of evaluation is. In my teaching and scholarship, I promote the notion that evaluation is about building community, about diverse stakeholder groups working through visions of both what is and what ought to be, that through this process we can arrive at more fair, more just, more caring communities within which we live and work.
If this is good for others, I reasoned, I should be committed enough to the idea that it is good for me, too. Good not just from my perspective as an evaluator, but also from my perspective as a citizen.
NIMBYs are folks who advocate for all sorts of social goods, in principle— organic pest control, community group homes, reduction in fossil fuel use— but when it impacts their lives personally these goods lose their value. When the rainy summer produces an infestation of mosquitoes or when a community home for AIDS patients moves in next door, the NIMBYs’ social responsibility is subverted to individual rights. Vowing not to be a NIMBY, I and my students embarked on the evaluation of the Afterschool Program.
PRIVATE VERSUS PUBLIC
The notion of public and private domains is discussed in cultural anthropology, especially of other cultures, and often related to gender (see, e.g., Rosaldo & Lamphere, 1974).
One common manifestation of this distinction is the domains of work (as public) and home (as private). This distinction interrelates with evaluation practice in a number of ways. Most important for this discussion is the observation that as professional evaluators doing evaluation we operate in a public domain; this is our work life. Others in an evaluation, such as service recipients, may be operating in a private domain—-for example, when a mental health program strives to transition hospitalized, mentally ill patients back into the community by providing support with ﬁnances, ﬁnding a place to live, shopping for clothes, and so on. Such issues are more connected to home life.
Culturally speaking, both the private and public domains are rule-governed and open for inquiry, although what is apparent from ethnographies of other cultures is the sometimes greater difﬁculty in accessing the private domain, such as when one is interested in Moslem women, for example. It is called the private domain because it is a realm of social life within 32 AMERICAN JOURNAL OF EVALUATION, 22(1), 2001 which we can somehow be ourselves, not having to provide explanations and rationales to outsiders for who we are, what we believe, and how we act. We have the veil of privacy.
Evaluation practice, it seems, is part of the cultural public domain, but intrudes on the private domain of some, at least some of the time. The meeting of the public and private domains for me personally became clear in the Afterschool Program evaluation. My role as the overseer of this evaluation is clearly in the public domain, but my role as a parent is clearly in the private domain.
What I realized in this evaluation is the much greater sense of vulnerability that “genuine” stakeholders experience when an evaluation ventures into their private domain.
Matters that we assume to be private become public, and possibly open to criticism. For example, our expectations about what afterschool care ought to be may reﬂect a range of possibilities in child-rearing practices, for example, how discipline ought to be handled. One issue in this evaluation was the overuse of “time-out” as the means for disciplining kids.
While it was not my sense that my son was subjected to time-outs, I perceived the possibility that he could be, and as such perceived (along with many other parents) the use of time-out to be excessive and indiscriminate. Other parents did not share this view— one father said, during a deliberative forum, that time-out made absolutely good sense because this is a practice used in all American homes, and therefore kids are familiar with the form of discipline. Time-out is never used in my home. Rather than using time-out, I might suggest a strategy of helping kids ﬁnd alternative activities that are engaging and that take them out of the situation that is creating the misbehavior.
An evaluator operating within the public domain can and does deal with such differences in perception, perhaps by collecting more data or mediating these differences, to come to a sense of what might be. As a parent and evaluator in this situation, the importance of these differences is apparent to me in a new way. The “time out is OK” view is a threat to my parental conception of good child-rearing practices—for my child especially. I, as a parent, want to defend and promote my position; I, as an evaluator, want all stakeholders to deliberate about discipline, with an eye to mutual respect and ultimately to agreement about how to handle discipline. I assume these intents need not be mutually exclusive; but this experience of being a genuine stakeholder has created for me a new appreciation for the complexity of how and to what end stakeholders are asked to share and make sense of their own and others’ perspectives.
It is clear from just this one example that otherwise noncontroversial issues often mask more deeply held beliefs. For example, a behaviorist perspective on learning and development may underlie the views of the parent who felt comfortable with the use of time-out, and a more humanist perspective may underlie the strategies I as a parent would feel comfortable with. As evaluators promoting dialogue and deliberation among stakeholders, we need to recognize that we will venture into potentially different foundational beliefs—a very substantial challenge. The lesson here is not to avoid or fear this likelihood, but rather to anticipate that deliberation will be about these values and to be respectful but open about that as the focus. Although our deliberative forum did not venture into a discussion of perspectives on learning and development, it could have. Perhaps it should have. This experience suggests to me these foundational beliefs are a critical element in creating a community that has a reasonably coherent view of what it will accept, with little grumbling, as good practice.
As an evaluator, operating within one’s private domain is a clarifying and humbling experience— one that is instructive for all of us who advocate participatory forms of evaluation.
Participatory Evaluator as a ‘Genuine‘ Stakeholder 33
SILENCING AS A PARENT/ADVOCATING AS AN EVALUATOR
On an afternoon while we were conducting the evaluation, I arrive to pick up my son.