«Taylor Dimsdale Mark Kutner Meeting of the Minds Practitioner-Researcher Symposium December 2004 American Institutes for Research ...»
Becoming an Educated
Consumer of Research:
A Quick Look at the Basics of Research
Methodologies and Design
Meeting of the Minds
American Institutes for Research
In recent years, the debate about the utilization of research studies that have
been used to inform educational practice has been steadily growing. This debate is
primarily the result of frustration; despite the investment of billions of dollars, learner outcomes have not significantly improved. One possible explanation is that most education programs and specific interventions are not informed by the same type of scientifically based research (SBR) used in medical and other scientific fields. There is a growing call for concentrating future research investments in the field of education on experimental SBR because other methods, including quasi-experimental and non- experimental studies, cannot identify effective practices and programs. Proponents of scientifically based research in education argue that education is a science, like chemistry, medicine, or psychology, and therefore is subject to the same rules of evidence as any other applied field.
Federally supported adult education and literacy research activities are increasingly emphasizing experimental studies. The landmark No Child Left Behind statute highlights the importance of experimental research that can identify educational programs that have demonstrated their effectiveness through scientifically based research.
Research in the adult education and literacy field is no exception. The Department of Education’s Institute for Education Sciences (IES) has just commissioned an impact evaluation of direct literacy instruction for adult ESL students.
The National Institute for Child Health and Human Development (NICHD) has commissioned a series of projects that use an experimental design to assess the impact of decoding and phonic instructional approaches for adults. The National Center for the Study of Adult Learning and Literacy (NCSALL) has been operating two “Lab Schools” in Portland, Oregon, and New Brunswick, New Jersey, to provide stable environments for conducting high quality research, facilitate close collaborations between researchers and practitioners, allow for systematic innovation, experimentation and evaluation of promising new instructional 1 methods, materials and technologies, and create knowledge that improves our understanding of adult learning and literacy and improves prac
Undoubtedly in the coming years, the amount of scientific research in education will continue to expand. The drive to produce, digest, and incorporate scientific research has already been taken up by endeavors such as the Department of Education’s What Works Clearinghouse, the National Center for Education Evaluation, and the Coalition for Evidence-based Policy. Yet not all policymakers, researchers, and practitioners have bought into what is viewed as a stampede in support of experimental research projects to the exclusion of other quasi-experimental and non-experimental methods.
Those who are more cautious about, or downright hostile to, experimental research in education contend that too many contextual variables are involved in education issues for experiments to be useful; that, unlike molecules or chemicals, for example, human beings do not always act or respond in predictable patterns of behavior; and that humans come from such a wide variety of backgrounds that no two schools or programs would respond to the same treatment in the same way. Those who also value quasi-experimental and non-experimental methods posit that teachers learn best practices from years of trial and error in the classroom; they claim that data are an unacceptable and cold way to describe students and accuse researchers of trying to explain and dictate education policy far from the classroom by using numbers rather than personal experiences.
In an effort to address some of these issues, the National Research Council (NRC) commissioned a study to investigate the nature of scientifically based research in education. Its 2002 report, Scientific Research in Education, reached a wide range of conclusions, perhaps none more integral than this: “At its core, scientific research is the same in all fields” (Shavelson & Towne, 2002). Although the NRC does not discount the importance of quasi-experimental or non-experimental methods, including “professional” or “folk” wisdom in education, the report concludes that scientific research should be a vital part of the study of education in the future, in the same way it is considered vital to research in related fields. Although researchers in education most certainly face unique challenges that will necessitate new and uncommon approaches, the report contends that SBR can and should be part of the solution instead of being branded part of the problem.
2Challenges Practitioners Face as Consumers of Research
So what is a practitioner to do? Practitioners face many challenges in interpreting research studies and their conclusions. How can you figure out whether and for what purposes you should be able to use research studies to inform practice? The proliferation of research is both encouraging and problematic because it cannot be assumed that a study’s conclusions are valid simply because it is “published.” The sheer volume of research produced is daunting for any consumer, as is the variety of sources in which it can be found. Whereas the majority of research used to be published almost exclusively in scholarly journals, research is now published by think tanks and partisan organizations, on websites, and by advocacy organizations. Each source is subject to different sets of standards and varying levels of peer review.
Unfortunately, bad research does get published and is not always given a “red flag” by the research community (McEwan & McEwan, 2003).
In considering research, practitioners must be careful to understand how and for what purposes they can use the findings to inform practice. All research studies using an experimental and scientifically based research approach are not necessarily well designed and well implemented. Scientific jargon and explanations of advanced statistical procedures that accompany many research studies often add yet another obstacle to the process of understanding because practitioners do not always have training in advanced research methods to determine research quality. Also, practitioners cannot always easily determine the appropriateness of the specific data collection and analysis methods, a necessary step before interpreting the quality of research findings.
Becoming an Educated Consumer of Research
Despite these challenges, with the appropriate knowledge practitioners can rely on research studies as effective and important sources of information to inform practice.
Well-designed studies using an experimental and scientifically based approach are extremely useful tools for teachers and administrators alike in improving educational practices. Other types of research projects that use quasi-experimental and nonexperimental designs also can provide useful information and should not be dismissed simply because of the lack of a control group.
3 This paper is a primer from which practitioners can draw when they are faced with new and unfamiliar research. It is an attempt to flesh out the most important aspects of quality research and to explain how those not trained in advanced research methods can be effective consumers of education research. The paper describes the
• The continuum of research methods so that you as a practitioner will have some rudimentary understanding of the features of each of these methods. The amount of certainty that research findings are both valid and generalizable to other settings depends on the type of research method used. Because of the emphasis increasingly being placed on experimental and quasi-experimental designs, they receive the most attention in this paper. We do, however, also describe nonexperimental designs at some length and do not discount them as useful tools in education research.
• Two aspects of research designs—the research question and research findings—that are important to consider when assessing the quality of research studies.
Research methods are reviewed first because the methodology that is used is a good indicator of how much trust consumers of research can have about the generalizability of findings, although a relationship clearly exists among the research questions being investigated, the research method used, and the type of findings that result. Appended to the paper is a glossary of research terms with which consumers of research need to be familiar.
Understanding the Continuum of Research Methods
Consumers of research need to do more than read the research question and then skip directly to the study findings and conclusions or implications. Although these elements of a research study can immediately inform the reader about what was investigated and whether there were significant findings, it is also important for consumers of research to be certain that the studies are methodologically appropriate for both the research questions guiding the study and the findings resulting from the 4 study. Interpreting research would be a much easier process if it could always be safely assumed that studies were methodologically sound.
The methods section of a research study should be used to judge whether the study findings can be supported by the study methodology. This section also gives the reader important insights into how to use the results and in what contexts the results may be applicable. A practitioner interested in using a piece of research should first understand the methodology used and its associated strengths and weaknesses. In this
paper, we examine three main types of research designs:
• Experimental designs;
• Quasi-experimental designs;
• Non-experimental or descriptive designs.
In examining these methods, with special attention paid to experimental and
quasi-experimental designs, we attempt to touch on four key issues:
• The type of questions that can be addressed;
• The features of the design;
• The types of data collection and analysis used;
• The implications for practice and policy.
Over the past 5 years, the application of scientifically based research to the field of education has received considerable attention. In addition to the research requirements cited in the No Child Left Behind statute, the U.S. Department of Education’s IES has made it a primary goal of the agency to “operate consistently with the standards of a science-based research agency…so that the research, evaluation, and statistical activities we fund lead to solving problems and answering questions of high relevance to education policy” (Whitehurst, 2002).
6 Experimental studies are highly quantitative studies that attempt to empirically test a hypothesis by using “hard” data and statistical techniques. This methodology is especially attractive in examining social issues, including education, because it seeks to produce definitive conclusions and measurable results. Results from experimental studies are primarily concerned with whether or not a treatment works. Experimental studies do not emphasize understanding how contextual factors and conditions influence outcomes.
Data and findings produced by experimental studies provide the most definitive conclusions possible and thus are very appealing to policymakers, researchers, and stakeholders. Experimental or “true” designs are the gold standard for research because these studies attempt to establish causal relationships between two or more factors. A well-designed experimental study allows the researchers to answer a research question with a high degree of certainty because their conclusions are backed up by concrete data.
The primary question an experimental study allows a researcher to answer is:
Did the treatment or intervention have a significant effect on the treatment group? Or, put another way, does the treatment work? In the adult education and literacy field, for example, an experimental study can answer the question: Is phonics-based instruction an effective instructional approach for adult education and literacy learners?
Features of Experimental Research
Experimental studies can establish or disprove a causal relationship between two variables because random assignment has ensured that the groups being compared have the same socio-economic and education characteristics (and are thus equivalent) prior to the study. These are, however, complex and complicated studies that must be carefully designed and executed. Essential features of an experimental research design
that practitioners should focus on when evaluating the quality of a study follow:
• Control group – All experimental studies are performed by comparing two groups: a treatment group that is exposed to some kind of intervention and a control group that does not receive the intervention. The control group provides 7 the groundwork for a comparison to be made and is perhaps the most essential feature of experimental research. It gives researchers an insight into how the treatment group would have theoretically behaved if group members had never received the treatment, and therefore it allows researchers to judge the effectiveness of the treatment.
• Sample selection – Identifying participants for the experimental and control groups is part of the sample selection process. In selecting samples it is essential for researchers to have a clear understanding of the characteristics of experimental and control group participants, such as age, race, economic status, and gender, among other things.