FREE ELECTRONIC LIBRARY - Dissertations, online materials

Pages:   || 2 | 3 | 4 | 5 |

«Taylor Dimsdale Mark Kutner Meeting of the Minds Practitioner-Researcher Symposium December 2004 American Institutes for Research ...»

-- [ Page 1 ] --

Becoming an Educated

Consumer of Research:

A Quick Look at the Basics of Research

Methodologies and Design

Taylor Dimsdale

Mark Kutner

Meeting of the Minds

Practitioner-Researcher Symposium

December 2004

American Institutes for Research



In recent years, the debate about the utilization of research studies that have

been used to inform educational practice has been steadily growing. This debate is

primarily the result of frustration; despite the investment of billions of dollars, learner outcomes have not significantly improved. One possible explanation is that most education programs and specific interventions are not informed by the same type of scientifically based research (SBR) used in medical and other scientific fields. There is a growing call for concentrating future research investments in the field of education on experimental SBR because other methods, including quasi-experimental and non- experimental studies, cannot identify effective practices and programs. Proponents of scientifically based research in education argue that education is a science, like chemistry, medicine, or psychology, and therefore is subject to the same rules of evidence as any other applied field.

Federally supported adult education and literacy research activities are increasingly emphasizing experimental studies. The landmark No Child Left Behind statute highlights the importance of experimental research that can identify educational programs that have demonstrated their effectiveness through scientifically based research.

Research in the adult education and literacy field is no exception. The Department of Education’s Institute for Education Sciences (IES) has just commissioned an impact evaluation of direct literacy instruction for adult ESL students.

The National Institute for Child Health and Human Development (NICHD) has commissioned a series of projects that use an experimental design to assess the impact of decoding and phonic instructional approaches for adults. The National Center for the Study of Adult Learning and Literacy (NCSALL) has been operating two “Lab Schools” in Portland, Oregon, and New Brunswick, New Jersey, to provide stable environments for conducting high quality research, facilitate close collaborations between researchers and practitioners, allow for systematic innovation, experimentation and evaluation of promising new instructional 1 methods, materials and technologies, and create knowledge that improves our understanding of adult learning and literacy and improves prac

–  –  –

Undoubtedly in the coming years, the amount of scientific research in education will continue to expand. The drive to produce, digest, and incorporate scientific research has already been taken up by endeavors such as the Department of Education’s What Works Clearinghouse, the National Center for Education Evaluation, and the Coalition for Evidence-based Policy. Yet not all policymakers, researchers, and practitioners have bought into what is viewed as a stampede in support of experimental research projects to the exclusion of other quasi-experimental and non-experimental methods.

Those who are more cautious about, or downright hostile to, experimental research in education contend that too many contextual variables are involved in education issues for experiments to be useful; that, unlike molecules or chemicals, for example, human beings do not always act or respond in predictable patterns of behavior; and that humans come from such a wide variety of backgrounds that no two schools or programs would respond to the same treatment in the same way. Those who also value quasi-experimental and non-experimental methods posit that teachers learn best practices from years of trial and error in the classroom; they claim that data are an unacceptable and cold way to describe students and accuse researchers of trying to explain and dictate education policy far from the classroom by using numbers rather than personal experiences.

In an effort to address some of these issues, the National Research Council (NRC) commissioned a study to investigate the nature of scientifically based research in education. Its 2002 report, Scientific Research in Education, reached a wide range of conclusions, perhaps none more integral than this: “At its core, scientific research is the same in all fields” (Shavelson & Towne, 2002). Although the NRC does not discount the importance of quasi-experimental or non-experimental methods, including “professional” or “folk” wisdom in education, the report concludes that scientific research should be a vital part of the study of education in the future, in the same way it is considered vital to research in related fields. Although researchers in education most certainly face unique challenges that will necessitate new and uncommon approaches, the report contends that SBR can and should be part of the solution instead of being branded part of the problem.

2Challenges Practitioners Face as Consumers of Research

So what is a practitioner to do? Practitioners face many challenges in interpreting research studies and their conclusions. How can you figure out whether and for what purposes you should be able to use research studies to inform practice? The proliferation of research is both encouraging and problematic because it cannot be assumed that a study’s conclusions are valid simply because it is “published.” The sheer volume of research produced is daunting for any consumer, as is the variety of sources in which it can be found. Whereas the majority of research used to be published almost exclusively in scholarly journals, research is now published by think tanks and partisan organizations, on websites, and by advocacy organizations. Each source is subject to different sets of standards and varying levels of peer review.

Unfortunately, bad research does get published and is not always given a “red flag” by the research community (McEwan & McEwan, 2003).

In considering research, practitioners must be careful to understand how and for what purposes they can use the findings to inform practice. All research studies using an experimental and scientifically based research approach are not necessarily well designed and well implemented. Scientific jargon and explanations of advanced statistical procedures that accompany many research studies often add yet another obstacle to the process of understanding because practitioners do not always have training in advanced research methods to determine research quality. Also, practitioners cannot always easily determine the appropriateness of the specific data collection and analysis methods, a necessary step before interpreting the quality of research findings.

Becoming an Educated Consumer of Research

Despite these challenges, with the appropriate knowledge practitioners can rely on research studies as effective and important sources of information to inform practice.

Well-designed studies using an experimental and scientifically based approach are extremely useful tools for teachers and administrators alike in improving educational practices. Other types of research projects that use quasi-experimental and nonexperimental designs also can provide useful information and should not be dismissed simply because of the lack of a control group.

3 This paper is a primer from which practitioners can draw when they are faced with new and unfamiliar research. It is an attempt to flesh out the most important aspects of quality research and to explain how those not trained in advanced research methods can be effective consumers of education research. The paper describes the


• The continuum of research methods so that you as a practitioner will have some rudimentary understanding of the features of each of these methods. The amount of certainty that research findings are both valid and generalizable to other settings depends on the type of research method used. Because of the emphasis increasingly being placed on experimental and quasi-experimental designs, they receive the most attention in this paper. We do, however, also describe nonexperimental designs at some length and do not discount them as useful tools in education research.

• Two aspects of research designs—the research question and research findings—that are important to consider when assessing the quality of research studies.

Research methods are reviewed first because the methodology that is used is a good indicator of how much trust consumers of research can have about the generalizability of findings, although a relationship clearly exists among the research questions being investigated, the research method used, and the type of findings that result. Appended to the paper is a glossary of research terms with which consumers of research need to be familiar.

Understanding the Continuum of Research Methods

Consumers of research need to do more than read the research question and then skip directly to the study findings and conclusions or implications. Although these elements of a research study can immediately inform the reader about what was investigated and whether there were significant findings, it is also important for consumers of research to be certain that the studies are methodologically appropriate for both the research questions guiding the study and the findings resulting from the 4 study. Interpreting research would be a much easier process if it could always be safely assumed that studies were methodologically sound.

The methods section of a research study should be used to judge whether the study findings can be supported by the study methodology. This section also gives the reader important insights into how to use the results and in what contexts the results may be applicable. A practitioner interested in using a piece of research should first understand the methodology used and its associated strengths and weaknesses. In this

paper, we examine three main types of research designs:

• Experimental designs;

• Quasi-experimental designs;

• Non-experimental or descriptive designs.

In examining these methods, with special attention paid to experimental and

quasi-experimental designs, we attempt to touch on four key issues:

• The type of questions that can be addressed;

• The features of the design;

• The types of data collection and analysis used;

• The implications for practice and policy.

–  –  –

Over the past 5 years, the application of scientifically based research to the field of education has received considerable attention. In addition to the research requirements cited in the No Child Left Behind statute, the U.S. Department of Education’s IES has made it a primary goal of the agency to “operate consistently with the standards of a science-based research agency…so that the research, evaluation, and statistical activities we fund lead to solving problems and answering questions of high relevance to education policy” (Whitehurst, 2002).

6 Experimental studies are highly quantitative studies that attempt to empirically test a hypothesis by using “hard” data and statistical techniques. This methodology is especially attractive in examining social issues, including education, because it seeks to produce definitive conclusions and measurable results. Results from experimental studies are primarily concerned with whether or not a treatment works. Experimental studies do not emphasize understanding how contextual factors and conditions influence outcomes.

Data and findings produced by experimental studies provide the most definitive conclusions possible and thus are very appealing to policymakers, researchers, and stakeholders. Experimental or “true” designs are the gold standard for research because these studies attempt to establish causal relationships between two or more factors. A well-designed experimental study allows the researchers to answer a research question with a high degree of certainty because their conclusions are backed up by concrete data.

Questions Addressed

The primary question an experimental study allows a researcher to answer is:

Did the treatment or intervention have a significant effect on the treatment group? Or, put another way, does the treatment work? In the adult education and literacy field, for example, an experimental study can answer the question: Is phonics-based instruction an effective instructional approach for adult education and literacy learners?

Features of Experimental Research

Experimental studies can establish or disprove a causal relationship between two variables because random assignment has ensured that the groups being compared have the same socio-economic and education characteristics (and are thus equivalent) prior to the study. These are, however, complex and complicated studies that must be carefully designed and executed. Essential features of an experimental research design

that practitioners should focus on when evaluating the quality of a study follow:

• Control group – All experimental studies are performed by comparing two groups: a treatment group that is exposed to some kind of intervention and a control group that does not receive the intervention. The control group provides 7 the groundwork for a comparison to be made and is perhaps the most essential feature of experimental research. It gives researchers an insight into how the treatment group would have theoretically behaved if group members had never received the treatment, and therefore it allows researchers to judge the effectiveness of the treatment.

• Sample selection – Identifying participants for the experimental and control groups is part of the sample selection process. In selecting samples it is essential for researchers to have a clear understanding of the characteristics of experimental and control group participants, such as age, race, economic status, and gender, among other things.

Pages:   || 2 | 3 | 4 | 5 |

Similar works:

«*** Free EBOOK*** Your Journey of Personal Transformation Begins HERE: An Introduction to Understanding Your Human Design Chart by Evelyn Levenson, MBA Certified Human Design Specialist Your Success By Design Coach Your Journey Begins HERE www.HumanDesignforSuccess.com This Report is for your personal use. You MAY SHARE it, in its entirety with all author and contact information. However, You May NOT Trade, Sell or Resell This Intellectual Property in Any Way. You may not reprint any part of it...»

«Tiedemann, K. B., D. F. Ventura and C. Ades. 1986. Spectral sensitivities of the eyes of the orb we b spider Argiope argentata (Fabricius). J. Arachnol., 14 :71-78. SPECTRAL SENSITIVITIES OF THE EYES OF THE ORB WE B SPIDER ARGIOPE ARGENTATA (FABRICIUS ) Klaus B. Tiedemann, Dora Fix Ventura and Cesar Ade s Department of Experimental Psychology Institute of Psychology University of Sao Paul o Sao Paulo, Brazil ABSTRACT Spectral sensitivity curves to light between 425 and 650 nm for the...»

«355 366. YTTERVALLA, FRUSTUNA SN, DAGA HD.366. Yttervalla, Frustuna sn, Daga hd. I en källare S om boningshuset i den östra gården i Yttervalla by finnas två mindre stycken av en runsten inmurade i källartaket. Källaren är enligt uppgift av gårdens ägare (herr Alb. Andersson) byggd år 1877. Stenmaterialet var delvis fört från en gammal grund i Långlid, delvis hopsamlat i backarna N om byn. Möjligen finnas i v'ággoch takmurarna flera rester av den sönderslagna runstenen, ehuru nu...»

«TECHNISCHE UNIVERSITÄT MÜNCHEN Lehrstuhl für Aquatische Systembiologie Bioindication tools for measuring the success of stream restoration Joachim Wolfgang Pander Vollständiger Abdruck der von der Fakultät Wissenschaftszentrum Weihenstephan für Ernährung, Landnutzung und Umwelt der Technischen Universität München zur Erlangung des akademischen Grades eines Doktors der Naturwissenschaften genehmigten Dissertation. Vorsitzender: Univ.-Prof. Dr. A. Melzer Prüfer der Dissertation: 1....»

«Nordic journal of working life studies Volume 3 ❚ Number 3 ❚ August 2013 Habituating pain: Questioning pain and physical strain as inextricable conditions in the construction industry Jeppe Z. N. Ajslev1 ❚❚ Ph.D. fellow, Centre for Working Life Research, Roskilde University Henrik L. Lund ❚❚ Associate professor, Centre for Working Life Research, Roskilde University Jeppe L. Møller ❚❚ Ph.D. fellow, Centre for Working Life Research, Roskilde University Roger Persson ❚❚...»

«From Soulard’s Notebooks Assistant Editor: Kassandra Soulard Poetry by Ric Amante 1 Notes from New England [Commentary] by Raymond Soulard, Jr. 9 Poetry by Judih Haggai 27 Poetry by Joe Ciccone 32 Paelo Redemption [Travel Journal] by Charlie Beyer 35 Poetry by Joe Coleman 45 Baby Steps Up into the Sky [Travel Journal] by Nathan D. Horowitz 55 Many Musics [Poetry] by Raymond Soulard, Jr. 59 Psychedelics and Lucid Dreaming: Doorways in the Mind [Essay] by A.S. Kay 91 Poetry by Tom Sheehan 99...»

«Acceptance and Commitment Therapy (ACT) Introductory Workshop Handout Dr Russell Harris, M.B.B.S, M.A.C. Psych. Med. Phone: 0425 782 055 website: www.actmindfully.com.au e-mail: russharris@actmindfully.com.au © Russ Harris 2007 This handout consists of: 1. Definitions of mindfulness (p1) 2. Brief notes on ACT (pp 2 -10) 3. Scripts for mindfulness exercises, and tips for creating your own (pp 11-16) 4. More mindfulness scripts that can also be client handouts (pp 17-18) 5. Client handouts (pp...»

«On the clarity of the music stave Arnold J Wilkins and Isobel Kiff Department of Psychology University of Essex Abstract The staves of 63 scores from 50 music publishers were measured. The staves had similar height but with lines that varied in thickness by a factor of 4 from 0.1-0.4mm. Evidence from visual psychophysics suggests that when the stave has thick lines, perceptual distortions are likely to affect the clarity of the score adversely. Music students were asked to sight read scores...»

«ÉTUDES ROMANES DE BRNO 35, 1, 2014 SAMUEL BIDAUD SUR LES SIGNIFIES DES MOTS ITALIENS CHE ET COSA Introduction Si elle est aujourd’hui très largement utilisée dans la linguistique francophone, la psychomécanique du langage est reçue de façon inégale à l’étranger. Si l’on excepte les travaux de quelques linguistes comme Alvaro Rochetti, Louis Begioni, Alberto Martone, Alberto Manco, Roberto Silvi ou Sophie Saffi, et le fait que les Principes de linguistique théorique et Temps et...»

«Pandemic Influenza Preparedness Framework for the sharing of influenza viruses and access to vaccines and other benefits (PIP Framework) Questions and Answers September 2011 Contents: Page I. General 3 II. Advisory Group 4 III. Global Influenza Surveillance & Response System (GISRS) 6 IV. Pandemic Influenza Benefit Sharing 7 V. Partnership Contribution 7 VI. Standard Material Transfer Agreements 8 VII. Intellectual Property Rights 9 VIII. Influenza Virus Traceability Mechanism (IVTM) 10 This...»

«The Three Symbolic Supports of the Dharma: Teaching on the Benefits of Constructing Statues and Stupas A Teaching by Khenchen Tsewang Gyatso Rinpoche Given at Orgyen Dorje Den (circa 1992) Orgyen Dorje Den, 2244 Santa Clara Ave Alameda, CA 94501 Table of Contents The Three Symbolic Supports of the Dharma: Opening Comments: The Importance of Motivation The Benefits in Four Parts Part 1 The General Benefits of Building the Three Supports Part 2 The Benefit of Building Buddha Statues: The Supports...»

«COGNITIVE SCIENCE 8, 1-26 (1984) Discrimination Nets as Psychological Models* LAWRENCE BARSALOU W. Emory University GORDONH. BOWER Stanford University Simulations of human cognitive processes often employ discrimination nets to model the access of permanent memory. We consider two types of dixrimina. ~ tion nets-EPAM and positive-proper&only nets-and argue that they have insufficient psychologi&l validity. ~hei;deficiencies arise from negative properties, insufficient sensitivity to the...»

<<  HOME   |    CONTACTS
2016 www.dissertation.xlibx.info - Dissertations, online materials

Materials of this site are available for review, all rights belong to their respective owners.
If you do not agree with the fact that your material is placed on this site, please, email us, we will within 1-2 business days delete him.