«Formn Approved I REPORT DOCUMENTATION PAGE OMB No. 0704-0188 Public reponring burden for this collection of information is estimated to average 1 ...»
REPORT DOCUMENTATION PAGE OMB No. 0704-0188
Public reponring burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching e0Msbng data sources, gathering and maintaining the
data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-
4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if itdoes not display a currently valid OMB control number, PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.
1. REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE 3. DATES COVERED (From - To) July 28, 2003 Final Report Jan-July' 2003
4. TITLE AND SUBTITLE Sa.CONTRACT NUMBERN00014-03-M-0083 Measuring Multi-tasking Ability 5b. GRANT NUMBER 5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S) 5d. PROJECT NUMBER Susan C. Fischer, Kevin A. Morrin, and Susan Joslyn 5e. TASK NUMBER
While multi-tasking (MT) may increase productivity, it also carries a tremendous downside including error, decreased morale, high training costs, high turnover rates, and attrition. This research shows that it is now possible to develop a test that would measure how individuals vary in their ability to concurrently perform multiple tasks under time limited conditions. The purposes of the present research were to (1) investigate complex real-world MT environments, (2) investigate existing measures of MT, and (3) begin development of a practical test of the ability. Current standards for educational and psychological tests served to guide the process of test development. Based on testing standards, a plan for development of an MT ability test was created. The initial phases of test development were also completed. The purpose, scope, and framework for the test are described in the report and the test specifications currently supported by empirical research are also given. This report also describes the additional research necessary for further development of a test of MT ability.
This research has greatly benefited from the contributions of several individuals. We would first like to thank the project's program officer, Susan Chipman for her continuing pursuit of a robust test of multi-tasking ability. Science is only beginning to understand the ability to concurrently perform multiple tasks in complex real-world settings. A reliable and predictive measure of the individual differences that underlie performance in laboratory and real-world multi-tasking environments would greatly promote both scientific and applied interests. Susan Chipman's continued interest in multi-tasking has made this project a reality, and we are grateful to have had the opportunity to contribute to knowledge about this topic. We would also like to thank her for her insights about multi-tasking, directing us to important literature areas, and her able management of the project.
We would also like to thank the technical and support staff at Anacapa Sciences. We are grateful for the support we received from Amy Newsam, Barbara Gates, and Peggy Liborio Willis, who helped us assemble and format this report. We are also indebted to Bill Campsey and Jack Stuster, who were invaluable in helping us to arrange interviews with subject matter experts who work in multi-tasking jobs. Finally, we greatly appreciate the excellent review of this report provided by Doug Harris.
We have learned a great deal about multi-tasking in this research. Most of what we have learned has been taught to us by the individuals who perform jobs that require this ability. Each of the jobs that we investigated demand an almost unbelievable degree of multi-tasking skill. We were truly amazed at what these individuals accomplish in their daily work. We would like to thank the ICU and floor nurses, chefs, Army combat leaders, LCAC Craftmasters and LCAC Navigators who educated us about multitasking in the working world.
111iv EXECUTIVE SUMMARY
INTRODUCTIONWe live and work in a world that frequently requires the performance of multiple tasks within a limited time period, requiring a capability that has become known as multi-tasking (MT). While MT may not be present in everything that we do, it is getting more difficult to find work environments in which MT is not at least part of the job.
Both military and civilian work environments require MT. For example, the crew aboard the Navy's Landing Craft Air Cushion (LCAC) is tremendously busy performing multiple tasks within a short period of time. Nurses, air traffic controllers, and chefs are examples of civilian positions that place heavy demands on MT ability.
While multi-tasking may increase productivity and reduce overall costs, it also carries a tremendous downside. The negative consequences of MT come in several forms, one of which is increased probability of error. When the human information processing system is used to capacity, as is often the case when multi-tasking, a likely outcome will be error. Unfortunately, human error in decision-making under timelimited situations has been the cause of several disasters in each of these types of jobs.
The air collision in German airspace in 2002 that was the result of air traffic control (ATC) error is only one example.
Another negative consequence of MT in the workplace is decreased morale, which nearly always leads to high levels of burnout, turnover rates, and attrition. MT is, by its very nature, stressful. Hence, many jobs that require MT also have high turnover rates and attrition. These jobs often require extensive training, and organizations invest a great deal of money to train selected applicants only to lose them later because of the stressful nature of the work.
MT not only increases the probability of error, burnout, stress, attrition, and training costs. Every time an individual switches to another task, it takes a small amount of time to reorient to the new task. While it may seem like productivity is increased by reducing staff and increasing task load, overall performance may actually be slowed by MT.
Despite the problems associated with MT, not every air traffic controller or nurse experiences stress, burnout, or makes a large number of errors. Some individuals seem resistant to the negative effects of MT, and even seem to thrive on the challenge. Some individuals are much more able to perform well in multi-tasking environments than others. In psychological terms, there may be a general ability to concurrently organize and perform more than one task, which allows some people to perform well in MT environments.
Recent research supports this hypothesis, showing that normal adults vary in how well they perform laboratory tasks requiring the simultaneous performance of multiple tasks under time-limited conditions (Joslyn & Hunt, 1998). What is even more impressive is that an abstract laboratory task used in this research predicts ultimate performance on laboratory simulations of jobs that require multi-tasking (emergency dispatching, emergency call answers, and air traffic control (ATC)) (Joslyn & Hunt, 1998).
V If individuals truly vary in their ability to multi-task, it should be possible to measure that ability and use the assessment to predict future performance in MT environments. In other words, it should be possible to develop a measurement instrument (a test) that could be used to screen individuals for positions that demand high levels of MT ability. Joslyn's and Hunt's work strongly suggests that development of such a test is possible. Indeed, their laboratory task, the Abstract Decision-Making (ADM) task may be a direct measure of MT ability.
A test that could reliably measure MT ability and could predict job performance in a variety of MT environments would be highly useful. Training costs for many MT jobs could be reduced by using the test to select those individuals who would perform well on the job. However, this report will also show that, while previous research has produced a great deal of knowledge about MT in relatively simple, controlled, laboratory settings, little is known about MT in complex real-world environments. To create a reliable and valid predictor of MT ability in real-world settings, a better understanding of complex environments is needed. The similarities and differences among MT environments have not been studied. As a result, we do not yet understand the kind of realworld performance a test of MT ability should predict. Moreover, there are no existing tests of MT ability for normal populations. The literature does include various laboratory tasks and paradigms that might form the basis of a future test of MT. However, usable tests have not been developed.
PURPOSE OF THIS RESEARCHThe purpose of the present research was twofold. The first purpose was to begin to dose the gap in our knowledge of real-world MT. Second, this research also began the process of developing a usable and practical test of MT ability. A two-pronged approach was taken to better understand (1) complex MT environments and (2) existing measures of MT. Four MT environments were studied to begin to understand the cognitive operations they demand. A preliminary ontology of cognitive operations required by MT was developed and used to analyze the environments. The results of the analysis of MT environments were used to establish preliminary requirements for a predictive test of performance in those settings.
A review of the literature was also conducted to (1) identify current measures that could potentially be used to predict MT performance in real world settings, and (2) analyze those measures to determine the kinds of cognitive operations they measure.
To begin the process of developing a usable and practical test of MT ability, current standards for educational and psychological tests were studied. Based on four phases of test development prescribed by the standards, a plan for development of an MT ability test was created. Following the plan, the initial phases of test development were completed. This report also describes the additional research necessary for further development of a test of MT ability. A set of studies was designed to lay the requisite empirical groundwork for test development and to examine the construct and predictive validity of the resulting test. These studies are fully described in Chapter Six of this report. In this executive summary, we provide an overview of the findings of this study.
INVESTIGATION OF MULTI-TASKING ENVIRONMENTSGenerally stated, the purpose of investigating MT environments was to gain knowledge about the kind of performance a test of MT ability should predict. This research constitutes an initial examination of the criterion performance the proposed test seeks to predict. A better understanding of the similarities and differences among MT environments is imperative to development of a test that can predict performance in a wide variety of MT environments.
Several issues were important to this study. First, how similar and how variable are MT environments in terms of the kinds of cognitive requirements they place on individuals who work in them? Do they all require the capacity to remember lots of information, for example? Do they all require the interleaving of tasks, and hence the ability to use prospective memory? Is the ability to prioritize important to all MT environments? Which cognitive capabilities make someone good at MT jobs?
METHOD Participants.Based on several criteria, two military MT environments were selected for study: operation of the Navy's Landing Craft Air Cushion (LCAC) and Army combat unit command. Both the Craftmaster and the Navigator positions aboard the
LCAC were investigated. Two civilian MT environments were also selected for study:
restaurant food preparation/chef and nursing. Both of these civilian environments experience high turnover rates and financial losses in training costs due to burnout.
Nine professionals who worked in four different MT environments participated in the interviews. Each of the participants was highly experienced and qualified in their own field.
Materials. A standard set of questions was designed to probe the cognitive requirements of work environments, regardless of the particular field of work or job content.
The questions were designed for use in the context of a critical incident of MT that the participant had experienced as part of his or her work.
Procedure. Interviews were conducted with each participant. The interviewer first described the purpose of the study and asked questions about the participant's qualifications and background. Then participants were asked to describe a critical incident that they had experienced in their work. They were asked to think of an incident that heavily demanded MT performance. After describing the incident, the interviewer asked a series of questions pertaining to six different topics related to the cognitive requirements of the job including issues of memory, task prioritization, decisionmaking, knowledge and experience, the work environment, and relationships among the components tasks.
RESULTS Each of the four MT environments are described in detail in this report. The results indicated the each environment possesses eleven characteristics of MT settings originally specified by Burgess (2000) and further elaborated in this report.
The results also indicated that the jobs varied somewhat in the kinds of cognitive operations they required. The memory requirements they place on workers were very vii similar. All of the jobs require required STM storage of information (e.g., headings for LCAC navigators and operators, vital signs for nurses). LTM retrieval of domainspecific knowledge learned in training or on-the-job experience was also necessary in each of the jobs we studied. Most, but not all, jobs required prospective memory.