«Clouds Make Nerds Look Good: Field Evidence of the Impact of Incidental Factors on Decision Making URI SIMONSOHN* The Wharton School-University of ...»
Journal of Behavioral Decision Making
J. Behav. Dec. Making (in press)
Published online in Wiley InterScience
(www.interscience.wiley.com) DOI: 10.1002/bdm.545
Clouds Make Nerds Look Good: Field
Evidence of the Impact of Incidental
Factors on Decision Making
The Wharton School-University of Pennsylvania, Pennsylvania, USA
Abundant experimental research has documented that incidental primes and emotions are capable of inﬂuencing people’s judgments and choices. This paper examines whether the inﬂuence of such incidental factors is large enough to be observable in the ﬁeld, by analyzing 682 actual university admission decisions. As predicted, applicants’ academic attributes are weighted more heavily on cloudier days and non-academic attributes on sunnier days. The documented effects are of both statistical and practical signiﬁcance: changes in cloud cover can increase a candidate’s predicted probability of admission by an average of up to 11.9%. These results also shed light on the causes behind the long demonstrated unreliability of experts making repeated judgments from the same data. Copyright # 2006 John Wiley & Sons, Ltd.
key words naturalistic decision making; incidental emotions; priming; college admissions; weather; feature priming; bootstrapped experts; ﬁeld data
INTRODUCTIONA growing body of research has studied the impact of incidental and irrelevant factors on judgment and decision-making. Within this literature, two somewhat independent streams have studied the role of incidental cognitive primes and incidental emotions. In terms of the former, people’s behavior has been shown to be inﬂuenced by the presentation of primes in a manner that’s consistent with them. In a well-known study, for example, subjects primed with words associated with the elderly approached the elevator outside the lab where the study took place at a slower pace than a control group (Bargh, Chen, & Burrows, 1996). For a review of this literature see (Ferguson & Bargh, 2004).
A related line of work has documented that priming people’s identity inﬂuences their choices. For example, LeBoeuf and Shaﬁr (2003) ﬁnd that subjects whose ‘‘academic self’’ was primed were more likely to choose an ‘‘academic’’ magazine (e.g., The Economist) and Mandel (2003) ﬁnds that subjects primed with their interdependent self (i.e., their reliance on others) become more risk seeking with ﬁnancial decisions and * Correspondence to: Uri Simonsohn, 500 Huntsman Hall, 3730 Walnut Street, Philadelphia, PA 19104, USA.
E-mail: firstname.lastname@example.org Copyright # 2006 John Wiley & Sons, Ltd.
Journal of Behavioral Decision Making more risk averse with social ones. Another line of work has documented a phenomenon labeled ‘‘feature priming,’’ which consists of primed attributes receiving greater weight in multiatribute decisions or judgments (Mandel & Johnson, 2002; Yi, 1990).
A mostly independent and much more voluminous research stream has documented the impact of (incidental) emotions on judgment and choice. For reviews see Forgas (1995), Loewenstein and Lerner (2002), and Schwarz (2000) and/or the special issue of this journal from April of 2006.
Three main mechanisms have been proposed for the inﬂuences of emotions (incidental or otherwise) on judgment and choice. First, emotions inﬂuence how information is processed. Most importantly for the present research, happy moods induce more heuristic and sad moods more analytical information processing, (for a review see Schwarz, 2002). Second, emotions enhance accessibility of mood-consistent memories, and third, they provide information (that can be misattributed to the wrong cause if the actual one is not salient).
This latter mechanism is often referred to as mood-as-information.
Summarizing any one of these three lines of research would require an entire paper, but the following examples of each of the mechanisms are illustrative: (i) Bodenhausen (1993) ﬁnds that subjects in happy moods are more likely to rely on stereotypes in the formation of judgments, (ii) Bower (1981) ﬁnds that subjects better recalled words learnt under their current mood, and (iii) Schwarz and Clore (1983) ﬁnd that respondents interviewed on sunnier days express higher levels of overall happiness.
Ultimately, however, such inﬂuences of incidental factors are of practical importance only to the extent that they have a sizeable inﬂuence on how people make decisions in their everyday lives. If people are inﬂuenced by incidental factors only when making hypothetical or low-stake decisions in contrived environments artiﬁcially created by an experimenter, but not when making (i) real and important decisions, (ii) in their natural environments, (iii) where they have incentives to make correct choices, and (iv) where experience has given them an opportunity to learn how to ignore irrelevant factors, normative theories of choice may still be our best tool for explaining behavior outside the lab.
This paper seeks to assess whether the impact of incidental factors is sufﬁciently large to be observable and relevant in such a setting, and furthermore, to shed light on the size of the effects they generate in everyday decision-making. It seeks, in other words, to test the statistical and practical signiﬁcance of incidental factors in the ﬁeld.
To this end, this paper assesses the impact of an ever-present, irrelevant, and random incidental factor— cloudiness—on an important and repeated decision, made by professionals in their everyday work environment: university admissions. In particular, this paper analyzes the admission recommendations made for 682 undergraduate applications and assesses the impact of cloud cover the day an application happened to be reviewed, on the weight the reviewers placed on the academic and non-academic attributes of the applicants.1 Cloud cover has often been studied as a natural manipulator of mood. Prior research, for example, has shown that sunshine increases tipping (Rind, 1996; Rind & Strohmetz, 2001), is positively correlated with returns in the stock-market (Hirshleifer & Shumway, 2003), and leads to increase of self-reported levels of happiness (Schwarz & Clore, 1983).
Based on these ﬁndings, in Simonsohn (2005), I examined the role of cloud cover during college visits of prospective students on their likelihood to enroll in the visited school. Contrary to initial expectations, visitors on cloudier days proved signiﬁcantly more likely to enroll. I hypothesized that this result may be driven by the fact that cloud cover not only inﬂuences people’s moods, but also acts as a cognitive prime, increasing accessibility to mental constructs which tend to be active during cloudy weather.
Since mellow activities like reading or studying are more appealing and common under cloudy weather, and recreational and social activities under sunny weather; it was hypothesized that cloudy weather may 1 Note that the data consist only of recommendations. Data on ﬁnal decisions are not available.
prime the former and sunny weather the latter. Because of feature priming, in turn, visitors during cloudy days would weight the school’s forte more heavily, academics, while visitors on sunny days would pay more attention to its much weaker social life and entertainment opportunities.2 Support for the hypothesis that cloudy and sunny weather are associated with those two different categories of mental constructs was obtained in a follow-up experiment where participants were randomly assigned to a cloudy or sunny weather-forecast prime, and then took part in a word-fragment completion task.
Subjects primed with a cloudy forecast were better at solving academic related words like book or student but not neutral words like carpet and girl.
Based on this hypothesized link between cloud cover and academics versus non-academic mental constructs, paired with the notion of feature priming, it was predicted that college admission reviewers would increase the weight placed on the academic attributes of applicants evaluated on cloudier days and increase it for the non-academic attributes of those evaluated on sunnier ones. As is discussed in detail below, furthermore, two of the three mechanisms by which emotions inﬂuence choice make the same prediction.
First, in terms of the inﬂuence of mood on processing style, the literature generally shows increased analytic processing under sad moods with greater focus in detail (Schwarz, 2002). Happy moods, in contrast, foster increased heuristic processing, broader categorizations, and the consideration of a wider range of inputs. This mechanism also predicts, therefore, that on cloudier/sad/focused days, reviewers will place more weight on attributes more closely related to the decision (i.e., academic attributes) while on sunny/happy/ inclusive days they will increase their attention to non-academic attributes.
In terms of the priming role of emotions, since cloud cover inﬂuences mood, we should expect that high levels of cloudiness will increase accessibility of mental constructs typically experienced under sad moods. To the extent that there is an association between a more mellow emotional state and mental constructs related to academics and/or a more happy/aroused mood and social/fun/non-academic ones (a plausible though untested possibility), this mechanism (paired with feature priming) would also predict that reviewers will place additional weight on applicants academic attributes on cloudy days and on their non-academic ones on sunny ones. Cloud cover, then, may prime academics both directly and indirectly via mood.
The mood-as-information mechanism does not make any obvious predictions in terms of attribute weighting. It would possibly predict that reviewers, after misattributing their sadder moods to candidates evaluated on cloudy days and their happier moods to candidates evaluated on sunny days, would be less likely to admit students on cloudier days. Daily admission rates would hence be predicted to be negatively correlated with cloud cover (as we shall see, however, this prediction was not supported by the data as cloud cover has no main effect on admission rates).
Documenting an inﬂuence of cloud cover on attribute weighting in actual decisions made by experts would not only demonstrate the practical importance of incidental factors research, but also contribute to the literature investigating the unreliability of expert judgment. Abundant research has shown that experts make inconsistent judgments when making repeated analyses of the same data (for a review see Ashton, 2000). It is typically assumed that such unreliability is caused by unpredictable factors like fatigue, boredom, and distraction (Dawes, Faust, & Meehl, 1989). The results from this paper demonstrate that in addition to the random noise provoked by these elements, incidental factors introduce systematic biases which even more strongly argue for the employment of systematic information integration (e.g., simple linear models).
Although the identity of the school cannot be disclosed, a recent college guide’s description is telling of its strengths and weaknesses:
‘‘Friends, Sleep, Work, choose two.’’
Data description The dataset consists of a sample of 682 paper forms used in the admission process by the university that facilitated the data. These forms are used by admissions’ personnel to summarize information about the
applicants. Each form contains:
(i) Sixteen 1–4 scores summarizing the applicant’s attributes. These ratings are categorized into academic (e.g., GPA), social (e.g., leadership) and special consideration (e.g., outstanding athlete) categories (ii) The admission recommendation of each of two reviewers assigned to review the application, and (iii) The date when the application was reviewed by each of the two reviewers.
Variables Recommendations. The sample contains reviews by at least 15 different reviewers.3 Any given application was evaluated by a subset of two of them. Each of the 682 applicants in the data, then, received two separate admission recommendations for a total of 1,364 observations. Reviewers disagreed on 119 of the 682 applications. Admission recommendations were coded as 1 when a reviewer recommended admission and 0 otherwise.
The total number of applications reviewed per day was not correlated with cloud cover (r ¼ 0.062, p ¼ 0.598), suggesting that an inﬂuence of cloud cover on attribute weighting is not mediated by effort or fatigue. Another concern is a possible systematic difference in the cloud cover experienced by different reviewers. The F-test from a regression with admission recommendations as the unit of observation, cloud cover as the dependent variable, and reviewers’ identities as the only predictors failed to reach signiﬁcance (p ¼ 0.41), however, which means that different reviewers worked experiencing the same average levels of cloud cover.
Finally, it is worth mentioning that the ofﬁces where applications are reviewed all have windows, providing ample opportunity for cloud cover to be perceived by reviewers.
Cloud cover. Cloud cover data for the city where the university is located was downloaded from the National Oceanic and Atmospheric Administration (NOAA) website. Cloud cover is measured on a discrete scale from 0 to 10, where 0 is clear skies and 10 is complete overcast. The cloud cover dataset was matched to the admissions dataset based on the date when applications were reviewed. All 11 different values of cloud cover were observed in the sample. The average cloud cover in the data was 7.91 with a standard deviation of 2.32.
Considering that some of the analyses will concentrate on differences in cloud cover experienced by two reviewers of the same application, it is worth noting that reviewers receive stacks of several applications at a time which they pass on to other reviewers only once they have all been reviewed. A given application is hence examined by different reviewers on different days.