«Prepared for the U.S. Department of Energy under Contract DE-AC05-76RL01830 CCSI Risk Estimation: An Application of Expert Elicitation DW Engel AC ...»
Prepared for the U.S. Department of Energy
under Contract DE-AC05-76RL01830
CCSI Risk Estimation: An Application
of Expert Elicitation
This report was prepared as an account of work sponsored by an agency of the
United States Government. Neither the United States Government nor any agency
thereof, nor Battelle Memorial Institute, nor any of their employees, makes any
warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof, or Battelle Memorial Institute. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.
PACIFIC NORTHWEST NATIONAL LABORATORYoperated by BATTELLE for the
UNITED STATES DEPARTMENT OF ENERGY
ph: (865) 576-8401 fax: (865) 576-5728 email: firstname.lastname@example.org Available to the public from the National Technical Information Service, U.S. Department of Commerce, 5285 Port Royal Rd., Springfield, VA 22161 ph: (800) 553-6847 fax: (703) 605-6900 email: email@example.com online ordering: http://www.ntis.gov/ordering.htm This document was printed on recycled paper.
(9/2003) PNNL-21785 CCSI Risk Estimation: An Application of Expert Elicitation DW Engel AC Dalton October 2012 Prepared for the U.S. Department of Energy under Contract DE-AC05-76RL01830 Pacific Northwest National Laboratory Richland, Washington 99352 Contents
2.0 Expert Elicitation Background and Significance
3.0 CCSI Element 7 Expert Elicitation Approach
4.0 Status Report and Next Steps
Figures Figure 1 CCSI Decision-Making Framework
Figure 2 CCSI Risk Factor Expert Elicitation Approach
Another focal point of the expert elicitation literature intersects with psychological research on heuristics and biases, and explores the effects of human cognitive patterns on elicitation. Heuristics refer to the effort-reduction methods employed by decision makers to alleviate judgmental burdens, and judgment biases refer to prejudicial mental behavior that undermines rational decision making (Arnott 1998; Shah and Oppenheimer, 2008; Simon 1990; Tversky and Kahneman 1974). Prominent classes of heuristics include representativeness (assessments based on similarity between events), availability (assessments based on one’s ability to recall past events), and adjustment and anchoring (assessments bounded by initial judgment) (Tversky and Kahneman 1974). Remus and Kottemann (1986) grouped heuristics based on their relevance to data presentation and information processing (Remus and Kottemann 1986);
Hogarth (1987) categorized heuristics based on the four stages in decision-making ranging from “acquisition, processing, output, to action” (Arnott 1998, p. 28; see Arnott 1998 for a comprehensive review). In probabilistic risk elicitation, the most widely observed effects of judgmental biases are systematic under- or over-estimation and overconfidence (Mosleh et al. 1988). Approaches to reducing systematic bias and overconfidence include calibration training (Lichtenstein et al. 1982) and the encouragement of considering both affirmative and contradicting evidence in experts’ decision making (Spetzler and von Holstein 1975). Moreover, improving the formulation of decision problems may prove useful for enhancing the quality of elicitation. For example, problem decomposition, the idea of breaking a decision problem into multiple smaller decision problems, allows experts to assess probabilities for each component of the decision question, which is then synthesized into an overall probability estimate.
Additionally, transforming a probability elicitation question into a less statistically esoteric format, such as a betting game or pairwise comparison of decision options, may be more appropriate for experts with limited statistical knowledge.
Another key component in the research on expert elicitation documents techniques of aggregating input from multiple experts (Clemen and Winkler 1990; Winkler 1981). Generally, combination can be implemented behaviorally and mathematically. In the absence of expert interaction, an opinion pool can help aggregate assessments. Elicitors can assign different weights to different experts, especially according to the accuracy of their assessments from training/calibration sessions. For a detailed discussion on mathematical combination methods, see Clemen and Winkler (1999). If interaction is permitted among experts, the elicitor can distribute the assessments among the experts to achieve a group consensus. An example of this approach is the Delphi technique, which seeks to derive a consensus through iterative deliberation among all the experts. Note, however, the interactive group consensus method may be influenced by dominant personalities in the group. In terms of performance, some evidence suggests that mathematical aggregation outperforms behavioral aggregation (Cooke and Goossens 2004; Clemen and Winkler 1999; Hora 2009) while others suggest the determination of aggregation approach should be dictated by the decision tasks (DeWispelare et al. 1995).
Expert elicitation has been applied to a broad range of domains, ranging from environmental protection (Salvi and Gaston 2004), infrastructure vulnerability (Vrijling et al. 2004), accident consequence modeling (Cooke and Goossens 2000), nuclear waste regulations (DeWispelare et al. 1995), medical diagnosis (Lau and Leung 1999) to carbon capture and storage technology (Chan et al. 2011). Regardless of the variations in elicitation approaches and the application domains, for expert elicitation to maintain its scientific rigor as a consensus methodology, it is imperative that the elicitation methods, processes, and tools are transparent and free of biases; the selection of expertise is based on relevant expertise to ensure proper domain coverage while minimizing bias; and the analysis approach and elicitation results are repeatable (Cooke and Goossens 2000).
CCSI Element 7 Expert Elicitation Approach 3.0 The fundamental purpose of CCSI expert elicitation activities is to enable and facilitate the sharing of relevant, diverse, and highly specialized knowledge to collectively inform the risks associated with the development and deployment of carbon capture technologies. Engagement of the CCSI risk analysis team with experts for risk elicitation is critical. First, carbon capture technology is relatively new and highly complex with a host of potentially latent risk factors that might compromise the success of the technology if left unaddressed. Thus, leveraging the diverse expertise on the project team can help identity and address these latent risk factors in addition to the risks captured by the financial risk, technical risk, and technical maturity assessments. Furthermore, given the focus on simulation and modeling, some system features might not be modeled due to their fine granularity. These under-specified system characteristics might present potentially significant risks, thus demanding a finer-grained risk analysis of the modeling assumptions, parameters, operational risks, and those characteristics that cannot be captured by formal modeling but nonetheless have operational impact. Additionally, the engagement of experts and stakeholders through elicitation early in the technology development process provides an excellent opportunity for continuous and iterative risk communication to help them better understand and address risks as the technology development unfolds, resulting in greater risk reduction and technology acceptance by the stakeholders.
The CCSI Element 7 is developing an elicitation approach that combines both qualitative risk rankings and quantitative risk assessments to capture the under-specified and not-yet-modeled risk factors in carbon capture technologies. Figure 2 provides a step-by-step illustration of the elicitation activities.
Figure 2 CCSI Risk Factor Expert Elicitation Approach The elicitation plan will build on the previous solid sorbent CO2 capture technology risk elicitation framework (see Appendix ). This framework identified five main risk drivers, including (1) knowledge maturity and development capacity, (2) uncertainties related to financial/regulatory/public opinion, (3) technical performance in testing and deployment, (4) costs and efficiency, and (5) compatibility between simulation tools and physical systems. An extensive list of potential risk factors (totaling 98) was organized into three major risk dimensions: technical risk, economic risk, and regulatory/public risk. Each risk category was further operationalized by multiple sub risk factor categories as shown in the following
Using this risk factor list, the CCSI Element 7 team will organize a qualitative risk factor identification exercise with a panel of experts representing all the task teams within the initiative as well as from industry partners. This exercise will accomplish two objectives: (1) to eliminate irrelevant risk factors or risk categories; and (2) to incorporate relevant factors/risk categories that are missing from the list. With a streamlined list of potential risk factors, the experts will participate in a risk ranking elicitation where they will be asked to rank each risk factor on a numeric scale. The risk rankings will produce relevance weights for each risk factor, and help sharpen the focus of the follow-up elicitation effort. Once the priority risk factors are identified based on rankings, the elicitation team will develop contextualized risk statements for these factors and these statements will be pooled to produce a new CCSI risk factor expert elicitation instrument to be used in a systematic quantitative risk assessment. Risk is characterized by the severity of potentially adverse events and the likelihood of the occurrences of these events. To that end, we will distribute the instrument and ask expert respondents to provide a likelihood assessment and related magnitude assessment for each risk statement on a numerical scale (for an example instrument, see Hunton and Williams 2008). Mathematical aggregation algorithms will be employed to produce a combined probability distribution for each risk factor (Clemen and Winkler 1999). Next, the Element 7 team will create future risk scenarios using the identified risk factors to forecast failure rates over varying time horizons. For example, two programmatic performance criteria, a minimum 90% carbon capture efficiency rate; and a 30% cost increase ceiling for carbon capture technology in long-term operations, can be used in risk scenario development to compare the current and the desired level of technical capabilities. Again, experts will each provide a probabilistic risk forecast for these scenarios and their input will be aggregated. The resulting probability distributions from the scenarios as well as from the elicitation questionnaire will be integrated into the overall risk analysis framework to improve the quality of the risk assessment of carbon capture technology as a whole. A final step in this risk assessment effort requires the development of a targeted risk mitigation strategic plan to address the risks and reduce their adverse effects on CCSI program outcomes. It is important to note that the CCSI expert elicitation effort will be iterative in nature, and will closely follow the progress in carbon capture technology simulation and testing.
Status Report and Next Steps 4.0 The success of expert elicitation critically depends on the active collaboration across CCSI task teams and the support from the external stakeholders. To enhance the collaboration within CCSI for the expert elicitation, effort is under way to engage scientists from other Element teams to help modify and refine the technology maturity levels (TRL) questionnaire. The CCSI E7 Team has created a wiki-based elicitation and model development tool to enable novel, probabilistic technology maturity assessment by CCSI project teams (E1, E2, E3, and E97) as well as external experts. This collaborative effort will pave the way for more elaborate and involved elicitation activities described in this report. With respect to engagement with external experts, the Element 7 team conducted a site visit to Eastman Chemical Company in August 2012. This visit offered an invaluable learning opportunity to understand how risk analysis is routinely performed in business organizations. It also signals the establishment of a solid CCSI-industry partnership crucial for expert elicitation tasks in the near future. It is anticipated that in the coming program year, greater attention will be invested in the elicitation-driven risk assessment to compliment the financial and technical risk analyses and technology maturity risk modeling that are the focus of the present program year. If successful, the expert-driven risk elicitation will contribute to a better understanding of the potential vulnerabilities and risks in developing, testing, deploying, and commercializing carbon capture technologies, and provide mitigation strategies to reduce risk and uncertainty, and help the technology reach the stage of commercialization more reliability and rapidly.
Reference 5.0 Arnott, D. 1998. A Taxonomy of Decision Biases. Unpublished Technical Report. Monash University School of Information Management & Systems.