«An Engineering Educator’s Decision Support Tool for Improving Innovation in Student Design Projects NUR OZGE OZALTIN MARY BESTERFIELD-SACRE AND ...»
Advances in Engineering Education
An Engineering Educator’s Decision Support Tool for
Improving Innovation in Student Design Projects
NUR OZGE OZALTIN
RENEE M. CLARK
University of Pittsburgh
Learning how to design innovatively is a critical process skill for undergraduate engineers in the 21 century. To this end, our paper discusses the development and validation of a Bayesian network st decision support tool that can be used by engineering educators to make recommendations that positively impact the innovativeness of product designs. Our Bayesian network model is based on Dym’s design process framework and actual design process data collected from 26 undergraduate engineering capstone teams over multiple terms. Cross validation using all available outcomes data and a sensitivity analysis showed our model to be both accurate and robust. Our model, which is based on data from teams that produced both innovative and non-innovative products, can be used to formatively assess the process used by a design team and the level of innovativeness, thereby contributing to more innovative final design outcomes.
Key Words: decision support tool, Bayesian network model, engineering design education
INTRODUCTIONTo capture and retain market share in the modern business environment, today’s organizations must meet or exceed customer expectations through product innovation (Shen et al., 2000).
Innovation includes the introduction of new materials, components, and/or manufacturing pro- cesses; design changes employed to reduce manufacturing costs; and new applications of existing technology (Schumpeter, 1934).
SUMMER 2015 1
ADVANCES IN ENGINEERING EDUCATIONAn Engineering Educator’s Decision Support Tool for Improving Innovation in Student Design Projects Solid knowledge of innovative design is critical for successful, professional contribution within the workforce; therefore, participation in student design projects such as senior capstone experi- ences is an integral part of the undergraduate engineering curriculum. These activities are invalu- able learning experiences and equip future engineering professionals with the ability to design new products and services with innovative features in rapidly changing and highly competitive markets.
Thus, instructors who teach innovative design techniques would benefit from a decision sup- port tool for guiding teams to carry out those activities and processes that best contribute to the production of innovative products. To thisend, our paper describes the development of such a tool that allows engineering design instructors to 1) predict the innovativeness of a design artifact given the team’s activities as well as 2) provide guidance on activities that would contribute to a more innovative design outcome. This tool, implemented as a Bayesian network model, was developed based on empirical process-level activity data collected from undergraduate teams that produced both innovative and non-innovative capstone design products. Specifically, the activities of 26 senior bioengineering design teams were journaled using a secured, online survey system over a 23 to 24 week period as they progressed from an initial design concept to a working prototype.
Subsequently, using this process-level activity data and Dym’s design process framework, three separate Bayesian network models for the early, middle, and late phases of the design process were built using the GeNie decision modeling software. Bayesian networks allow for determining the probability that an upstream event or activity occurred given that a downstream event, such as an innovative outcome on a capstone design project, also occurred (Genie Documentation, 2014).
Thus, with a Bayesian network and ultimately Bayes Theorem, on which this network is based, we can determine the probability that a certain design process activity was performed to a certain degree in a certain project phase given that the design was ultimately rated as “innovative” or “non-innovative.” This type of analysis can lead to data-driven recommendations from instructors to students. Our work began as a doctoral dissertation, with subsequent refinement and analysis by the additional authors (Ozaltin, 2012).
Engineering Design and Design Frameworks Design is a central and distinguishing engineering activity (Simon, 1996). It is a complex process versus a single isolated action that has a collectivist nature (Okudan & Mohammed, 2006). There is also no universally agreed-upon definition of design (Hyman, 2003). The literature has many simplified step-by-step models and frameworks of the design process based on the ABET definition,
An Engineering Educator’s Decision Support Tool for Improving Innovation in Student Design Projects which presents the aspects of an ideal engineering design process (ABET Criteria, 2013). Hyman breaks the design process into the following nine steps: recognizing the need, defining the problem, planning the project, gathering information, conceptualizing alternative approaches, evaluating the alternatives, selecting the preferred alternative, communicating the design, and implementing the preferred design (Hyman, 2003). Another framework proposed by Pugh is based on a design core including the market, specification, concept design, detailed design, manufacture, and marketing (Pugh, 1990). Atman et al. compare freshman and senior engineering design processes and describe the design steps as follows: identification of the need, problem definition, gathering of information, generation of ideas, modeling, feasibility analysis, evaluation, decision, and communication (Atman et al., 1999). Dominick et al. define engineering design as an iterative process, and they segment it into the following four main phases: defining the problem, formulating solutions, developing models and prototypes, and presenting and implementing the design (Dominick et al., 2001). Several other engineering design models and frameworks can be found in the literature (Pahl et al., 2007;
Lewis & Samuel, 1989; French, 2010; Cross, 2001), although none of these models or frameworks is universally accepted by the engineering community (Hyman, 2003). We chose Dym’s engineering design framework, shown in Figure 1, as our theoretical model to generalize and thereby simplify Figure 1. Feedback and Iteration in Dym’s Design Process.
the large number of design activities used by our students (Dym & Little, 2004). We selected this model for two reasons. First, Dym et al.’s definition of engineering design matched the goals of the senior capstone project well. Their definition is as follows: “Engineering design is a systematic, intelligent process in which designers generate, evaluate and specify concepts for devices, systems, or processes whose form and function achieve clients’ objectives or users’ needs while satisfying a specified set of constraints” (Dym & Brown, 2012, p. 16; Dym et al., 2005, p. 104). In addition, Dym’s framework is flexible enough to be applied in different fields of engineering, including systems engineering, but still detailed enough to model important design activities, including iterations.
Dym’s model was also preferable to other design process models in the literature because it aligned with both the engineering design and engineering education disciplines and was well-suited to the data collected. Since our research was focused on both design and product realization, we expanded Dym’s model by adding the marketing and management categories, since many product realization projects incorporate these activities. Hence, our overall process was described by eight categories (i.e., the original six categories in Dym’s design process model and two product realization categories).
For this research, we adapted 89 design activities for collecting design process data throughout an academic year (Golish et al., 2008). These activities can be organized into engineering design stages based on the literature, such as opportunity identification, design and development, testing and preproduction, and introduction and production. In studies of design, it is common to generalize design activities into a smaller set of categories and/or cognitive operations. For example, the categories of exploration, generation, comparison, and selection have been used (Stempfle & Badke-Schaub, 2002). The challenge with this approach is its dependency on the information about the design activities and the fact that these activities often occur in cycles or iterations (Ha & Porteus, 1995; Krishnan et al., 1997).
Bayesian Networks and Their Applications Within the broad area of engineering design, our research focused specifically on creating a tool to influence innovative design outcomes. We ultimately used a Bayesian network (BN) to develop a decision support tool to increase the likelihood of innovative outcomes in design settings. A Bayesian network is a probabilistic graphical model that represents a set of variables as circular nodes and their conditional dependencies or interactions as arcs or arrows. A BN allows for forward and backward inference under uncertainty given known evidence and is useful for analyzing “what-if” scenarios, even those that are not observed in practice (Jensen & Nielsen, 2007; Yannou et al., 2013;
Genie Documentation, 2014). We used the GeNie software to create our Bayesian network model (Genie Documentation, 2014). This software provides a development environment for building
An Engineering Educator’s Decision Support Tool for Improving Innovation in Student Design Projects graphical decision models. Although Bayesian networks are formulated using only chance nodes, the “set evidence” property of GeNie allows a chance node to be treated as a “decision node” by setting the evidence to a chosen state.
Although the most popular application area for Bayesian networks is medical decision making, especially verification of a diagnosis, they have a wide range of applications in finance (e.g., market analysis), reliability (e.g., processor fault diagnosis), and defense (e.g., automatic target recognition).
Bayesian networks have also been applied to engineering design problems, including improvement of the early design stage by addressing uncertainties in component characteristics and compatibility (Moullec et al., 2013). This model also contributed to innovation by ensuring product feasibility and reducing the design risk. They focused on the early design stage (i.e., conceptual design) and determined the probabilities based on expert opinion. In contrast, our BN model encompasses the stages of conceptual design through prototype development and was built using actual design-team data to estimate the probabilities.
Another application of BN’s to engineering design was an evaluation of innovation by considering industrial contexts (Yannou et al., 2013). These authors performed an empirical study to identify the factors related to design and analyzed the influence of these factors on the quality of the problem setting and subsequently the problem-solving process as well as the quality of the innovative project outcome. In comparison, our model offers suggestions for the utilization levels of the design activities that may lead to more innovative design outcomes.
Data Collection The data used for developing our decision support tool was collected from bioengineering senior capstone design teams during the 2007-08 and 2008-09 academic years. Twenty-six teams participated, with 18 teams from an engineering school in the Mid-Atlantic region and eight teams from an engineering school in the Midwest. The design projects were similar in nature, in which all students had to design a biomedical product or device. Where possible, we minimized variability between the two institutions. The number of students per team varied from three to five, and the students were paid for their participation in the study. Students were surveyed twice per week through a secure online system to collect quantifiable data about their design activities. Within each of four design stages, a student could select up to three activities he/she had worked on. This number was arbitrarily set but was believed to be sufficient given the three to four day interval between the surveys. If the student had not worked on the project since the last survey, he/she could select
“I have not worked on the design.” Each student completed the survey up to a total of either 45 or 48 times, depending on his/her school. As discussed, the entire set of activities was based upon the work of Golish et al. and was further refined by the capstone instructors (Golish et al., 2008). Students were trained in the meaning of the activities and were provided with a definition sheet for easy reference.
It was assumed that students selected the activities and answered the open-ended questions in good faith. It is our belief for multiple reasons that students were honest in providing data. During their initial training session, students were informed that their answers would not be shared with the instructors and would not affect their grades. We utilized the training session and the assurances we gave to students at the time as a means to alleviate the Hawthorne effect (McBride, 2010).
Students had the option to select “I have not worked on the design,” which was chosen 129 times during the project timeframe. Further, while reviewing the data, students appeared to be selecting logical activities and writing detailed reflections. Their responses did not appear to be cursory in any manner.