Volume 2, Number 2, Special Issue 2011

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach Robert E. Kleine, III * and John-David Yoder Abstract – A process for developing and assessing programming -- curricular, co-curricular, or extracurricular -- intended to cultivate the entrepreneurial mindset in engineering students is described. Steps of the process are to 1) define learning-outcomes; 2) create a rubric for each learning-outcome; 3) develop activities that generate student work; 4) collect student work; 5) conduct direct assessment of student work by applying the rubrics to student work generated by the activities; 6) interpret and report assessment results; and 7) modify learningoutcomes, rubrics, activities or other program elements in response to assessment results.

1. Introduction and Background Stakeholders at educational institutions worldwide (e.g., students, parents, boards of trustees, state legislators, accrediting agencies, granting agencies and foundations, employers) increasingly expect evidence that an institution’s educational programs are effective (Arum and Roska, 2011). The proliferation of disruptive new business models for delivering educational experiences, enabled by advances in information technologies, increases the pressure on institutions to document their ability to develop and deliver programs that achieve claimed outcomes (Christensen et al., 2008). Current economic conditions, specifically declining tuition and investment revenues experienced by many institutions, further increases the pressure to document the degree to which programs are effective. These forces apply equally to engineering entrepreneurial education programs. Effective programs, courses, or class activity designs require clarity on the learning-outcomes to achieve and on the tools that will be used to assess them (Fink, 2003; Smith, 2008). Learningoutcomes are essential as they guide creation of programs, courses, class activities, and/or extraor co-curricular activities. Learning-outcomes also provide the benchmarks against which student achievement is measured. Such assessment of student performance reveals program effectiveness and guides program improvement efforts. Designers of engineering entrepreneurship programs must specify carefully the learning-outcomes that motivate their programs. Business creation (Bhide, 2000), if specified as a program outcome, will imply a program very different from a program designed to cultivate the entrepreneurial mindset, which emphasizes cognitive or attitudinal outcomes (Riffe et al., 2010). This article grew out of work with the Kern Entrepreneurship Education Network (KEEN). The KEEN program, an initiative of the Kern Family Foundation, is a network of schools working collaboratively to cultivate entrepreneurial engineers characterized by an entrepreneurial mindset (Kriewall and Mekemson, 2010).

*Associate Professor of Marketing, Pharmaceutical Business Program Coordinator James F. Dicke College of Business Administration, Ohio Northern University 57

R. E. Kleine, III * and J.D. Yoder Designing and implementing engineering education to cultivate students’ entrepreneurial mindset requires clarity on three key issues: 1. What is the entrepreneurial mindset (EM)? The EM must be defined in a way that is appropriate for development of specific learning-outcomes. 2. How can the entrepreneurial mindset be operationalized in a way that is appropriate for guiding the design of educational activities intended to cultivate the entrepreneurial mindset? The EM learning-outcomes must be expressed in a way useful for cultivating EM in students. 3. How can those activities be assessed to determine whether or not they have had the intended effects upon students’ entrepreneurial mindsets? We must be able to identify what activities effectively cultivate the EM and those that do not. Effectiveness insights are essential for enabling continuous program improvement. This article leverages work completed by KEEN to define the entrepreneurial mindset to propose a learningoutcome driven program development and assessment process model. The model is a framework intended to guide anyone involved in cultivating the EM in students, whether an individual faculty member, program coordinator, dean, or the director of a multi-school consortium (such as KEEN). The next section provides a brief overview of the proposed model of the learning-outcome driven program development and assessment process. Subsequent sections unpack model components. We conclude with suggestions for implementing the proposed model so as to maximize cultivation of the entrepreneurial mindset among our students.

2. An EM Program Development and Assessment Process Model A normative model of the learning-outcome driven program development and assessment process for cultivating the entrepreneurial mindset is depicted in Figure 1. This model reflects inspiration from several sources, most specifically Fink’s model (2003) for creating significant learning experiences and the Association of the American Colleges and Universities (AAC&U) initiative to develop normative models to guide assessment of undergraduate general education (Association of American Colleges and Universities, 2011). The process is inherently adaptive and recursive. Experiences applying each component of the model may precipitate modifications intended to more closely align the system with desired objectives. The following sections describe each of the model’s six key elements.

2.1. Learning Outcomes Learning outcomes that reflect the entrepreneurial mindset are the foundation to this approach. Learning outcomes are typically developed by each institution (general education learning outcomes, for example) or units within an institution such as colleges or departments (e.g., learning outcomes for a major or for individual classes; see Otter (1995) for a nice overview of learning outcomes). The process through which the EM learning outcomes were developed is unique in that the KEEN schools collaborated to develop a set to apply across institutions. Although a full chronology of the process employed by the KEEN schools is beyond the scope of this paper, it is important to note that this inter-institutional approach to development of learning outcomes is uncommon. The AAC&U LEAP initiative, which is developing a common set of general education learning outcomes is an exemplary exception. The EM learning outcomes were developed through a yearlong, multi-step process. The process included representatives from the various KEEN institutions, all private institutions with ABET accredited engineering programs, who met on several occasions with the goal of achieving consensus on a set of learning outcomes. 58

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach The resulting suite of seven EM learning outcomes defines the Entrepreneurial Mindset for the KEEN schools (see Table 1).

Figure 1. A Model of the Entrepreneurial Mindset Learning-outcome Driven Program Development and Assessment Process. Table 1. Entrepreneurial Mindset Learning-outcomes Developed by the Kern Entrepreneurship Education Network A student should be able to: 1. 2. 3. 4. 5.

Effectively collaborate in a team setting Apply critical and creative thinking to ambiguous problems Construct and effectively communicate a customer-appropriate value proposition Persist through and learn from failure. Effectively manage projects through appropriate commercialization or final delivery process. 6. Demonstrate voluntary social responsibility. 7. Relate personal liberties and free enterprise to entrepreneurship.

Volume 2, Number 1 Winter 2011

59

R. E. Kleine, III * and J.D. Yoder This paper focuses on the role of learning outcomes in the context of developing and assessing programs to cultivate the entrepreneurial mindset. Explication of how to construct these is beyond the scope of this paper. For purposes of EM program development and assessment, learning outcome statements provide direction to development and assessment of programming intended to cultivate specific outcomes in students. When used in an inter-institutional context, as with the KEEN schools, learning outcomes are also a vehicle for focusing the collective energies of disparate institutions toward a cultivating a common set of student abilities. It is important to realize that the suite of seven learning outcomes applies to the totality of a program. An individual activity, or even a single course, cannot cultivate all of the abilities implied by the EM learning- outcomes. A specific activity, such as an in-class egg case design exercise, strives to cultivate a focused subset of the EM learning outcomes. The myriad EM cultivating activities deployed by a program, in aggregate, yield the total impact of a program. One challenge with learning- outcomes, even those that are well crafted , is that they are often subject to multiple interpretations. Different interpretations of learning outcomes by individual faculty members or by disparate institutions induce variance in what abilities students develop with regard to a particular learning outcome. Rubrics are a tool commonly used to enhance uniformity of understanding of learning outcomes.

2.2. Rubrics to Operationalize the Entrepreneurial Mindset A rubric is an explicit set of criteria used for assessing a particular type of work or performance (The TLT Group, 2011). Rubrics serve several vital functions. This paper emphasizes the utility of rubrics for EM program development and assessment. Rubrics can serve other important functions. For example, rubrics can function as a communication tool among faculty within an institution or across institutions, as was mentioned above. When shared with students, rubrics afford an efficient tool for conveying performance expectations. Applied in the assessment process, rubrics serve as measuring devices for systematizing the assessment of student work. Rubrics are especially helpful when, as is often the case, the learning outcome has a subjective component. A rubric is a tool that helps objectify the assessment of subjective phenomena. A rubric provides a set of criteria developed to operationalize a learning outcome. The unit of application must be identified when developing learning outcomes and hence rubrics. The entrepreneurial mindset rubrics (reproduced in the Appendix) were developed for the purpose of assessing the work of an individual, as opposed to groups of students. The rubric is applied to a unit of work that can be identified as having been produced by a specific student. Individual student performance on a group activity can be assessed if the individual’s contribution is distinct and separable from the work of the other group members. For example, effective verbal communication for an individual could be determined from their part of a group presentation if there is a portion of the presentation during which the individual is the only group member speaking. In contrast, a group-produced written report would not be an appropriate artifact for assessing written communication effectiveness for individual students. Assessment rubrics should be shared with students at the time relevant activities are assigned. The rubrics thus afford students a useful guide for how to approach the activity. It is also productive to instruct students to self-assess their work with the relevant rubrics. The selfassessment might afford the foundation for a reflection paper. The self-assessment might also be directed toward closing the self-identified weaknesses in the work.

60

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach 2.2.1. A Recipe for Rubric Creation A general recipe for constructing an assessment rubric is as follows: a) Dimensionalize the learning outcome according to the distinct student tasks or abilities specified in the learning outcome. Rubrics are commonly summarized in matrix form. The dimensions typically form the rows of a rubric. This step can be tricky when a learning outcome includes a fundamental ability that is an implied or assumed but not explicitly stated. For example, consider the learning outcomes ―Persist through and learn from failure.‖ This learning outcome contains two explicit dimensions: persist through failure and learn from failure. Implicit in this learning-outcome statement is the ability for a student to recognize when a failure has occurred. An individual must recognize that a failure has occurred before they find evidence of persistence and learning. Consequently, the rubric for this learning outcome has three dimensions: ―failure recognition,‖ ―persist through failure,‖ and ―learn from failure‖ (see the Appendix). For this reason, the individual(s) creating a rubric must be proficient in the domain addressed by the rubric. b) Determine the number of performance levels. Performance levels typically form the columns of a rubric matrix. Performance levels delineate qualitatively distinct levels of student performance. The performance levels progress monotonically and typically range from absence of the desired ability (at the low end) to proficiency (at the high end). The number of intermediate performance levels included must strike a balance between making distinctions that capture incremental variation in qualitatively distinct levels of student performance and discriminating the results by all who will use the rubric. In general, ease of use declines as the number of performance levels increases. Although we have not found research on this topic, four levels of performance appear to strike the balance between capturing essential performance variance and ease of use. Four performance levels are used for the entrepreneurial mindset learning-outcome rubrics: Does not meet expectations, Developing, Meets Expectations, and Proficient.    

―Does not meet expectations‖ characterizes a student performance that does not display any of the desired target activity. ―Developing‖ characterizes a student performance that evidences some of the desired target activity. ―Meets expectations‖ designates student performance that evidences the minimal level of ability expected. ―Proficient‖ designates student performance that exceeds ―meets expectations‖ and evidences mastery of the target activity.

c) Identify student abilities that reflect performance characteristic of each dimension. It is useful to ask the question: What would provide evidence that a student has mastered this dimension? The resulting answer can form the foundation for describing the ―proficient‖ level of the dimension. For example, proficiency at the ―failure recognition‖ dimension was determined to be indicated by two activities: recognition that a failure has occurred and the ability to properly identify (categorize) the failure. Bloom’s taxonomy (Bloom et al., 1956) is a useful aid for articulating the desired behavior characteristic of proficiency. The act of describing what constitutes ―proficient‖ behavior may reveal that a dimension is very complex and should be disaggregated into sub-dimensions to fully capture the breadth of the domain encompassed by the learning outcome. The Effective Communication rubric reflects the complexity of the learning outcome. This rubric includes two levels of sub division. First, it

Volume 2, Number 1 Winter 2011

61

R. E. Kleine, III * and J.D. Yoder differentiates three types of communication—written, oral, and visual--on recognition that each is a unique behavioral domain. Second, written and oral share the ―overall organization‖ dimension—the behaviors that reflect that behavior differ by communication modality. The rubric descriptors reflect these differences. Each cell in the rubric matrix presents a description of the student behavior characteristics of the specific ability level along that dimension. The ability levels increase monotonically. Each ability level subsumes the performance abilities of the levels below it. Pragmatically, it is often easiest to write the description of the highest-level indicator and then back fill the lower levels. For example, proficiency of failure recognition is characterized by evidence of three conceptually related events: failure recognition, failure identification, and the ability to think critically about the consequences of the failure. Conceptually, recognition that a failure has occurred must occur before identification and apprehension of the consequences. Put differently, if a student is unaware that a failure has occurred, none of the subsequent behaviors can be present. Consequently, the descriptors progress from the following behaviors: not evident; evident at a low level (recognition that failure occurred); evident at a qualitatively distinct higher level (recognition that failure occurred and proper identification of the failure); evident at a level that reflects mastery of the dimension (recognition that failure has occurred, correctly identifying the failure, and explaining the consequences of the failure). The specific student behaviors that reflect each level will depend on the context in which they are applied. For example, evidence of failure recognition in the context of a freshman engineering activity, in which team constructed bridges that were stress-tested, will differ from the evidence of failure recognition if the student was analyzing a business model. Rubric construction can be expedited or avoided altogether by first searching for existing rubrics constructed by others (rubistar.com and tltgroup.org provide inventories of rubrics and tools for rubric construction). When developing rubrics for the EM learning outcomes, we leveraged the fact that several of the learning outcomes overlap with general education learning outcomes developed by the AAC&U and Ohio Northern University (ONU). The AAC&U have published a set of general education learning outcomes and rubrics with the intent that institutions use them as meta-rubrics to inspire institution- or program-specific rubrics. The AAC&U rubrics were developed through the assistance of faculty from institutions around the world. The rubrics developed by Ohio Northern University (2009), some of which were inspired by the AAC&U efforts, were created to support ONU’s new general education program. ONU rubrics were constructed by teams comprised of ONU faculty with relevant expertise via a modified Delphi technique facilitated by EduMetry (http://edumetry.com/), a consultancy that specializes in higher education program assessment. The use of teams, comprised of faculty from across the ONU campus, was employed to increase the likelihood that the resulting rubrics, as they result from faculty engagement, would be more readily accepted and adopted across disciplines. 2.2.2. EM Rubrics: An Overview A brief overview of the rubric developed for each EM learning outcome follows. It is important to keep in mind that rubrics are typically part of a suite of tools that faculty, programs, or foundations might use to assess the effectiveness of a program. Also, these rubrics are intended to provide a starting point. It is expected that the rubrics will be modified as application data becomes available. These overviews are intended to provide background on the origins and logic underlying each of the EM learning-outcome rubrics. The rubrics themselves are presented in the Appendix.

62

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach Effectively collaborating in a team setting – This rubric built on the foundation afforded by the AAC&U teamwork rubric, specifically, the AAC&U rubric’s five dimensions: contributes to team meetings, facilitates the contributions of team members, the individual’s contributions outside of team meetings, and fosters constructive team climate. Working in collaboration with two ONU management faculty, the target behaviors were adjusted in ways that better align with salient target attributes based on their understanding of the teamwork literature and experience with senior capstone groups. Emphasis was placed on ―soft skill‖ attributes that are sufficiently objective such that their presence or absence can be reliably detected. The ―contributes to team meetings‖ dimension captures the degree to which an individual effectively engages during team meetings. ―Facilitates the contributions of team members‖ focuses on the degree to which team members respect and enable contributions by other team members. ―Individual’s contribution outside of team meetings‖ captures the degree to which an individual completes his/her team assignments in accordance with the agreed upon time schedule and assures that the work done is of a quality that advances the project effort toward its goals. ―Fosters constructive team climate‖ focuses on the degree to which an individual respects, motivates, encourages, and supports team members. Conflict is common in group activities. How individuals handle conflicts can determine whether a group becomes dysfunctional or succeeds and proceeds forward. The ―responds to conflict‖ dimension captures the degree to which an individual engages and resolves conflict in an appropriate way. Applying critical and creative thinking to ambiguous problems – Critical and creative thinking are considered by many to be the holy grail of learning outcomes; cultivating critical and creative thinking is regarded by many as the primary purpose of an undergraduate college education (Arum and Roska, 2011). The ability to think critically and creatively is fundamental to entrepreneurial activity. Ambiguous problems refer to the ill-structured nature of the situations to which the entrepreneurial mindset is applied. Ill-structured problems contrast with well-structured problems. Well-structured problems have an unambiguously correct solution. For example, ―what is the load bearing capacity of a bridge made from carbon fiber of a specified density and dimensionality?‖ In contrast, ―what is the best way to study for my statistics exam?‖ is an illstructured problem. There are myriad possible solutions. The challenge is to apply critical and creative thinking to identify the solutions that are most appropriate given the constraints and resources available. The rubric presented here was developed by a team of ONU faculty for application with ONU’s general education program. The learning outcome is elaborated as the following: students are able to correctly identify the underlying problems or issues in both theoretical and practical realms; apply appropriate analytical and creative skills to develop feasible alternatives while considering multiple perspectives; and provide creative and logical solutions. The ONU team process was seeded by two AAC&U rubrics: critical thinking and creative thinking. The resulting rubric has three sections: problem identification, research and analysis, and solution development. Constructing and effectively communicating a customer-appropriate value proposition – Innovation and entrepreneurial activity are complementary yet distinct activities. Entrepreneurial activity concerns innovation harnessed in the service of providing value to an identified market. This learning outcome has two distinct components: constructing a customer-appropriate value proposition and the ability to effectively communicate that value proposition to key constituencies (e.g., end users, purchasers, investors). A search failed to yield a rubric appropriate for the customer-appropriate value proposition learning outcome. SRI International’s NABC model—Need, Approach, Benefits relative to costs,

Volume 2, Number 1 Winter 2011

63

R. E. Kleine, III * and J.D. Yoder and Competition—for formulating a value proposition (Carlson and Wilmont, 2006) was thus adopted to operationalize this learning outcome for three reasons. 1) The NABC model has a proven record of success (Carlson and Wilmont, 2006). 2), The NABC model has proven useful as a pedagogical aid in Kleine’s Product Development class. NABC provides a framework students grasp readily and can apply productively when assessing and expressing a value proposition. 3) The KEEN community, the ecosystem in which the rubric is to be deployed, is familiar with the SRI model. This familiarity is conducive to accelerating adoption of the rubric across the KEEN schools. The rubric operationalizes the key dimensions of the NABC formulation:      

Is an important customer and market need addressed? Is the proposed approach meaningfully unique or distinctive relative to alternatives? Does the proposed approach offer distinctive benefits relative to costs? How do the benefits/costs of the proposed solution compare to existing alternatives? Are the benefits/costs of the proposed solution quantitatively superior to existing alternatives? Are key constituencies identified and an appropriate variation of the value proposition formulated for each? Key constituencies might be prospective investors, channel partners, and end-users. The value proposition appropriate for an end-user would not be appropriate for a channel partner or a prospective investor.

This rubric implicitly subsumes myriad sub-activities necessary to effectively support each level. For example, the ability to identify key constituencies and identify and quantify market needs presumes market research and analysis are conducted. The rubric is crafted such that a comprehensive business plan is not mandatory. The rubric also does not presume that a working prototype is crafted, but rather the emphasis is on making the case for the potential of an idea. Effective communication is one of ONU’s general education learning outcomes. In the interest of efficiency, ONU’s rubric for effective communication general was adopted. This also illustrates that the rubrics proposed are modular. If an institution has effective communication as a general education or other learning-outcome and has a rubric developed for assessing it, that rubric could be adopted to leverage existing infrastructure on campus. ONU’s effective communication learning outcome has three major components: effective written communication, effective verbal communication, and effective nonverbal communication. Persisting through and learning from failure – An individual’s ability to overcome adversity, to stick with a problem until an effective solution is found, to ―fail-forward‖ effectively (Maxwell, 2000), is characteristic of ―growth mindset‖ individuals (Dweck, 2000). Others, such as Dwek’s ―fixed mindset‖ individuals are less likely to try again upon experiencing failure. Dwek’s research shows that ―fixed mindset‖ individuals can be taught how to approach situations using a growth mindset approach. This research suggests that the ability of an individual to persist through and learn from failure is a skill and not a personality characteristic inherent in individuals. Consequently, individuals can cultivate their ability to negotiate failure. This learning outcome has three fundamental dimensions that build on one another: failure recognition, persist through failure, and learn from failure. The ability to recognize that a failure has occurred is essential as it sets the stage for corrective action. Consequently, the first row of the rubric captures the ability to identify, categorize, and apprehend the consequences of the identified failure. Implicit in this formulation is the possibility that a recognized failure is

64

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach misidentified. Improper identification or categorization of the failure will have implications for the individual’s ability to effectively mitigate the failure. Identification that a failure has occurred sets the stage for figuring out how to deal with the failure. Persisting through failure is operationalized in terms of an individual’s ability to identify and articulate a course of action that could mitigate, reduce or eliminate the consequences of the identified failure. Effectively managing projects through appropriate commercialization or final delivery process – This learning outcome recognizes that a good customer-appropriate value proposition must be translated into reality. That translation requires understanding of the process through which an idea is transformed into a valuable entity. The rubric has four primary dimensions: 1) project level, 2) task level, 3) evaluation and 4) deliverables. The project level dimension captures activities relevant to overall project management: goal setting, steps to achieve goals, optimal use of resources to achieve those goals, and the ability to create and maintain a budget. This rubric was developed with the recognition that different programs teach different process models. For example, some programs emphasize the stage-gate model (Cooper, 2011) for managing the product development process. Consequently, students would be expected to evidence understanding of the stage-gate model in how they approach project management. Embedded within the overall structure of a project are the myriad tasks that must be identified, defined, sequenced and implemented in a way that advances the project toward its ultimate goals. These task-level project activities include defining and sequencing appropriate tasks, calculating the time a task will require, executing the task, and managing the task process, which may require delegating and/or coordinating with team members. Evaluation captures the ability to effectively assess progress toward the identified goal(s). This includes the ability to articulate procedures to use to assess project effectiveness into the future. Deliverables, the materials that must be created and supplied to designated constituencies by an identified time form the final dimension. This dimension reflects that projects typically require generation of materials that provide evidence of project activity. The specific materials to be delivered will depend on the project and its goals. For example, one course may have creation of a development plan as its key deliverable. Another course may entail a working prototype with supporting documentation as expected deliverable materials. Demonstrating voluntary social responsibility – This learning outcome parallels ONU’s general education learning outcome for ―informed and ethical responses to personal, civic and global needs. ― These learning-outcomes parallel the AAC&U ―Civic Engagement‖ learning outcome. However, the KEEN learning outcome is not isomorphic with these two learning outcomes. Accordingly, the rubric (see the Appendix) leverages those foundations, yet embraces the uniqueness of the KEEN learning outcome. The target behaviors are that students display an understanding of personal, civic and social needs; identify underlying ethical issues in specific situations; and make informed ethical responses to those needs. This parsing strives to embrace the spirit of the KEEN learning outcome while maintaining as much overlap as is possible with the ONU and AAC&U general education rubrics so as to maximize utility to schools that may apply it. The rubric has three dimensions that operationalize the three parts of the learning outcome. The first dimension concerns the degree to which the individual evidences an understanding of needs

Volume 2, Number 1 Winter 2011

65

R. E. Kleine, III * and J.D. Yoder that are grounded in data as opposed to personal opinion. The second dimension concerns an individual’s ability to discern the ethical issues that arise from social needs. The third dimension concerns the individual’s ability to respond to identify needs in an appropriate way by applying appropriate analytical frameworks. Relating personal liberties and free enterprise to entrepreneurship – This learning outcome highlights individuals’ understanding of personal liberties and free enterprise and how each is reflected in and enabled by entrepreneurship (Schramm, 2006). As such, this learning outcome focuses on the intersection of values, government regulation, and individual initiative directed toward creating and delivering value. The rubric’s three dimensions flow naturally from the learning outcome. Entrepreneurship, the first dimension, focuses on the student’s understanding of entrepreneurial behavior and attributes of successful entrepreneurial behavior. The second dimension taps the student’s ability to provide evidence of a connection between personal liberties and entrepreneurship. The third dimension focuses on individuals understanding of free enterprise and entrepreneurial behavior. 2.3. Developing curricular, co-curricular, and extra-curricular activities, and programs that cultivate the EM An outcomes-based approach to program development implies that learning outcomes are simultaneously the beginning and the result of the education cycle. Rubrics, by providing an elaboration of each learning outcome, afford a tool useful for aiding the design of educational activities. ―Activity‖ is used here as a generic term for curricular, extra-curricular and/or cocurricular activities intended to further a learning outcome. Student abilities are typically cultivated via modular units such as majors, minors, courses, extracurricular activities such as student organizations, and co-curricular activities such as idea pitch competitions. The smaller the modular activity unit, the more specific the learning outcomes the activity will develop. In the case of the KEEN program, for example, all seven of the entrepreneurial mindset learning-outcomes apply. Program-level design involves asking the question: how will we cultivate the abilities of our students with the goal of them achieving proficiency? As with all unstructured problems, there is no single best answer. What is a best approach for one school may not be appropriate for another school. 2.3.1. The Program Matrix A Program Matrix is a visual tool useful for guiding program development. Form a Program Matrix by expressing the learning outcomes as columns. Express the various EM cultivating activities available on a campus as the rows. Place a check mark or other symbol in the table cells to designate the specific EM learning outcome(s) addressed by a particular activity. The result is a convenient visual summary of an institution’s EM cultivating initiatives. Review the summary relative to the objectives established for your institution’s EM initiative. Identify learning outcomes lacking coverage. These voids should be addressed. Similarly, review the summary to identify over-emphasized learning outcomes. Over emphasized areas could be scaled back. The resources freed up (e.g., class time) can be reallocated to develop activities that cultivate the underserved learning outcomes. As a first pass, the matrix may include only the seven learning outcomes. As the program design becomes better specified, expand the learning outcomes into their constituent dimensions and rubric rows. Review the resulting Program Matrix to identify voids. The goal is for each learning outcome to map onto at least one activity.

66

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach 2.3.2. Aligning Activities with Learning Outcomes, Rubrics and Rows Activities should be closely aligned with the EM learning outcomes and rubrics. Activity/rubric alignment requires addressing two questions: 1) What learning outcome(s) will this activity cultivate? This step identifies the relevant rubrics. 2) What row(s) of the relevant rubrics apply? Typically, an activity will cultivate one or two of the EM learning outcomes and specific rows of the relevant rubrics. A single assignment created for a particular course might focus on parts of one or more learning outcomes. For example, an activity that requires the student to develop and deliver an elevator pitch might be used to develop a student’s ability to construct a customer appropriate value proposition and her oral communication skills. When developing the activity, care must be taken to align the activity task with the behaviors emphasized by the relevant EM rubric rows. Rarely will a single activity suffice for bringing students to the target proficiency level for a specific learning outcome. One activity will generally impact incrementally the degree to which participating students are proficient. In many cases, multiple activities that cultivate a single ability will be required for students to achieve proficiency. For example, critical and creative thinking might be developed throughout a student’s undergraduate program. A single activity might simultaneously cultivate multiple learning outcomes. For example, a group activity focused on developing criteria for screening ideas, may simultaneously cultivate critical thinking to solve ambiguous problems, team collaboration skills, and an element of constructing a customerappropriate value proposition. It is useful to prioritize the intended learning. It is also possible that an activity will cultivate specific parts of a learning outcome. For example, an activity that requires students to prepare a slide deck to support defense of a selected alternative might be identified as an activity that cultivates visual communication skills. The rubrics for the learning outcomes (and sub-parts) to be emphasized in a particular activity provide useful guidance on the types of work the students might do to cultivate the outcome. They provide guidance on the specific skills that should be emphasized or that may be developed prior to commencing an activity. The rubrics can also be used as a way to check activities to verify they are appropriate for the targeted learning outcomes. As a result of engaging the activities created to cultivate learning outcomes, students generate artifacts or work. The artifacts provide data useful for assessing proficiency at the student level, the activity level, and the program level. 2.4. Accumulating evidence of student ability Data that provides direct evidence of student ability is fundamental to the proposed model. Program accrediting agencies, such as ABET, expect direct assessment of student work. Evidence of student EM abilities is termed an artifact. An artifact reflects the work of an individual student that is relevant to one or more rows of the EM learning-outcome rubrics. An artifact may take myriad forms: a reflection paper written by a student; a video of a student presentation; a student’s response to an exam question; or a term paper or project report and more. A system or systems for accumulating and storing artifacts is needed. Although the details of artifact archive systems are beyond the scope of this paper, we outline some basic parameters. Procedures for collecting EM artifacts also need to be established. Many artifacts will be generated in response to a class assignment. These artifacts can be collected using normal class procedures, which may range from students handing a tangible assignment to the instructor to uploading the assignment to a class management system (CMS) such as BlackboardTM. The system used to collect and archive student artifacts should be compatible with the assessment

Volume 2, Number 1 Winter 2011

67

R. E. Kleine, III * and J.D. Yoder system (see below). As with all educational materials, the procedures developed should be compliant with the Federal Educational Records Privacy Act (FERPA). The artifact collection system should be designed such that artifacts can be retrieved when needed by those responsible for EM assessment. 2.5. Assessment Assessment is about gaining insights into whether or not program learning outcomes are being realized. Assessment can be applied to the totality of the EM program or to each activity that cultivates EM abilities. Assessment, in the proposed model, involves applying the rubrics to the student artifacts. This may involve applying specific rubric rows, rather than a rubric in its entirety. The rubric rows applied will be the those designated when constructing the activity that generated the student artifact. Artifact scoring involves applying the appropriate rubric row(s) to an artifact. For each applicable rubric row, the artifact is scored into one of the performance levels. For the entrepreneurial mindset rubrics, that would entail categorizing an artifact as ―does not meet expectations,‖ ―developing,‖ ―meets expectations,‖ or ―proficient.‖ In Blackboard and in other LMS systems, rubrics can be enabled via grading forms to streamline the scoring process. A grading form (rubric) can be attached to an assignment. LMS based approaches such as this one automatically capture how an artifact is scored on each rubric row and aggregate the scoring data. EpsilenTM (www.epsilen.com) is the only CMS of which we are aware that enables sharing of rubrics across courses, programs, and institutions. Rubric sharing makes it possible for one source to publish rubrics that can then be easily used by many scorers at different institutions, as well as those teaching different courses. The outcome of the artifact scoring process is a matrix of students by rubric rows. Assessment of an activity or a program is based on aggregated data. Data aggregation involves determining the frequency for each rubric row at which each proficiency column was evident in the student artifacts. These summaries provide a useful snapshot for answering the question, ―how are we doing?‖ Interpretation is simplified if the scores are normalized as percentages and displayed visually as bar charts. A preponderance of scores in the ―meets expectations‖ and ―proficient‖ categories for a rubric row would suggest students are performing well at that ability. A preponderance of scores in the ―does not meet expectations‖ or ―developing‖ categories signals ability on which more development is needed. Low ability is expected, for example, of freshman students making their first attempts at creating a customer appropriate value offering. A pattern of low ability evident in the work of senior standing students, on the other hand, could point to a gap in program design or a weakness in relevant program activities. Figure 2 illustrates three scenarios based on simulated data for n=20 artifacts. Each bar represents the number of student artifacts scored at each of the four proficiency levels. Let us assume that the goal is for at least 85% of artifacts to be scored as ―meets expectations‖ or ―proficient.‖ In the left-most scenario, 15 of the 20 artifacts were scored ―does not meet expectations.‖ Only 15% of the artifacts reach the goal levels. This distribution reflects a lack of proficiency. In the middle scenario, the scores are concentrated in the ―developing‖ and ―meets expectations‖ categories. The middle scenario provides evidence that some progress is being made to achieve the learning outcome, however, only 45% of the artifacts meet the minimum expectations. Clearly there is room for improvement. The bar set on the right reveals that the 60% of artifacts are scored at ―meets expectations‖, and 25% of the artifacts are scored as ―proficient.‖ Thus, the goal is realized as 85% of the artifacts are scored as ―meets expectations‖ or ―proficient.‖

68

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach

Figure 2. Three assessment scenarios Who scores the artifacts? Artifact scoring should be objective, repeatable, and performed by knowledgeable individuals familiar with the artifacts and the rubric(s) they are applying. Units vary in terms of how artifact scoring is handled. Faculty in the College of Engineering at ONU, for example, are responsible for scoring artifacts generated by assignments in classes that they teach. In contrast, ONU’s new outcome-driven general education program places the responsibility for scoring artifacts under the jurisdiction of the University Assessment Committee. Companies such as EduMetry provide an artifact scoring services. Departments and colleges will need to develop artifact-scoring procedures that fit with the resources available. A database of artifact scores should be designed to allow assessment of student ability at key program points (e.g., freshman vs. seniors). The ability to assess intra-student development over time is also a desirable feature, as it would allow tracking student proficiency changes over time. Artifact score data can also be compared across institutions. An example of how assessment data might be aggregated is provided in Figure 3. Figure 4, which presents simulated data, illustrates how assessment data might be reported for each learning outcome. Each bar represents the percentage of artifacts that meet or exceed the identified minimum expectation. Reprising the standard used earlier of at least 85% of artifacts scoring at the ―meets expectations‖ level or better, the example (Figure 4) suggests that program targeted proficiency is achieved on all but the first two learning outcomes. This suggests that changes may be needed in regards to how student collaboration and critical thinking abilities are cultivated.

Volume 2, Number 1 Winter 2011

69

R. E. Kleine, III * and J.D. Yoder

EM Learning-outcome

Does Not Meet Expectations

Developing

Meets Expectations

Proficient

Effectively collaborate in a team setting Apply critical and creative thinking to ambiguous problems Construct and effectively communicate a customerappropriate value proposition Persist through and learn from failure. Effectively manage projects through appropriate commercialization or final delivery process. Demonstrate voluntary social responsibility. Relate personal liberties and free enterprise to entrepreneurship. Figure 3. Example EM Program Dash Board

Figure 4. Example Program Summary Graphic 70

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach 2.6. What should be changed? Assessment enables continuous improvement. When there is evidence that student performance is not achieving or exceeding target levels, those areas of underperformance should be examined and adjusted. Our hypothetical score card data revealed failure to achieve target performance levels for the effective collaboration and critical thinking learning outcomes. These failures may be due to the activities developed to cultivate the learning outcome, problems with the rubric, or specific rows of the rubric, or with the learning outcome. Each is discussed. 2.6.1. Reviewing then modifying existing or create new activities Lack of performance may reflect problems with the activities designed to cultivate the learning outcome. Examine the portfolio of activities developed to cultivate the learning outcome. The Program Matrix, described above, will be very useful for this task. Perhaps, examination of the Program Matrix reveals an absence of activities designed to cultivate the learning outcome: no activities equals no development of relevant student abilities. Activity development is the obvious next step. If the performance matrix reveals activities intended to cultivate abilities relevant to the learning outcome, then drill down in the assessment data to determine whether failure to achieve the target performance level can be isolated to specific rows of the rubric. Determine whether activities exist to cultivate the identified problem rows. If no activities exist, they should be created or existing activities should be modified to include them. If all rubric rows are addressed by existing activities, those activities should be reviewed and modified or replaced. Activities should be reviewed to ensure they are aligned with the abilities emphasized by the rubric. Activities not aligned with the rubric should be modified to bring them into alignment. It is possible that each activity will be modified in some way in response to its application and assessment. 2.6.2. Modify the rubric? It is possible that failure to achieve target performance levels is due to errors, omissions, or misspecifications in the rubric. It is possible that the level of performance formalized into the rubric is too ambitious for an undergraduate program, or not possible given available resources, for example. It is also possible that individuals applying the rubric are misinterpreting the language describing the performance levels. This may be discovered through conversation with individuals that score the artifacts. These exemplify some of the reasons that assessment data may suggest modifying a rubric to remove these sources of error or bias. After making modifications to a rubric, the activities relevant to that rubric should be reviewed and modified, if necessary. The assessment process should enable and inform continuous refinement of the rubrics. 2.6.3. Modify the learning-outcome? Modifying the learning outcome is a third area of possible change. Failure to achieve performance levels on a specific learning outcome may reflect a learning outcome that is too broadly defined or unattainable. The assessment data patterns may suggest ways to modify the learning outcome to one that is well defined or attainable. Changes to learning outcomes may also be changed to reflect shifting priorities or new philosophies. Of the three areas of possible change, modifications to learning outcomes are the least frequent.

3. Conclusions and Future Work This article describes a learning-outcome driven approach to developing and assessing programs to cultivate the entrepreneurial mindset in undergraduate students. The entrepreneurial mindset is characterized by seven learning outcomes. Each learning outcome is operationalized via a rubric.

Volume 2, Number 1 Winter 2011

71

R. E. Kleine, III * and J.D. Yoder The rubrics afford specific guidance to faculty, program coordinators and others in their efforts to create educational activities intended to develop specific abilities reflective of the entrepreneurial mindset. The rubrics also provide a basis for assessing student performance so as to ascertain the effectiveness of an individual activity, a course, or a program. The rubrics as presented represent an initial effort to operationalize the seven KEEN learning outcomes. Future work is needed to refine and validate the rubrics. This requires that faculty employ the rubrics and provide feedback on their experiences. We emphasize that the rubrics as presented are intended as a starting point. It is anticipated that the rubrics will evolve and improve through use across KEEN. Further, it is possible that individual institutions may see fit to modify one or more of the rubrics to better reflect local approaches to cultivating aspects of the entrepreneurial mindset. At the time of this writing, rubrics for three of the seven EM learning outcomes are undergoing field trial by faculty at participating KEEN institutions. Feedback received through this process will be used to revise the rubrics. It is expected that two rubrics will be subjected to field trial each academic term on an ongoing rotation. Faculty members around the world are creating activities that cultivate some aspect of the entrepreneurial mindset. It would seem desirable to create an easily accessible EM Activity Repository of these activities. The repository should designate each activity according to the entrepreneurial mindset learning outcome(s) it is designed to cultivate and also allow designation of the rubrics and the rows of the rubrics that should be used for assessing student work generated by the activity. A tool such as this would allow faculty or program designers, perhaps after reviewing their Program Matrix, to quickly identify activities that cultivate specific student abilities. Faculty members seek EM cultivating activities that are effective. It would be desirable for the EM Activity Repository to also accumulate assessment results from faculty who have used the activities. It would also seem desirable that the Repository would allow faculty to submit updated versions of an activity that reflects modifications based on experience applying the activity. An assessment dashboard would be desirable. The dashboard would display a metric that reveals the degree to which students in a program are realizing each of the EM learning outcomes. To gain understanding of the effectiveness of a multi-school initiative, dashboard data could be aggregated across institutions. Ideally, the dashboard would provide a summary at the learningoutcome level and also allow fine-grained analysis of program effectiveness for each learningoutcome sub-area. A dashboard that allows drilling down to the rubric-row level would yield such fine-grained insights.

References Association of American Colleges and Universities. (2011). Valid assessment of learning in undergraduate education. http://www.aacu.org/value/rubrics/ (last accessed February 2011). Arum, R. and Roska J. (2011). Academically Adrift: Limited Learning on College Campuses. Chicago: University of Chicago Press. Bhide, A.V. (2000). The Origin and Evolution of New Business. New York: Oxford. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals. Handbook I: Cognitive Domain. New York: Longmans, Green, 1956. Carlson, C.R. and Wilmot W. (2006). Innovation: The Five Disciplines for Creating What Customers Want. New York: Crown. 72

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach Christensen, C.M., Horn, M.B., and Johnson, C.W. (2008). Disrupting Class: How Disruptive Innovation Will Change the Way the World Learns. New York: McGraw-Hill. Cooper, R.G. (2011). Winning at New Products: Creating Value Through Innovation. New York: Basic Books. Dweck, C.S. (2000). Self-Theories: Their Role in Motivation, Personality, and Development. Florence, KY: Psychology Press. Fink, L.D. (2003). Creating Significant Learning Experiences: An Integrated Approach to Designing College Courses. San Francisco, CA: Jossey-Bass. Kriewall, T.J., and Mekemson, K. (2010). Instilling the entrepreneurial mindset into engineering undergraduates, The Journal of Entrepreneurial Engineering, 1(1): 5-19. Ohio Northern University. (2009). Final Report of the General Education Committee. http://www.onu.edu/files/academic_affairs_general_education_final_report_i_23april2009_a nd_rubrics_-_updated_03252011.pdf (last accessed August 2011). Otter, S. (1995). Learning-outcomes in higher education. In J. Burke, ed., Outcomes, Learning, and the Curriculum: Implications for NVQ’s, GNVQ’s and Other Qualifications. Bristol, PA: Falmer Press, 271-283. Maxwell, (2000). Failing forward: Turning mistakes into stepping stones for success. Nashville, TN: Thomas Nelson, Inc. Riffe, W.J., Tavakoli, M., and Harris, M. (2010). ―Design, implementation and assessment of entrepreneurial workshops for an entire university faculty.‖ NCIIA Open 2010, http://nciia.org/sites/default/files/conf2010papers/riffe.pdf (Last accessed August, 2011). Schramm, C.J. (2006). The Entrepreneurial Imperative: How America’s Economic Miracle will Reshape the World (and Change Your Life). New York: Harper Collins. Smith, R.M. (2008). Conquering the Content: A step-by-Step Guide to Online Course Design. New York: Jossey-Bass. The TLT Group: Teaching, Learning, and Technology. (2011). ―Rubrics: Definition, Tools, Examples, References,‖ http://www.tltgroup.org/resources/flashlight/rubrics.htm (Last accessed February 2011).

Volume 2, Number 1 Winter 2011

73

R. E. Kleine, III * and J.D. Yoder Appendix Draft Rubrics Developed to Operationalize the Entrepreneurial Mindset

KEEN Learning Outcome: Effectively collaborate in a team setting Student is able to contribute to team meetings in ways that advances the group’s work, facilitates the contributions of other team members, contributes to the project effort outside of team meetings, fosters a constructive team climate and responds effectively to conflict that may arise within the team effort.

Facilitates the Contributions of Team Members

Contributes to Team Meetings

Does Not Meet Expectations

74

Developing

Meets Expectations

Proficient

Offers new ideas or solutions to advance the work of the group. Does not share ideas or solutions that advance the work of the group.

Offers alternative ideas or solutions or courses of action that build on the ideas of others. Helps the team move forward by articulating the merits of alternative ideas or solutions. Engages team members by taking turns and listening to others without interrupting. Engages team members in ways that facilitate their contributions to meetings by restating the views of other team members and/or asking questions for clarification.

Does not engage team members by taking turns and listening to others without interrupting.

Engages team members in ways that facilitate their contributions to meetings by constructively building upon or synthesizing the contributions of others. Notices when someone is not participating and invites them to engage.

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach

Fosters Constructive Team Climate

Individual Contributions Outside of Team Meetings

KEEN Learning Outcome: Effectively collaborate in a team setting (Continued) Completes all assigned tasks by deadline Work accomplished advances the project. Work accomplished is thorough and comprehensive.

Does not complete all assigned tasks by deadline.

Does not support a constructive team climate; does none of the following:

Proactively helps other team members complete their assigned tasks to a similar level of excellence. Supports a constructive team climate by doing any two of the following:

Supports a constructive team climate by doing any three of the following:

Supports a constructive team climate by doing all of the following:

Treats team members respectfully by being polite and constructive in communication. Uses positive vocal or written tone, facial expressions, and/or body language to convey a positive attitude about the team and its work. Motivates teammates by expressing confidence about the importance of the task and the team's ability to accomplish it. Provides assistance and/or encouragement to team members. Identifies and acknowledges conflict.

Responds to Conflict

Redirects focus toward common ground, toward task at hand (away from conflict). Stays engaged with conflict until it is resolved. Passively accepts alternate viewpoints/ideas/ opinions; fosters conflict.

Directly and constructively helps to manage/resolve conflict in a way that strengthens overall team cohesiveness and future effectiveness.

Derived from the AAC&U Teamwork Value rubric (http://www.aacu.org/value/rubrics/Teamwork.cfm)

Volume 2, Number 1 Winter 2011

75

R. E. Kleine, III * and J.D. Yoder KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems Students are able to correctly identify the underlying problems or issues in both theoretical and practical realms, apply appropriate analytical and creative skills to develop feasible alternatives while considering multiple perspectives, and provide creative and logical situations. Section 1 of 3: Problem Identification

Considers the situation from multiple perspectives

Identifies key issue(s) and converts to a problem statement

Does Not Meet Expectations

76

Developing

Meets Expectations

Proficient

Does not show comprehension of the situation to be able to identify the key issue(s)

Shows comprehension of the situation and correctly identifies some key issue(s) but does not articulate a reasonable problem statement

Shows good comprehension of the situation and correctly identifies most key issue(s) and articulates reasonable problem statement(s)

Comprehends situation to be able to identify key issue(s) and articulates valid problem statement(s)

Does not consider the problem(s) from different perspectives to view it comprehensively

Brings out some of the perspectives applicable to the issue(s), but leaves out some aspects; hence, problem identification suffers from this bias/skew

Considers the issue(s) using a few applicable alternate perspectives, but does not use this to ―frame‖ the key issue(s) realistically/ originally

Considers the issue(s) from multiple perspectives and uses this to ―frame‖ the key issue(s) realistically/ originally

Shows evidence of combining some of the alternate perspectives to present well rounded issue(s)

Is able to combine different perspectives and relate them to each other effectively to comprehend the issue(s) more comprehensively

Doesn't show the ability to combine these different perspectives to view the issue(s) more comprehensively

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems (Continued)

Analyzes information using appropriate conceptual framework/tools

States plausible reasoning for position or actions Demonstrates ability to identify and described evaluate information

Section 2 of 3: Research and Analysis Does Not Meet Developing Expectations

No evidence of search, selection or source evaluation skills

Demonstrates ability to search and select information sources, but evaluation of sources is not quite adequate

Meets Expectations Demonstrates adequate skills in searching, selecting and evaluating information sources to meet information needs

Proficient Evidence of search, selection and source evaluation skills to meet information needs Displays ability to identify uniquely salient sources

Selects and applies data to situation or rejects it without any justification

Analysis is inappropriate, inadequate and superficial, at best

Does not explain which specific tools or methods are relevant to the issue(s) at hand

Applies some rudimentary reasoning in consideration and application of data to situations, which is often incorrect/ inadequate

Spells out the reasoning applied in consideration and application of data to situations with few lapses in adequacy of data

Spells out the reasoning applied in consideration and application of data to situations

Uses some analytical tools/ methods

Analytical tools and methods employed are appropriate and about accurate, but not the most suitable ones, against the need

Analytical tools and methods employed are appropriate and adequate for the need

Analysis somewhat clarifies the issues and facilitates decision-making

Analysis clarifies the issues and facilitates decision-making

Partly explains why specific tools or methods are relevant to the issue(s) at hand

Explains why specific tools or methods are relevant to the issue(s) at hand

Analysis is minimal against the need and does not fully clarify the issues and facilitate decision-making Does not clearly and explain which specific tools or methods are relevant to the issue(s) at hand

Volume 2, Number 1 Winter 2011

77

R. E. Kleine, III * and J.D. Yoder KEEN Learning Outcome: Apply critical and creative thinking to ambiguous problems (Continued)

Makes convincing arguments for recommended option, aligned to Displays ability to explore alternate the given context options to the identified problem

Section 3 of 3: Solution Development

78

Does Not Meet Expectations

Developing

Meets Expectations

Proficient

Does not come up with alternate feasible and original options

Makes an effort to minimally explore alternate solutions but plays safe, does not risk uncertainty associated with original solutions

Explores alternate solutions to the problem but these are not very original, tends to play safe to an extent

Explores and comes up with original, relevant alternate solutions to the problem

Fails to identify conclusions, implications, and consequences of the issue

Does not clearly identify conclusions, implications, and consequences of the issue

Analyzes conclusions, implications, and some consequences of the issue

Analyzes conclusions, implications, and the majority of consequences of the issue

Doesn’t propose solution to problem

Proposes solution to problem(s) without much investigation and analysis

Proposes solution to problem(s) with some investigation and analysis

Proposes solution to problem(s) based on detailed analysis

Justifies the recommended option and recognizes the implications

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach KEEN Learning Outcome: Construct a customer-appropriate value proposition This rubric is based on the NABC model for expressing value propositions as presented in Carlson and Wilmot’s book Innovation: The Five Disciplines for Creating What Customers Want.

What is the unique approach for addressing the need?

Important customer and market need (opportunity)

Does Not Meet Expectations Customer and market needs not identified.

Developing

Meets Expectations

Proficient

Prospective customer and market needs and characteristics are identified and described. Identified customer and market needs and characteristics are quantified. All key constituencies are identified (e.g., target customer, investors); the above are replicated for each key constituency.

No approach for addressing the need is provided.

An approach for addressing the need is described.

The approach is appropriate relative to the identified customer and market needs.

The approach evidences incremental uniqueness relative to existing alternatives.

Volume 2, Number 1 Winter 2011

The approach has a distinguishable advance in design, technology, process, or business model.

The approach has an important advance in some combination of design, technology, process, or business model.

79

R. E. Kleine, III * and J.D. Yoder

KEEN Learning Outcome: Construct a customer-appropriate value proposition (Continued)

How are these benefits/costs superior to available alternative solutions for solving the problem?

What are the specific Benefits/Costs that result from this approach

Does Not Meet Expectations

80

No customer benefits per costs are provided.

Developing

Meets Expectations

Proficient

Key customer benefits of the approach are identified using specific language. Customer costs of the approach are identified using specific language. The identified key customer benefits are quantified in an appropriate way. The identified customer costs of the approach are quantified in an appropriate way. Benefits relative to costs of the approach are clearly expressed.

No alternative solutions are mentioned.

Key direct alternative solutions are identified and the benefits and costs of each are quantified and explained in specific language. Key indirect alternative solutions are identified and the benefits and costs of each are quantified and explained in specific language. The superiority of the proposed alternative’s benefits and costs relative to key direct and indirect alternatives is quantified and explained in specific language.

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach KEEN Learning Outcome: Construct a customer-appropriate value proposition (Continued)

Multiple value How are the benefits/costs of the proposed solution propositions for multiple superior to the Competition? constituencies

Does Not Meet Expectations No competition is mentioned.

No constituencies identified.

Meets Proficient Expectations Key direct competitors are identified and the benefits and costs of each are quantified and explained in specific language. Key indirect competitors are identified and the benefits and costs of each are quantified and explained in specific language. The superiority of the proposed alternative’s benefits and costs relative to key direct and indirect competitors is quantified and explained in specific language. Developing

All key constituencies are identified.

Volume 2, Number 1 Winter 2011

An effective value proposition is developed for each of some of the identified key constituencies.

An effective value proposition is developed for each key constituency.

81

R. E. Kleine, III * and J.D. Yoder ONU General Education Learning Outcome: Effective communication of a customer appropriate value proposition Students demonstrate the written, oral and visual communication skills necessary to communicate professionally and effectively as responsible members of their organizations and their communities. Section 1 of 3: Written Communication

Written Language Mechanics

Format and Style

Paragraph Development

Overall Organization

Does Not Meet Expectations

82

Developing

Meets Expectations

Demonstrates improper use of appropriate document formatting guidelines.

Provides a thesis statement that is somewhat developed Provides a structure and organization that is somewhat cohesive and coherent Writes paragraphs that are developed inconsistently, with topic sentences that are present, but not fully supported Demonstrates inconsistent use of appropriate document formatting guidelines.

Integrates quotations and exhibits inappropriately

Integrates quotations and exhibits inconsistently

Integrates quotations and exhibits appropriately

Writes sentences that are unclear or indirect

Writes sentences that are occasionally unclear, indirect or grammatically incorrect

Writes sentences that are generally clear, concise, and direct.

Thesis statement is unclear or absent Provides a structure and organization that is not cohesive or coherent Writes paragraphs that are underdeveloped, with topic sentences that are missing or unsupported

Proficient

Provides a thesis statement that is generally clear

Provides a strong, clear thesis statement

Provides a structure and organization that is generally cohesive and coherent

Provides a structure and organization that is strongly cohesive and coherent

Writes paragraphs that are generally well-developed, with topic sentences that are present and supported

Writes paragraphs that are welldeveloped, with strong, focused topic sentences that are fully supported

Demonstrates appropriate use of appropriate document formatting guidelines.

Demonstrates excellent use of appropriate document formatting guidelines. Integrates quotations and exhibits effectively into the analysis Writes sentences that are consistently clear, concise, and direct and grammatically correct

Minor problems with grammar

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach ONU General Education Learning Outcome: Effective communication of a customer appropriate value proposition (Continued)

Argument and Rhetoric

Overall Organization of presentation

Section 2 of 3: Oral Communication Does Not Meet Developing Expectations The central idea is not clearly stated and remains unclear or unidentified.

The central idea is implied but not explicitly stated or communicated.

The presentation moves from one point to the other in a disconnected manner, without regard to the objective.

The organizational pattern of the presentation does not adequately meet the audience’s needs or develop the thesis appropriately.

Key ideas are easily overlooked and important aspects of the thesis are not addressed.

There is lack of clarity regarding the purpose and only a few of the key ideas have been addressed.

Presentation lacks an identifiable central message.

Presentation contains an identifiable central message, but it argues from a false premise or attempts to prove the null hypothesis.

Presentation does not rely on credible evidence, or form a valid and true outcome.

Argument is not fully supported by credible evidence; the outcome is either invalid or valid but untrue. Unable to answer basic objections to their conclusions

Volume 2, Number 1 Winter 2011

Meets Expectations The central idea of the presentation is completely and clearly presented. The organization of the presentation is generally logical and meets the audience’s needs. Presentation begins with a clear sense of purpose, but the presence of irrelevant content in some parts leads to digression from the central idea. Presentation contains a significant central message, with valid and true outcome adequately supported by evidence and logic. Able to answer basic objections to their conclusions.

Proficient

Proposes a clear central idea.

The entire presentation is a cohesive piece in which the ideas developed support the thesis.

The presentation leads naturally to a strong conclusion that summarizes the key points and leaves the audience with a clear message.

Presentation contains a compelling central message, with a valid and true outcome fully supported by evidence and logic.

Able to answer basic and (at least some) more advanced/sophisticated objections to their conclusions.

83

R. E. Kleine, III * and J.D. Yoder ONU General Education Learning Outcome: Effective communication of a customer appropriate value proposition (Continued)

Argument and Rhetoric

Overall Organization of presentation

Section 2 of 3: Oral Communication Does Not Meet Developing Expectations

84

The central idea is not clearly stated and remains unclear or unidentified.

The central idea is implied but not explicitly stated or communicated.

The presentation moves from one point to the other in a disconnected manner, without regard to the objective.

The organizational pattern of the presentation does not adequately meet the audience’s needs or develop the thesis appropriately.

Key ideas are easily overlooked and important aspects of the thesis are not addressed.

There is lack of clarity regarding the purpose and only a few of the key ideas have been addressed.

Presentation lacks an identifiable central message.

Presentation contains an identifiable central message, but it argues from a false premise or attempts to prove the null hypothesis.

Presentation does not rely on credible evidence, or form a valid and true outcome.

Argument is not fully supported by credible evidence; the outcome is either invalid or valid but untrue. Unable to answer basic objections to their conclusions

Meets Expectations The central idea of the presentation is completely and clearly presented. The organization of the presentation is generally logical and meets the audience’s needs. Presentation begins with a clear sense of purpose, but the presence of irrelevant content in some parts leads to digression from the central idea. Presentation contains a significant central message, with valid and true outcome adequately supported by evidence and logic. Able to answer basic objections to their conclusions.

Proficient

Proposes a clear central idea.

The entire presentation is a cohesive piece in which the ideas developed support the thesis.

The presentation leads naturally to a strong conclusion that summarizes the key points and leaves the audience with a clear message.

Presentation contains a compelling central message, with a valid and true outcome fully supported by evidence and logic.

Able to answer basic and (at least some) more advanced/sophisticated objections to their conclusions.

The Journal of Engineering Entrepreneurship

Operationalizing and Assessing the Entrepreneurial Mindset: A Rubric Based Approach KEEN Learning Outcome: Demonstrate voluntary social responsibility

Make Informed, Ethical Responses to Those Needs

Identify Underlying Ethical Issues in Specific Situations

Understanding of personal, civic, and social needs

Students display an understanding of personal, civic and social needs; identify underlying ethical issues in specific situations; and make informed, ethical responses to those needs. Does Not Meet Meets Developing Proficient Expectations Expectations Bases Bases Bases Does not base understanding of understanding of understanding of understanding of the needs of the needs of the needs of the needs of persons, civil persons, civil persons, civil persons, civil societies, or the societies, or the societies, or the societies, or the global global global global community on a community on a community on a community on a substantive and limited body of substantive body body of pertinent diverse body of pertinent of pertinent information. pertinent information. information. information. Discerns the Identifies ethical underlying issues; Does not identify Identifies ethical needs, values, recognizes that ethical issues in issues in and perspectives these ethical personal, civic, personal, civic or that create issues shape or global life. global life. ethical issues in personal, civic, personal, civic, or global life. or global life. Responds to ethical issues by Responds to considering Responds to ethical issues by pertinent Responds to ethical issues by considering information; ethical issues considering pertinent defends chosen without regard to pertinent information; position within pertinent information; does not defend an applicable information; defends chosen chosen position analytical does not apply position within within an framework; an analytical an applicable applicable evaluates framework. analytical analytical assumptions and framework. framework. implications of divergent ethical perspectives.

Volume 2, Number 1 Winter 2011

85

R. E. Kleine, III * and J.D. Yoder KEEN Learning Outcome: Relate personal liberties and free enterprise to entrepreneurship.

Relate free enterprise to entrepreneurship

Relate personal liberties to entrepreneurship

Entrepreneurship

The student is familiar with defining attributes of entrepreneurial behavior, can explain the relationship between personal liberties and entrepreneurship, and can connect the concept of free enterprise to entrepreneurial behavior. Does Not Meet Meets Developing Proficient Expectations Expectations No defining attributes of Key defining attributes of entrepreneurial behavior are entrepreneurial identified. behavior are identified. Can identify attributes of successful entrepreneurial behavior. Provides relevant examples to successful entrepreneurial behavior. Some evidence No evidence of of relating relating personal Can describe the relationships personal liberties liberties and between personal liberties and and entrepreneurial entrepreneurial behavior. entrepreneurial behavior. behavior. Provides relevant examples to illustrate the relationships among personal liberties and entrepreneurial behavior. No evidence of Some evidence relating free of relating free Can describe the relationships enterprise and enterprise and between free enterprise and entrepreneurial entrepreneurial entrepreneurial behavior. behavior. behavior. Provides relevant examples to illustrate the relationship between free enterprise and entrepreneurial behavior.

86

The Journal of Engineering Entrepreneurship