1.4.4 Measuring Quality in Design

by Dan Cordon (Mechanical Engineering, University of Idaho),
Steven W. Beyerlein (Mechanical Engineering, University of Idaho). and
Denny Davis (Bioengineering, Washington State University)

This module is a resource for creating tools to measure performance on design projects. Because working knowledge and problem-solving are usually the areas targeted for measurement, it is convenient to base these tools on rubrics. This module presents holistic and analytical rubrics for measuring individual and team skills that add value to design products. The holistic rubric is useful in visualizing overall design performance on a continuum from novice to expert. The analytical rubric can be used as a library of items for customized rubrics that examine a diversity of design issues.

Need for Transferable Tools

To accurately assess or evaluate the quality of a design activity, one must consider two complementary types of outcomes: learner development and solution development (Davis, 2006). For many reasons, it is difficult to provide just-in-time feedback because many different projects are underway at the same time; projects are often of a long-duration; and within individual projects, the development of the learner, as well as the solution, progress from a state of fragmented understanding and ideas to a more mature state of integrated understanding and design solutions (2.4.12 Creating a Capstone Course). Furthermore, such activities typically involve multiple participants and stakeholders whose levels of prior experience vary dramatically, as do their levels of engagement in different phases of the design process. For these reasons, there is a widespread need and interest in having transferable tools with which to measure design performance in multiple contexts and with different performers (McKenzie, 2004).

Definition of Quality

Design is a systematic, interactive, and iterative development of a process, system, or product to meet a set of specifications. It often involves working with clients to meet their perceived needs within a set of explicit and implicit constraints. It is frequently team-based and initially involves a detailed needs analysis, definition of the problem, deployment of prior knowledge and past experience, generation and analysis of possible solutions, and the exploration of alternative solution paths. As the process moves forward, designers identify optimal solutions, implement selected designs, document progress, and communicate elements of the design and its process of development to multiple audiences. Often alternate solutions are derived from the need to optimize various resources or to work optimally under certain constraints, to recycle components of previous designs, and to integrate solutions to the main problem with solutions to sub-problems. As they evolve, design solutions are regularly validated against performance criteria and timetables that are either agreed upon at the beginning of the process or re-negotiated at specific milestones. Throughout the entire process, participants are committed to continuous improvement in design methodology, design tools, and personal/team capabilities that are valuable to future clients.

Holistic Rubric

Faculty attending a workshop on performance measures at Western Michigan University identified the top ten attributes of quality design performance. These attributes were then grouped into five related pairs as shown in Table 1. Pair 1 focuses on the setup and closure of the design process. Pair 2 is concerned with interactions, both internal and external, to the design team. Pair 3 focuses on component and thorough idea generation. Pair 4 recognizes the use of iteration and assessment to make wise choices. Pair 5 looks at the organized, systematic nature of design. Five skill levels associated with being a successful designer were identified and labeled. Titles were selected to avoid negative connotations and stereotyping. These titles and the five attribute pairs from Table 1 were used to create the holistic rubric shown in Table 2.

Analytical Rubric

Each of the ten attributes in Table 1 is broken down into five sub-components that provide detail and definition of the attribute. A single word or short phrase is used instead of a lengthy description for each of the sub-components. Expert profiles are very helpful in generating ideas for sub-components and validating completeness of the final rubric (2.4.3 Development and Use of an Expert Profile).

The analytical rubric can be used for assessment purposes (4.1.9 SII Method for Assessment Reporting) as well as evaluation purposes (2.4.10 Course Grading Systems). In different circumstances, design instructors will want to use a subset of items from the analytical rubric. Different performance areas should be emphasized depending on the focus of measurement; whether it be learner development or solution development, whether the target audience is an individual or a team, or whether the project is in its initial or final stages. It is a good rule of thumb to select and use only a dozen sub-items at one time. Table 4 identifies performance areas for investigation in different design project settings.

Concluding Thoughts

The holistic and analytical rubrics provided here can be adapted for use in assessing and evaluating learner development and solution development in any project. They are works-in-progress that can save you time in creating new measurement instruments or in updating existing instruments such as those maintained by the Engineering Education Research Center at Washington State University (EERC, 2007). The holistic rubric is a vehicle through which to motivate high-level performance and to promote reflective practice. The sub-items in the analytical rubric are prompts to remind students and project managers about elements of quality in design. Feel free to combine and reconfigure these to match specific needs in the design process.

References

Davis, D., Beyerlein, S., Trevisan, M., Thompson, P., & Harrison, K. (2006). A conceptual model for capstone engineering design performance and assessment. Proceedings of the 2006 American Society for Engineering Education Annual Conference.

Engineering Education Research Center (EERC). Transferable assessments for capstone engineering design. Retrieved May 25, 2007, from <http://eerc.wsu.edu/ASA/index.shtml>

McKenzie, L, Trevisan, M., Davis, D., & Beyerlein, S. (2004). Capstone design courses and assessment: A national study. Proceedings of the American Society for Engineering Education Annual Conference.

Table 1 Attributes of Quality Design

Pair 1

Problem Definition: Interviewing client, needs analysis, identifying issues and constraints, developing specifications, define functions/uses, perception check

Solution Validation: Interpreting and mediating requirements, creating an appropriate test plan, measuring for success, satisfying societal requirements, upholding professional responsibilities, customer acceptance

Pair 2

Communication: Informing client/teammates, personal documentation, oral presentations, written reports, visual and graphic elements, professionalism

Teamwork: Belonging, commitment, ownership, cooperation, performing within roles, managing conflicts, efficiently using/creating resources

Pair 3

Prior Knowledge: Locating and reviewing available resources, applying existing knowledge/principles/solutions, evaluating external information, depth and breadth of internalized knowledge, identifying learning needs

Divergent Thinking: Brainstorming, generating alternative solutions, quantity, uniqueness, novelty, problem reformulation

Pair 4

Decision Making: Inclusion of stakeholders, evaluating alternatives against criteria, justifying selection, cost-effectiveness, level of consensus attained

Iterating & Assessing: Ongoing, captures and applies lessons learned from prior iterations, assess solutions, assess process followed, oriented toward continuous improvement.

Pair 5

Creation/Follow-through of Plan: Tasks and milestones clearly defined, logical and organized plan/timetable, taking action, documentation of progress

Professional Analysis: Appropriate tool selection, tool proficiency, parameter identification, system modeling, sensitivity analysis, estimating, experimenting, optimization

Table 2 Holistic Rubric for for Measuring Quality in Design

Apprentices in the Discipline

1. Present clear thesis statements that interest readers somewhat

2. Introduce a few key ideas that connect and support these ideas with key references

3. Discuss the key concepts, but do not necessarily connect them clearly to a conclusion

4. Present a few competing or complementary ideas and are able to recognize the dilemmas they present

5. Are familiar with the writing conventions of the discipline but use jargon to attempt to gain credibility and to mask flaws in understanding

Project Engineer

Achieves a creative solution encompassing customer and societal needs, while meeting all applicable professional requirements

Supports the efficient and timely achievement of project milestones, deployment of team resources, and extraordinary group dynamics, including the clear, thorough, and useful communication of project needs, activities, findings, and recommendations

Masterfully uses divergent thinking, in-depth knowledge, and existing processes to develop multiple creative solutions

Always exercises professional judgment, expert knowledge, and design criteria to evaluate and make recommendations on potential solutions, continuously improving the design at all process stages

Consistently applies excellent analytical skills that produce professional plans that are easy to follow

Design Engineer

Identifies, analyzes, and validates customer and societal needs and professional requirements

Directs others and coordinates communication that supports the achievement of project milestones, deployment of team resources, and healthy group dynamics

Routinely uses divergent thinking, in-depth knowledge, and existing processes to develop multiple high quality solutions

Incorporates relevant professional standards and design criteria in evaluating alternative designs, leading to identifiable design improvements

Can apply extensive analytical skills that lead to solid design plans that can be followed by others

Apprentice Designer

Identifies and analyzes most customer and societal concerns, along with key professional requirements

Contributes to project activities on request, assuming assigned roles, assisting in meeting most project milestones, and supporting team activities under some supervision

Generates several feasible solutions in a mechanical way based on prior experience and knowledge

Performs design tasks within well-defined project areas, and suggests incremental design changes

Possesses sufficient analytical skills to produce marginally successful plans that are difficult to follow

Engineering Intern

Collects information requested on customer needs, societal concerns, and professional requirements

Assists in meeting some project and reporting milestones under extensive supervision

Uses narrow knowledge base to build a limited set of alternatives

Follows established procedures for gathering data and decision-making, occasionally leading to incremental design changes

Can apply limited analytical skills as directed in plans made by others

Pre-engineer

Often ignores customer needs, societal concerns, and professional requirements

Frequently works outside the team environment, leading to irregular and undependable progress toward project milestones. Communication activities are minimal or do not contribute to documenting or furthering project outcomes

Locks onto first idea when exploring options, existing solutions, or considering alternatives

Uses capricious and arbitrary standards to select among possible designs, and is sporadic in producing suggestions for design modification

Often displays inferior analytical skills and seldom plans or follows plans

Table 3 Analytic Rubric for Measuring Quality in Design

Performance Area	Project Engineer	Design Engineer	Apprentice Designer	Engineering Intern	Pre-Engineer
Problem Definition
Identify needs	contrasted	ranked	complete	sees one	perfunctory
Establish requirements	comprehensive	nearly all	many	some	uncertain
Identify constraints	systematically	in groups	in pairs	one by one	disregards
Define functions	exhaustively	comprehensively	with detail	basically	vaguely
Perception check	facilitates	fulfills role	values role	occasionally	myopically
Prior Knowledge
Locate/Review resources	rich set	focused set	standard	basic	haphazard
Use of principles	customized	complete	well-known	obvious	inappropriate
Evaluate external info	correctly	critically	partially	by coaching	incorrectly
Internalized knowledge	authority	extensive	adequate	partial	superficial
Professional growth	continuous	proactive	receptive	reactive	unconcerned
Divergent Thinking
Quantity of ideas	extensive	large	several	some	few
Distinctively unique	many	several	some	few	rare
Out of the norm	often	some	occasional	seldom	few
Causes redefinition	often	some	occasional	seldom	accidentally
Outside the box	consistently	often	occasional	seldom	irrelevant
Professional Analysis
Use key parameters	distills essence	in context	frequently	occasionally	rarely
Toolbox/Usage	tool builder	power user	standard	limited	ineffectual
Estimating	valuable	useful	approximate	variable	inaccurate
Experimentation	creative	appropriate	adaptive	imitative	trial/error
System modeling	integrative	thoughtful	complete	partial	simplistic
Decision-making
Include stakeholders	empathetically	all	most	primary ones	unaware
Achieve consensus	consistently	often	occasionally	seldom	rarely
Cost-effective	efficient	controlled	reasonable	uncontrolled	oblivious
Use design criteria	consistently	frequently	periodically	minimally	sporadically
Justification	always	frequently	dependably	occasionally	randomly
Create and Follow Plan
Define tasks	with vision	thoughtfully	partially	minimally	whimsically
Outline milestones	originates	modifies	executes	short range	unaware
Organized & logical	directs	solid	mechanical	scattered	confusing
Track & revise plan	assesses	implement	support	passively	disjointedly
Document progress	comprehensive	complete	methodical	perfunctory	incoherent
Iterate and Assess
Frequency	in parallel	continuous	consistent	methodical	irregular
Review previous cycles	integrates	extends	combines	most recent	seldom
Assess design process	continuously	all steps	trouble steps	big steps	when done
Assess design solutions	to optimize	to revise	to finish	when done	after failure
Iterate effectively	masterfully	purposefully	frequently	occasionally	awkwardly
Validate Solutions
Interpret requirements	exceeds	meets	knows all	knows some	oblivious
Mediate requirements	effectively	successfully	somewhat	minimally	unaware
Build test plan	elegant	solid	generally	sketchy	attempts
Test against criteria	conclusively	reliably	inconclusively	incompletely	erratically
Stakeholder acceptance	assures	leads	supports	understands	unaware
Communication
Written reports	comprehensive	informative	mechanical	superficial	unintelligible
Oral communication	persuasive	purposeful	understandable	in context	inconsistent
Project documentation	thorough	substantive	useful	disorganized	incoherent
Visuals & graphics	interpretive	illustrative	supportive	elementary	confusing
Professionalism	polished	consistent	attempts	uncomfortable	unaware
Teamwork
Use of resources	mentors	recruits	requests	follows	misuses
Manages conflicts	grows	resolves	assists	observes	generates
Shared consensus	shapes	molds	sees	accepts	disregards
Belonging/Commitment	committed	enrolled	believes	compliant	erratic
Performs roles	facilitates	fulfills	values	follows	misconstrues

Table 4 Performance Areas Linked to Different Design Assessments

Assignment Type

Performance Areas

Personal Growth

Prior Knowledge

Professional Analysis

Create & Follow Plan

Iterate & Assess

Team Processes

Communication

Teamwork

Create & Follow Plan

Iterate & Assess

Solution
Requirements

Communication

Problem Definition

Prior Knowledge

Decision-Making

Solution Assets

Communication

Divergent Thinking

Professional Analysis

Validate Solutions

1.4.4 Measuring Quality in Design

by Dan Cordon (Mechanical Engineering, University of Idaho),
Steven W. Beyerlein (Mechanical Engineering, University of Idaho). and
Denny Davis (Bioengineering, Washington State University)

Table 1 Attributes of Quality Design

Table 2 Holistic Rubric for for Measuring Quality in Design

Apprentices in the Discipline

Project Engineer

Apprentice Designer

Engineering Intern

Pre-engineer

Table 3 Analytic Rubric for Measuring Quality in Design

Table 4 Performance Areas Linked to Different Design Assessments

Pair 1	Problem Definition: Interviewing client, needs analysis, identifying issues and constraints, developing specifications, define functions/uses, perception check Solution Validation: Interpreting and mediating requirements, creating an appropriate test plan, measuring for success, satisfying societal requirements, upholding professional responsibilities, customer acceptance
Pair 2	Communication: Informing client/teammates, personal documentation, oral presentations, written reports, visual and graphic elements, professionalism Teamwork: Belonging, commitment, ownership, cooperation, performing within roles, managing conflicts, efficiently using/creating resources
Pair 3	Prior Knowledge: Locating and reviewing available resources, applying existing knowledge/principles/solutions, evaluating external information, depth and breadth of internalized knowledge, identifying learning needs Divergent Thinking: Brainstorming, generating alternative solutions, quantity, uniqueness, novelty, problem reformulation
Pair 4	Decision Making: Inclusion of stakeholders, evaluating alternatives against criteria, justifying selection, cost-effectiveness, level of consensus attained Iterating & Assessing: Ongoing, captures and applies lessons learned from prior iterations, assess solutions, assess process followed, oriented toward continuous improvement.
Pair 5	Creation/Follow-through of Plan: Tasks and milestones clearly defined, logical and organized plan/timetable, taking action, documentation of progress Professional Analysis: Appropriate tool selection, tool proficiency, parameter identification, system modeling, sensitivity analysis, estimating, experimenting, optimization

*Assignment Type*	*Performance Areas*
Personal Growth	Prior Knowledge Professional Analysis Create & Follow Plan Iterate & Assess
Team Processes	Communication Teamwork Create & Follow Plan Iterate & Assess
Solution Requirements	Communication Problem Definition Prior Knowledge Decision-Making
Solution Assets	Communication Divergent Thinking Professional Analysis Validate Solutions