In the process of creating a quality program assessment system, all stakeholders need to collaborate in the creation of clear, concise performance criteria that can be used to guide assessment of the program. This module offers a series of steps that stakeholder teams can follow to generate high quality performance criteria for a program. These steps involve brainstorming current and future program qualities, identifying qualities that will have the largest bearing on the future design of the program, and selecting critical areas for measurement. Key findings are ultimately distilled into a finite set of readable statements that express the essential nature of the program being assessed, along with key indicators of how its success will be measured. These statements about performance of a program are the performance criteria for the program.
Table 1 Methodology for Writing
|
|
|
Role of Stakeholders
Accrediting bodies expect programs to involve their constituents (students, faculty, administration, alumni, and industry supporters) in the establishment and maintenance of the program objectives (Accreditation Board for Engineering and Technology, 2002). Beyond the accreditation visit, these statements can be used to share program intentions with other faculty, campus administrators, student applicants, and potential donors. These statements have greatest meaning when they are used to align administrative and instructional decisions with program intentions.
Before a program can be implemented, stakeholders need to come to consensus about what the program is attempting to achieve and how that achievement can be defined, accomplished, and measured in specific “performance” (things that are done). These well-articulated descriptions become the measuring sticks for program effectiveness. They provide essential reference points to which all participants can return, time and again throughout the review process, to check on the clarity of their thinking and to ensure consistency in analyzing a program.
Criteria for a Program
The writing of performance criteria for a program parallels the process used to write performance criteria for a course or activity. What is important in any of these cases is determining which qualities or characteristics are absolutely essential to the program, course, or activity in question. The work of identifying these features enables team members to then define the performance criteria that will determine how those program qualities will be achieved. In other words, if a key quality of the program is “commercially talented artists,” the performance criteria statement should spell out specifically how that program quality can become evident. The following methodology will help assessors identify, verify, clarify, prioritize, and analyze these qualities. These preliminary steps will then be used to develop statements of performance criteria that incorporate the most important ideals that have emerged from this collaborative thinking-sharing-writing process.
Determining Qualities to Select for a Program
Many designers of continuous quality improvement emphasize the need for team effort to fully understand and appreciate a program or system (Scholtes, 1993; Productivity-Quality Systems, 1992) and stress the importance of buy-in from all key players (Badiru & Ayeni, 1993). Deming advocates the need for the inclusion of all classes of stakeholders in all these steps (Deming, 1982) and emphasizes the need for the entire team to brainstorm all the “knowables.” In so doing, they can create a comprehensive or profound system of knowledge about the program, though there will always be “unknowables” which create system variance (e.g., the next year’s enrollments, budget, political developments). Still, writing key performance objectives effectively demands that participants begin by reflecting on what is most essential to their program, bringing to the endeavor as much information and insight as they can.
Methodology for Writing Performance Criteria
Step 1—Review previous design work.
In performing a program assessment, you will be creating your own design document that captures your work as you progress. After writing a one-sentence statement that captures the “essence” of your program, you will identify its goals and processes, as well as its scope and shareholders. With these documents in hand and with the collaborative experience of producing them behind the team, the participants will be ready to proceed to the steps outlined in this module.
Step 2—Brainstorm a list of current qualities.
Next, the team’s task is to brainstorm a list of characteristics that account for significant aspects of program quality. These qualities appear across products, processes, and other components of the program. Overall quality results from a set of specific program “qualities,” i.e., those things that characterize the program in a positive sense. The team should consider those aspects that make the program unique and give it an identity, are critical to the program’s success, match characteristics found in other quality programs, and are attributes that stakeholders find special. These descriptions can best be captured in the form of adjectives or adverbs connected with nouns (e.g., dedicated faculty, research-based, empowering).
Additionally, assessors should explore stakeholders’ perspectives, program resources, graduates, and program-related events to determine which features are most valuable to the program. It might be helpful to ask, for instance, “When recruiting students or faculty, how would you describe your program to them? How would you describe your program to someone at a conference?” Examine written materials about your program (e.g., marketing materials) to see what they say or imply about your program. The flow of information and insights from this array of resources will provide an excellent pool from which to select key ideas for writing performance criteria statements.
Examples:
innovative community-based scholarly challenging
rigorous highly desired graduates applied friendly
success-oriented world-class faculty open access technical
responsive adaptable value-added efficient
Step 3—Brainstorm a list of future program qualities.
The next question to consider concerns the direction in which you would like your program to move. What key qualities would you like to see as outcomes of your ideal program in the future? What capabilities do you find in superior graduates or expert practitioners that you would like those in your program to emulate? (Mattingly & Fleming, 1992) How would you like to enhance your current program? Are there characteristics lacking in your graduates that reveal defects that are somehow embedded in the very design of your program? (Newble & Hejka, 1991) What attributes would you like to build into your program for the future?
By determining the difference between the current and future status of your program, you can identify the areas that need attention. This type of analytically derived information will be invaluable to program leadership as they begin to map out future priorities of the program and of the institution it serves. The future program qualities that your team articulates, therefore, should reflect anticipated or perceived shifts; those changes should be reflected in planning. Brainstorm these ideas with your design team, remembering to include material from the “products” and “processes” your group has completed in the earlier phase of the program assessment system.
Examples:
state-of-the-art student-centered
assessment-oriented empowering
evidence-based problem-based fully inclusive 24/7 access
life-changing transformational
diverse environment resource rich
heavily endowed well-funded trend-setting cutting edge
Step 4—Determine key qualities that are missing.
Combine the lists from Steps 2 and 3 to aggregate the collective qualities. Check the new listing to determine if there are any important characteristics/qualities missing or if any gaps exist. Investigate programs similar to yours (e.g., those of competitors, peers, and exemplars) and consider why they are viewed as being strong (or of high quality). Determine which of their characteristics you desire, and decide whether they are applicable to your program. It is extremely important to facilitate the participation of all stakeholders, such as board members, students, and representatives of the community, in this process. It is also important that the resulting set of qualities identified represents all critical areas of the program and captures the essence of it. Contact collaborative partners outside your program (e.g., funding agencies, peers who produce significant contributions to their professional organizations) and get their opinions and feedback. Ask them to feed back to you their perceptions about what is special about your program; ask them to express in their own words their perception of who you are and how you contribute to their efforts or serve their needs.
Examples:
highly selective highly employable
heavily endowed resource rich
learner-centered job-ready graduates
high technology high retention
community visible
Step 5—Rank the top ten integrated current and future qualities.
The next step involves ranking the characteristics you have just identified in the previous steps. Begin by selecting the qualities that are most important. This is an excellent stage in which to enlist the assistance of community and alumni advisory groups for validation. This is also a good point at which to cross-reference selected qualities with additional requirements, such as accrediting bodies, state regulator boards, certification examination criteria, community needs, and college initiatives and priorities (James A. Rhodes State College, 2002). Make a first pass at ranking the list by labeling criteria from low to high (on a scale from 1 to 5). Then, sort the scores. Next, starting at the bottom of the list, see if you can justify moving a characteristic higher up in the list. Move to the next highest ranked item and determine if it can be moved up. Two to three passes through the list will help ensure that no truly critical item is overlooked.
Step 6—Analyze these qualities to pull out redundancy and overlap.
In this step, you need to double-check for possible redundancy in your final list of qualities. Are all your program strengths represented? Additionally, check this listing against characteristics of other programs in your institution. Have you included anything that is actually covered by other programs or college departments (e.g., advising, marketing)? Are you still operating within your stated scope or boundaries? Do a perception check and ask whether, collectively, the qualities (the program characteristics you have identified) cover every aspect of your program. In other words, do they fully describe the unique traits that make it what it is, that give it a special identity?
Examples:
heavily endowed well-funded
resource rich innovative
cutting edge state-of-the-art
Step 7—Prioritize qualities; select the top six to eight.
You now need to examine the special characteristics of your program in terms of your overall institutional strategic priorities and initiatives. All aspects of your program (both academic and non-academic) should ultimately feed into student learner outcomes and be kept in alignment with the institutional mission (Higher Learning Commission, 2003). Are you still targeting the most significant areas? A program should select six to eight criteria. The number of criteria chosen depends on the length of time the program has existed and on its magnitude or complexity. In general, the longer or more complex the program, the greater the number of criteria it will need. One of the tools commonly used in continuous quality improvement systems is the “pareto” diagram, which arranges data into categories for easy visualization. Charting selected qualities with this tool can help create a holistic view of your chosen qualities (Productivity-Quality Systems, 1992). McNamara (2002) reminds program designers of the “20-80” rule, which claims that 20 percent of effort generates 80 percent of the results.
Deming says 85 percent of quality problems are due to system design; clearly identifying quality areas will enable all stakeholders to get more systematic control of the program (Deming, 1982).
Step 8—Analyze the qualities to find three to five aspects of each quality.
What makes each of your qualities unique? Why are they important in defining your program? Analyzing each of the qualities, describe in different phrases what each one means. Ask what is meaningful or significant in a given area of performance or if other possible meanings need to be considered. What important things must exist for this program characteristic to be true? For instance, if you claim a quality of “computer literacy,” is the institutional infrastructure in place to support it?
Examples:
Quality:
student-centered
Aspects:
students define their own learning objectives,
faculty identify student learning needs,
students are engaged in active learning, and
faculty and students assess student performance.
Quality:
success-oriented
Aspects:
needs are being met
outcomes produced
external affirmation
rewarding
minimal failures
Step 9—Write the performance criteria as statements.
The performance criteria are thoughtfully expressed performance expectations that are mutually understood by all stakeholders. They demonstrate the importance of key performance areas to the overall effectiveness of your program. They delineate the specific aspects of a performance and describe how they are tied to a larger integrated performance. They also provide direction about what programs need to do specifically to satisfy the goals that have been previously set out in much more global terms. Performance criteria and qualities have a critical two-way relationship. The performance criteria you write must deliver the specific qualities that have been selected. For example, if you have specified that your program needs to recruit more students, the performance criteria need to spell out how that will be achieved.
Try to visualize the integrated performance that you are seeking. Now put together a sequence of steps or actions to get the job done, checking to see that the plan is coherent and fluent. Describe and then imagine putting it into a real context. For instance, is it reasonable to expect that you can increase student enrollment by 10 percent in the next year or by 15 percent in the next two years? Will your plan achieve the qualities you had identified earlier as being descriptive of your program’s unique character?
Concluding Thoughts
The writing of performance criteria is facilitated by strong writing prompts that identify the qualities that matter for program effectiveness. Once these qualities are visualized and captured, the task of writing the performance criteria statements that flow from them becomes easier. Key processes and products can then be highlighted and made apparent to all stakeholders. A road map for the design specifications for your program will emerge from this process. A systematic approach for measuring program progress will be presented in the next module.
References
Accreditation Board for Engineering and Technology. (2000). Engineering Criteria 2000. <http://www.abet.org>
Badiru, A., & Ayeni, B. (1993). Practitioner’s guide to quality and process improvement. London: Chapman & Hall.
Deming, W. E. (1982). Quality, productivity, and competitive position. Cambridge: Massachusetts Institute of Technology Press.
Higher Learning Commission. (2003, February). The criteria for accreditation and the operational indicators. Retrieved June 1, 2004 from <http://www.ncahigherlearningcommission.org/restructuring/newcriteria> [No longer active.]
Mattingly, C., & Fleming, M. F. (1992). Clinical reasoning: Forms of inquiry in therapeutic practice. Philadelphia: F. A. Davis.
McNamara, C. (2002). Basic guide to program evaluation. The Management Assistance Program for Nonprofits. Retrieved June 1, 2004 from <http://www.mapnp.org/library/evaluatn/fnl_eval.htm>
Newble, D. I., & Hejka, E. J. (1991). Approaches to learning of medical students and practicing physicians: Some empirical evidence and its implications for medical education. Educational Psychology, 11, 3-4.
Productivity-Quality Systems Inc. (1992). Improvement tools for educators. Miamisburg, OH: Author.
Scholtes, P. (1993). The team handbook. Madison, WI: Joiner.