The Evaluation Methodology is a tool to help one better understand the steps needed to do a quality evaluation. By following this process, a faculty member can learn what he or she needs to know to determine the level of quality of a performance, product, or skill. The discussion and examples of the use of this methodology are geared toward evaluation of student learning. Much of the terminology used in this methodology is taken from the module 1.4.6 Overview of Evaluation.
Evaluation Methodology
The Evaluation Methodology consists of four main steps along with a set of sub-steps.
1. Define the parameters of the evaluation.
The client
2. Design the methods used for the evaluation.
determines the need for the evaluation.
determines the use for the results of the evaluation.
determines what should be reported to the evaluatee and by whom.
determines what the evaluator needs to report to the client.
chooses guidelines to follow to implement the evaluation.
(This step is skipped if an already existing evaluation tool is used.)
The evaluator alone (or the evaluator with the client)3. Set standards and collect evidence.
chooses criteria to use for the evaluation based on the guidelines in Step 1e.
determines the evidence that will be collected for each chosen criterion.
determines the sample that will be used, if appropriate.
determines ways to collect the evidence.
sets up a plan to collect the evidence.
The evaluator informs the client of the scales that will be used to determine quality.
The client develops the decision-making process to use based on the evaluated performance quality.
The client sets the standards that will be used in decision-making.
After the client has set the standards, the evaluator
4. Report and make decisions.
collects the evidence.
documents the findings.
The evaluator reports the information in Step 1d to the client.
The client checks the quality against the standards set in Step 3c.
The client makes and implements decisions based on the findings.
The client and/or evaluator documents the results appropriately.
Either the evaluator or the client reports the findings to the evaluatee, if appropriate.
Discussion of the Evaluation Methodology
1. Define the parameters of the evaluation.
The first step in setting up an evaluation is to determine the need for the evaluation. If there is no need to collect evidence and no decisions will be made based on the evidence, there is little point in an evaluation. Having the purpose in place gives a framework for the design process.
Before designing an evaluation process, the client must decide how the results will be used and what decisions need to be made. Knowing these requirements in advance facilitates finding reliable criteria and determining the evidence needed.
Using the rationale for the evaluation, the client can determine what results, if any, need to be reported to the evaluatee. Often during indirect evaluations, in which the level of one group’s performance is used to evaluate or assess another groups’ performance, no feedback is given to the evaluatees (1.4.6 Overview of Evaluation). On the other hand, when decisions are being made that affect the evaluatee directly, a plan should be in place, before the outcome is known, for reporting the information to the evaluatee.
The client will make decisions based on the evaluation, so the client must know what evidence he or she needs to make a decision. This evidence might be an average, such as an ACT score, an individual score, or an annotated report. This information must be provided to the client by the evaluator.
Before planning the methods of evaluation, it is important to sketch out the time for completion, checks for reliability, and lists of needed and unacceptable quality criteria. Developing these guidelines ahead of time ensures that the evaluation will align with the client’s needs.
2. Design the methods used for the evaluation.
The parameters set in Step 1 should be used to help determine the criteria used in the evaluation process.
Once the criteria are set, the evaluator determines what evidence should be collected. The time required for collection, the cost, and the usefulness of the evidence must be considered in deciding what evidence to collect.
Often, a sample of performance is used to determine quality, particularly in indirect evaluation. This sample must be unbiased and large enough for the results to be useful (1.4.6 Overview of Evaluation).
When one determines how to collect data, one must decide what form of evidence will be used (such as test questions, observation of behavior, performance) and how it will be collected.
The evidence collection plan includes how, when, and where the information will be collected.
3. Set standards and collect evidence from a performance or outcome.
4. Report and make decisions.
Once it is determined what evidence will be collected, a scale must be set to describe how the quality will be judged. This scale could be a numeric scale, a rubric, or a description.
For the process to be unbiased, the client must decide before he or she receives the information from the evaluation what consequences will result from the outcome of the evaluation. This could be a simple process: for example, a high-quality performance may bring positive consequences; a low-quality performance may yield negative consequences; and for performance quality that falls between these two, other evaluative methods will be used to determine the consequences.
Once the scale and the decisions to be made are known, the client can set standards for decisions. These standards define “quality” based on what will be reported by the evaluator.
Following the plan in Step 2, the evaluator collects the needed information and analyzes it.
The evaluator documents what has been found in the form of individual scores, level of performance, averages, or narratives. Results should be documented in a way that helps the evaluator write the requested report to the client, using the scales that informed the setting of standards.
For criteria-based standards, after the client has set the standards, the evaluator can give the report to the client, and the standards will not be biased by the evaluative outcome. For norm-based standards, the evaluator must include summaries of outcomes of all evaluations in the report.
Once the client has set the standards and has received the evaluator’s report, the client can check the evaluative outcomes against the standards that have been set.
The client makes the decision based on Step 4a and evaluative outcomes, and develops plans to implement it.
Anything that needs to be documented for future use should be done at this point.
If appropriate, the contents of a report to the evaluatee should be guided by the results of Step 1e.
Examples of Evaluating
Winfield College decides which students to admit and to whom to grant non-need-based scholarships based in part on ACT scores.
1. Define the parameters of the evaluation.
The purpose of the evaluation is to find the level of knowledge skills of a college applicant.
The results of the evaluation will be used to deny admission to students with weak knowledge skills and to offer scholarships to students with knowledge skills above expectations.
Acceptance and/or scholarship decisions, based on decisions made by the client and the level of quality of knowledge-based skills by the evaluator, should be reported to the evaluatee.
The evaluator needs to report to the client the overall score and subscores for each applicant.
These are the guidelines: the level of quality of a student must be known the spring before fall matriculation; the cost must fall on the applicant, not the college; and there must be a way to compare two students.
Client: Director of Admissions Office Evaluator: ACT’s administrator Evaluatee: student applicant
2. Design the methods used for the evaluation.
In this case, the client decides to use the ACT test to test the students’ knowledge skills needed for college. Since this test already exists, this step is complete.3. Set standards and collect evidence.
ACT (evaluator) informs the client of the scales. In this case, the ACT uses a norm-based scale on which 36 is “perfect” and 20 is “average.” The scores that are one standard deviation above or below the mean are set at 16 and 26 respectively.
The client develops the decision-making process to use based on the evaluated performance quality: a student with weak skills will not be considered for admission; a student with very strong skills will be considered for a non-need-based scholarship; a student with average skills will be accepted, or not, based on other criteria.
These are the standards: a student with a composite ACT score below 16 will not be considered for admission; a student with a composite ACT score above 26 will be accepted to the college unless his or her score on all other acceptance criteria are very poor; a student with a composite ACT score above 28 will be considered for a non-need based scholarship. Other evaluative tools will be used for admission for a student with a composite score between 16 and 26.
The applicant completes the ACT test and the evaluator collects the evidence (the completed test).
ACT scores the test and documents the results.
4. Report and make decisions.
The applicant scores a 21 ACT composite score, which is sent to Winfield College by ACT at the evaluatee’s request.
This score eliminates the possibility of a non-need-based scholarship, but opens the door to admission, assuming other evaluative findings support the decision.
The client informs the financial aid office that the student will not be receiving a non-need-based scholarship. He then looks at the student’s high school record, which will be used to determine whether or not the student is accepted.
The client puts a note in the applicant’s file that indicates that the applicant is ineligible for a scholarship, but that the ACT scores do not prohibit admission. If appropriate, either the evaluator or the client sends a report of the findings to the evaluatee. ACT has already informed the student of the scores. The client will contact the student once an admission decision has been made. Because non-need-based scholarships are awarded, not denied, the client will say nothing to the student.
The program developed for economics majors at State College includes a first-year course in economics. Professor Backer, a second year faculty member, is teaching this course for the second time. Professor Chadler, chair of the Economics Department and one of the developers of the new program, wants to evaluate the learning of the students in Professor Chadler’s course this year, particularly since students have complained often about his teaching in the last year. The results will be used to help Professor Chadler either draft a letter to the tenure board that supports Professor Backer’s quest for tenure or draft a letter to Human Resources justifying the termination of Professor Backer.
1. Define the parameters of the evaluation.
The purpose of the evaluation is to determine the quality of Professor Backer’s teaching (indirectly) and the quality of students’ learning of concepts (directly).
The results can provide job security for Professor Backer.
The students (direct evaluatees) will get a grade for completing an assignment that will be used to evaluate their learning. The students will get a report that indicates how well they have learned the introductory concepts. Dr. Backer (indirect evaluatee) will get a report that indicates how well his students learned the introductory concepts, and a summary of how well all students learned the introductory concepts. Dr. Backer will receive a report from the client to explain any decisions made based on the evaluation.
Professor Chadler will receive information for each individual student indicating which instructor the student had in the introductory course and how well he or she understands the introductory concepts.
All students should understand the concept of supply and demand after finishing an introductory economics course. The information on quality must be received before October 15th in view of termination and tenure date deadlines. The cost must be minimal.
Performance: teaching Type of evaluation: indirect Evaluator: Professor Chadler Evaluatee: students (directly), Professor Backer (indirectly) Client: Professor Chadler2. Design the methods used for the evaluation.
The criteria, based on the guidelines, are the ability to explain and exemplify the concept of supply and demand.
There will be a pretest given in the second economics course. The pretest establishes the extent to which each student has met each criterion. The pretest asks the student to:
1. State the principle of supply and demand.
2. Give an example of supply and demand.
3. Given a change in demand, what would be the change in the cost?
3. Set standards and collect evidence.
All students who take the second term of economics will be evaluated. This could cause a certain bias, since many of those least likely to have learned the concepts are also those most likely to have stopped taking economics. Dr. Chadler is willing to allow this bias since he believes it will be the same for all faculty teaching introductory students. However, he will look at the dropout rate for each faculty member to find out if his assumption is true.
Collect the data and other evidence using pretest questions in a subsequent course.
Collect data on the first day of the next term.
4. Report and make decisions.
These are the scales used to determine quality:
i. Stating the principle. All points stated:
A Basics in place with some points missing:
C No evidence of knowledge:
F ii. Giving an example of the principle. Example supports the principle:
A Example supports only part of the principle:
C Example has no relationship to the principle:
F iii. Give examples of changes in supply, stating the effect on cost. Completely correct answer:
A Some minor errors, but basic understanding present:
C No evidence of understanding:
F
If Dr. Backer’s students’ average grade falls below the set standard, Dr. Chadler will request his termination. If the average grade is above the set standard, Dr. Chadler will use this evaluation as evidence for Dr. Backer’s tenure request.
If Dr. Backer’s students’ average grade is more than one letter grade below the other faculty members’ grades, Dr. Chadler will request his termination. If it is one letter grade above the other faculty members’ grades, he will use this evaluation as evidence for his request for tenure.
Dr. Chadler collects the answers to the exam questions and grades them.
He finds that Dr. Backer’s students’ average grade on the questions was “D.” The average grade of the other professors was “A.”
Concluding Thoughts
Students who had Professor Backer had a grade average of D on the questions. The average for all faculty was a “B.”
Dr. Backer’s students do not appear to have learned the concept of supply and demand nearly as well as the average of all instructors’ students. This result falls below the standard set for continuing employment.
Dr. Chadler decides to terminate Dr. Backer’s contract.
Dr. Chadler puts the evaluation report in Dr. Backer’s employment file. He also contacts all the people on campus who need to know that Dr. Backer will not be rehired.
Dr. Chadler notifies Dr. Backer that his contract will not be renewed.
ReferencesAs shown in these examples, decisions that have serious ramifications can be made with confidence and documentation when a systematic, well-planned evaluation process is used. The planning process reduces the occurrence of bias from evaluations and justifies decisions, even difficult decisions. The quality of confidence in results from a quality evaluation process clearly justifies the effort to perform the process systematically.
Angelo, T. A., & Cross, K. P. (1993). Classroom assessment techniques: A handbook for college teachers. San Francisco: Jossey-Bass.
Marzano, R., Pickering, D., & McTighe, J. (1993). Evaluating student outcomes: Performance evaluation using the dimensions of learning model. Alexandria, VA: Association for Supervision and Curriculum Development.
Wiggins, G. (1998). Educative evaluation: Designing evaluations to inform and improve student performance. San Francisco: Jossey-Bass.
Wiggins, G. P., & McTighe, J. (2005). Understanding by design (2nd expanded edition). Baltimore: Association for Supervision & Curriculum Development.