Please note: This site's design is only visible in a graphical browser that supports Web Standards, but its content is accessible to any browser or Internet device. To see this site as it was designed please upgrade to a Web standards compliant browser.

Program-Based Review and Assessment: Tools and Techniques for Program Improvement

Download PDF version of the handbook

Assessment Methods and Strategies

What Should You Remember When Selecting Assessment Methods?

When developing assessment methods, make sure your selections:

adapted from California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999)

What are Some Ways to Assess the Undergraduate Major?

Existing Information

Think about the ways in which you can use one source of information for a variety of individual student and program-level purposes. Such data could include:

Assessment and Grading

When the issue of assessment is raised, faculty members often say, "I already do assessment. I grade student assignments." Grades are indeed one measure of student achievement. There are significant drawbacks, however, to using grades to meet assessment's primary goal ­ to improve teaching and learning. Grades are global evaluations that represent the overall proficiency of students. They don't tell you about student performance on individual (or specific) learning goals. Because grades don't tell you about student performance on individual (or specific) learning goals or outcomes, they provide little information on the overall success of the program in helping students attain specific and distinct learning objectives of interest.

New Information

In addition to accessing data that are already available, your department can collect new data specific to student learning in your program and designed to address departmental goals and objectives. These data sources might include information collected through:

What are Some Guidelines for Selecting Assessment Methods?

Focus on Key Questions

The evidence you collect depends on the questions you want to answer. In thinking about program assessment, consider four key questions:

  1. Does the program meet or exceed certain standards?
  2. How does the program compare to others?
  3. Does the program do a good job at what it sets out to do?
  4. How can the program experience be improved?
Use Multiple Methods

Use multiple methods to assess each learning outcome. Many outcomes will be difficult to assess using only one measure.

Include Direct and Indirect Measures

Direct methods ask students to demonstrate their learning while indirect methods ask them to reflect on their learning. Direct methods include some objective tests, essays, presentations and classroom assignments. Indirect methods include surveys and interviews.

Assess Strengths and Weaknesses

Choose assessment methods that provide both positive and negative feedback.

Include qualitative as well as quantitative measures

All assessment measures do not have to involve quantitative measurement. A combination of qualitative and quantitative methods can offer the most effective way to assess goals and outcomes. Use an assessment method that matches your departmental culture.

adapted from Volkwein, J., Program evaluation and assessment: What's the question (1996)

Be selective about what you choose to observe or measure

As you work through this process, remember that:

Include passive as well as active methods of assessment

Assessment measures are also available that allow you to analyze assessment information without direct student contact or effort. Some passive methods include:

Use capstone courses or senior assignments to directly assess student learning outcomes

If you use this method, however, care should be taken that:

Enlist the assistance of assessment specialists when you create, adapt, or revise assessment instruments

Staff in the Office of Academic Planning and Assessment are happy to assist you in finding the appropriate resources. Areas in which you might want to seek assistance include:

Use established accreditation criteria to design your assessment program

Using established criteria will help you respond more effectively to accreditation requirements.

Examples on are adapted from University System of Georgia: Task Force on Assessing Major Area Outcomes, Assessing Degree Program Effectiveness (1992); Western Carolina University, Assessment Resources Guide (1999).

What Assessment Methods Can You Use?

As you consider which methods might be most appropriate for your departmental culture and your assessment questions, it might be useful to use a Method Selection Criteria Matrix [Table 8]. This matrix allows you to evaluate the appropriateness of the methods you are considering based on criteriaof importance to the department.

In the next example [Table 9], the learning objectives under consideration are listed in the first column and methods are outlined along the top. Completing this matrix will help you link learning objectives to specific measures that can be used to assess these objectives. Think about whether each measure is direct or indirect and note that in the appropriate column (in this example, "D" and "I"). You can also rate the extent to which each measure appropriately represents the objective, using pluses and minuses or other indicators with meaning for you.

How do you link outcomes, methods, and results?

When you have identified the outcomes you will assess, have decided on the methods you will use to collect the data, and have tried to anticipate the results you might see, it is important to link these components together to most effectively articulate and operationalize your assessment plan. The following examples [Table 10] can help you outline assessment objectives and methodology, and mark out a timeline for the plan. Remember that for program improvement purposes, all data do not have to be collected every year, since there will probably not be much change from year to year unless you have made substantial changes in your delivery system or curriculum.

Sample Departmental Assessment Timeline

Glossary of Assessment Methods

[Table 11]

ALUMNI SURVEYS
Description:

Surveying department alumni can provide a wide variety of information about program satisfaction, how well students are prepared for their careers, what types of jobs or graduate degrees majors have gone on to obtain, starting salaries for graduates, and the skills that are needed to succeed in the job market or in graduate study. These surveys provide the opportunity to collect data on which areas of the program should be changed, altered, improved or expanded.

Strengths and Weaknesses:

Alumni surveying is usually a relatively inexpensive way to collect program data from individuals who have a vested interest in helping you improve your program as well as offering the opportunity for improving and continuing department relationships with program graduates. However, without an easily accessible and up-to-date directory of alumni, they can be difficult to locate. It also takes time to develop an effective survey and ensure an acceptable response rate.

adapted from Palombo et al. Ball State University, Assessment Workbook (2000).

CULMINATING ASSIGNMENTS
Description:

Culminating assignments offer students the opportunity to put together the knowledge and skills they have acquired in the major, provide a final common experience for majors, and offer faculty a way to assess student achievement across a number of discipline-specific areas. Culminating assignments are generally designed for seniors in a major or field to complete in the last semester before graduation. Their purpose is to integrate knowledge, concepts and skills that students are expected to have acquired in the program during the course of their study. This is obviously a curricular structure as well as an assessment technique and may consist of a single culminating course (a "capstone" course) or a small group of courses designed to measure competencies of students who are completing the program. A senior assignment is a final culminating project for graduating seniors such as a performance portfolio or a thesis that has the same integrative purpose as the capstone course.

Strengths and Weaknesses:

Many colleges and universities are using capstone courses to collect data on student learning in a specific major or in general education or core requirement programs. Putting together an effective and comprehensive capstone course can be a challenge, however, particularly for those programs that mesh hands-on technical skills with less easily measurable learning outcomes. Also, there is a great deal of start-up time to developing appropriate and systematic methods for assessing these or other culminating experiences. See Content Analysis and Primary Trait Analysis below for further information.

adapted from the University of Wisconsin, Madison, Outcomes Assessment Manual (2000).

Additional Resources:
Southern Illinois University website: www.siue.edu/~deder/assess
CONTENT ANALYSIS
Description:
Content

analysis is a technique that looks at a group of students, such as majors in a program or department, and assesses samples of written work that are produced by this group. This assessment method uses outcomes identified as important prior to the analysis or as the analysis proceeds. For example, you might want to determine how well majors in your department write. To use content analysis to assess their writing skills, you will need a representative sample of the writing. Analysis may look at what students actually write or at the underlying meaning of their writing. Results are generally presented in written form giving averages and examples of specific categories of outcomes (e.g., spelling errors). Primary trait analysis, which identifies important characteristics of specific assignments and assigns levels of competency to each trait, can be particularly effective in identifying student learning.

Strengths and Weaknesses:

Content analysis allows you to assess learning outcomes over a period of time and can be based on products that were not created for program assessment purposes. Because writing samples can be re-examined, content analysis also makes it easier to repeat portions of the study and provides an unobtrusive way to assess student learning. However, accuracy of the assessment is limited to the skill of the person(s) doing the analysis. Data is also limited by the set of written work and may not be relevant to technical skills valued by a particular field or major that involve hands-on performance. Pre-testing coding schemes, using more than one analyst per document, and concrete materials and coding schemes can improve the reliability of this technique.

adapted from the California State University Bakersfield, PACT Outcomes Assessment Handbook (1999).

COURSE-EMBEDDED ASSESSMENT
Description:

Course-embedded assessment refers to methods of assessing student learning within the classroom environment, using course goals, objectives and content to gauge the extent of the learning that is taking place. This technique generates information about what and how students are learning within the program and classroom environment, using existing information that instructors routinely collect (test performance, short answer performance, quizzes, essays, etc.) or through assessment instruments introduced into a course specifically for the purpose of measuring student learning.

Strengths and Weaknesses:

This method of assessment is often effective and easy to use because it builds on the curricular structure of the course and often does not require additional time for data collection since the data comes from existing assignments and course requirements. Course-embedded assessment does, however, take some preparation and analysis time and, while well documented for improving individual courses, there is less documentation on its value for program assessment.

adapted from the University of Wisconsin, Madison, Outcomes Assessment Manual (2000), and the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999).

Additional Resources:
Classroom Assessment Techniques. (1999). Center for Excellence in Learning & Teaching. www.personal.psu.edu/celt/CATs.html
CURRICULUM ANALYSIS
Description:

Curriculum analysis involves a systematic review of course syllabi, textbooks, exams, and other materials to help you clarify learning objectives, explore differences and similarities between course sections, and/or assess the effectiveness of instructional materials. It offers a way to document which courses will cover which objectives and helps in sequencing courses within a program. Also see Matrices.

Strengths and Weaknesses:

Using curriculum analysis as an assessment tool can be a valuable way of tracking what is being taught where. It can provide assurance that specific learning goals and objectives are being covered in the program and can pinpoint areas where additional coverage is needed. This method, however, can be time-consuming, particularly in large departments with many courses and different instructors, and there may be little consistency between how learning objectives are addressed in one course and how they are taught in another.

adapted from the Ball State University, Assessment Workbook, 1999 and The University of Wisconsin, Madison, Outcomes Assessment Manual I (2000).

DELPHI TECHNIQUE
Description:

The Delphi technique is used to achieve consensus among differing points of view. In its original form, a team of experts, who never actually meet, are asked to comment on a particular issue or problem. Each member's response is reviewed and a consensus determined. Any member whose response falls outside of the consensus is asked to either defend or rethink the response. The anonymity provided by this technique offers more junior members of the team an equal chance to get their ideas out, as well as permits a challenge to the ideas of senior members that might never take place in an open forum. More recently, the Delphi technique has been modified so that teams of individuals are brought together to discuss an issue or problem face-to-face and reaching a consensus at the meeting. For instance, a team of faculty members might meet to review possible goals and objectives for their department in an effort to develop a set of goals and objectives on which they can agree.

Strengths and Weaknesses:

The Delphi technique can be useful in bringing together diverse opinions in a discussion forum. This technique fails, however, when the facilitator lacks objectivity or when the participants feel unsafe or insecure in voicing their real opinions. For instance, a faculty member discussing intended goals and objectives might not be comfortable in disagreeing with the department head. For this technique to succeed, care must be taken to appoint an impartial facilitator and to convince participants that differing opinions are welcome. Returning to the original design of this technique, with an anonymous team who never meet, might ensure more honest and open input.

Additional Resources:
Armstrong, M. A. (1989). The Delphi technique. Princeton Economic Institute. http://www.pei-intl.com/Research/MARKETS/DELPHI.HTM. Cline, Alan. (2000). Prioritization Process using Delphi Technique. www.carolla.com/wp-delph.htm
EMPLOYER SURVEYS
Description:

Employer surveys help the department determine if their graduates have the necessary job skills and if there are other skills that employers particularly value that graduates are not acquiring in the program. This type of assessment method can provide information about the curriculum, programs and student outcomes that other methods cannot: on-the-job, field-specific information about the application and value of the skills that the program offers.

Strengths and Weaknesses:

Employer surveys provide external data that cannot be replicated on campus and can help faculty and students identify the relevance of educational programs, although, as is true in any survey, ambiguous, poorly-worded questions will generate problematic data. Additionally, though data collected this way may provide valuable information on current opinion, responses may not provide enough detail to make decisions about specific changes in the curriculum or program. Also, it is sometimes difficult to determine who should be surveyed, and obtaining an acceptable response rate can be cost and time intensive.

adapted from the Ball State University, Assessment Workbook (1999), the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999), and the University of Wisconsin, Madison, Outcomes Assessment Manual I (2000).

FOCUS GROUPS
Description:

Focus groups are structured discussions among homogeneous groups of 6-10 individuals who respond to specific open-ended questions designed to collect data about the beliefs, attitudes and experiences of those in the group. This is a form of group interview where a facilitator raises the topics for discussion and collects data on the results. Emphasis is on insights and ideas.

Strengths and Weaknesses:

Focus groups can provide a wide variety of data about participants' experiences, attitudes, views and suggestions, and results can be easily understood and used. These groups allow a small number of individuals to discuss a specific topic in detail, in a non-threatening environment. Data collected in this way, however, is not useful for quantitative results, and qualitative data can be time-consuming and difficult to analyze because of the large amount of non-standardized information. Ultimately, the success of this method depends on a skilled, unbiased moderator and appropriate groups of participants.

adapted from Palombo et al. Ball State University, Assessment Workbook (2000); and the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999).

INSTITUTIONAL DATA
Description:

A variety of departmental and student data are routinely collected at the university level. These data can enhance and elaborate on data you collect in the department. Institutional data can tell you whether the program is growing, what the grade point average is for majors in the program, and what the retention rate is for your students.

Strengths and Weaknesses:

Institutional data are generally easily accessible and readily available. On the UMass Amherst campus, you can access this data through the Office of Institutional Research (OIR), located in 237 Whitmore. Student and departmental data are collected on a systematic and cyclical schedule that can offer you both current and longitudinal information. On the other hand, these data sets are generally large and may be difficult to sort through, particularly for those individuals who are not used to working through large databases.

The data may be less useful to specific departments or programs because the information collected is very often general (age, gender, race, etc.) and may not directly relate to program goals and objectives.

adapted from the Ball State University, Assessment Workbook (1999).

MATRICES
Description:

At its most basic, a matrix is a grid of rows and columns used to organize information. For assessment purposes, a matrix can be used to summarize the relationship between program objectives and course syllabus objectives, course assignments, or courses in a program or department. Matrices can be used for curriculum review, to select assessment criteria or for test planning. A matrix can also be used to compare program outcomes to employer expectations.

Strengths and Weaknesses:

Using a matrix can give you a good overview of how course components and curriculum link to program objectives, can help you tailor assignments to program objectives, and can lead to useful discussions that in turn lead to meaningful changes in courses or curricula. However, because a matrix can offer a clear picture of how program components are interconnected and can reveal where they are not, acknowledging and responding to discrepancies may involve extensive discussion, flexibility and willingness to change.

adapted from the Ball State University, Assessment Workbook, revised April (2000), and the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999).

OBSERVATIONS
Description:

Observation as a method of assessment is an unobtrusive tool that can yield significant information about how and why students learn. You may choose to observe any relevant interactive event, such as classes, club meetings, or social gatherings. This tool is generally used when you are interested in how students study, are concerned about the effectiveness of study sessions or other supplementary activities, or when you are focusing on the relationship between out-of-class behavior and in-class performance. Data collected through observation can be correlated with test scores and/or course grades to help provide further insight into student learning.

Strengths and Weaknesses:

Data collected through observation can yield important insight into student behavior that may be difficult to gauge through other assessment methods. This method is typically designed to describe findings within a particular context and often allows for interaction between the researcher and students that can add depth to the information collected. It is especially useful for studying subtleties of attitudes and behavior. Observed data, however, is not precise and cannot be generalized to larger populations. Conclusions may be suggestive rather than definitive, and others may feel that this method provides less reliable data than other collection methods.

adapted from the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999).

PERFORMANCE ASSESSMENT
Description:

Performance assessment uses student activities to assess skills and knowledge. These activities include class assignments, auditions, recitals, projects, presentations and similar tasks. At its most effective, performance assessment is linked to the curriculum and uses real samples of student work. This type of assessment generally requires students to use critical thinking and problem-solving skills within a context relevant to their field or major. The performance is rated by faculty or qualified observers and assessment data collected. The student receives feedback on the performance and evaluation.

Strengths and Weaknesses:

Performance assessment can yield valuable insight into student learning and provides students with comprehensive information on improving their skills. Communication between faculty and students is often strengthened, and the opportunity for students' self-assessment is increased. Performance assessment, like all assessment methods, is based on clear statements about learning objectives. This type of assessment is also labor-intensive, is sometimes separate from the daily routine of faculty and student, and may be seen as an intrusion or an additional burden. Articulating the skills that will be examined and specifying the criteria for evaluation may be both time-consuming and difficult.

adapted from the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999).

PORTFOLIO EVALUATIONS
Description:

Portfolios are collections of student work over time that are used to demonstrate student growth and achievement in identified areas. Portfolios can offer information about student learning, assess learning in general education and the major, and evaluate targeted areas of instruction and learning. A portfolio may contain all or some of the following: research papers, process reports, tests and exams, case studies, audiotapes, videotapes, personal essays, journals, self-evaluations and computational exercises. Portfolios are often useful and sometimes required for certification, licensure, or external accreditation reviews.

Strengths and Weaknesses:

Portfolios not only demonstrate learning over time, but can be valuable resources when students apply to graduate school or for jobs. Portfolios also encourage students to take greater responsibility for their work and open lines of discussion between faculty and students and among faculty involved in the evaluation process. Portfolios are, however, costly and time-consuming and require extended effort on the part of both students and faculty. Also, because portfolios contain multiple samples of student work, they are difficult to assess and to store and may, in some contexts, require too much time and effort from students and faculty alike.

adapted from the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999), and the University of Wisconsin, Madison, Outcomes Assessment Manual I (2000).

PRE-TEST/POST-TEST EVALUATION
Description:

This method of assessment uses locally developed and administered tests and exams at the beginning and end of a course or program in order to monitor student progression and learning across pre-defined periods of time. Results can be used to identify areas of skill deficiency and to track improvement within the assigned time frame. Tests used for assessment purposes are designed to collect data that can be used along with other institutional data to describe student achievement.

Strengths and Weaknesses:

Pre-test/post-test evaluations can be an effective way to collect information on students when they enter and leave a particular program or course, and provide assessment data over a period of time. They can sample student knowledge quickly and allow comparisons between different students groups, or the same group over time. They do, however, require additional time to develop and administer and can pose problems for data collection and storage. Care should be taken to ensure that the tests measure what they are intended to measure over time (and that they fit with program learning objectives) and that there is consistency in test items, administration and application of scoring standards.

adapted from the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999), and the University of Wisconsin, Madison, Outcomes Assessment Manual I (2000).

REFLECTIVE ESSAYS
Description:

Reflective essays may be used as an assessment tool to gauge how well students are understanding class content and issues. They are generally short essays (5 to 10 minutes) on topics related to the course curriculum and may be given as in-class assignments or homework. Reflective essays may be voluntary or required, open-ended questions on surveys required in student portfolios or capstone composition courses.

Strengths and Weaknesses:

Reflective essays as an assessment tool can offer data on student opinions and perspectives at a particular moment in a class. Essays will provide a wide array of different responses and might lead to increased discussion among faculty and students. On the other hand, poorly worded, ambiguous questions will yield little, and opinions and perceptions may vary in accuracy. Analysis of essay content also takes additional time and expertise.

SCORING RUBRICS
Description:

Scoring rubrics are typically grids that outline identified criteria for successfully completing an assignment or task and establish levels for meeting these criteria. Rubrics can be used to score everything from essays to performances. Holistic rubrics produce a global score for a product or performance. Primary trait analysis uses separate scoring of individual characteristics or criteria of the product or performance.

Strengths and Weaknesses:

Scoring rubrics allow the instructor to efficiently and consistently look at complex products or performances and to define precise outcomes and expectations. They also are easily shared with students. However, developing an effective rubric can be time-consuming and often requires ongoing edits to fine tune criteria and anticipated outcomes. Training raters to use the scoring rubrics in a consistent manner also involves a significant time commitment.

adapted from the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999).

Additional Resources:
Southern Illinois University: www.siue.edu/~deder/assess
STANDARDIZED AND LOCAL TEST INSTRUMENTS
Description:

Selecting a standardized instrument (developed outside the institution for application to a wide group of students using national/regional norms and standards) or a locally-developed assessment tool (created within the institution, program or department for internal use only) depends on specific needs and available resources. Knowing what you want to measure is key to successful selection of standarized instruments, as is administering the assessment to a representative sample in order to develop local norms and standards. Locally-developed instruments can be tailored to measure specific performance expectations for a course or group of students.

Strengths and Weaknesses:

Locally-developed instruments are directly linked to local curriculum and can identify student performance on a set of locally-important criteria. Putting together a local tool, however, is time-consuming as is development of a scoring key/method. There is also no comparison group and performance cannot be compared to state or national norms. Standardized tests are immediately available for administration and, therefore, are less expensive to develop than creating local tests from scratch. Changes in performance can be tracked and compared to norm groups and subjectivity/misinterpretation is reduced. However, standardized measures may not link to local curricula and purchasing the tests can be expensive. Test scores may also not contain enough locally-relevant information to be useful.

adapted from the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999), and the University of Wisconsin, Madison, Outcomes Assessment Manual I (2000).

Additional Resources:
National Post-Secondary Education Cooperative (NPEC) Assessment Tests Review. http://www.nces.gov/npec/evaltests
STUDENT SURVEYS AND EXIT INTERVIEWS
Description:

Surveys and interviews ask students to respond to a series of questions or statements about their academic experience. Questions can be both open-ended (respondents create answers) and close-ended (respondents answer from a list of simple and unambiguous responses). Surveys and interviews can be written or oral (face-to-face) or phone. Types of surveys include in-class questionnaires, mail questionnaires, telephone questionnaires, and interviews. Interviews include structured, in-person interviews and focus group interviews.

Strengths and Weaknesses:

Surveys can be relatively inexpensive and easy to administer, can reach participants over a wide area, and are best suited for short and non-sensitive topics. They can give you a sense of what is happening at a given moment in time and can be used to track opinions. Data is reasonably easy to collect and tabulate, yet the sample may not be representative of the population (particularly with a low response rate). Ambiguous, poorly written items and insufficient responses may not generate enough detail for decision making. An interview can follow-up on evasive answers and explore topics in-depth, collecting rich data, new insights, and focused details. It can, however, be difficult to reach the sample and data can be time-consuming to analyze. Information may be distorted by the respondent, who may feel a lack of privacy and anonymity. The success of the interview depends ultimately on the skills of the interviewer.

adapted from the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999), and the University of Wisconsin, Madison, Program Assessment Tool Kit (1998).

SYLLABUS ANALYSIS
Description:

Syllabus analysis (as well as systematic review of textbooks, exams and other curricular material) involves looking at the current course syllabus (written or oral assignments, readings, class discussions/projects and course expectations) to determine if the course is meeting the goals and objectives that the instructor or deparment has set for it.

Strengths and Weaknesses:

Use syllabus analysis when you want to clarify learning objectives; explore differences and similarities between sections of a course; or assess the effectiveness of instructional materials. Syllabus analysis can provide invaluable information to enhance any assessment plan. However, this review is time consuming and, as there may be more than one reviewer, there may not be adequate consistency in collecting and analyzing the data.

TRANSCRIPT ANALYSIS
Description:

Transcript analysis involves using data from student databases to explore course-taking or grade patterns of students. This tool can give you a picture of students at a certain point in their academic careers, show you what classes students took and in what order, and identify patterns in student grades. In sum, transcript analysis gives you a more complete picture of students' actual curricular experiences. Specific information can be drawn from transcripts to help answer research questions, and course pattern sequences can be examined to see if there is a coherence to the order of courses taken.

Strengths and Weaknesses:

Transcript analysis is an unobtrusive method for data collection using an existing student database. This information can be linked to other variables such as sex or major, or used to measure outcomes. It is important to keep in mind, however, that course patterns may be influenced by other variables in students' lives that don't show up on their transcripts. Also, solutions that arise from results of the analysis may not be practical or easily implemented. It is critical to have specific questions whose answers can lead to realistic change before conducting the analysis.

adapted from the California State University, Bakersfield, PACT Outcomes Assessment Handbook (1999), and the Ball State University, Assessment Workbook (1999).

 
Assessment Consulting

Assessment