Assessing Students’ Integrative Learning Across Curricular and Co-Curricular Experiences
Through the example of an integrative learning assessment project tied to the introduction of an “Integrative Experience” (IE) General Education requirement, this Research Brief addresses three particularly challenging aspects of student learning assessment: (1) Implementing centralized assessment activities that both serve as learning experiences for students and offer evidence useful in informing General Education and other campus-based programming and interventions; (2) How to conduct systematic analysis of complex qualitative evidence; and (3) Documenting students’ integrative (or connected) learning. Using evidence the Office of Academic Planning and Assessment (OAPA) collected as a part of a series of focus groups with sophomores, juniors, and seniors, this brief analyzes the types of connections students map across their experiences at the University and the impact these connections have on their learning and development.
The results of this analysis highlight the number and variety of connections undergraduates make across their learning experiences, when prompted to do so. Of particular importance for advocates of General Education, connections between General Education and other aspects of the student experience are the most prevalent connections—highlighting the role Gen Ed experiences play in informing students’ experiences in their major and contributing to students’ personal and interpersonal development. The results also illustrate the important role integrated learning can play in students’ development along a whole host of areas. As a result of the connections they identify, students report gains in skill development, personal growth, career preparation, knowledge gain and transfer, and a number of other outcomes.
This study provides further evidence of the value for students of purposeful reflection about, and opportunities to integrate, their learning in higher education. It also offers empirical support for the contributions the General Education experience makes to student learning and development.
An earlier version of this paper was presented at the Association for the Assessment of Learning in Higher Education (AALHE) Annual Meeting June 11, 2019. The author is grateful for the thoughtful suggestions and observations of her AALHE colleagues during the presentation and has included many of these observations in the discussion of study results.
Introduction and Overview
Integrative and reflective student learning outcomes have received increasing emphasis in higher education in recent years, inspired by growing evidence of the value of reflective and integrative thinking in fostering student learning and development. The literature has also highlighted the importance of educational institutions building intentional opportunities for students to practice those types of thinking (see, for example, Association of American Colleges and Universities, 2005; Grindley, 2010; Huber & Hutchings, n.d.; Lardner & Malnarich, 2009; Rogers, 2001). In 2010 in response to this research and in conjunction with a review and revision to the campus’s General Education program, the University of Massachusetts Amherst implemented an upper-division General Education requirement, the Integrative Experience (IE), designed to provide students with opportunities to reflect upon and integrate their General Education (and other) experiences with their Major experiences. More on the University’s IE requirement and assessment results is available at the Integrative Experience website, here: http://www.umass.edu/oapa/program-assessment/general-education-assessment/integrative-experience-ie.
The campus assesses the effectiveness of this requirement in the early years of implementation in a number of ways, including student surveys that focus on the alignment of the course/student experience with the IE learning objectives, direct assessment of samples of student work for evidence of integrative and reflective thinking, interviews with IE instructors and students about how IE assignments and activities foster (or don’t foster) the integrative and reflective learning objectives, and a five-year cycle of Gen Ed Council (GEC) review (the GEC “Quinquennial Review” of all IE offerings). The introduction of the IE also offered the impetus for assessment into what integrative (or connected) learning on campus looks like for students more generally—and what outcomes are associated with these connections.
This Research Brief focuses on this latter assessment project, the analysis of students’ general integrative thinking and experiences using results from a series of focus groups, organized by class level (sophomores, juniors, and seniors). A total of 9 focus groups were conducted (3 each for each class level) and generated participation of a total of 62 students.
A key component of these focus groups was a pre-discussion mapping exercise all students were asked to complete when they entered the room where the Focus Group would be conducted. Students were supplied with a copy of their transcript and asked to review it and identify particularly important/memorable courses they have taken (in both General Education and their Major). They were also asked to identify important co-curricular, professional/career, and personal development experiences. Students were instructed to list these experiences and then spend time considering how/if there were connections across these 5 areas of experience and, when they saw a connection, to identify the impact or outcome of that connection. (See both a blank version and a completed sample of this mapping exercise as Appendix A).
The intent in beginning the focus groups with the mapping exercise was to prompt students’ focused attention to the concept of integrative/connected learning prior to discussing their experiences together. The mapping tool would also provide us with additional evidence of the kinds of connections students make on their own (the evidence that informs the analyses in this Research Brief). A somewhat unanticipated additional outcome was that the exercise also became an educational intervention (or stimulus for reflection) where we observed students’ “Aha” moments as new connections and scaffolded learning experiences emerged for them through completing the mapping exercise.
Relevance to Current Assessment Challenges
This research project and analysis addresses three important priorities for the assessment of student learning:
- Implementing centralized assessment activities that both serve as learning experiences for students and offer evidence useful in informing general education and other campus-based programming and interventions. Since very early in the assessment movement, assessment leaders have emphasized the value of authentic assessment processes for students, faculty, and campus professionals alike (Cross & Steadman, 1996; Walvoord & Anderson, 1998; Wiggins, 1998). These efforts include, for example, using systematic scoring of course-based student work, student learning artifacts drawn from students’ capstone demonstrations of their cumulative learning, and evidence drawn from regularized feedback of internship or other applied learning experiences. Making survey or focus group methodologies authentic learning experiences is particularly challenging. However, in this project, we were able to create a hybrid data collection and a learning/reflection opportunity for students through the use of our mapping technique, an approach that supports emerging emphases on blending reflection with assessment (Ludvik, 2018).
- Designing methods for analyzing complex qualitative evidence that can inform campus-wide curricular and programmatic improvements. Assessing student learning is a complex enterprise, requiring multiple evidentiary lenses to fully represent those complexities. While some critics characterize assessment as “bean counting” using narrow and reductionist standardized quantitative measures and mindless counting, the fact of the matter is that comprehensive student learning assessment should and often does incorporate both quantitative and qualitative sources of evidence. Among the ongoing challenges is how to fully incorporate qualitative evidence in assessment analyses in ways that are both relatively efficient and systematically informative (Kimbal & Loya, 2017; Maki, 2004; Suskie, 2009; Werder & Otis, 2010). This project demonstrates a method of representing students’ integrative mapping that combines an inductive coding process with a quantitative descriptive technique.
- Assessing integrative thinking and connected learning. With the advent of cognitive research highlighting the importance of integrative and reflective thinking has come the challenge of how to document and assess the outcomes of these complex processes (Association of American Colleges and Universities, 2005; Cuevas, Mateev, & Miller, 2010; Huber & Hutchings, n.d.; Ludvik, 2018; Manilla, 2005; Rogers, 2001; Toppo, 2019). The mapping tool introduced in this Brief and the results of the analysis of students’ mapping responses offer important insights into the kinds of connections students experience across their curricular, co-curricular, and personal development experiences in higher education.
There are two elements to the analysis: (1) developing a quantitative translation of student-identified connections and (2) qualitative (inductive) coding of students’ descriptions of the kinds of connections and the impact of those connections.
Quantitative Translation of Connections: As the mapping example in Appendix A makes clear, students identified a number of connections across the three University-structured curricular/co-curricular areas (Gen Ed, the Major, and Co-Curricular experiences) and the two additional areas of student life and development (Personal & Interpersonal Life and Professions/Career Life). An Excel spreadsheet was developed for each University-structured category to capture the scale and patterns of these connections. For each of the three University-structured areas, each type of connection was documented for each student record. Specifically, for each student ID, the number of connections for each combination was entered. The entry also included the number of Gen Ed courses, Major courses, or co-curricular activity (as relevant) the student listed for connections.
Qualitative Inductive Coding of Types of Connections and Impacts. The spreadsheet also included the results of inductive coding of the kinds of co-curricular activities students’ identified as being connected to the other components and the outcome of the connections across all structured and unstructured areas in the mapping. As an example, a student notes a connection between co-curricular experiences and their major. They list in the co-curricular box “working on the student newspaper.” They then draw a line between that activity and the major box, noting on the line the co-curricular activity gave them practice in writing skills that helped them in the major. (See the mapping example in Appendix A for an illustration of students’ notations.) In this case, the co-curricular code would be “Focused Activity” (like choir, athletics, etc.) and the impact code would be “Skill Development.” This type of coding was done for all connections where students provided the details.
An excerpt from the Gen Ed connections Excel workbook (Data Excerpt 1) shows what these spreadsheets looks like in practice. Note that there can be multiple entries/rows per student ID depending upon the number of connections they identify.
Data Excerpt 1.
Sample from Excel Spreadsheet For Recording Students’ Reported Connections to their Gen Ed Experience.
Once the connections were all entered in separate spreadsheets for each structural experience, they were combined using student ID and an SPSS data set was created, making it possible to study the connections across all students and all types of connections.
The results of this analysis highlight the rich landscape from which educators can draw in helping students develop their integrative and reflecting thinking capacities. The results also, interestingly, affirm the contributions General Education makes to students’ education and development, both within the major and more broadly. The value of these maps is illustrated in the varied layers of evidence they offer.
Before looking at the aggregate results, it is important to point out that each student map tells its own story and gives us a new way “in” to understanding how students make sense of their own educations. These maps have been shared with workshop participants and IE instructors to illustrate the value of connections in individual students’ lives. The Writing Program also asked for samples of maps that illustrate how students connect their first-year writing course to other aspects of their undergraduate experience. In addition, the mapping tool itself is also serving as an educational intervention. There are IE instructors who are using their own adaptations of the tool in their courses to facilitate students integrative and reflective thinking (two primary objectives of the IE).
The results of the mapping exercise also offer the University insights into the kinds of connections students make most often—and the outcomes/impacts these students report as a result of those connections. Table 1 offers a summary of the patterns of connections students identify.
Count of Types of Dyadic Connections Undergraduates Make and the Reported Outcome/Impact of those Connections.
Note first the frequency of the varied connections made (with the total for each connection dyad in the column on the far right) and the frequency of the types of outcomes/impact of those connections (with the total for each outcome in the bottom row). The most frequently identified connections (the rows highlighted in red) both represent connections with General Education: one between Gen Ed and the Major (at n=69 connections) and the other between Gen Ed and Personal & Interpersonal Life (at n=60 connections). Advocates for General Education often assert the opportunity Gen Ed courses offer to help students identify or reinforce their major and to foster student intellectual and personal development and enrichment. In these maps, we see a confirmation of these possibilities. It is worth highlighting the fact that the mapping exercise did not explicitly ask students to focus on Gen Ed connections, nor did the Focus Group invitation emphasize a focus on Gen Ed. Given that Gen Ed was not promoted as the emphasis of the discussion, these results are all the more striking.
The types of outcomes or impacts of these connections also offer interesting patterns. Those with the most mentions (> 25) are represented in the columns highlighted in blue. Skill development as a result of these connections is mentioned most frequently and emerges out of a whole range of dyadic connections (again, with the most coming from Gen Ed x Major connections). Career Preparation, Personal Growth, and Self-awareness are also mentioned frequently illustrating the ways in which connections support students’ professional/career life and their personal/interpersonal development. Finally, knowledge and knowledge transfer are also mentioned frequently—particularly related to General Education connections. Here is another area proponents of Gen Ed emphasize—the opportunity for students to be introduced to new topics and new ways of thinking as well as applying that knowledge to other areas of their learning.
While students mentioned specific Gen Ed and Major courses in their maps, these responses have not been coded to discern specific patterns. Although such an analysis might provide additional insights, the wide variety of courses named and students’ sometimes peculiar notations in naming courses makes such an analysis more complex than was possible here.
However, students also named the specific co-curricular activity they connected to other aspects of their education and these were coded to identify the broad range of co-curricular
activities mentioned. Graph 1 shows the frequency of 6 different kinds of co-curricular activities connected to Gen Ed, the Major, or personal or professional life. Both focused co-curricular activities (those that require an ongoing time commitment, focused on a specific task) and various club activities are identified most often. It is somewhat surprising that residential life activities receive little mention—this might have to do with students seeing their residential life activities (e.g., serving as a Resident Assistant) as a job as opposed to co-curricular activity.
Graph 1. Categories of Co-Curricular Activities Connected
The results in Table 1 illustrate the overall patterns of connections and outcomes students report. However, this table doesn’t fully illustrate what the connections patterns look like for individual students. It is possible that the overall counts mask important differences across the 62 students. For example, the high number of connections between Gen Ed and the major might really only reflect the active engagement of half of the students, while the other half make no connections between Gen Ed and the major. The analysis represented in Graph 2 explores the patterns of connections at the individual student level. The first graph (with the red background) represents the patterns of student connections with Gen Ed. As the box to the right of the donut graph explains, of the 62 students who completed a mapping exercise, 90 percent (n=56) identified at least one connection with Gen Ed (clearly illustrating that the connections represented in Table 2 are across a range of students—not driven by a particularly engaged minority/sub-set). The percentages in the donut reflect the proportion of Focus Group participants that identified each kind of Gen Ed connection; of the 62 participants, 66% identified a connection between Gen Ed and their Major and 63% identified a connection between Gen Ed and their Personal lives.
What is particularly interesting about the patterns in Graph 2 is (1) that the vast majority of students (74 percent or more) identify connections that include each of the three University-structured experiences that are the focus of the mapping exercise and (2) that connections with Gen Ed are the most common form of connection. It is also worth noting how the patterns of connections are different across these three structured experiences. While there are few co-curricular connections with Gen Ed, there are twice as many with the major, and the proportion of students who identify a connection between their personal life and Gen Ed is almost double what the connection is between personal and co-curricular experiences. Perhaps not surprisingly, the highest proportion of connections to Professional/Career life are with the Major, although the percentage of those who identify connections between their Professional/Career life and Co-Curricular activity (42%) and Gen Ed (32%) are noteworthy areas for further attention as the University continues to enhance its support for students’ career/professional preparation.
Graph 2: Undergraduate Integrative Learning: Patterns of Individual Student Connections
These results illustrate the benefit of asking students to consider their educational experiences and how they contribute to their learning and development across structured and unstructured college experiences. With the aid of a mapping tool, some prompting, and dedicated time set aside to reflect and consider one’s experiences (including a review of one’s transcript), students identify a wide array of connections that span across varied aspects of their education. The impacts of these connections are also varied, ranging from skill and personal development to knowledge gain and transfer to enhanced awareness of self and others. These results are particularly compelling given the relatively short period of time students were given at the start of the Focus Group to complete the mapping exercise. Just think what kind of insights and development might occur with longer term and scaffolded opportunities to reflect and integrate the student experience. The efforts of some IE instructors and some departments to reinforce this kind of work either over the course of a semester or over the course of the student’s career at UMass offer important promise for further enhancing students learning at UMass.
The contribution that General Education experiences make to the other aspects of undergraduates’ connected learning and development are also noteworthy. While those who advocate for the value of General Education experiences may not find these results surprising (since the patterns observed here have long been represented in campus-based and national arguments for the importance of General Education), they should certainly find them affirming. At a recent presentation of some of these results at the 2019 Association for the Assessment of Learning in Higher Education (AALHE) Annual Conference, a participant said, “First-year students should be shown these results—they should see the ways their peers say Gen Ed contributes to their learning!” Indeed, these results offer a useful resource to students, faculty, and administrators alike on the potential value and contribution of the Gen Ed experience.
Another AALHE participant noted that the outcomes (or impacts) of the connections that students identify represent the heart/essence of what higher education institutions and their representatives assert are the outcomes of a University education. These are also the very outcomes institutions are being called to document and demonstrate through assessment. The fact that these impacts emerge from students’ descriptions of their connected learning, unprompted by forced choice survey items or direct questioning of researchers (or instructors) offers an important confirmation of the promise not only of integrative learning, but of having students participate in structured opportunities to reflect on that learning.
The pedagogical usefulness of the mapping tool itself should not be overlooked. In addition to its value in facilitating students’ self-reflection and integration, the tool can help instructors consider the design of their courses as well. In previous workshops, IE instructors have been asked to complete the map for themselves based on their college experiences as a way of introducing them to the tool. Another participant in the AALHE session suggested an additional use—to have Gen Ed instructors map the connections they hope students will make to/from their Gen Ed course (after these instructors see the kinds of connections students are capable of making). Indeed, having this summary of how students use the mapping tool, and the insights their responses provide, offers us a whole new way to think about not only reporting the results of student integration, but also how to use this information and the tool itself for teaching and learning improvements.
The General Education Council (GEC) is currently in the process of conducting its required Quinquennial Review of all Integrative Experience (IE courses) with feedback to departments about their IE offerings anticipated in fall 2019. As a follow-up to this review, OAPA will once again conduct a series of focus groups to better understand integrative learning and the IE. OAPA will replicate this mapping exercise to deepen the data base we have on the particulars of connected learning. In addition, GEC, in collaboration with OAPA and the campus’s Center for Teaching and Learning, will offer workshops and consultation on teaching for integrative and reflective learning to help support departments and individual IE instructors in their efforts to more fully meet the expectations of the IE requirements. The results outlined in this paper, and the mapping tool itself, will be incorporated into instructor workshops and the Gen Ed Council feedback to individual departments.
FOCUS GROUP INTEGRATIVE MAPPING TOOL – WITH AN EXAMPLE OF COMPLETED MAP
The Outcomes of the Connections College Students Make Across their Learning Experiences
The Office of Academic Planning and Assessment (OAPA) conducted a series of 9 Focus Groups where Undergraduates (n=62) were asked to map the connections they saw across their curricular (General Education, the Major) and co-curricular experiences at the University. For those connections they identified, students were also asked to describe the outcomes or impact of that connection. This table identifies the range of impacts students describe and the proportion of all outcomes named that each categories represents. Note how these outcomes range from skill development, to intellectual growth, preparation for the future, and individual development and growth. (Total number of outcomes identified by the 62 Focus Group participants, n=317.)
APPENDIX C: Integrative Learning Focus Group Participation
Includes students representing 29 majors: Accounting, Anthropology, Bachelor's Degree of Individualized Concentration (BDIC), Biochemistry, Biology, Comparative Literature, Communication, Communication Disorders, Economics, Electrical Engineering, English, Environmental Science, History, Hotel Tourism Management, Kinesiology, Legal Studies, Linguistics, Management, Marketing, Nutrition, Operations Information Management, Political Science, Physics, Psychology, Public Health, Social Thought and the Political Economy, Sociology, and Sustainable Field and Farming.
Martha L. A. Stassen, Ph. D.
Associate Provost, Assessment and Educational Effectiveness and
Director, Office of Academic Planning and Assessment
University of Massachusetts Amherst
Association of American Colleges and Universities (Summer/Fall 2005). Integrative Learning. Peer Review, 7, 3.
Cross, K. P. & Steadman, M. H. (1996). Classroom Research: Implementing the scholarship of teaching. San Francisco: Jossey Bass Publishers.
Cuevas, N. M., Mateev, A. G., Miller, K. O. (Winter 2010). “Mapping General Education Outcomes in the Major: Intentionality and Transparency.” Peer Review, 12, 1. Pp. 10-15.
Grindley, C. J. et. al. (Winter 2010). “Pulling it all together: Connecting liberal arts outcomes with departmental goals through General Education.” Peer Review, 12, 1. Pp. 27-29.
Huber, M. T. and Hutchings, P. (n.d.). “Integrative Learning: Mapping the Terrain.” A background paper for Integrative Learning: Opportunities to Connect, an initiative of The Carnegie Foundation for the Advancement of Teaching and Association of American Colleges and Universities (AAC&U).
Kimbal, E. & Loya, K.L., eds. (2017) Using Qualitative Research to Promote Organizational Intelligence. New Directions for Institutional Research, 174. San Francisco, CA: Jossey Bass, Inc.
Lardner, E. and Malnarich, G. (September/October 2009). “When Faculty Assess Integrative Learning.” Change Magazine, pp. 29-35.
Ludvik, M. B. (July-August 2018). “The Neuroscience of Learning and Development: What does that mean for assessment and evaluation?” Assessment Update, 30, 4, pp 3, 13-16.
Maki, P. L. (2004). Assessing for Learning: Building a sustainable commitment across the institution. Sterling, VA: Stylus.
Mansilla, V. B. (January/February 2005). “Assessing Student Work at Disciplinary Crossroads.” Change Magazine, pp. 15-21.
Rogers, R. R. (2001) “Reflection in Higher Education: A concept analysis.” Innovative Higher Education, 26, 1, pp. 37-57.
Suskie, L. (2009). Assessing Student Learning: A common sense guide, 2nd Edition. San Francisco: Jossey Bass Publishers.
Toppo. G. (February 4, 2019) “Colleges experiment with experiential transcripts. Inside Higher Education (Retrieved 2/4/2019 https://www.insidehighereducation.com).
Walvoord, B. E. & Anderson, V. J. (1998). Effective Grading: A tool for learning and assessment. San Francisco: Jossey Bass Publishers.
Werder, C. & Otis, M. M., eds. (2010) Engaging Student Voices In the Study of Teaching and Learning. Sterling, VA: Stylus.
Wiggins, Grant. (1998). Ensuring authentic performance. Chapter 2 in Educative Assessment: Designing Assessments to Inform and Improve Student Performance. San Francisco: Jossey-Bass, pp. 21 – 42.
 A random sample of sophomore, junior, and senior students were sent an email invitation to participate in the focus group and offered $15.00 for their participation. Volunteers were asked to sign a participation agreement and to give permission for OAPA to access their college transcript. See Appendix C for Focus Group participant demographics. The population was disproportionately female and white. Future Focus Group efforts will work to improve the representativeness of participation.
 A note in interpreting these results: students could identify a number of connections to each structured experience, all of which would be counted here. Therefore, the count is duplicative of students across the types of connections they make within each structured experience (i.e., Gen Ed, Major, Co-Curricular).