Results

This page reports current results as of April 2020 in four sections:

1. Overall Summary of Results Across All Measures
2. Fall 2017 Reflective Writing Results
3. Findings from Miville-Guzman Universality-Diversity Scale: Fall 2017 and Spring 2018
4. Spring 2017 Pilot Semester Results

Overall Summary of Results Across All Measures

This section summarizes the major trends that we have seen in our data analysis across Spring 2017 and Fall 2017 semesters. For more on our mixed-methods, multi-measure assessment model, please see our Research Methods & Assessment page.

1. Intercultural development along the DMIS scale is not linear: we see many students who exhibit a pendulum model of IC across the semester, particularly when they encounter intercultural challenges. We also note that students can be demonstrating evidence of multiple worldviews at the same time.
2. Our paired sections seem to influence the cultural experiences of domestic students more than that of international students.
3. While international students benefit from the intercultural competence focus, they seem more profoundly affected by the writing focus of the class. This is likely because norms of American college academic writing are new and culturally unfamiliar.
4. While we see intercultural skill gains via the exposure to and interaction across difference, most students do not make leaps and bounds in their intercultural competence during a semester. Those who do tend to integrate their experiences in our class with concurrent experiences in other settings.
5. The most significant intercultural competence-related changes we see in students’ reflective writings include:
   • A more nuanced understanding of culture and cultural difference
   • Stronger recognition of and ability to critically analyze their home culture
   • Positive emotional responses to the experience of cross-cultural interaction from students in both the international and mainstream sections. 

Fall 2017 Reflective Writing Results

Our reflective writing analysis is twofold: first we apply this grounded theory coding scheme to each sentence in each piece of reflective writing. We are able to trace the frequencies of skills and themes across the semester, enabling us to do highly detailed analysis of trends and changes in student writing. Secondly, we complete holistic mapping of each reflective journal on the DMIS scale, to trace the overall trajectory of students’ intercultural competence development.

Here are the main trends from the grounded theory coding:

1. During the treatment semester, we saw a substantial increase in students’ ability to perform critical evaluation.
2. While most students relate some pre-course experiences to the course content during the first six weeks of the semester, international students do this more than domestic students. It is easier for students to make such connections when learning contexts greatly differ (in the case of international students). On a cognitive level, it is more difficult to draw upon these relationships when educational contexts appear similar — i.e., transitioning from American public high school to American public college (in the case of domestic students). 
3. At the beginning of the semester, domestic students focus on more cultural exposure, cultural interaction, and cultural identity in their reflective writing than do international students. This aligns with the quantitative results — the international students come in with higher IC behaviors (though not higher overall IC) and the domestic students spend the first part of the semester catching up and reflecting on newly encountering diversity that they have not previously experienced.
4. Students in the international sections reflect on writing skills much more frequently than do students in the mainstream sections.

In student reflective writing, we observe a significant amount of transfer into the class of prior experiences. As a result of connecting their prior experiences to learning during the first third of the semester, students find deeper intercultural meaning and understanding of previous experiences.

Reflective writing yields better self- and cultural understanding. We see transfer among concurrent contexts during the semester fairly frequently; students will connect concurrent experiences to the class and connect cultural learning in our classes with cultural learning in other places.

However, we see less explicit ability to transfer out of the intervention semester. Imagining specific future transfer is abstract, whereas transfer in and transfer among are concrete. Finally, to truly measure whether students are able to transfer skills from our course into other settings, post-semester, we will need follow-up measures. Unfortunately, none of our Fall 2017 participants chose to complete the follow-up interview. 

Here are the main trends from DMIS mapping:

Throughout the semester, we notice a pendulum pattern of intercultural competence development. (See also Acheson & Schneider-Bean, 2019.) Many students, depending on their relative level of comfort and challenge with respect to the intercultural interventions, swing back and forth among stages. When an intervention significantly challenges a student, we often see that they temporarily retreat to an ethnocentric phase as they cope with an intercultural challenge.

We have noticed that even within DMIS phases, students may be engaging the phase at a more basic or more complex level. This is particularly true for minimization and acceptance. Thus, when conducting intercultural competence assessment, we believe it is important to attend to development within DMIS phases. We are working to identify ways to demarcate low and high acceptance, for instance. What’s also important to remember is that the DMIS is a worldview scale. Our research identifies written, cognitive, and behavioral indicators of DMIS phases, strengthening connections between IC worldviews and how individuals live out these worldviews in real situations.

Through the DMIS mapping, we have observed students working across multiple phases of the scale at once (also documented by Acheson & Schneider-Bean, 2019). It is not surprising that some participants show evidence of  acceptance and reverse defense in the same reflective journal, because certain reflection scenarios can portray the developmental process of learning through multiple stages as participants attempt to bridge cultural differences. Acheson and Schnedifer-Bean (2019) illustrate the successful bridging between cultural differences as resolving “the dissonance between self and other, achieving a dialectic of similarity and difference” (p. 50). This argument further magnifies the complex nature of the construct and the fluid and intricate process of development.

Because this course has a dual focus — academic writing skills and intercultural competence — in some reflective writings, students chose to address their writing development. Although we can track students’ growth in reflective ability (which is applicable to IC), if their assignment addresses writing skills rather than intercultural skills we indicate that it is unmappable. An unmappable journal does not necessarily indicate that a student is unengaged or resistant. In most cases, it simply means that the student was focused on their writing development.

Findings from Miville-Guzman Universality-Diversity Scale: Fall 2017 and Spring 2018

The MUGDS measures three indicators of intercultural competence: 

• Cognitive: Relativistic Appreciation of Difference
• Affective: Comfort with Difference
• Behavioral: Diversity of Contact

We administered the MGUDS at the beginning and end of the second and third semester of curricular implementation. For both FA 17 and SP18, we had 100 survey participants total. The trends for the entire data set were as follows:

1. International students scored higher than domestic students for the behavioral indicator of intercultural competence.
2. International students scored lower than domestic students for the affective indicator of intercultural competence.
3. Difference for both the behavioral and domestic indicators is significant (p < 0.05)
4. There was very little difference between international and domestic students for the cognitive indicator of intercultural competence.
5. Difference for the cognitive variable is not significant.

International and domestic students report similar levels of appreciation of difference: they perceive themselves as equally interested in and appreciative of people who do not share the same cultural background. International students report stronger behavioral indicators of IC, which makes sense given that they are living, every day, in an environment that requires behavioral adjustment and nuance given cultural differences between themselves and other groups on the host campus.

However, international students report lower comfort with difference. Our international students have much more concrete experience with culturally diverse environments, and this practical experience gives many a deeper understanding of the complexities of interacting across difference, and this a lower comfort with difference score. Our mainstream section student sample had less practical experience with culturally diverse environments. Because difference is more theoretical for those students, they may report higher comfort with difference as they are imagining how they think they would feel. That might change if these students had substantial experience living in an unfamiliar cultural environment.

At the aggregate level, we did not observe significant change in any of the factors across the semester (little or no difference in pre and post scores). This may be a limitation of the measure: MGUDS has rare use as a pre/post measure. Additionally, because we did not pair participants in pre- and post- administrations, our inferential statistical testing ability was limited, a further impediment to understanding change at the individual level. In the future, we will trace pre/post scores from individual students (rather than looking only at the whole sample) in order to allow better specific analysis of individual change and of demographic factors. 

Spring 2017 Pilot Semester Results

The first semester of curricular implementation was a small-scale pilot: we ran two sections of the course. Eight students submitted writing for the research project, and two completed follow-up interviews. Half of the participants were international (China, Thailand, and Japan). Half of the participants were domestic (including two students with international parents).

• 6 of 8 participants showed substantive intercultural development
• Students who engaged with the curriculum developed intercultural competence or maintained an already high level of IC.
• Thus, there are students for whom this curriculum achieves its outcomes
• It turns out, there are many paths to intercultural development.
• Qualitative analysis helps us see how engagement with curriculum develops intercultural competence and writing proficiencies.

A small-scale pilot (while limiting the initial results in some ways) was helpful because it:

• allowed us to examine data recursively, necessary for the development of a solid grounded-theory coding scheme
• offered the opportunity for a deep-dive into different student profiles
• gave us a closed set of 40 reflective texts with which we could come to intercoder agreement
• directed minor curricular revisions before training more teacher-researchers to implement the curriculum the following year

We notices three key trends in our pilot data grounded theory coding:

• Critical evaluation increases across the semester, and parallels the DMIS advancement. Critical evaluation skills are instrumental for building cultural knowledge (Deardorff, 2004).
• Students have a high frequency of the prior conditions code during their first journal, demonstrating a consistent effort to draw connections between life experiences before the course and the content of the course.
• In the final course reflection, students tended to emphasize writing skills more than intercultural skills, demonstrating an awareness of the university/departmental context of the interventions.