Research Methods & Assessment

Our Research Questions:

  • How can first-year writing curricula effectively develop all students’ intercultural competence and better promote social and academic adjustment for international and diverse domestic students?
  • How can we assess the effects of the curriculum on improving students’ intercultural competence?

Thus, in our pedagogical, empirical research project, we seek not only to develop students’ intercultural competence, but also to develop methods of assessment that work for both intercultural competence and writing skills. This dual-purpose assessment is reflected in our coding scheme. Additionally, we seek to measure skill levels as well as trace the paths by which students develop these skills.

Methods

We employ a mixed-methods approach for our project. The data collected comes from demographic surveys, student reflective writing, semi-structured follow-up interviews, and teacher-researcher experiences as well as from a standardized measure of intercultural competence– the Miville-Guzman Universality-Diversity Scale Short Form (MGUDS-S). Our analytical methods integrate qualitative and quantitative analysis to offer a nuanced, thorough understanding of participants’ writing skills and intercultural competence development during the introductory composition course.

We have developed a rigorous grounded-theory coding scheme for student reflective writing. This coding scheme enables detailed analysis of cultural themes and meta-cognition in student writing. While our coding scheme is primarily qualitative, we are also able to trace frequencies of codes across documents to have a quantitative view of course outcomes and cultural competencies.

After this first level of coding, we map students’ documents onto the Developmental Model of Intercultural Sensitivity (DMIS) to understand larger-scale changes across the entire semester. Our grounded theory coding offers insight into the different ways that students move through the DMIS phases.

The phases of the Bennett Developmental Model of Intercultural  Sensitivity: Denial, Defense, Minimization, Acceptance, Adaptation,  Integration.
The phases of the Bennett Developmental Model of Intercultural Sensitivity.

To map the students’ writings on the DMIS scale, we each assigned the piece of writing a particular phase or phases, and justified that decision. We then met to resolve any differences in evaluation by discussing the evidence together. We share an in-depth look at how we evaluated and came to an agreement on a student’s DMIS profile here.

In our second and third rounds of curriculum implementation (Fall 2017 and Spring 2018), we collected pre- and post-course measures of intercultural competence via the MGUDS-S. The MGUDS-S is a 3-factor measure that uses a 6-point Likert scale. The three subscales measure participants’ level of curiosity about other cultures, empathy, which is the relative appreciation of difference in other cultures, and openness i.e. comfort with difference. These results are analyzed in the aggregate to trace the development of the classes as a whole.

See below for a sample of items from the MGUDS-S.

A sample of items from the MGUDS-S. The first question, "I would like to join an organization that emphasizes getting to know people from different countries," factors into the diversity of contact/curiosity subscale. The second, "Persons with disabilities can teach me things I could not learn elsewhere," is part of the relativistic appreciation/empathy subscale. The third,  "Getting to know someone of another race is generally an uncomfortable experience for me," is part of the comfort with difference subscale. See the whole scale here.
A sample of items from the MGUDS-S. The first question factors into the diversity of contact/curiosity subscale. The second is part of the relativistic appreciation/empathy subscale. The third is part of the comfort with difference subscale. See the whole scale here.

Transparency about our data collection and analytical procedures is important to us. Our coding scheme can be found here:

If you are interested in learning more about our approach to data coding, please contact us at info@writeic.org.

References

Bennett, M. J. (1993). Towards ethnorelativism: A developmental model of intercultural sensitivity. In R. M. Paige (Ed.), Education for the intercultural experience (pp. 21–71). Yarmouth, Me: Intercultural Press.

Bennett, Milton J. (1986). A developmental approach to training for intercultural sensitivity. International Journal of Intercultural Relations, 10(2), 179–196. https://doi.org/10.1016/0147-1767(86)90005-2

Fuertes, J. N., Miville, M. L., Mohr, J. J., Sedlacek, W. E., & Gretchen, D. (2000). Factor Structure and Short Form of the Miville-Guzman Universality-Diversity Scale. Measurement & Evaluation in Counseling & Development, 33(3), 157.