In this post, I present an overview of a DBR process using Nelson et al.’s Design-based Research Strategies for Developing a Scientific Inquiry Curriculum in a Multi-User Virtual Environment and explain the issues and implications with the work.
Image from Investigating the Impact of Individualized, Reflective Guidance on Student Learning in an Educational Multi-User Virtual Environment by Brian Nelson © 2006.
Social and historical context and Author Info.:
Nelson, Ketelhut, Clarke, and Bowman worked with Dede at Harvard on the River City Project, which was funded by the National Science Foundation. Additional information about the authors can be found here. In his article on DBR, Dede (2004) suggests that the DBR community should engage in substantive collective reflection on setting standards that improve the quality of DBR and on refining innovations so that they matter to the audiences for our research (p. 114). Without standards for determining when to abandon a design approach as unpromising, the DBR field risks being seen as a venue for suboptimal educational strategies endlessly tweaked by their proponents in the hopes of an unlikely breakthrough (p. 108).
Goal of Paper:
Provide a guide to others that want to use DBR methods to investigate a design, development, and implementation process (Nelson et al., 2005, p. 2).
Goal of Nelson et al.’s Multi-User Virtual Environment (MUVE):
Emphasis on MUVE research is in increasing student motivation, self-efficacy, and scientific literacy. MUVE helps to narrow gaps among students by helping all learners reach their full potential (Nelson et al., 2005, p. 13-14).
- Social constructivist environment
- Middle school is the development level where girls tend to lose interest in science (AAUW, 1999; Butler, 2000) so the designers created a lead female role model for girls
- Girls prefer environments that are collaborative in nature (Clark, 1999) so the designers created a world where students work collaboratively to solve health problems in River City (Nelson et al., 2005, p. 3-4).
Can virtual environments simulate real world experimentation and provide students with engaging, meaningful learning experiences that enhance scientific literacy? (Nelson et al., 2005, p. 2).
Implementations/Cycles of DBR:
- First Implementation – Method
- Examined usability, student motivation, student learning, and classroom implementation issues using two public school sixth- and seventh-grade classrooms in urban Massachusetts in 2003
- used Patterns for Adaptive Learning Survey (Midgley, 2000) and a pre- and post-test for content
- teachers responded to a pre- and post-questionnaire on methods and comfort with technology
- Researchers used this data and own observations to analyze the learning outcomes for students and to inform the understanding of how MUVE worked, thus providing information on how to refine the design and reflect on the theoretical foundations (Nelson et al., 2005, p. 4-5)
- MUVE was motivating, e.g. 6/7 experimental students improved to average or above on content, compared to 2/5 control students
- five different hypotheses about health problems emerged, showing that MUVE enabled ability to engage in inquiry in an authentic setting
- MUVE seemed to have the most positive effects for students with high perceptions of their own thoughtfulness of inquiry (Nelson et al., 2005, p. 5)
- Researchers found that gender was not a significant predictor of success in MUVE
- however, on pretest, 6/11 lowest performing students were female
- science self-efficacy of these 6 females increased 7% over the 2-week implementation, also, motivation increased (Nelson et al., 2005, p. 5)
- In initial design, students could not interact with citizens of town. This was changed based on student feedback so they could ask What’s new?
- Students expressed the interest to cover more area quickly, so the ability to teleport to different locations was created
- Students were given the ability to choose their avatar, to enable more self-expression in the world
- All three were ways of increasing students’ psychological immersion in the MUVE, by adding new types of actions, social situations, and participation in the LE (Nelson et al., 2005, p. 6)
- Second Implementation – Method
- Evaluated the student responses to the changes (interactive residents, teleporting map, and ability to choose and change avatar) in a focus group in December 2003
- Observed student interactions, solicited focus group suggustions and conducted exit interview (Nelson et al., 2005, p. 6-7)
- Students needed time to experience the world, they were easily confused by connection of digitized Smithsonian artifacts, they became easily lost, they wanted to access the books in the virtual library, and wondered why their avatars were not also getting sick (Nelson et al., 2005, p. 7)
- Discovered that changes were positive but additional modifications were needed:
- Reorganization of lab book to allow students time to explore the world
- A new section that guides the understanding of digitized artifacts
- Permanent link on interface to interactive map
- Clickable volumes in virtual library to allow students to locate background information on disease
- health meter that would impact on avatar based on location (Nelson et al., 2005, p. 7)
- Third Implementation – Method
- Conducted similar pre- and post-tests to first implementation with two full-scale pilot studies in January and February 2004
- first pilot study was conducted in informal after-school program
- second pilot study was conducted in west coast university laboratory school (Nelson et al., 2005, p. 8)
- Alterations made significant improvements in student engagement and learning outcomes
- When given time to explore, students were able to use the lab book to guide their investigation more readily than previously
- Student increased interaction with digitized artifacts, increasing involvement with curriculum
- Teleporting map increased mobility and allowed students to access more of curriculum, but still complained it was difficult to locate themselves on map
- The virtual library became a popular place to get more information
- The health meter allowed students to have additional information for disease
- Teachers commented that students should be able to conduct experiments in the world (Nelson et al., 2005, p. 8-9)
- Researchers realized the MUVE enabled situated learning, ability to conduct an authentic task, rather than constructivism
- Discovered two variations of MUVE: GSC – guided social constructivist – model of learning in which guided inquiry experiences in the MUVE alternate with in-class interpretive sessions vs. EMC – expert modeling and coaching – model of learning in which expert agents embedded in the MUVE and experts collaborating with teachers in facilitating the in-class interpretive sessions (Nelson et al., 2005, p. 9-10)
- Fourth Implementation – Method
- Randomly assigned treatments to students within a single classroom scaled up to 11 teachers and more than 1000 students in spring 2004
- Included 8 hours of prof. development for teachers, focused on content review, alternative pedagogical strategies based on different theories of learning, facilitation strategies when students are in MUVE, and interpretive strategies for leading class discussion (Nelson et al., 2005, p. 10)
- Preliminary findings show that of 300 students analyzed, students in experimental treatments improved biological knowledge by 32% for GSC and 35% for EMC, while control was 17%
- Knowledge and application of scientific processes improved by 20% for control, 18% for GSC and 16% for EMC
- After conducting experiment, students are asked to write a letter to mayor, more of the lower-performing test students met criteria of providing suggested interventions or future research than students who scored higher which shows that intricate patterns of learning require more appropriate assessment for authentic activities
- Students overwhelmingly positive about tools and avatar choice (Nelson et al., 2005, p. 11-12)
- New LPP – legitimate peripheral participation – model of learning needs to be tested in new implementation
- Use of experiments in MUVE were successful – a mosquito catcher, blood tests, and throat swabs for samples of bacteria
- New map that tracks student movement in world
- Assessing learning needs to be re-conceptualized
- PD feature made available for teachers (Nelson et al., 2005, p. 13)
Issues with this study:
Research team is still planning on conducting formal randomized experimental trials (Nelson et al., 2005, p. 14). However, isn’t DBR supposed to replace formal randomized experimental trials, not run in tandem?
What are the criteria for success in declaring a design finished?
In the MUVE, substantial parts of the design have remained unchanged while other parts have changed based on feedback from initial studies. Findings in study show that students, both boys and girls, have comparable experiences in MUVE. Therefore, additional improvements in the avatar do not appear to be a condition for success. So, Nelson et al. determine that aspect of the design finished (Nelson et al., 2005, p. 14-15).
Nelson et al. provide a good overview of their design process using DBR methodology. Researchers can model future studies off of the process shown here.
American Association of University Women. (1999). Gender gaps: Where schools still fails our children. New York: Marlowe.
Butler, D. (2000). Gender, girls, and computer technology: What’s the status now? The Clearing House, 73, 225-230.
Clark, J. V. (1999). Minorities in Science and Math. Columbus: ERIC Clearinghouse for Science, Mathematics, and Environmental Education.
Dede, C. (2004). If design-based research is the answer, what is the question? A commentary on Collins, Joseph, and Bielaczyc; diSessa and Cobb; and Fishman, Marx, Blumenthanl, Krajcik, and Soloway in the JLS special issue on design-based research. The Journal of the Learning Sciences, 13(1), pp. 105-114.
Midgley, C., Maehr, M. L., Hruda, L. Z., Anderman, E., Anderman, L., Freeman, K. E., Gheen,
M., Kaplan, A., Kumar, R., Middleton, M. J., Nelson, J., Roeser, R., & Urdan, T. (2000). Manual for the Patterns of Adaptive Learning Scales (PALS), Ann Arbor, MI: University
Nelson, B., Ketelhut, D. J., Clarke, J., Bowman, C., and Dede, C. (2005). Design-based research strategies for developing a scientific inquiry curriculum in a multi-user virtual environment. Educational Technology, 45(1), 21-27.