Issue Seventeen

Grinnell College students examine a double-pen slave cabin in Vacherie, Louisiana.
0

Using Virtual Reality to Expand Teaching and Research in the Liberal Arts

Abstract

Grinnell College has established a lab for teaching undergraduate liberal arts students the hard and soft skills necessary to develop extended reality (XR) experiences. This lab helps the College respond to external social and economic pressures while retaining its core liberal arts values. Within the lab, students develop the metacognitive skills, technical training, and problem-solving strategies that will make them competitive candidates in a global twenty-first–century marketplace. For other institutions interested in implementing an XR lab on their campuses, we provide key takeaways in the following areas: how we launched our lab, the funding instruments that support lab activities, the hardware and software used to develop XR experiences, the development team structure and member responsibilities, lessons learned from the pilot project, and projects currently in development.

Background

Grinnell College, like many small liberal arts colleges, has questioned how to remain robust and relevant in a digital age (Selingo 2013; 2017). We value knowledge for its own sake, social justice, and critical thinking; yet, we accept responsibility for equipping our students with the skills that allow them to adapt to a world of rapidly changing professional opportunities. We refused to sacrifice the former for the latter. Instead, we created a learning environment to promote both our traditional values and practical job skills. In our lab, when students research, create, and evaluate extended reality (XR) experiences, they develop the technical, social-awareness, and problem-solving skills that make them attractive candidates for twenty-first–century jobs while exhibiting liberal arts sensibilities. By developing marketable skills within the framework of core liberal arts questions and experiences, the College moves toward a future in which our educational offerings are both highly relevant and eminently sustainable.

Various characteristics of and cultures within the institution have influenced how the College has responded to the pressures of a changing academic and digital landscape. Grinnell College is a small, residential, undergraduate-only liberal arts college in rural central Iowa. The College was established in 1846 with a basis in individual intellectual pursuit for the betterment of humankind that has remained strong to today and is in evidence with the individually advised curriculum. The teaching culture is centered around small, face-to-face, discussion-based classes that explore topics according to professor interests. The College includes disciplines in the arts, social sciences, and natural sciences; but we do not have professional programs such as journalism, business, and nursing, perhaps because corporate or practical pursuits are viewed as less intellectually rigorous. The College also functions with a conservative curriculum and traditional views of faculty, who are the College employees and experts primarily responsible for helping students to grow in their own knowledge. Challenges arise when new developments conflict with traditional conditions. For example, we have seen the professionalization of College staff, with highly educated, non-faculty employees taking on more significant roles in students’ educational experiences. Additionally, we have seen changes in what students need and want from their college experience to help them succeed beyond school. Similar to other institutions and labs developing projects in XR, the College wrestles with how to remain true to our essential values while accommodating emerging needs (Szabo 2019).

The Grinnell College Immersive Experiences Lab (GCIEL) emerged from discussions at the administrative level, which identified a need to synthesize a twenty-first–century liberal arts education using emerging digital visualization technologies. GCIEL is an interdisciplinary community of inquiry and practice that allows students, faculty, and staff at the College to explore the liberal arts through XR technologies (Brown, Collins, and Duguid 1989; Wenger 1998; Wenger, McDermott, and Snyder 2002). XR is an umbrella term encapsulating immersive technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR). Of these technologies, lab activity started with a focus on developing VR experiences that completely immerse the user in a simulated three-dimensional (3D) environment (Bailenson 2019; Greengard 2019; Rubin 2018; Jerald 2015); we plan on expanding into AR and MR in the future.

Participating in the hands-on process of developing VR experiences has resulted in educational benefits for students. First, students gain critical-thinking and technical skills. When working in project teams to create immersive digital content, students experience an authentic development environment using industry-standard hardware and software, which prepares them to succeed in a rapidly changing job market. From a liberal arts perspective, the development process challenges students to explore deep questions and make interdisciplinary connections. The research required for developing culturally sensitive, ethical, and historically accurate immersive digital content is both demanding and comprehensive. Compared to research methodologies privileging linear subject matter presentations, such as a term paper or a video, research for VR projects compels students to consider how elements of their chosen topic function together as an interconnected, object-oriented activity system (Engeström, Miettinen, and Punamäki 1999; Jonassen and Rohrer-Murphy 1999). To do this, students must consider multiple context-specific variables for the system they investigate, how these variables interact within historical, spatial, and social contexts, and how end users will ultimately interact with the variables in an VR environment. Second, students develop soft skills including communication and collaboration. Interdisciplinary teamwork between students, faculty, and staff is a key feature of the problem-solving experience and establishes a collaborative knowledge-generation framework. The faculty role shifts from a lecturer focused on content coverage to a coach who guides students as they navigate the “real world” challenges they encounter. Staff member roles shift from assistant to technical advisors and mentors. Student roles shift from being passive recipients of knowledge to co-creators in the learning experience. These shifts allow team members to learn from each other as they integrate their own discipline knowledge and methods into the project.

Narrative

Pedagogical approaches

Inspired by Jonassen’s concepts about teaching for solving ill-structured problems and active learning (Jonassen 2000), GCIEL’s pedagogical practices guide students through a problem-solving process in which they integrate several content domains and negotiate the unpredictable paths that emerge along the way. Jonassen, Carr, and Yueh (1998) conceptualize technology as knowledge construction “Mindtools” that students learn with, not from. Using this framework, GCIEL allows learners to function as designers using VR technologies to explore their subject matter, critically evaluate the content they are studying, and represent their knowledge in a meaningful way. This approach challenges certain traditional liberal arts attitudes about what kinds of learning are valued. While the liberal arts shies away from anything that resembles “vocational” training, GCIEL fully embraces training in practical hard and soft skills as an integrated part of content knowledge acquisition and critical thinking. We recognize skills such as software and hardware competence, digital file management, project and time management, troubleshooting, and team communication as foundations for the higher-order thinking skills that liberal arts college graduates will need throughout their lives. Thus, we intentionally teach these competencies alongside more traditional humanities topics rather than hope that learners acquire them incidentally through trial and error. In this way, GCIEL builds effective learning experiences that result in students thinking critically about VR technologies and using these technologies to examine, interrogate, and represent core liberal arts topics.

GCIEL seeks to optimize learning by maintaining a flexible, inclusive, and student-centered educational environment in which instructors “pay close attention to the knowledge, skills, and attitudes that learners bring” (National Research Council 2000, 23) to the research and development experience. By treating learners “as cocreators in the teaching and learning process, as individuals with ideas and issues that deserve attention and consideration” (McCombs and Whistler 1997, 11), GCIEL allows students to take an active role in reinventing their liberal arts experience. Heeding advice that “supplementing or replacing lectures with active learning strategies and engaging students in discovery and scientific process improves learning and knowledge retention” (Handelsman et al. 2004, 521), GCIEL emphasizes hands-on, authentic learning. Students develop products aligned with their interests and wield digital technologies in socially conscious ways within widely-ranging content domains. Students, in a focus group interview, viewed the experience as highly beneficial to their overall education. One student team member particularly valued the opportunity to learn “interdisciplinary communication on a long-term project” of a scale and duration that far exceeded what could be done within just one semester of a class (GCIEL Focus Group 2018). Another student observed that one of the most important parts of the project was how, “It feels like we’re on a team with our bosses…instead of it being very much top down” (GCIEL Focus Group 2018).

When developing VR experiences in GCIEL, Grinnell College students cultivate skills which help them adapt to rapidly-changing professional opportunities and contribute to others’ learning. Because the student-developed VR products are released as open educational resources (OER) under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License, students anywhere in the world can augment their education by using and contributing to the custom-built immersive experiences. As an educational tool, VR is particularly useful for enhancing spatial knowledge representation, promoting experiential learning opportunities, increasing motivation and engagement, and contextualizing the learning experience (Dalgarno and Lee 2010; Steffen et al. 2019). The embodied experiences in VR have been found to promote empathy (Herrera et al. 2018; van Loon et al. 2018) and perspective taking (Ahn, Le, and Bailenson 2013; Yee and Bailenson 2006), both of which are particularly important within liberal education contexts that focus on preparing students to deal with complexity, diversity, and change and to promote social responsibility (“What Is Liberal Education” n.d.). The VR projects developed in GCIEL (detailed below) offer new ways to engage students in various learning experiences across widely ranging domains from history (Ijaz, Bogdanoych, and Trescak 2017; Taranilla et al. 2019; Wood, William, and Copeland 2019; Yildirim, Elban, and Yildirim 2018), second language and culture acquisition (Blyth 2018; Dolgunsoz, Yildirim, and Yildirim 2018; Legault et al. 2019), and mathematics (Sundaram et al. 2019; Nathal et al. 2018; Putman and Id-Deen 2019).

Funding instruments

Dr. David Neville, a Digital Liberal Arts Specialist at Grinnell College, spearheaded the GCIEL initiative. Dr. Neville’s background in instructional technology and design, digital game-based learning, 3D modeling, and Unity development gives him the expertise to serve as the director of the lab and act as the technical advisor on all GCIEL projects. In Fall 2016, Dr. Neville received a $10,000 planning grant from Grinnell College’s Innovation Fund (IF) to investigate the feasibility of implementing a VR lab at the College. He used the grant funds to educate faculty and staff, bring in external experts, purchase equipment, and hire students with the following financial breakdown: First, about 45% of the IF monies supported participant stipends for a summer workshop led by Dr. David Neville and Dr. Damian Kelty-Stephen. This workshop helped 10 faculty and staff members at Grinnell College learn how to use VR technologies in a curricular setting. Tweets about the workshop are archived under the #gcielsw17 hashtag. Because more people showed interest in the topic than originally anticipated, the Center for Teaching, Learning, and Assessment provided an additional $1,920 to support the extra participants who registered for the workshop. Second, about 4% of the funds paid for VR experts to present their research at the workshop. Dr. Joel Beeson, Associate Professor in West Virginia University’s Reed College of Media, presented his work on the Bridging Selma Project and the Fractured Tour app. Dr. Glenn Gunhouse, Senior Lecturer of Art History in the School of Art and Design at Georgia State University, presented a general introduction to his cultural heritage projects in virtual reality, with observations about how the technology can provide access to otherwise inaccessible objects of study (Sinclair and Gunhouse 2016). Third, roughly 15% went towards purchasing new VR hardware and software (e.g., Dell Precision 5810 with NVIDIA Quadro M5000 GPU, Oculus Rift, and Wacom tablet). Finally, about 37% of IF monies paid wages for students working on the development team for the lab’s first VR project. Supporting student development work on this project, the Institute for Global Engagement at Grinnell College contributed $6,200 to fund a one-week visit to Louisiana for site-based research.

In Fall 2018, GCIEL received $144,000 for a three-year pilot project IF grant. These monies were utilized in ways which allowed the lab to expand its influence on campus and widen its project portfolio. First, about 10% of the IF award supported a new XR speaker series, which involved bringing academics and industry representatives to Grinnell College. These experts presented on the current state of XR in their fields, shared their vision for how XR will grow in the future, and demonstrated how a liberal arts education can prepare students for a career in XR. Students gained networking opportunities with these influential thought and industry leaders. Second, about 78% of the award paid personnel costs for the development teams, including student wages (72%) for four development teams and site-based research costs (6%). Finally, GCIEL used the remaining 12% of IF monies to purchase software and hardware necessary for developing VR experiences. These included software licenses, online training, digital assets, an additional VR-capable workstation with associated hardware, and an HTC Vive. This IF support ends in Summer 2021, at which time the College will consider whether to provide permanent institutional funding for GCIEL.

Team structure and technology pipeline

After confirming faculty interest in VR at the summer workshop, we began to assemble a VR development team for a pilot project. Forming the team proved that our small liberal arts college had sufficient resources and talent on site to shoulder an ambitious digital project. This was a considerable achievement considering that larger game design studios typically have development teams with hundreds of members, each contributing deep subject-matter knowledge, software and programming expertise, visual and 3D design capabilities, technical support, and project guidance. Echoing the development team experience that students might encounter in the XR industry, we envisioned our scaled-down version of the team to include a faculty adviser, a technical staff member, and students functioning as Subject-Matter Experts (SMEs), 3D Artists, and Unity Developers. The faculty adviser would come from a field related to the project’s topic and focus on helping students learn the subject matter. The technical staff member would help students manage the project and learn essential technological skills. Each student role had unique requirements.

Typically, we recruited the project SME through the faculty member, who invited an advanced undergraduate student major from their discipline. This student may have demonstrated relevant skills while working with the faculty member on prior academic projects. Only the SME had a personal invitation to join the VR development project, unlike the 3D Artist and Unity Developer, who were recruited using an open application and interview process through the student employment portal. The SME was responsible for (a) finding, evaluating, and utilizing resources to guide project development; (b) disseminating research findings to other team members in an understandable manner; and (c) leading the team’s process and progress. We considered giving SMEs more responsibilities in directing and managing a project in order to offset the marginalization that SMEs from humanities fields may feel during the coding-heavy portions of the project when they lack technical experience compared to their teammates. This may require the SME to learn and apply instructional design theories and models, Agile software development methods (e.g., Scrum), and the Unified Modeling Language (UML) to the project.

We selected a 3D Artist based on this individual’s technical experience or interests. The Artist needed to be able to use software such as Autodesk 3ds Max, Substance Painter, and Adobe Creative Cloud software platforms (e.g., Illustrator and Photoshop), and also be willing to engage in 3D modeling and texturing, UV mapping and unwrapping, model rigging and animation, developing concept art, and storyboarding. We chose 3ds Max because it is an industry-recognized tool, and familiarity with this system should better prepare students for internships and employment opportunities. The Artist is primarily responsible for 3D asset development and animation in 3ds Max and texture creation in Substance Painter. Artists may also contribute to other aspects of the project such as writing entries for a project blog, creating turntable animations of project assets for the GCIEL YouTube channel, or presenting to students and faculty about the lessons learned during project development. The Artist’s workflow included (a) evaluating primary and secondary resources identified by the Subject-Matter Expert and any data collected through site-based research; (b) utilizing these resources and data to create 3D models and animations in 3ds Max for the VR experience; and (c) importing the FBX file of the models into Substance Painter and Unity. Within Substance Painter, the Artist uses the physically based rendering and shading (PBR) capabilities of the software platform to create albedo transparency, specular smoothness, normal, occlusion, and emission texture maps. Within Unity, the Artist creates materials with a standard specular shader and then applies the texture maps to the 3D models. The Artist may also create lighting and particle effects for the VR experience inside Unity.

We selected Unity Developers based on their technical experience or interests in the Unity integrated development environment (IDE), object-oriented design and programming principles, Unity script writing in the C# programming language, and version control and collaboration with Git and GitHub. The Unity Developers were primarily responsible for writing the code that drives the VR experience; the information provided from the SME and the team’s site-based research informs how the Unity Developer programs the functionality of the experience. The Unity Developer also needed to be familiar with or willing to learn the SteamVR Unity plugin, which allows Unity to interact with and receive input from attached VR hardware (e.g., Oculus Rift-S and HTC Vive). The workflow for the Unity Developers entailed (a) brainstorming the interactivity in the VR experience; (b) bodystorming the experience with the team to flesh out what the user experience (UX) should look and feel like and how users would potentially interact with the experience; (c) utilizing whiteboxing and method stubbing to quickly make experience prototypes; (d) running through prototype tests of the VR experience to elicit user feedback; and (e) producing a minimal viable product (MVP) that could be used to secure external grant funding or to gather data in research experiments. The MVP is a version of the VR experience with just enough features to demonstrate proof of concept and provide feedback for future product development. We uploaded major versions of the VR experiences and their MVPs to the lab GitHub repos to serve as our backups, include in students’ portfolios, and share open source resources with other educational institutions interested in developing VR experiences.

Pilot project

Dr. Sarah Purcell, the L. F. Parker Professor of History at Grinnell College, and Dr. David Neville, Director of GCIEL, launched the pilot project in late Spring 2017. They hired four students for the project development team including history student Sam Nakahira as the SME, studio art student Rachel Swoap as the 3D Artist, and computer science students Zachary Segall and Eli Most as the Unity Developers. The project began as an ambitious attempt to build a VR experience of the Uncle Sam Plantation, a nineteenth-century Louisiana sugar plantation. Unfortunately, the project had an unintentionally slow, rolling start as two team members went to study in Europe for a semester. In Summer 2017, Sam Nakahira worked with Dr. Sarah Purcell to research and write about the Uncle Sam Plantation and its inhabitants, the 19th-century sugar production methods, and the historical context that would guide the team’s development process. During Fall 2017, Zachary Segall began prototyping the VR experience, deepening proficiency in the Unity IDE, and choosing a VR interaction system for the project. Based on development problems at the time with the Virtual Reality Toolkit (VRTK), Zachary Segall chose SteamVR v. 1.2.3 as the VR interaction system. With all the team members back on campus by early 2018, the full development team visited Louisiana in January 2018 for site-based research (see Figure 1). They met immediately afterwards to begin building the VR experience. At this point, we encountered a brand new series of challenges.

Grinnell College students examine a double-pen slave cabin in Vacherie, Louisiana.
Figure 1. Site-based research. Members of the GCIEL student development team (from left to right: Sam Nakahira and Zachary Segall) conduct site-based research of a double-pen slave cabin at Laura Plantation in Vacherie, Louisiana (January 2018). Photo by David Neville.

Initially, the team intended to simulate the 19th-century sugarhouse and steam-powered sugar mill that had operated on the Uncle Sam Plantation. The team could access the plot plan and survey data of the plantation mansion and larger outbuildings (see Figure 2); however, we had difficulty locating documentation for the sugarhouse and sugar mill. Additionally, modeling and animating the sugar mill exceeded the skill level of our 3D Artist, who was new to the 3ds Max modeling software. We soon realized the project’s scale was far beyond what we could reasonably handle with our current resources and timeframe; so, we opted to start small and then iterate toward the larger-scale goal.

Plot plan of the Uncle Sam Plantation made by the Historic American Buildings Survey (HABS) in 1940.
Figure 2. Plot plan of the Uncle Sam Plantation. Plot plan of the Uncle Sam Plantation (Leimkuehler 1940) made by the Historic American Buildings Survey (HABS) in 1940 and one of the historical documents utilized by the GCIEL student development team for developing the VR experience.

To provide a common ground for historical understanding across team members, all participated in Dr. Sarah Purcell’s two credit-hour guided-reading course on the history of American slavery that focused on Louisiana, museum curation, and public history theory. Course readings inspired the new direction for our project. To honor the humanity of the enslaved people who lived on the plantation, the team decided to refocus the project on teaching users how to interpret the home life of the enslaved. Having agreed on a new approach, the team began recreating a double-pen slave cabin, which our site-based research provided sufficient data for a digital model (see Figures 3 and 4), and designing plans for structuring the VR experience itself (see Figures 5 and 6).

The 3ds Max interface showing a high quality render of a double-pen slave cabin.
Figure 3. The 3ds Max interface. This screenshot shows a high quality render of the double-pen slave cabin currently in development. The render uses the NVIDIA Mentalray Renderer with Sunlight and Daylight Systems set to 7 AM on 21 October 1868 in Baton Rouge, Louisiana. A turntable render of this 3D model is available on the GCIEL YouTube channel. Screenshot and model by David Neville.
The Unity interface showing models of the double-pen slave cabin and the plantation mansion.
Figure 4. Importing models into the Unity game engine. GCIEL student development team members import the models they developed in 3ds Max into the Unity game engine for programming user interactivity. The HABS plot plan is used as a reference image to ensure proper scale of the VR experience and approximate distances between its features. Screenshot by David Neville.
Students on the GCIEL development team discuss the Uncle Sam Plantation VR project.
Figure 5. Development team discussion. GCIEL student development team members (from left to right: Sam Nakahira, Zachary Segall, and Rachel Swoap) reflect on how to reconstruct the lived spaces of the plantation complex as authentically and sensitively as possible, and brainstorm possible directions that a VR experience could take. Photo by David Neville.
Experience flowchart for the Uncle Sam Plantation VR project.
Figure 6. VR experience flowchart for the proposed structure of prototype Uncle Sam Plantation VR experience. Image by David Neville.

We came to four critical insights as we found ourselves frequently adjusting our development pipeline. First, we needed to design the curricular content around the problems arising in the project. We initially held the course meetings separate from project-development meetings to prevent talk about the project’s technical details from overshadowing discussion about the historical topics. However, we discovered that the course topics could easily become divorced from and less relevant to the specific historical challenges that emerged naturally from the project work. We actually needed to let the project work and the historical topics inform one another in real time. Second, working together closely as an interdisciplinary team to identify problems and brainstorm solutions was essential. At first, everyone worked on their own and within their own disciplinary perspective in a disconnected divide-and-conquer approach. This left little overlap for noticing how the separate parts were not quite fitting together as a whole. Had the team been working together more closely, we could have saved time by realizing sooner that researching the sugar production was a dead end. Third, we needed alignment between the project goals and the team members’ skills, especially for technology-heavy projects. If the team members did not already have the skills when they started, the team needed to re-think the goal or to devote time and resources to help the team members acquire the necessary skills.

Fourth, and perhaps most crucially, we discovered that team members must adapt themselves to different disciplinary expectations and research styles. In particular, the approaches used in computer science and history were quite different and led to some tension. Computer science professionals reduce a design problem into small, manageable components and then rapidly iterate through prototypes to find the most effective and efficient solution. In contrast, history professionals start with library and archival research to shape the research questions, then they produce a polished document with the conclusions about the subject of inquiry. Risking oversimplification, it was as if the computer science approach tried building a complex whole from smaller, simpler parts and the history approach tried contemplating a complex whole to extract a few smaller, concrete understandings. Puzzling over how to merge these distinctly different problem-solving approaches, we began implementing a new project workflow based loosely on Scrum with two-week sprints (Ashmore and Runyam 2015; Deemer et al. 2009; Rubin 2013). This process provided a common framework for approaching the problem by breaking the whole project into smaller chunks so the SME would have a more narrow issue to explore and the Unity Developers had more tangible components to start building.

Scrum is a software development project framework that embraces iterative and incremental practices, collaborative teamwork, and self-organization. A Scrum sprint is a fixed space of time in which a product of the highest possible value is created. The sprint began with team members meeting in the GCIEL space to brainstorm and assign project tasks (see Figure 7). Members tracked their progress on these tasks using Trello, a web-based project management platform, and a whiteboard located in the team space and collaboratively addressed questions as they arose (see Figures 8 and 9). At the end of the sprint, team members met to debrief, identify new areas that needed to be developed, and reflect on what they learned with regard to both the historical subject matter and project technical skills. At appropriate stages in developing the VR experience, the development team included prototype testing in their workflow to ensure the end-users would have a favorable experience (see Figure 10). By involving all team members in this process, we improved the interdisciplinary communication and problem solving.

Students on the GCIEL development team launch a Scrum sprint for the Uncle Sam Plantation VR project.
Figure 7. Two-week Scrum sprint. The start of a two-week Scrum sprint utilized the community-building spaces of the Digital Liberal Arts Lab (DLAB) at Grinnell College, as well as the Media:Scape technology available there. GCIEL student dev team members (clockwise around the table): Rachel Swoap, Sam Nakahira, Zachary Segall, and Eli Most. Photo by David Neville.
The Trello interface showing lists and cards used for managing the Uncle Sam Plantation VR project.
Figure 8. High-tech project management. Trello, a web-based project management platform, was critical for implementing a Scrum framework that included brainstorming new ideas for the project and who was in charge of completing assigned tasks. Screenshot by David Neville.
The whiteboard in the GCIEL workspace functions as a Scrum board.
Figure 9. Low-tech project management. In addition to Trello, a Scrum board located in the GCIEL space helped student development team members keep track of project-related tasks, who they were assigned to, and their status. Photo by David Neville.
Prototype testing a VR experience in the GCIEL workspace.
Figure 10. Prototype testing. Zachary Segall tests a prototype VR experience with an unidentified Grinnell College computer science student. User testing allows GCIEL development teams to think critically about their own work. Photo by David Neville.

Second-generation projects

Having learned valuable lessons about the VR design process through the pilot project, GCIEL moved forward with three new VR projects spanning the liberal arts disciplines at Grinnell College, including recreating a Viking meadhall, creating an environment to help students visualize mathematical ideas, and creating an immersive experience to teach German language and culture.

Dr. Tim D. Arner, Associate Dean and Associate Professor of English, and Dr. David Neville lead the Envisioning Heorot Project that is building a VR experience of Heorot, the meadhall from the Old English poem Beowulf where much of the narrative happens. This immersive experience is modeled on archeological excavations of meadhalls in Denmark, England, and Iceland (see Figure 11) and on accounts from historical and poetic records from the early Middle Ages. Grinnell College students involved in the project include Ethan Huelskamp, Joseph Robertson, Maddy Smith, Anna Brew, Brenna Hanlon, Zoe Cui, Tal Rastopchin, and Michael Andrzejewski. The team plans to fill the VR meadhall with people and objects from the poem in order to help the participants exploring the space sense how the room’s layout contributes to its function as a political and social arena. The Envisioning Heorot Project will help student researchers and people reading Anglo-Saxon poetry, especially Beowulf, to understand how such civic spaces functioned in Anglo-Saxon and medieval Scandinavian culture and helped shape Anglo-Saxon social structures. While building or exploring this virtual space, students will learn to analyze how the meadhall functions in Beowulf and its analogues, to locate northern European cultures within a global network of trade and cultural influence, and to consider how movement through physical space is defined by and reinforces social roles in a particular cultural context.

Grinnell College students conducting site-based research at the Reykjavik City Museum, Iceland.
Figure 11. Site-based research in Iceland. Site-based research in Iceland and Denmark has been invaluable for students working on the Envisioning Heorot Project: Development work in 3ds Max and Substance Painter has been strongly influenced by findings and impressions made on these trips. Here students (from left to right) Ethan Huelskamp, Joseph Robertson, Maddy Smith, and Megan Gardner, examine a Viking hearth in Iceland with a representative from The Settlement Exhibition at the Reykjavik City Museum, Iceland. Photo by Tim Arner.

Dr. Chris French, Professor of Mathematics, and Dr. David Neville lead the Math Museum Project, which allows participants to explore and interact with mathematical ideas in VR. Grinnell College students involved in this project are Nikunj Agrawal, Ziwen Chen, Alexander Hiser, Yuya Kawakami, HaoYang Li, Robert Lorch, Tal Rastopchin, Lang Song, Charun Upara, and Hongyuan Zhang. This project is inspired by the mathematical models from the late 19th century when mathematicians partnered with industrialists to model new kinds of surfaces out of plaster, cardboard, or wire. These models brought new developments in algebraic geometry and new notions of non-Euclidean geometry. Immersed within the virtual Math Museum, students can interact with visualized mathematical concepts thereby experiencing greater enjoyment and comprehension of mathematical ideas.

In one room of the virtual museum, players walk around on a large ellipsoid surface, so they experience the shape in much the same way as an insect might move around on a plaster model. The player can find the umbilic points of the shape by using a tool that measures the curvature of the ellipsoid at the current location whenever the player triggers the measuring device. Another room is inspired by models created by the German mathematician Kummer. In this space, the player can manipulate a surface by adjusting certain parameters and then can watch how the surface evolves. The player’s task is to find the values for the surfaces that Kummer built. In a third room, the player must assign colors to the vertices of a graph consisting of edges and vertices so adjacent vertices take different colors. The goal is to use the minimal number of colors. This activity teaches the notion of the chromatic number of a graph. Also, students are currently developing another room in which the player learns about graph isomorphisms by manipulating the vertices of a graph to make it look like another graph.

Dr. David Neville leads the German VR Project, a game for teaching environmentalism in authentic German linguistic and sociocultural contexts. Originally developed as a flat screen 3D game focusing on glass recycling and waste management systems in German public spaces, Zachary Segall and Eli Most ported the game in 2018 to create an alpha-level VR prototype (see Figure 12). Grinnell College students involved in the project include Savannah Crenshaw, Martin Pollack, Yinan Hui, Bojia Hu, Jin Hwi, Tal Rastopchin, and Michael Andrzejewski. Research on the 3D game found that goal-directed player activity provided learners of a second language and culture with a more nuanced view of the activity systems that constitute a target culture, and also apparently influenced how learners invoked and structured language in order to describe these systems (Neville 2014). The VR version of the game will expand the scope of the 3D game by including more narrative to situate the user in an authentic German cultural situation and more in-game tasks related to recycling and waste management practices. We hope that increased immersion and sense of presence in a completely virtual environment will target greater learning outcomes in second language and culture acquisition, and perhaps even realize outcomes that were not discovered in the 3D version of the game.

Screen capture from the German VR project showing a German public space, a beer bottle, and a VR hand controller with directions in German.
Figure 12. Screen capture from the German VR project. The German VR project situates second language and culture acquisition within authentic sociocultural contexts and activities. Screenshot by David Neville.

Next steps

We are currently refining the teams’ workflows to use Scrum methods for project management and incorporating problem-based learning theory to intentionally teach metacognitive skills (Barrows 1996; Edens 2000; Hmelo-Silver 2004; Dunlop 2005; Yew and Schmidt 2011). A victim of our own success, we face a number of challenges while scaling up the lab to support multiple VR projects simultaneously. It has been difficult to find a dedicated physical space on campus which can support a growing community of practice. As a result, GCIEL’s work remains somewhat decentralized. It also remains to be seen how much these discrete cross-curricular VR projects will transform Grinnell College’s core curricula. Likely, GCIEL’s future projects will rely on external grant support, and it may be difficult for small-scale liberal arts teams to compete with large R1 research and development labs for funding. While we are excited to see our established team members graduate and move on to high-powered tech jobs and graduate schools, this leaves recurring gaps in our project teams, so we must constantly train new students to join the project teams. Successful project teams need consistent faculty and staff time and attention; yet, College employees find themselves increasingly burdened with competing responsibilities. Overcoming these challenges depends on our ability to convince the College to change some traditional structures and to provide sufficient time and resources for experimentation. Success is not guaranteed, but we believe the effort is worthwhile.

The future of GCIEL beyond our grant funding is still in discussion. As a well-resourced institution with an individually advised curriculum, Grinnell College has a few options that we can harness to secure GCIEL’s future. For example, the Writing Lab pays student writing mentors out of their general operations budget and these students do not receive academic credit, though they do take an introductory writing course to ensure they have the necessary skills. GCIEL could adopt a similar model and teach an VR basics course to develop a pool of potential student employees as VR mentors. Another possibility is integrating the lab into existing or emerging curricular structures. VR project development would fit most seamlessly into the Mentored Advanced Project (MAP) structure as a group research project supervised by a faculty member. These MAP experiences allow students to register for 2- or 4-credit MAP research credits and work closely with faculty advisers on independent research projects. We might also be able to utilize the “Plus 2” option, which allows professors and individual students enrolled in a regularly scheduled course to plan work that would go beyond the standard syllabus. GCIEL and student VR projects may also find a place within the emerging Digital Studies Concentration or the new Film and Media Studies Program. Grinnell College’s concentrations typically involve a cross-departmental listing of various courses that meet the concentration’s themes and goals, but GCIEL could provide the seed for a concentration-specific seminar that is listed as a requirement or additional way for the students to complete credits towards the concentration. Ultimately, we want to find ways to leverage the benefits of housing GCIEL within the curriculum (e.g., rewarding students with class credits and guaranteed team members) along with the benefits of being independent from the curriculum (e.g., freedom from semester limits and ability to form multidisciplinary collaborations with skilled students, staff, and faculty). Fortunately, Grinnell College has a history of offering student learning opportunities that take many forms, including those that exist outside of traditional classroom environments.

We think all these efforts will pay off in the long run. Opening the traditional classroom format to integrate technological expertise and domain-specific content across disciplinary divides will expand student assessments beyond term papers to include scholarly products that will excite and engage a new generation of scholars in the twenty-first century. We will also have to ask: what is the best way to assess learning outcome achievement for interdisciplinary projects related to creating VR experiences? Can we identify meaningful learning outcomes we should expect of all students, such as project management and effective communication? Do we need to assess students on their domain-specific skills and knowledge, such as software troubleshooting, graphical design, or archival research? Who would be responsible for designing and evaluating these assessments? How do we more closely integrate staff and faculty roles in collaborative curriculum design, which breaks down the traditional barriers between faculty and staff roles? How do we challenge College organizational structures to harness staff expertise alongside faculty domain knowledge?

Learning from the successes of vocational and professional schools, we can reinvigorate liberal arts education with hands-on cooperative training, yet retain the focus on our traditional values that makes us unique. This new model could help to transform liberal arts institutions into laboratories for innovation in solving twenty-first–century problems. In the end we believe liberal arts graduates can—and should—have the best of both worlds: knowledge and the skills to apply it. 

Key Takeaways

  • Complex projects, especially ones using technology, require teams consisting of people with different technical and subject-matter competence. These projects provide excellent opportunities for interdisciplinary collaboration and teaching.
  • To develop transferable skills and knowledge, model the project experience on “real-world” structures. This includes treating student collaborators as equals who participate in decision-making and receive compensation (e.g., stipends or academic credit).
  • Time-intensive projects will require focused, concentrated effort by team members. These projects may require institutional support for faculty involvement (e.g., reassigned time) and students to commit at least 10 hours a week to project development.
  • Long-term, complex projects benefit from a permanent physical space that is equipped to support the technology, comfortably hold team meetings, and accommodate team members’ work styles, including access outside of business hours.
  • The project curriculum must provide team members with the necessary prerequisite technical and subject-matter knowledge to start the project, and it must also be flexible enough in time and resources to adapt to questions that emerge during project development. As VR projects require new ways of configuring faculty-staff-student interaction and budgets to support developments, they provide excellent opportunities for institutional growth and external funding.
  • When properly configured teams work on developing well-designed VR experiences, students learn valuable skills related to communication, self-directed learning, attention to detail, problem solving, negotiation, and time management.
  • Development team members need to be well-versed in the ethical, psychological, and pedagogical affordances of VR and how these impact the project.
  • Start small with complex projects and iterate towards larger goals.
  • Open lines of communication between all team members—staff, faculty, and students—are essential to project success. Avoid isolation by encouraging teammates to pair up, even when working on components that traditionally involve many hours of individual work, such as archival research or programming. In this way, teammates can learn from the others’ processes. This supports cross-training and allows cross-pollination from diverse backgrounds/expertises. Web-based project management platforms, when used appropriately, help to facilitate this communication.
  • To truly transform, institutions will have to examine deep structures: curricula, staff/faculty time, majors, and funding.

Bibliography

Ahn, Sun Joo (Grace), Amanda Minh Tran Le, and Jeremy Bailenson. 2013. “The Effect of Embodied Experiences on Self-other Merging, Attitude, and Helping Behavior.” Media Psychology 16, no. 1: 7–38.

Ashmore, Sondra, and Kristin Runyan. 2015. Introduction to Agile Methods. New Jersey: Pearson Education.

Bailenson, Jeremy. 2019. Experience on Demand: What Virtual Reality Is, How It Works, and What It Can Do. New York: W. W. Norton & Company.

Barrows, Howard. 1996. “Problem-Based Learning in Medicine and Beyond: A Brief Overview.” New Directions for Teaching and Learning 68 (Winter): 3–12.

Blyth, Carl. 2018. “Immersive Technologies and Language Learning.” Foreign Language Annals 51: 225–232.

Brown, John Seely, Allan Collins, and Paul Duguid. 1989. “Situated Cognition and the Culture of Learning.” Educational Researcher 18, no. 1 (January–February): 32–42.

Dalgarno, Barney, and Mark J. W. Lee. 2010. “What are the Learning Affordances of 3-D Virtual Environments?” British Journal of Educational Technology 41, no.1 (January): 10–32.

Deemer, Pete, Gabrielle Benefield, Craig Larman, and Bas Vodde. 2012. The Scrum Primer: A Lightweight Guide to the Theory and Practice of Scrum. Version 2.0. http://scrumprimer.org.

Dolgunsoz, Emrah, Gurkan Ylidirim, Serkan Yildirim. 2018. “The Effect of Virtual Reality on EFL Writing Performance.” Journal of Language and Linguistic Studies 14, no. 1: 278–292.

Dunlop, Joanna C. 2005. “Problem-Based Learning and Self-Efficacy: How a Capstone Experience Prepares Students for a Profession.” Educational Technology Research and Development 53, no. l (March): 65–85.

Edens, Kellah M. 2000. “Preparing Problem Solvers for the 21st Century through Problem-Based Learning.” College Teaching 48, no. 2: 55–60.

Engeström, Yrjö, Reijo Miettinen, Raija-Leena Punamäki. 1999. Perspectives on Activity Theory. New York: Cambridge University Press.

GCIEL Focus Group. 2018. Interview by Vanessa Preast. Report of Interviews with Student Team Members. Grinnell College, April 2.

Greengard, Samuel. 2019. Virtual Reality. Cambridge, Massachusetts: MIT Press Essential Knowledge Series.

Handelsman, Diane Ebert-May, Robert Beichner, Peter Bruns, Amy Chang, Robert DeHaan, Jim Gentile, Sarah Lauffer, James Stewart, Shirley M. Tilghman, and William B. Wood. 2004. “Scientific Teaching.” Science 304, no. 5670 (April): 521–22.

Herrera, Fernanda, Jeremy Bailenson, Erika Weisz, Elise Ogle, and Jamil Zaki. 2018. “Building Long-term Empathy: A Large-scale Comparison of Traditional and Virtual Reality Perspective-taking.” PLoS ONE 13, no. 10: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0204494.

Hmelo-Silver, Cindy E. 2004. “Problem-Based Learning: What and How Do Students Learn?” Educational Psychology Review 16, no. 3 (September): 235–66.

Ijaz, Kiran, Anton Bogdanovych, and Tomas Trescak. 2017. “Virtual Worlds vs Books and Videos in History Education.” Interactive Learning Environments 25, no. 7: 904–929.

Jerald, Jason. 2015. The VR Book: Human-Centered Design for Virtual Reality. Williston, VT: Morgan & Claypool Publishers.

Jonassen, David H. 2000. “Toward a Design Theory of Problem Solving.” Educational Technology Research and Development 48, no. 4 (December): 63–85.

Jonassen, David H., Chad Carr, and Hsiu-Ping Yueh. 1998. “Computers as Mindtools for Engaging Learners in Critical Thinking.” TechTrends 43, no. 2 (March): 24–32.

Jonassen, David H., and Lucia Rohrer-Murphy. 1999. “Activity Theory as a Framework for Designing Constructivist Learning Environments.” Educational Technology Research and Development 47, no. 1 (March): 61–79.

Legault, Jennifer, Jiayan Zhao, Ying-An Chi, Weitao Chen, Alexander Klippel, and Ping Li. 2019. “Immersive Virtual Reality as an Effective Tool for Second Language Vocabulary Learning.” Languages 4, no. 13: https://www.mdpi.com/2226-471X/4/1/13.

Leimkuehler, F. Ray, field team supervisor. 1940. Uncle Sam Plantation. From the Library of Congress, Historic American Building Survey. https://www.loc.gov/pictures/item/la0030.

McCombs, Barbara L. and Jo Sue Whistler. 1997. The Learner-Centered Classroom and School: Strategies for Increasing Student Motivation and Achievement. San Francisco: Jossey-Bass Publishers.

Nathal, Karla Liliana Puga, María Eugenia Puga Nathal, Humberto Bracamontes del Toro, Marco Antonio Guzmán Solano, and Juan Carlos Martínez Sandoval. 2018. “The Immersive Virtual Reality: A Study in Three-dimensional Euclidean Space.” American Journal of Educational Research 6, no. 3: 170–174.

National Research Council. 2000. How People Learn: Brain, Mind, Experience, and School: Expanded Edition. Washington, DC: The National Academies Press. https://www.nap.edu/catalog/6160/how-people-learn-brain-mind-experience-and-school.

Neville, David. 2014. “The Story in the Mind: The Effect of 3D Gameplay on the Structuring of Written L2 Narratives.” ReCALL: The Journal of the European Association for Computer Assisted Language Learning 27, no. 1: 1–17.

Putman, Shannon, and Lateefah Id-Deen. 2019. “I Can See It! Math Understanding through Virtual Reality.” Educational Leadership 76, no. 5 (February): 36–40.

Rubin, Kenneth. 2013. Essential Scrum: A Practical Guide to the Most Popular Agile Process. New Jersey: Pearson Education.

Rubin, Peter. 2018. Future Presence: How Virtual Reality Is Changing Human Connection, Intimacy, and the Limits of Ordinary Life. New York: Harper Collins.

Selingo, Jeffrey. 2013. College (Un)bound: The Future of Higher Education and What It Means for Students. New York: Houghton Mifflin Harcourt.

———. 2017. There Is Life after College. New York: William Morrow.

Sinclair, Bryan and Glenn Gunhouse. 2016. “The Promise of Virtual Reality in Higher Education” EDUCAUSE Review: https://er.educause.edu/articles/2016/3/the-promise-of-virtual-reality-in-higher-education.

Steffen, Jacob, James E. Gaskin, Thomas O. Meservy, Jeffrey L. Jenkins, and Iopa Wolman. 2019. “Framework of Affordances for Virtual Reality and Augmented Reality.” Journal of Management Information Systems 36, no. 3: 683–729.

Sundaram, Shirsh, Ashish Khanna, Deepak Gupta, and Ruby Mann. 2019. “Assisting Students to Understand Mathematical Graphs Using Virtual Reality Application.” In Advanced Computational Intelligence Techniques for Virtual Reality in Healthcare, edited by Deepak Gupta, Aboul Ella Hassanien, and Ashish Khanna, 49–62. Studies in Computational Intelligence, vol 875. Cham, Switzerland: Springer.

Szabo, Victoria. 2019. “Collaborative and Lab-Based Approaches to 3D and VR/AR in the Humanities.” In 3D/VR in the Academic Library: Emerging Practices and Trends, edited by Jennifer Grayburn, Zack Lisher-Katz, Kristina Golubiewski-Davis, and Veronica Ikeshoji-Orlati, 12–23. Council on Library and Information Resources Report 176. https://www.clir.org/wp-content/uploads/sites/6/2019/02/Pub-176.pdf.

van Loon, Austin, Jeremy Bailenson, Jamil Zaki, Joshua Bostick, and Robb Willer. 2018. “Virtual Reality Perspective-taking Increases Cognitive Empathy for Specific Others.” PLoS ONE 13, no. 8: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0202442.

Villena Taranilla, Rafael, Ramón Cózar-Gutiérrez, José Antonio González-Calero, and Isabel López Cirugeda. 2019. “Strolling through a City of the Roman Empire: An Analysis of the Potential of Virtual Reality to Teach History in Primary Education.” Interactive Learning Environments. https://www.tandfonline.com/doi/full/10.1080/10494820.2019.1674886.

Wenger, Etienne. 1998. Communities of Practice: Learning, Meaning, and Identity. Cambridge: Cambridge University Press.

Wenger, Etienne, Richard McDermott, William M. Snyder. 2002. Cultivating Communities of Practice. Cambridge, MA: Harvard Business Press.

“What Is Liberal Education.” n.d. Association of American Colleges & Universities, accessed March 03, 2020, https://www.aacu.org/leap/what-is-liberal-education.

Wood, Zebulon M., Albert William, and Andrea Copeland. 2019. “Virtual Reality for Preservation: Production of Virtual Reality Heritage Spaces in the Classroom.” In 3D/VR in the Academic Library: Emerging Practices and Trends, edited by Jennifer Grayburn, Zack Lisher-Katz, Kristina Golubiewski-Davis, and Veronica Ikeshoji-Orlati, 39–53. Council on Library and Information Resources Report 176. https://www.clir.org/wp-content/uploads/sites/6/2019/02/Pub-176.pdf.

Yee, Nick, and Jeremy Bailenson. 2006. “Walk a Mile in Digital Shoes: The Impact of Embodied Perspective-taking on the Reduction of Negative Stereotyping in Immersive Virtual Environments.” Proceedings of PRESENCE 2006: The 9th Annual International Workshop on Presence. Cleveland, Ohio, August 24–26. https://vhil.stanford.edu/pubs/2006/walk-a-mile-in-digital-shoes-the-impact-of-embodied-perspective-taking-on-the-reduction-of-negative-stereotyping-in-immersive-virtual-environments.

Yew, Elaine H. J. and Henk G. Schmidt. 2011. “What Students Learn in Problem-Based Learning: A Process Analysis.” Instructional Science 40, no. 2 (March): 371–95.

Yildirim, Gürkan, Mehmet Elban, and Serkan Yildirim. 2018. “Analysis of Use of Virtual Reality Technologies in History Education: A Case Study.” Asian Journal of Education and Training 4: 62–69.

About the Authors

David O. Neville (PhD, Washington University in St. Louis; MS, Utah State University) is a Digital Liberal Arts Specialist and Director of the Immersive Experiences Lab at Grinnell College.

Vanessa Preast (PhD, Iowa State University; DVM, University of Florida) is Associate Director of the Center for Teaching, Learning, and Assessment at Grinnell College.

Sarah J. Purcell (PhD, Brown University) is the L.F. Parker Professor of History at Grinnell College.

Damian Kelty-Stephen (PhD, University of Connecticut-Storrs) is Assistant Professor of Psychology at Grinnell College.

Timothy D. Arner (PhD, Pennsylvania State University) is Associate Dean of Curriculum and Academic Programs and Associate Professor of English at Grinnell College.

Justin Thomas (MFA, University of Maryland) is Associate Professor of Scenic and Lighting Design and Chair of the Theatre and Dance Department at Grinnell College.

Christopher P. French (PhD, University of Chicago) is Professor of Mathematics at Grinnell College.


360° photograph of Junipero Serra statue and campus lawn, displayed in Google Tour Creator interface with digital annotation icons
0

Representing Indigenous Histories Using XR Technologies in the Classroom

In this article, we describe the major assignments from our team-taught course, Virtual Santa Clara, which drew on the affordances of extended reality (XR) technologies and public memory scholarship from the fields of rhetoric and anthropology to represent Native Ohlone history and culture on our campus. Based on our experience, we argue for the affordances of producing small-scale XR projects—using technologies such as 360° images and 3D models—to complement and contribute to larger-scale XR digital projects that are founded on deep community collaboration. In a landscape where exciting technological work so often tends to entail thoroughly developed, large-scale projects, we argue for the value of more modest contributions, both as scaffolded pathways into technology work for teachers and students and as a means of slowing down the process of technology adoption in order to better respond to ethical, humanistic, and decolonial considerations. Our own incremental process enabled us to proceed with more care, more caution, and, ultimately, a more collaborative framework going forward. Read more… Representing Indigenous Histories Using XR Technologies in the Classroom

Image shows the interior living room area with sofa, coffee table, wall hangings and rugs seen in the background. The virtual client is seen sitting on the couch. Dialogue prompts and options are shown.
0

Developing Virtual Reality Modules Aimed to Enhance Social Work Students’ Skills and Reinforce Knowledge

Abstract

The use of virtual simulations has steadily increased within the social work master’s level curriculum and its benefits have been noted in varied research studies. This paper aims to describe the development of two virtual simulations that are innovative, educational, and responsive to the needs of social work skill development and knowledge reinforcement. The first module is a 360 virtual reality (VR) tool that allows social work students to navigate areas of the Lower East Side of New York City. The second module allows social work students to conduct a home visit and assess a virtual client within an immersive 3D VR environment utilizing a framework built on top of a commercial game engine. Each simulation is individually described and the processes taken to create the scripts, assets, and framework are detailed. Their pedagogical value to social work is noted and next steps in VR development, assessment and research are outlined and discussed.

Introduction

Social work masters-level education integrates both theoretical and practice based curricula. Traditionally, each facet has been delivered to students within the classroom and Field placement setting. Technology, however, has broadened the systems of delivery and has facilitated and advanced these pedagogical methods.

The use of virtual simulations has steadily increased within social work masters-level education and its benefits have been noted. Generally, simulation based learning has been shown to be effective for adult students because it is problem-focused, self-directed and it has relevant applicability to real settings (Washburn, Bordnick, and Rizzo 2016). Virtual simulations allow for greater diversity and variability in Field settings and in patient demographics that may not be found in “real” Field placement outlets. It also provides students with “immediate” feedback and knowledge from their interactions within an assortment of settings and with diverse clients. Admittedly, the virtual interaction will be limited in some ways, but it will nonetheless provide students the advantage of gaining additional Field/practice perspectives that cannot be feasibly offered in vivo. Virtual simulations may be designed to be accessible throughout the day and will therefore be able to accommodate multiple student connections at one time. Virtual environments grant students “safe spaces” to make mistakes and to hone in on skill sets that they may not ordinarily be processing in “real” settings (Boulos, Hetherington, and Wheeler 2007). Research has indicated that students feel more comfortable and less pressure to perform in virtual settings as opposed to working with face-to-face role plays, a more traditional method of simulation (Fitch, Canada, Cary, and Freese 2016). Within the safety of the virtual simulation, students have the ability to rehearse and reflect upon the methods to use for the presented issues, without jeopardizing the protective standards of practice and client safety (Olson, Lewis, Rappe, and Hatley 2015). To facilitate learning, students may pause a simulation so that they can revisit a scenario multiple times or attempt to answer questions more than once. These systemic techniques will improve reflective skills (Boulos et al. 2007), which ultimately lead to better process and understanding.

The use of simulation training, particularly when incorporating standardized patient information, can in fact be utilized to measure and assess a student’s proficiency levels in the areas of practice and Field learning (Olson et al. 2015; CSWE 2015), both prior to and post the Field placement experience. This is an improvement over current proceedings, where unstandardized assessment modes are often used to evaluate student competencies (Washburn et al. 2016). Social work techniques are not always as clear and as easily defined as those in other disciplines; the delivery and operationalization of certain interpretive and evaluative skill sets are more abstract (Rishel and Majewski 2009) and they encompass various facets of social work subject matters. Therefore, it is important that the virtual simulation address these factors and the intersectionality of the social work curricula so that students may be evaluated on various levels: engagement, assessment, intervention, and critical thinking.

This paper aims to describe the development of two virtual simulations at New York University’s (NYU) Silver School of Social Work (Silver); the content of which is both innovative and educational. The first module is a 360 virtual reality (VR) program that allows social work students to navigate areas of the Lower East Side of New York City. The second, allows social work students to have an immersive experience of conducting a home visit to assess a client. Each of the virtual simulations operationalize critical thinking and attempt to concretize social work competencies in a holistic manner (CSWE 2015). Silver will endeavor to integrate each of the modules within courses to prepare students to enter community environments, and to hone social work practice skills, particularly related to client engagement and assessment.

Module One: Social Environment Immersion

Social work educators, particularly in Practice and Human Behavior related courses emphasize the correlations that exist between the social environment and an individual’s physical, cognitive, and emotional development. Most classes incorporate the biopsychosocial perspective as a tool to understand and process knowledge about a person’s biological, psychological and social interactions. This perspective views the person in the context of the environment and takes into consideration the challenges and stressors that might influence development throughout the life cycle. However, having students apply this perspective with clients as they work with them in their respective neighborhoods and communities can be challenging. Abstract theoretical principles are not easily transferable to real environmental circumstances, and students often struggle to recognize the theoretical frames in social contexts and then apply them to the populations that they are serving. An extensive body of research in the learning sciences supports the efficacy of learning activities situated in authentic practice and scaffolded within a community of practice (Lave and Wenger 1991; Collins 1987; Brown et al. 1989). Research on metacognition and instruction suggests that the incorporation of reflection opportunities into learning experiences improves the potential for knowledge transfer to future practice (Brown 1978; Flavel 1976; Bransford and Schwartz 1999). The 360 VR experience builds on models of cognitive apprenticeship by incorporating opportunities for corrective action based on reflection from students and feedback from instructors.

The Lower East Side (LES) is an area of New York frequented by social work master’s students completing the Field requirements of their curriculum. The LES was chosen as a pilot focus because of its demographic diversity, immigration history, social resources and its connection to the settlement house movement; each are important facets within social work (Citizens’ Committee for Children of New York 2015).

Due to budgetary restrictions on the acquisition and use of virtual simulation (VS) software, the authors adopted WONDA VR, a prosumer-oriented 360 video-editing platform. Wonda enabled our team to incorporate interactive elements such as hotspots, buttons, information panels, branching narratives, and a variety of other multimedia experiences relevant to the learning. In addition to accommodating the learning objectives and scope of the intended project, Wonda VR presented a robust alternative to more costly professional software platforms on the market.

Storyboarding

Prior to the start of video production, Silver’s educational technologists designed a storyboard/scripting template that faculty could use for inputting content and visualizing the multiple branches of the narrative and user interactions. The process of designing this template was iterative; faculty and designers worked closely to develop a prototype using a basic spreadsheet that included columns describing each scene, associated competencies, learning objectives, action, and dialogue. Designers and faculty also collaborated on framing the substantive content within the context of the objectives and ensuring that it would succinctly coincide with each video still.

A seven column table describes the process of coordinating each scene of the Lower East side, the learning objectives associated with the scene and the description of the related “hot spots.

Figure 1. Google sheet storyboard template.

Video Production

Our educational technology team videotaped each scene of the 360 video using a Nikon KeyMission 360 camera and standard tripod set up. Guided by the storyboard, the camera operator positioned the camera at predetermined locations on the Lower East Side, capturing two minutes of video footage per scene. Each scene of video footage was stitched together using Adobe Premiere software and an actor was subsequently enlisted to record voice-over audio of the script. The VR simulation took approximately nine months to complete.

Our budget included funding for a 360 video production consultant with advanced Wonda expertise who was brought on to complete the video during the final phase of post-production. With a background in fine art and documentary video production, the consultant worked iteratively with faculty and the education technology team to align his aesthetic sensibilities with the learning objectives, and multimedia cognitive design principles for engagement and learning (Mayer and Johnson 2010; Plass 2009).

Walkthrough

The 360 VR simulation starts with the disembarkation from the F train, at the Forsythe subway station and ending with a sidewalk view of Orchard Street.

=""

Figure 2. First scene from 360 VR: Forsythe train station with individuals on train platform. “Hotspot” is shown with the map of the Lower East Side.

The VR simulation is approximately 12 minutes in duration, and includes an instructional guide, learning objectives, content chapters, an introduction, and 17 navigable and interactive scenes (environments). Each scene contains verbal and textual dialogues, and a situational map. Reflective exercises and “hot spots” are presented throughout to guide students in thinking critically about the population inhabiting the Lower East Side, the community landscape, and data points, including demography, mental health, and housing. These key components were derived from the storyboard texts, which required collaborative editing by both designers and educators.

=""

Figure 3. Street scene from 360 VR of the Lower East Side. “Hotspot” indicates the number of police stations, fire stations and EMS resources in the area.

Pedagogical Integration

The Silver School of Social Work aims to use the 360 VR tool within two courses and their respective curricula: Human Behavior in the Social Environment I (HBI) and Introduction to Social Work Education and Practice in the United States (ISWEPUS). The first, HBI, is a core course that all graduate students are required to take in their first year of study. It examines the different systems (individual, family, and communities) in the environment and the relationship that exists between them. Lessons two and three of the HBI course are specifically centered on the social environment, with topics that include risk and resilience factors within communities that impact human behavior, and social work interventions designed to develop and promote community resources. Faculty would use the VR tool to help students better understand how to recognize the resources that are both present and deficient within the community. Students would then critically think about the implications that their presence and absence poses to community residents. The VR’s verbal and written guide, scripted by faculty, poses reflective questions throughout the virtual journey, prompting students to think about community circumstances and their implications to target populations.

The ISWEPUS course is purposely created to prepare international students for coursework and Field instruction in the graduate social work program and within the greater context of the United States (US). In part, lectures and experiential exercises are constructed to help International students develop an understanding of the background of social work in the US, the core values of social work, and the nuances of the biopsychosocial perspective. Lessons four and five are centered on social work practice in the US. Students are given a foundation of social work history, where they are taught the legacies of the social change movement and they are introduced to perspectives on social reform. Within the context of these lessons, students will have an opportunity to engage with the 360 VR tool. Since the VR setting is the LES, which is historically significant to social work, immigration, and social reform, it will be shown to students via WONDA Spaces. This new platform allows multiple users to enter the virtual space simultaneously, enabling faculty to actively guide the students within the environment. The interactive class exercise will help students proactively apply theoretical ecologically centered models to communities, while also getting students to understand the impact that community environments have on individuals.

Before the school disseminates the 360 VR tool within the HBI and ISWEPUS courses, first-year students enrolled in the Masters of Social Work program at Silver School of Social Work are being asked to participate in a small IRB approved study. The project will assess the VR tool’s pedagogical value and student reactions to VR use, generally. Recruited students are contacted individually to schedule an appointment to view the VR simulation and to complete the pre- and post-test questionnaires. At the scheduled appointment time, students complete the pre-questionnaire via a Qualtrics access link. The questionnaire is based on a measure created and provided by North Carolina State University. The original measure was modified to accommodate the simulation. It contains questions on areas of learning perception, learning experience, technology experience, and general attitudes toward the 360 simulation. Once the questionnaire is completed, students are offered the option to view the simulation via VR goggles or via a designated computer desktop. The research assistant ensures that students are given VR access, and they are present throughout the VR simulation to monitor for any disruption in VR service. They also note any spontaneous feedback that is provided by the participant. Upon completion of the simulation, participants are asked to complete the post-questionnaire via a Qualtrics access link. Data will be downloaded from the Qualtrics website into a Statistical Package for the Social Sciences (SPSS) software.

Initial data comparing pre- and post-tests of students engaging in the 360 VR tool indicate statistically significant learning gains and also that students value and appreciate using 360 VR.

Module Two: Virtual Engagement Utility

By 2030, there are projected to be 80 million older persons, over twice the number living in 2000 (Lowell 2015). Yet many social work students and indeed professionals are reticent to work with this population, which is often based on fear, bias, and ageist beliefs. Research has repeatedly noted that consistent exposure to the population and its nuances is the strongest predictor for future gerontological work (Wang and Chonody 2013). Working toward this objective, a VR tool was created to provide students an opportunity to practice with an avatar so that they might gain some understanding on how it is to engage and assess an older adult in their home.

Silver partnered with NYU’s Teaching and Learning with Technology Group (TLT) to pilot an in-development virtual reality framework known as the Virtual Engagement Utility (VEU). VEU will be provided to NYU as a service to streamline the creation of educational virtual experiences. The client visit scenario built with VEU is multi-layered and contains 11 different stages of interactions that start from a knock at the client’s door initiating the visit and assessment process, to the summarization and end of the in-home visit, where the student can generate an evaluation sheet of how they progressed through the exercise. Such exposure might induce them to work with the population within their Field internships. As will be described, the VEU development was quite detailed and it was one of the first collaborative VR related exercises undertaken. The project took 19 months from its inception to the final deliverable product. The VEU is designed with a reusable framework so it is anticipated that future projects will take shorter periods of time to develop.

Script Development

The client composite and the main artery of the script guiding the VR were based on the Field notes received from MSW students who were currently interning within agencies that primarily worked with older adults in the context of their homes. Issues cited by students within their notes served as a basis for the challenges that student users would face within the simulation. Once the main script was developed by faculty, it was then placed in storyboard and branching sequences were added to further enrich the interaction between student and avatar. The case dossier or file that was constructed for the client avatar was based on an assessment form used within a social work agency. VEU learning tasks and goals were based on learning objectives stipulated within core Practice courses within the social work graduate curriculum. The final script and storyboard was reviewed by faculty and a content expert.

Concurrent with script authoring, members of the Interactive Development team, part of NYU’s TLT group, created a branching narrative template. Template creation was an iterative process with input from faculty informing its clarity and ease of use. This template was used as a tool to both visualize and organize the complexity of a branching narrative, including interactions with the virtual environment, feedback on performance, and animation and audio cues for voice actors and software developers.

Three separate charts are connected via arrows that demonstrate the sequencing of the verbal dialogue between student and avatar. The charts illustrate option sequences that might be taken depending on the response of the student.

Figure 4. VEU branching narrative template.

Framework Development

VEU was developed in Unity, a cross-platform game engine. Virtual reality experiences created in the VEU framework are deployable to Google Cardboard and Google Daydream compatible devices, both smartphone based VR platforms. These platforms require VR viewers, which are available from both Google and a variety of third party vendors. VEU scenarios can also be deployed as a traditional interactive experience to PC and Mac platforms using a mouse and keyboard for interaction. VEU allows students to interact with virtual characters and their environment, complete tasks, save their progress, view feedback on their performance, and share their progress with faculty.

Asset Creation

Under guidance from faculty and their content consultant, members of the TLT media group created both a virtual environment and client avatar. Environmental 3D models were sourced or crafted internally using Autodesk Maya. Textures were created using Adobe Photoshop. The apartment layout and appearance went through several iterations informed by reference images from the content expert and feedback from faculty within the School of Social Work.

The client avatar was created with Autodesk Character Generator and further customized in Maya. The script was recorded by a voice actor as individual responses while the actor’s face was captured using a basic HD webcam. The raw audio and video was processed using Adobe Premiere to remove background noise and normalize audio levels. Faceware Analyzer was used to map the actor’s facial expressions to the face of the avatar. Body animation and generic responses such as frustration, smiling, or nodding yes and no, were manually animated in Maya. These assets were imported into Unity, creating the environment and avatar that students interact with.

The image shows an open concept kitchen and living room area. A refrigerator, stove and cabinets are seen in the kitchen area. A sofa, recliner chair, bookshelves and plants are seen in the living room area. The room is purposely cluttered with books, laundry and cleaning items.

Figure 5. Interior Image of VEU apartment.

Image shows the interior living room area with sofa, coffee table, wall hangings and rugs seen in the background. The virtual client is seen sitting on the couch. Dialogue prompts and options are shown.

Figure 6. VEU interior living room area with client.

Framework Customization and Functionality

Once assets were imported into Unity, the VEU framework was used to create an interactive experience. Dialogue between the student and virtual client was inputted into a node based dialogue tree. When a student clicks on the avatar using a pointer in VR, or the mouse on a PC, they are presented with a dialogue bubble and multiple options to respond, guiding the conversation. Most choices provide some form of feedback. Optimal decisions allow the conversation to move forward. Less appropriate responses provide feedback, then return to the previous choice, allowing students to try again. Students can be prompted to observe and interact with their environment, identifying objects relevant to the client visitation. At any time the client file can be accessed so students can refer back to their visitation goals and client background information. Access to controls such as saving or sharing assessment results was also customized to appear as a smartphone. These customizations serve to provide a more immersive experience.

The image shows the perspective of the user, sitting in a chair and looking down at their hands. The user has a file in one hand and their smartphone in the other. The virtual client is sitting on the sofa, a coffee table, and rug are seen in the background.

Figure 7. VEU student perspective.

 

Image shows an open file folder with a post-it note on the left side with the visit goals itemized. On the right side of the file folder, there is an image of an assessment form with client information.

Figure 8. VEU client file that can be accessed by students throughout the VR experience.

Tutorial Development

Outside of research and professional environments virtual reality is a relatively new technology. Many students (and faculty) who utilize the VEU tool may have no prior experience with VR. For this reason a custom tutorial scenario was developed with the assistance of Silver faculty. The tutorial was designed to cover the types of interactions students might encounter in their virtual visit. It demonstrates basic interactions such as pointing and clicking, interacting with a dialogue tree, and how to access the client file. It utilizes some assets from the client visit scenario so students will be familiar with the look and feel before beginning their assessment.

Pedagogical Integration

The School of Social Work aims to incorporate this simulation within the Integrated Social Work Practice Field Instruction I course (Practice I). This core class provides graduate students with generalist knowledge on working with a range of systems that include individuals, couples, families, agencies and communities. It also helps students develop the basic skills of engagement, assessment and goals setting. Lesson nine of this course is centered on teaching students how to conduct a multidimensional biopsychosocial assessment, which needs to be culturally sensitive and attuned to the client’s strengths and risks. The VEU tool would be an assignment that students would need to complete post-lesson. The VEU’s prescribed goals of engagement, assessment, and of identifying needs and resources are in keeping with lesson nine. The VEU also reinforces the prior Practice I lessons that are centered on engagement and the helping process. The internal evaluation that is generated by the VEU for each student will be assessed by faculty and then reviewed with the student. Students can share their VEU progress with faculty at any time. Faculty receive an email with the score for each learning sequence as well as the individual choices made. The value of each question can be weighted for greater control over performance evaluation. This data is anonymized. Students can choose to de-anonymize the data by providing their unique ID assigned when first activating the VEU tool. A reflective exercise post-VEU will also be assigned to better understand the student’s experience and to obtain knowledge on how to further support their learning. Students will have additional opportunities to engage their VEU experience through discussion with classmates in structured classroom activities, as well as a programmed opportunity to revisit the VEU later in the semester with the goal of interleaving their prior experience with later course learning experiences, such as Field work with clients. Research supports the educational value of interleaving multimodal approaches to similar content (Birnbaum et al. 2013), reinforcing prior knowledge, and contextualizing learning experiences within larger conceptual frameworks (Richland et al. 2005).

Future Developments

As was noted in an earlier subsection, the VR 360 tool is currently being assessed for its pedagogical value with students. Although initial results indicate there is overall positive reaction to the tool, further steps will be taken to purposely infuse the VR simulation within HBI and ISWEUS and then obtain student reactions both quantitatively and qualitatively. Similarly to the 360 VR simulation, the VEU simulation will be studied in a pre- and post-test research design in the coming months for its pedagogical value and whether students are comfortable navigating the VR tool. Also, social work faculty with Practice expertise will be asked to review the simulation and respond to questions related to learning objectives, subjectivity, content and overall quality. Results will inform the development of the current VR simulations as they pertain to content, structure, and design.

In addition, several steps will be taken to further improve the VEU simulation’s design and internal evaluative mechanism. Metrics can be reported such as tracking time spent in the experience or on particular questions, and whether the scenario was completed in one session or several. Engagement could be measured by checking if students skip the voiceovers of the avatar or if they completed optional assessments. These metrics could then be reported alongside the multiple choice performance, giving faculty deeper insight into the behaviors and learning styles of their students.

The VEU framework will be adapted to the Oculus Quest headset. In the process it will gain six degrees of freedom hand tracking, so students can reach out and interact with the environment in an intuitive manner. The Quest is also capable of room-scale tracking, which will allow students to walk around the apartment so they may perform closer inspections of the environment. Faculty can then integrate questions related to hazards and safety, while also integrating the assessment within the internal evaluative mechanism. Additional modes of interaction with the virtual client are also being explored, such as a “flow mode” where feedback is limited and the conversation proceeds naturally and with less interruptions. These features will hopefully lead to a more immersive and approachable learning experience for the student.

Bibliography

“2015 Educational Policy and Accreditation Standards.” 2015. Council on Social Work Education (CSWE). https://cswe.org/Accreditation/Standards-and-Policies/2015-EPAS.

Birnbaum, Monica S., et al. 2013. “Why interleaving enhances inductive learning: The roles of discrimination and retrieval.” Memory & Cognition 41, no. 3: 392–402.

Boulos, Maged N. Kamel, Lee Hetherington, and Steve Wheeler. 2007. “Second Life: an overview of the potential of 3‐D virtual worlds in medical and health education.” Health Information & Libraries Journal 24, no. 4: 233–45.

Bransford, John D., and Daniel L. Schwartz. 1999. “Chapter 3: Rethinking transfer: A simple proposal with multiple implications.” Review of Research in Education 24, no. 1: 61–100.

Brown, Ann L. 1978. “Knowing when, where and how to remember. A problem of metacognition.” In Advances in Instructional Psychology, Vol. I,  edited by R. Glaser, 77–165. Hillsdale, New Jersey: Erlbaum.

Brown, John Seely, Allan Collins, and Paul Duguid. 1989. “Situated cognition and the culture of learning.” Educational Researcher 18, no. 1: 32–42.

Citizens Committee for Children (CCC). 2015. “CCC Community Risk Ranking.” http://www.cccnewyork.org/wp-content/uploads/2015/01/CCCRiskRankingNYC.pdf.

Collins, Allan, John Seely Brown, and Susan E. Newman. 1987. “Technical Report No. 403: Cognitive apprenticeship: teaching the craft of reading, writing, and mathematics.” Center for the Study of Reading: U. Illinois at Urbana-Champaign.

Fitch, Dale, Kelli Canada, Suzanne Cary, and Rebekah Freese. 2016. “Facilitating social work role plays in online courses: The use of video conferencing.” Advances in Social Work 17, no. 1: 78–92.

Flavell, J. H. 1976. “Metacognitive aspects of problem solving.” In The Nature of Intelligence, edited by L.B. Resnick, 231–35. Hillsdale, New Jersey: Erlbaum.

Lowell Sun. 2019. “Elder Care–Giving: Will You Be Ready for the Challenge?” Lowell Sun, July 11.

Lave, Jean, and Etienne Wenger. 1991. Situated learning: Legitimate peripheral participation. Cambridge, UK: Cambridge University Press.

Mayer, Richard E., and Cheryl I. Johnson. 2010. “Adding instructional features that promote learning in a game-like environment.” Journal of Educational Computing Research 42, no. 3: 241–65.

Olson, Mark D., Melinda Lewis, Paula Rappe, and Sandra Hartley. 2015. “Innovations in Social Work Training: A Pilot Study of Interprofessional Collaboration Using Standardized Clients.” International Journal of Teaching and Learning in Higher Education 27, no. 1: 14–24.

Plass, Jan L., Bruce D. Homer, and Elizabeth O. Hayward. 2009. “Design factors for educationally effective animations and simulations.” Journal of Computing in Higher Education 21, no. 1: 31–61.

Richland, Lindsey E., Robert A. Bjork, Jason R. Finley, and Marcia C. Linn. 2005. “Linking cognitive science to education: Generation and interleaving effects.” Proceedings of the Twenty-Seventh Annual Conference of the Cognitive Science Society 27: 1850–55.

Rishel, Carrie W., and Virginia Majewski. 2009. “Student gains in self-efficacy in an advanced MSW curriculum: A customized model for outcomes assessment.” Journal of Social Work Education 45, no. 3: 365–83.

Um, Eunjoon, Jan L. Plass, Elizabeth O. Hayward, and Bruce D. Homer. 2012. “Emotional design in multimedia learning.” Journal of Educational Psychology 104, no. 2: 485.

Wang, Donna, and Jill Chonody. 2013. “Social workers’ attitudes toward older adults: A review of the literature.” Journal of Social Work Education 49, no. 1: 150–72.

Washburn, Micki, Patrick Bordnick, and Albert “Skip” Rizzo. 2016. “A pilot feasibility study of virtual patient simulation to enhance social work students’ brief mental health assessment skills.” Social Work in Health Care 55, no. 9: 675–93.

Washburn, Micki, and Shu Zhou. 2018. “Teaching Note—Technology-Enhanced Clinical Simulations: Tools for Practicing Clinical Skills in Online Social Work Programs.” Journal of Social Work Education 54, no. 3: 554–60.

About the Authors

Nicholas Lanzieri is an Associate Clinical Professor at NYU Silver School of Social Work and Faculty Lead for Field Learning & Community Partnerships. He also co-leads Silver’s Faculty Education Technology Board.

Henry Samelson is the Associate Director of Educational Technology at NYU Silver School of Social Work. He received his MA in Digital Media Design for Learning/Games for Learning from NYU Steinhardt School of Media, Culture and Communications.

Jon Bowen received his MS in Biomedical Visualization at the University of Illinois at Chicago and is currently a senior interactive developer at New York University. He creates games and simulations for education and is interested in emerging technologies such as extended reality (XR) and spatial computing.

Left, User testing the system with HoloLens headset in the historic home. Right, What the user sees through the HoloLens.
0

Blending Disciplines for a Blended Reality: Virtual Guides for a Living History Museum

Abstract

This article describes the early stages of a virtual guide for onsite museum experiences, a project undertaken at Rochester Institute of Technology (RIT) involving students and faculty in computer science, museum studies, art and design, and theatre in conjunction with Genesee Country Village & Museum, the third-largest living history museum in the US and the largest in NY state. Our work focuses on the use of augmented reality, where technology and devices are used to superimpose digital assets over real elements in physical spaces, to demonstrate potential for enhancing storytelling within a historic village context. We outline our process—involving students and faculty from three colleges within our university, and staff from the museum partner—from exploration, research, and design to capture, delivery, and testing. With four faculty leading a cross-disciplinary collaboration among more than eighty students, three additional faculty from RIT (theatre and music), and six museum staff members thus far, our interest lies in facilitating opportunities for incidental learning (Crawford and Machemer 2008). We are keenly interested in pushing the boundaries of Pomerantz’s “spirit of experimentation” (2019) among the students and instructors where the former learn about the technology and subject matter, while the faculty forfeit prescriptive outcomes in an effort to foster experimentation within the context of courses and assignments where this project is facilitated. Ultimately, we see this application of XR as a mode for the conception, creation, and dissemination of storytelling within the classroom experience that simultaneously shares attributes of constructivist learning proffered in education and museums.

Overview

This project emerged from an existing partnership between Rochester Institute of Technology (RIT) and Genesee Country Village & Museum (GCV&M). This perhaps unlikely pairing between a research university with more than 19,000 students and the largest living history museum in New York provides opportunities for faculty, staff, and students from virtually every college within RIT to foster collaborations from a variety of disciplines. Many projects and research areas have multi-disciplinary or cross-disciplinary foci. This project, “Blending Disciplines for a Blended Reality: Virtual Guides for a Living History Museum,” is one such example where an interdisciplinary, research-inspired question forges connection among multiple constituencies within the university with the museum as the site for developing tangible skills and undertaking projects that have scholarly reach and long-term, mutual benefit. Because of this partnership, and the trust and history of association between the two organizations, we, as faculty researchers, have the freedom and flexibility to foster interdisciplinary collaboration in a meaningful way and to engage our students in developing skills in storytelling, digital composition, and multimodal literacy. The museum contributes to, and benefits from, the research and output of this collaboration, thereby serving as a site where our research can thrive.

Process

While this concept at present involves faculty and students from several disciplines with the production geared toward realizing work around one historical person that we have developed and provided with historically accurate contextual narrative, our project began with a much broader framing that our students helped to refine. The collaboration began in 2018 between computer science and museum studies faculty who wanted to set a research problem at the museum, employ technology as a possible solution, and engage our students in the research and scholarship around this project. Inspired by ongoing research with intelligent virtual agents (IVA) (Norouzi et al. 2018), we pondered as to what role an IVA might play in the context of a living museum. We posed the question: “Could a stylized avatar, serving as a historical guide, be used to augment visitors’ physical experiences at Genesee Country Village & Museum?” Over two semesters, the faculty and students from museum studies and computer science, with the help of faculty and students from theatre, developed 38 historical narratives which were recorded via audio only or audio and motion capture. This earliest phase of exploration was evaluated by team members, Decker and Geigel, in November 2018 and in April 2019, which in turn enabled them to pivot the project in five ways over the past several months. The project team expanded to include collaborators from among art and design faculty. In turn, we began to focus on researching and developing one character initially; to create historically accurate clothing, props, and environment for the character; and to refine our workflow—all with the end goal of stacking historical narratives into a six-minute story, delivered in monologue form as a vignette to engage with the visitor. This article outlines the project over this entire span of 18 months, with primary focus on the past several months, which is the period of robust project development and testing by students and faculty.

Timeline Between Spring 2018 and Spring 2021 showing collaboration between 4 University programs and museum
Figure 1. Roles and tasks of AR storytelling team. Informal feedback will continue, and formalized user testing will be developed, through spring of 2021.

Exploration

The earliest iteration of this project was exploratory in order to see the viability of our launching a long-term project. Led by two of our four-person faculty team (Decker and Geigel) from spring 2018 through spring 2019, we tested the technology and research/script-writing as well as recording. In terms of technology, we chose a Microsoft HoloLens as a delivery platform as it provides an intuitive, hands-free interface as well as built-in voice recognition. Furthermore, the HoloLens has been shown to be an effective platform in other museum contexts (Hammady et al. 2019). Development was done using Unity, a 3D software platform for rapid prototyping of VR and AR applications. We developed three short sprints using prescribed, pre-loaded character types in Unity, writing an application in Unity to allow for placement of the virtual storyteller at specific, appropriate spots on our campus (in lieu of the museum). As the app was running on a HoloLens, users had the opportunity to interact with the application by asking pointed questions of the avatar, to which the storyteller would respond, making users feel like they were having a conversation with a real person.

Research

Initially, neither the text nor the visuals used in this exploratory phase were keyed to our historical site. However, simultaneous to the technology testing, we asked museum studies students to research the buildings situated at the museum and to develop “character types” who might be viable suggestions for developing an AR character for this project. Over two semesters of increasingly focused, exploratory research, the students created 38 one-to-three-minute monologues situated at 12 of the 68 historic structures at the museum. Each of these monologues was historically based and researched using resources from the museum as well as contextual sources (Bolger 1985). While only a small portion of this overall work was, in turn, used as part of our refined prototype (explored fully in this article), the initial research phase informed our workflow, as well as the decision to develop one character more fully to focus our team’s concept development and execution.

Over the summer of 2019, in consultation with the Genesee Country Village & Museum staff,[1] the team selected Dr. Frederick F. Backus (1794–1858) as the inspiration for our first fully developed character. First and foremost, Backus had myriad interests and connections to Rochester history, making his story rich with intersections that could, in turn, be amplified through research-informed narrative writing. Second, we chose this individual in order to tether our virtual character with his actual home, one of the first grand mansions in Rochester which Backus purchased in 1838. Third, and perhaps most interestingly in terms of creative output, no images exist of his appearance, thereby making him an opportunity to blend historical reality and interpretation.

After deciding upon a character, primary and secondary research guided the script-writing, with the immediate need to develop one narrative for this phase of testing. Immediately, the decision was made to situate the character a bit later in his life, so as to draw upon a wealth of experiences documented by Backus in letters. As the museum interprets the home to the year 1850, writing a script that would be situated at around the same time of the museum’s interpretation bolstered our ability to render a seamless integration between the AR experience and the museum environment.

Historically accurate assets were gathered as part of the research. These include professional and domestic contexts, including newspaper accounts from the years that Backus served in the New York State Senate and background information on Backus’s neighborhood gleaned from property records and maps of Rochester’s wealthy Third Ward. This portfolio of research was passed on to the museum studies students in the spring of 2020 to guide their development of academically and historically rigorous narratives for five characters (Backus and four additional characters).

The current student cohort (spring 2020) developed 15 monologues focused on individuals who lived in the region over the years that the museum interprets (Pioneer Settlement era of 1780s through the 1920s), with particular attention to the 1820s–1860s. These individuals included the aforementioned Frederick Fanning Backus (1794–1858); Candace Beach (1790–1850), a teacher at a one-room schoolhouse who lived through the historic “year without summer” that occurred in 1816, over a three-year period of climate change and uncertainty as a result of the eruption of Indonesia’s Mt. Tambora in the spring of 1815; John Carlin (1813–1891), a poet and painter who graduated in 1825 from Pennsylvania Institute for the Deaf and Dumb before traveling to England and France for a Grand Tour and returning to New York and picking up clients across the state; Austin Steward (1793–1869), who was born to enslaved parents in Virginia before moving to New York and becoming engaged in antislavery and temperance as well as the black convention movement, all the while being engaged as a merchant, publisher, and orator years before Frederick Douglass settled in this region; and Lavinia Fanning Watson (1818–1900), a Philadelphia socialite, with ties to the region, who was the first woman to commission a naval ship, the USS Germantown (1846). The monologues were sited at three of the buildings on the museum campus.[2]

Design: character, model, and rigging

Throughout the research phase, the faculty team had discussed how to proceed with the digital design phase. The decision process for creating the first 3D character (described below) would also inform projects and workflow for continued production, including the development of additional characters in spring 2020 and beyond.

The design process began with the choice to build a stylized avatar, rather than a realistic 3D animation, so as to avoid the “uncanny valley”: a feeling of unease and disconnect experienced when humans encounter robotic or audio/visual simulations that are too realistic. This key decision was informed by the work of Masahiro Mori who presented the theory of the uncanny valley five decades ago (Mori 2012). Mori posited that an individual’s feeling about a human-like robot would go from empathy to revulsion the closer the representation grew to reality, because the representation would naturally not achieve true realism. Mori’s premise has been applied to the development of digital characters as well, as the uncanny valley is often referenced vis-à-vis the film Polar Express (Noe 2012) and CGI characters that fail to achieve true realism and therefore alienate the viewer (Weschler 2011). For Weschler in particular, the “vacant” quality of the eyes and unrealistic movement are cited as features that foster the eeriness associated with the uncanny valley.

While some scholars are now exploring the ability of digital artists to create avatars realistic enough to foster trust and empathy, such production is at a level of digital artistry that requires mastery and extensive experience. Students would not have the expertise to overcome this valley and therefore we chose to pursue a stylized character. The choice meant the final agent would be distinctly unrealistic in an authentic historical environment. We accepted this anachronism as a way of attuning to the museum’s approach to onsite interpretation. GCV&M does not presume visitors are transported to 1850; it sets out to interpret and demonstrate the era authentically while acknowledging that the museum staff, chiefly the costumed interpreters and the guests, are inhabitants of the present. Additionally, we intended to utilize modern technology (HoloLens) to immerse the viewer in the experience of interacting with the character, further removing them from the idea of being transported to the past. Our digital agent, viewed through the HoloLens, would clearly be an AR animation and not an actual human interpreter, so the decision to opt for a stylized avatar meant students could design all aspects of the character with the burden of bridging the uncanny valley relieved.

The avatar needed to be approachable and warm in order to appeal to older adults and children alike. To avoid a sense of unease, certain attributes are exaggerated in digital human representations—most often the size of the head, hands, and feet. For continuity of design, the style developed would be carried through into additional avatars, to be executed by 3D digital design students.

As a character, Backus presented the unusual but fortunate position of having an actual historical figure for whom there is no visual record, only written references. With no extant images, the team was left to interpret his appearance through the use of his father’s portrait from Hamilton College and his own writings of his life experiences. The character design incorporates the physical input of Azel Backus as the subject’s father, with historic, social, and economic aspects of the 1851 time period. As we move forward with students to develop further avatars and agents for the museum, classes will follow the same pattern of character analysis in the design, regardless of any visual references we may have of subjects. Thorough research of the fashion of the period was balanced with the knowledge that Rochester, NY in 1851 was both rural and remote and therefore not on trend with the latest styles. It was also clear from the writings of our historic subject that he had traveled the area and experienced the hardships of practicing medicine in such a time and place.

Left, Artist engraving of Azel Backus from 1813. Right, Costume Design Sketch of Azel Backus
Figure 2. Portrait of Azel Backus and preliminary costume design.

It was also important to have knowledge of the character’s setting in the actual house and take color scheme into account. The AR device through which the character will be viewed will superimpose the image on the surroundings, so it was important to make sure the agent would stand out from the environment. The buildings at the Genesee Country Village & Museum are from several different decades and span a wide range of architectural styles. Everything from the number of windows in a building to the color trends and financial status of its residents will impact how well the avatar is seen in the setting. For Backus, this meant opting for cool, darker colors so he would be better distinguished amongst the tans, browns, and reds of the well-lit entryway.

We began with a rough sketch to outline the physical properties of the character before moving into 3D development. The 3D digital design program utilizes software from a variety of companies in order for students to experience the full range of programs in use throughout the professional industry. For this project we chose to use software from well-established and reliable companies with the idea that we will be able to upgrade and improve the designs as the software advances. In order to achieve the stylized character we had determined would best suit our needs, we utilized Character Creator by Reallusion, a 3D software that would allow us to morph realistic human proportions. This software utilizes an interface and key strokes that are common in several 3D programs, making it approachable and intuitive for students of 3D art. Facial and body features were exaggerated; the nasal, cheek, and chin areas were expanded to match historic drawings of Azel Backus, along with digitally sculpted hair and sideburns matching historic styles. The head and eyes were enlarged, as seen in many animated characters, to make them less realistic and more childlike. The avatar’s physique and appearance were also altered to better reflect that of an older gentleman of 1851.

Left, 3D model of exxagerated body propotions. Right, 3D model next to 2D patterns of costume garments.
Figure 3. Body and garment modeling.

We then used Marvelous Designer 8, a digital patterning and simulation software to build period-appropriate clothing for Backus. This software in particular is not only widely embraced by the 3D industry, but is advancing rapidly in its effectiveness and efficiency. As we develop further historic digital avatars for the museum, students will be utilizing this software to create historically accurate garments that are uncommon in the 3D world.

In selecting garments for Backus, as well as any future characters for the museum, it was important to keep in mind that clothing production was not yet industrialized, meaning it was not mass produced nor readily available (Holkeboer 1993; Gorsline 1994; Armstrong 1995; Tortora and Marcketti 2015). Most, if not all, of the garments worn by Backus would have been home or locally produced. Men’s shirts in particular were traditionally made by a wife or mother, but a man’s tailored waistcoat and frock coat would have been made by a skilled, male tailor. Additionally, the materials used would have been relatively expensive, so tailored menswear tended to be an investment that was worn for several years. Considering the remoteness of Rochester to any major metropolitan hub of 1851, it’s likely his garments could have been 5–10 years old at the time. To that end, we opted to dress Backus in a slightly dated frock coat with the soft, sloping shoulders and high back collar of the mid-1840s, and a waistcoat with a wide lapel and only slightly rounded hem of the 1840s. Here, we opted for a deep navy blue melton wool that would be a strong contrast to the wood staircase and tan wallpaper of the home’s entry. His trousers also bear the marks of the 1840s, with the relatively new center front fly closure, as opposed to the earlier fall front. Men’s trousers of this early Victorian era were tapered and narrow at the hem and tended towards large-scale patterns, especially plaids. We opted for a somewhat subdued gray wool plaid flannel as Backus was more an elder statesman than fashionable dandy. Students’ detailed character analysis informs these design decisions, and the design choices inform how the patterning software is used. These decisions, coupled with the research prepared by museum studies students in their development of monologues, inform the 3D students’ design choice, right down to the type of fabric used in a waistcoat and whether or not a collar is top stitched.

Capture

Transitioning from research and design to capture and render meant involving actors from performing arts faculty who, based on their vocal style, could offer a viable presentation of Dr. Frederick F. Backus. In order to preserve the legibility of the narrative in performance, the actor’s voice-over track was recorded in advance, which enabled the actors to adjust inflection and vocal emphasis of segments of the script in a sound-isolated recording booth. The performers then recreated the character’s movements in front of a motion capture system, using the audio playback as reference. Using Character Creator, the motions were then transferred to the avatar and any jitter was removed. Additionally, a digital face rig was created and lip-synched to the narrated audio track recorded earlier, and additional gestures were added as required.

Left, Actor and audio technician reviewing script outside audio recroding booth. Right, Actor performing for motion capture system
Figure 4. Audio and motion capture recording.

Delivery

The rigged and animated model was exported as an FBX file from Character Creator into the Unity game engine for use in the HoloLens, which would simultaneously display the character in the museum space and create the user experience of interaction with the virtual guide. (A user interacting with Backus in the museum setting and the view seen through the HoloLens are shown in Figures 8 and 9.) Using the speech-recognition capabilities of the device, the application can recognize key spoken phrases to which the character will respond with a predefined and prerecorded monologue.

Left, User testing the system with HoloLens headset in the historic home. Right, What the user sees through the HoloLens.
Figure 5. User interacting with the character in the museum setting.

In this way, the narrative is designed as contextually rich, narrative-driven storytelling delivered by a historical character, set in the home that the historical Backus did, at one time, inhabit. The AR character, house, and site are woven together in a storytelling construct that engenders historical information, situates a conversation between an agent and a visitor at a site that the character may have once visited, and presents an opportunity for incidental learning—the learning along the edges that Falk and Dierking proclaim as critical to the museum visitor experience (Falk and Dierking 2012). In short, the monologue as written, performed in audio and motion capture, and associated with the digital asset, which includes the character creation as well as historical treatment, are tethered and presented through the HoloLens. These facets come together to create an experience that provides an opportunity for visitors to engage with a person from the past that is only possible through this medium.

Informal feedback

Our ultimate goal is to deploy at GCV&M with visitors, wearing HoloLens, who have entered the threshold of the doorway of the Livingston-Backus house at the museum. While we have not yet deployed at the museum, as of November 2019, our team has reached a significant milestone of creating the Backus who can deliver a monologue and respond to voice commands from the user. As of this writing (March 2020), our research and design has continued with four new characters that will be captured in the coming months.
Our progress thus far has been informed by preliminary informal feedback in two phases in 2019, both working toward the goal of user testing this system and content onsite at Genesee Country Village & Museum.[3] These two iterations of informal feedback, while very different in design and nature, have offered us the opportunity to see an increase in ease of use and interest, as well as fulfillment. These facets will be measured again as we move into our third iteration of informal feedback, involving students from across the collaboration team, as well as museum staff. We plan to develop and conduct formalized user testing at the museum in the summer of 2020. Each of these feedback scenarios has enabled us to reflect on our work as faculty, and, in coordination with our students, to assess our pedagogical goals and structure our next advancement.

Left, Student learning navigation on HoloLens. Right, Student adjusting HoloLens head straps.
Figure 6. Students Lizzy (left) and Brie (right) learned how to use the HoloLens in order to facilitate informal feedback as part of our university’s annual AR/VR/XR symposium. November 22, 2019.

Authenticity and living history museums

The creation of a virtual museum guide may seem at odds with the history and context of our museum partner and our intended location for delivering the XR experience, Genesee Country Village & Museum, which belongs to the classification of living history museums. This genre grew out of world’s fairs and international displays in the 19th century that offered exhibitions arranged in village-like settings to provide viewers with an engaging sense of culture and history simultaneously (Alexander, Alexander, and Decker, 118). Founded in 1966 and open to the public a decade later, GCV&M has sought, from its earliest days, “an endeavor to visualize and interpret this bygone era…[and] has assembled authentic examples—functional buildings and artifacts of the period—from a score of area towns. It has not endeavored to recreate any specific village but to recapture and portray the character and atmosphere of the village era” (McKelvey in Bolger 1985, 2).

Walking through the gates of the toll house and entering into the historic village, visitors are treated to a vision of the past before their eyes. Such a treatment of living history museums affirms folklorist Jay Anderson’s (1985) definition of living history as “the simulation of life in another time.” Museum interpretation at living history museums is often mediated through the costumed interpreters who may take on a particular role, often with the premise that they are conveying what it was like to live in the past, and the modern visitor has encountered them in their daily life (Reid 2001; Roth 2005; Thierer 2010). Because they are the primary communicators with museum visitors, costumed interpreters are essential to the interpretation function of living history museums, which are entirely re-contextualized environments. Interpreters serve as the key factor of on-site engagement for visitors. They communicate with visitors through demonstration and conversation. As theorists Handler and Saxton (1988) argue, living history practitioners are keenly concerned with authenticity and that the role of the interpreter is to bridge past and present.

This connection between past and present while simultaneously seeking authenticity is key to our project which utilizes extended reality as a medium for the dissemination of a first-person narrative keyed to the identity of a known, historical person. These choices were made by the project team so as to distance the digital work and its outlay from the onsite, face-to-face, interpreter-to-visitor experience. In addition, we wanted to push the limits of this medium to see the extent to which our virtual tour guide can convey authenticity even while avoiding the aforementioned uncanny valley.

Whereas traditionally, onsite at the museum, visitors come into contact with costumed interpreters who staff approximately a dozen buildings and engage in third-person dialogue, meaning that they are dressed in historically accurate costume yet use contemporary language and are fully aware of the present, our virtual tour guide offers the opportunity to hear from a character speaking in first-person, performing a role for visitors, and speaking in paraphrases or direct quotes from diaries, notes, and primary sources. Both methods of interpretation—the third-person, interpreter-based and first-person, avatar-based—seek to serve as relevant, authentic, and historically accurate bridges between past and present for visitors.[4]

Pedagogy

Our project design has been informed by pedagogy, as this project was conceived from the outset as a collaboration among faculty and student researchers across several disciplines. Over the eighteen months of this project, students and faculty have been involved at every phase of our project (see figure 1, Roles and Tasks). Some aspects have been developed within the framework of a course assignment for museum studies students, including research, monologue development, participation in audio and motion capture, and collection of feedback. The early iteration of the virtual museum guide was developed by computer science students enrolled in Applications in Virtual Reality, a course focusing on the use of VR/AR technologies in creating unique mixed reality experiences. And, enhancements of the application have been taken on by several master’s students in computer science as part of their capstone projects. Other facets took place outside of the classroom assignment or context; students self-selected to become involved in that phase of the work. For instance, theatre students were involved as actors for motion and audio capture, a 3D design student facilitated the motion capture as part of advanced study toward her thesis project, and museum studies students facilitated informal feedback in November 2019.

The application of XR as a mode for the conception, creation, and dissemination of storytelling within the classroom experience shares attributes of constructivist learning of educational systems in general (Dewey 1998). Specifically, our project—involving students and faculty from three colleges at a research university, along with a museum partner—encourages discourse during knowledge construction. For instance, the collaboration necessary for success of this project provides a unique learning opportunity for computer science students. Though the focus of the work of the computer science students may be technical in nature, the design, implementation, and approach of the application development are shaped by the continual interaction with the creative team. Back and forth communication regarding the assets, both visual and aural, guides the development activities of the application, and, at the same time, directs the work of the design team creating the assets as they must assure proper formatting, timing, and synchronization of the models and animations to work on the HoloLens device.

Faculty have served as mentors to one another and students, but also have let go of prescriptive outcomes for classroom assignments or milestones of our project in an effort to foster experimentation within the context of our collaboration. We have embraced key facets of Pomerantz’s “spirit of experimentation” (2019) which contends that success can be measured by virtue of experimentation rather than meeting criteria on a traditional rubric. As Pomerantz notes, “Sometimes experimentation is the point.” As faculty, too, our learning experiences as collaborators and facilitators guiding our students’ work throughout this project have embraced this facet of experimentation.

Our blending of disciplines to create a blended-reality experience realizes constructivist pedagogy and further mirrors attributes of visitor experiences at museums, where knowledge is actively produced by the learner. For instance, throughout this project, students engage in incidental learning, which may be defined as “unplanned or tacit learning, stemming from the learner’s actions” which is “an often hidden aspect of higher education” (Crawford and Machemer 2008, 106, 109). These attributes are hallmarks of a “learner-centered environment” (104) and are key to understanding the pedagogical outcomes of our project.

Our conception of a virtual museum guide to be developed among a cohort of interdisciplinary researchers and their students intended, from the outset, for incidental learning to occur as a means of individual student work (as an assignment or other framework for involvement in this project). In fact, we found, in review of Crawford and Machemer’s characterization of 19 incidental learning skills associated with project-based learning, that students across the project were developing (and continue to develop) each of these skills at various points throughout the project.[5] In addition to particular facets skills gained by particular cohorts of students involved in our project, all students and faculty gained “teamwork skills,” “time management skills,” and “potential to apply what is learned here to other situations” (variables 2, 4, and 19 of Crawford and Machemer). Each of these attributes described above enabled the students to develop skills that were not part of the initial project requirements, indeed, but they also fostered a sense of real-world experience. That is, the workflow and processes defined above—with collaborators having domain knowledge and expertise entering into a project for a particular purpose and then exiting until called upon again—mirrors the work world of industry where various aspects of a large-scale project are completed independently in contribution of a larger whole. Importantly, the undergraduate students across all disciplines expressed an interest in continuing to be updated on the project’s progress, long after their semester or other engagement had come to an end, thereby affirming the pedagogical impact of this project.

While much of our decision-making was informed by pedagogical aims and aspirations for cross-disciplinary learning, we were collectively interested in how XR can inform storytelling practices. Our conception of storytelling based at a living history museum was informed by Bedford’s proclamation of storytelling as a key attribute of museum work (2001) and Lowe’s articulation of who defines stories as the “interpretive tales we craft” and narrative as “the way that we consciously and unconsciously shape those stories” (Lowe 2015, 45). Such a framing of the past impacts the process of meaning making. As David Allison notes, “The way that museums present the history and the prejudices and biases they bring to the design process [of living history interpretation] will affect the meaning that individuals construct for themselves” (2016, 29). Allison thus affirms Lowe’s assertion that particular institutions do a “much better job explaining the complexity of history making—the craft, the methods, and the narrative construction” and sees such places as sites of innovation where leveraging “the old, bad history” (Lowe 2015, 47, 52) can—through storytelling—foster multivocality and inclusive interpretations of the past. Such museum-focused outcomes cross over to our pedagogical aims of storytelling and our project’s framing by affirming the value, relevance, and importance of storytelling as a form of historical communication, bridging between past and present as well as opportunities for authenticity, empathy, and inclusion.

Notes

[1] Museum staff involved in this discussion included the museum director and curator of collections. In developing further characters, we also consulted the senior director of interpretation and interpretation office manager. Two costumed interpreters will be involved in motion capture in the spring of 2020.

[2] The buildings include: Livingston-Backus House, the Land Office, and the Schoolhouse. The Livingston-Backus House is a plausible location for the Backus monologues as well as those involving his niece, Lavinia Fanning Watson, and the painter John Carlin, who befriended the Backuses. The Land Office is a reasonable location for Steward, who worked for Henry Towar when the structure was onsite in Lyons, New York. The Schoolhouse (built in 1822) is a reasonable site for the monologues by Candace Beach who, although employed as teacher in the region before this structure was built, is positioned much later in life, as she reminisces on her years teaching. Our process of monologue creation has involved the expertise of the museum staff, taking cues from Maria Roussou et al.’s understanding and assessment of the importance of collaborative participatory creation (2015) while also being mindful of the developments, research, and outcomes of storytelling on mobile devices in the cultural heritage sector (Lombardo 2012).

[3] Our first testing took place in April 2019 within the context of a museum studies course where students were familiar with the project because each student had contributed to it by writing monologues for characters. The feedback at that time indicated that only 31.6% of the users felt that the experience fulfilled their desire of a museum experience (Decker 2019). Because the results were promising in terms of desirability of use and potential for engagement, we expanded the team to include collaborators from among art and design faculty; to focus on researching and developing one character initially; to create historically accurate clothing, props, and environment for the character; and to refine our workflow. Each of these facets was accomplished in the intervening months, leading to a second phase of informal feedback in November 2019, when we deployed the HoloLens with the Backus content as part of a demonstration at our university’s annual AR/VR/XR symposium (Carr and Johnson-Morris, 2019).

[4] Such a bridging of past and present is part of the living history tradition, as defined by Scott Magelssen who reads living history interpretation through the lens of performance practices and argues that living history has fallen into a comfort zone of merely “undoing history” and tracing time back to a past moment directly, and effortlessly, from today. Such homogeneity, Magelssen argues, is native to the work of museum professionals who may aspire to a linear format rather than addressing the ebbs and flows of history on the margins (2007, xiii, 59). Beyond the scope of this study is David Allison’s examination of museum staff who use costumed interpretation in museums that are not entirely living history museums, such as the Children’s Museum of Indianapolis which employed live, first-person accounts in the gallery for the program The Power of Children to tell the stories of Anne Frank, Ruby Bridges, and Ryan White. See David B. Allison, Living History: Effective Costumed Interpretation and Enactment at Museums and Historic Sites, Lanham: AASLH, 2016, 41–61.

[5] For instance, “communication skills” and “leadership skills” (variables 1 and 3 of Crawford and Machemer) were developed in particular by 3D student Hannah Chase who guided the theatre actors in the fall 2019 and communicated what the software needed from them in order to correctly/effectively acquire what we needed. Her directives not to cross hands over the body and how to gesture properly put her in the position of domain knowledge (motion capture) that would supersede domain knowledge from theatre by asking actors to act unlike actors in order to yield the results that we needed for the motion capture.

In addition, “understanding through social interaction” and “flexibility in day-to-day project management skills” (variables 13 and 5 of Crawford and Machemer) were gained by computer science student Kunal Shitut as he received the rigged and animated model from the design team which was used to create the application for the HoloLens. The navigation back-and-forth between computer science and 3D digital design guided the way the application was created and achieved an outcome that would not have been otherwise achieved if working on one’s own, without conversation and input from the art and design faculty.

Further, museum studies students gained the “ability to direct [their] own learning” and “ability to identify needs and tasks” (variables 15 and 16 of Crawford and Machemer), particularly through their research and writing of monologues. Finally, it is anticipated that 3D digital design costume students, in the spring 2020, will also gain incidental learning skills as they develop costumes for our virtual museum guide and additional characters that we will develop over the next several months.

Bibliography

Alexander, Edward P., Mary Alexander, Juilee Decker. 2017. Museums in Motion: An Introduction to the History and Functions of Museums. Lanham, MD: Rowman & Littlefield for the AASLH.

Allison, David B. 2016. Living History: Effective Costumed Interpretation and Enactment at Museums and Historic Sites. Lanham: AASLH.

Anderson, Jay. 1982. “Living History: Simulating Everyday Life in Living Museums.” American Quarterly 343: 290–306. https://doi.org/10.2307/2712780.

Armstrong, Helen Joseph. 1995. Patternmaking for Fashion Design. New York: HarperCollins Publishers.
Bedford, Leslie. 2001. “Storytelling: The Real Work of Museums.” Curator: The Museum Journal 441: 27–34. https://onlinelibrary.wiley.com/doi/abs/10.1111/j.2151-6952.2001.tb00027.x.

Bolger, Stuart. 1985. Genesee Country Museum: Scenes of Town & Country in the Nineteenth Century. Rochester, New York: Flower City Printing.

Carr, Lizzy, and Brienna Johnson-Morris. 2019. “User Testing at Frameless Labs Symposium, November 22.” Unpublished, anecdotal evidence.

Crawford, Pat, and Patricia Machemer. 2008. “Measuring Incidental Learning in a PBL Environment.” Journal of Faculty Development 22, no. 2 (May): 104–111.

Decker, Juilee. 2019. “MUSE 360 User Testing #4: Virtual Museum Assistant.” 19 respondents, April 9. Unpublished data set.

Dewey, John. 1998. The Essential Dewey, edited by Larry Hickman and Thomas M. Alexander. Bloomington, Indiana: Indiana University Press.

Falk, John H., and Lynn D. Dierking. 2012. The Museum Experience Revisited. London: Routledge.
Gorsline, Douglas. 1994. What People Wore: 1,800 Illustrations from Ancient Times to the Early Twentieth Century. New York: Dover Publications.

Hammady, Ramy, Minhua Ma, and Carl Strathearn. 2019. “User Experience Design for Mixed Reality: A Case Study of HoloLens in Museum.” International Journal of Technology Marketing 13, no. 3/4. http://www.inderscience.com/offer.php?id=104600.

Handler, Richard and William Saxton. 1988. “Dyssimulation: Reflexivity, Narrative, and the Quest for Authenticity in ‘Living History.’” Cultural Anthropology 33: 242–260.

Holkeboer, Katherine Strand. 1993. Patterns for Theatrical Costumes: Garments, Trims, and Accessories from Ancient Egypt to 1915. New York: Drama Book Publishers.

Lombardo, Vincenzo, and Rossana Damiano. 2012. “Storytelling on Mobile Devices for Cultural Heritage.” New Review of Hypermedia and Multimedia 18, no. 1/2: 11–35. doi:10.1080/13614568.2012.617846.

Lowe, Hilary Iris. 2015. “Dwelling in Possibility: Revisiting Narrative in the Historic House Museum,” The Public Historian 37, no. 2 (May): 42-60.

Magelssen, Scott. 2007. Living History Museums: Undoing History through Performance
Lanham, Scarecrow.

Mori, Masahiro. 2012. “The Uncanny Valley: The Original Essay by Masahiro Mori,” translated by Karl F. MacDorman and Norri Kageki. IEEE Spectrum. https://spectrum.ieee.org/automaton/robotics/humanoids/the-uncanny-valley. Previously published as Masahiro Mori, “The Uncanny Valley,” Energy 7, no. 4 (1970): 33–35.

Noe, A. 2012. “Storytelling and the Uncanny Valley.” NPR, January 20. https://www.npr.org/sections/13.7/2012/01/20/145504032/story-telling-and-the-uncanny-valley.

Norouzi, Nahal, Kangsoo Kim, Jason Hochreiter, Myungho Lee, Salam Daher, Gerd Bruder, and Greg Welch. 2018. “A Systematic Survey of 15 Years of User Studies Published in the Intelligent Virtual Agents Conference.” In Proceedings of the 18th International Conference on Intelligent Virtual Agents IVA ’18. Association for Computing Machinery, Sydney, NSW, Australia, 17–22. https://dl.acm.org/doi/10.1145/3267851.3267901.

Pomerantz, Jeff. 2019. “XR for Teaching and Learning: Year 2 of the EDUCAUSE/HP Campus of the Future Project,” Educause October 10. https://www.educause.edu/ecar/research-publications/xr-for-teaching-and-learning-year-2-of-the-educause-hp-campus-of-the-future-project/factors-that-influence-learning.

Reid, Debra Ann. 2001. Living History-Social History or Post-Modernism: Toward a Historiography of Open-Air Museum Interpretation in the United States. Charleston, IL: Eastern Illinois University.

Roth, Stacy Flora. 2005. Past into Present: Effective Techniques for First-Person Historical Interpretation. Chapel Hill: University of North Carolina Press.

Roussou, Maria, Laia Pujol, Akrivi Katifori, Angeliki Chrysanthi, Sara Perry, and Maria Vayanou, 2015. “The Museum as Digital Storyteller: Collaborative Participatory Creation of Interactive Digital Experiences.” MW2015: Museums and the Web 2015. Published January 31. https://mw2015.museumsandtheweb.com/paper/the-museum-as-digital-storyteller-collaborative-participatory-creation-of-interactive-digital-experiences/.

Thierer, Joyce M. 2010. Telling History: A Manual for Performers and Presenters of First-Person Narratives. Lanham, MD: AltaMira for AASLH.

Tortora, Phyllis G., and Sara B. Marcketti 2015. Survey of Historic Costume. New York: Fairchild Books.

Weschler, Lawrence. 2011. Uncanny Valley and Other Adventures in the Narrative. Berkeley: Counterpoint.

Acknowledgements

The authors are grateful to the staff of Genesee Country Village & Museum, in particular Becky Wehle and Peter Wisbey. At RIT, we thank students Hao Su, Kunal Shitut, Hannah Chase, Lizzy Carr, Brienna Johnson-Morris as well as a number of students from the following courses: MUSE 360/Visitor Engagement and Museum Technologies; MUSE 225/Museums and the Digital Age; FNRT 231/Fundamentals of Acting; DDDD 517/Costume Hair and Makeup; DDDD 521/Character Design and Rigging; and CSCI 715/Applications in Virtual Reality. In addition, we are grateful to faculty colleagues in theatre at RIT, Andy Head and David Munnell, and Katherine Collett, Archivist, Hamilton College Archives, for providing historical images.

About the Authors

Juilee Decker is an Associate Professor of Museum Studies at Rochester Institute of Technology (RIT). She has served as Editor of Collections: A Journal for Museums and Archives Professionals since 2008. She earned her PhD in 2003 from the joint program in Art History and Museum Studies at Case Western Reserve University and the Cleveland Museum of Art.

Amanda Doherty is an Adjunct Professor in the department of 3D Digital Design at Rochester Institute of Technology (RIT). She is a costume designer and historian who has been working principally in the entertainment industry and is now teaching character development and costume design for digital characters. She received her MFA in Design from Penn State University.

Joe Geigel is a Professor of Computer Science at Rochester Institute of Technology (RIT) and co-director of the CS Graphics and Applied Perception Lab there. He earned his DSc. in Computer Science from George Washington University in 2000. His research interests focus on mixed reality multimedia projects that combine computer science, real-time graphics, art, music, and theatre to create interactive, live experiences.

Gary D. Jacobs is an Assistant Professor of 3D Digital Design at Rochester Institute of Technology (RIT). He has designed public spaces, stage productions, and themed environments for over 15 years. He is a certified LEGO® Serious Play facilitator and leads Design Thinking workshops for creative teams. Gary received his MFA in Entertainment Design from Pennsylvania State University.

0

Immersive Pedagogy: Developing a Decolonial and Collaborative Framework for Teaching and Learning in 3D/VR/AR

Abstract

In June 2019, a cohort of CLIR postdoctoral fellows convened Immersive Pedagogy: A Symposium on Teaching and Learning with 3D, Augmented and Virtual Reality at Carnegie Mellon University. The symposium sought to bring together a multidisciplinary group of collaborators to think through pedagogical issues related to using 3D/VR/AR technologies, as well as to produce and disseminate materials for teaching and learning. This essay presents the Immersive Pedagogy symposium as a model for interrogating and developing pedagogical practices and standards for 3D/VR/AR; we offer a decolonial, anti-ableist, and feminist pedagogical framework for collaboratively developing and curating humanities content for this emerging technology by summarizing the symposium’s keynotes, workshops, as well as its goals and outcomes. Workshops, keynotes, and participant conversations engaged with decolonial and feminist methodologies, practiced accessible design for universal learning, offered templates for humanistic teaching, and illustrated the possibilities of using 3D/VR/AR to extend critical thinking. While 3D/VR/AR technologies demonstrate real possibilities for collaborative, multidisciplinary learning, they are also fraught with broader concerns prevalent today about digital technologies, as well as complex issues specific to 3D/VR/AR. There is a clear need to assemble academic practitioners on a regular basis in order to facilitate an ongoing discussion about 3D/VR/AR technology and its responsible, meaningful use in teaching and learning.

Introduction

As access to three-dimensional (3D) technologies has become increasingly available in academic venues, the desire to teach with these emerging technologies, particularly augmented reality (AR) and virtual reality (VR), has outpaced digital humanists’ abilities to provide meaningful support for immersive projects. There is a growing and ongoing need to produce shared and open pedagogical materials adaptable to the needs of teachers in various professions and disciplines and are accessible to students without significant coding experience. This need is partially driven by the contingencies of relatively new and rapidly updating technologies, as well as the fact that support for commercially-available immersive tools are tailored for industry purposes. Game-driven tutorials, for example, do not always take into consideration the needs of humanities practitioners seeking to integrate critical thinking with technical mastery. Contemporary contexts for emerging technologies can structure our interactions with 3D/VR/AR. Though not always visible to users, these can have the effect of naturalizing problematic historical and political narratives through selective access to resources and functionality.

Nonetheless, game engines that offer free educational licenses have been repurposed for academic inquiry and teaching over the past decade. For example, Unity Technologies’ Unity 3D game engine is utilized by over 4.5 million users and has been at the forefront of historical and archaeological 3D visualizations in scholarly research. First available in 2005, the Unity 3D game engine has been used to make approximately 60% of all AR/VR applications and is used by 90% of AR/VR companies (“Public Relations” 2019, np). Educational licenses are available for students and educators seeking to use the engine for scholarly or creative use. Its main competitor, the Unreal Engine, while initially inaccessible beyond professional and academic institutions with licenses, dropped its paywall for educational use in September 2014. VR headsets, once a hypothetical fantasy or niche short-lived technology, are now commercially viable and relatively inexpensive for institutions to purchase. In a few years, the financial barrier for individuals may diminish; in the meantime, Google Cardboards and other stereoscopic viewers with fewer interactive features currently provide alternatives for students with access to smartphones. However, students are also increasingly able to make use of interactive 3D/VR/AR technology within dedicated spaces in academic libraries, maker spaces, media studios, and community outreach centers. Yet, we would be remiss not to point out that access is still mediated by other social hierarchies; 3D/VR/AR technology is still not accessible in much of the Global South, or in marginalized communities across the world. These aforementioned developments still privilege students at institutions that dedicated staff or faculty to maintain and encourage use of 3D/VR/AR technologies and facilities.

This is all to say that in our current 3D/VR/AR moment, digital humanists have a lot to navigate. Current 3D/VR/AR pedagogy and projects can pose problems related to accessibility and long-term preservation of projects and assets, and often run afoul of minimal computing recommendations. Yet the technology offers rich possibilities for multidisciplinary research and collaboration; many virtual reality projects combine art production, computing, archival research, network theory, and data visualization, among other practices. Given its potential for scholarship and teaching, understanding how to use the technology responsibly necessitates engaging with active practitioners to identify what is now possible and what still needs to be done to facilitate productive use of 3D/VR/AR. As many key problems are likely to persist through subsequent permutations of the technology and its use in educational settings, this conversation needs to be ongoing and open. What humanists within and beyond the academy have to say about 3D/VR/AR will probably not be unique to humanistic inquiry. This dialogue will provide crucial critical approaches to the emerging technologies’ advantages and limitations that will be of use to industry professionals as well as the casual creative user. A vocal contingent of humanists seeking to think and learn with 3D/VR/AR may, in fact, fill a wider sociocultural need by addressing these issues.

This is the context in which a small cohort of 2017–2019 Council on Library and Information Resources (CLIR) Postdoctoral Fellows organized Immersive Pedagogy: A Symposium on Teaching and Learning with 3D, Augmented and Virtual Reality at Carnegie Mellon University on June 26 and 27, 2019. The CLIR cohort included Lorena Gauthereau (University of Houston), Jessica Linker (Bryn Mawr College), Eric Kaltman (Carnegie Mellon University), Emma Slayton (Carnegie Mellon University), Neil Weijer (Johns Hopkins University), Alex Wermer-Colan (Temple University), and Chris Young (University of Toronto). The goal of this symposium was to assemble a wide range of stakeholders to develop teaching materials and strategies that considered problems inherent and specific to immersive technologies, as well as to address problems that affect but are not unique to 3D/VR/AR. It is for this reason the symposium was so attentive to decolonial and feminist methodologies in thinking about appropriate pedagogical applications. Building on the previous work of scholars such as María Cotera, Elizabeth Losh, Tara McPherson, Angel Nieves, Roopika Risam, and Jacqueline Wernimont, we have advocated for an intersectional digital humanities that interrogates a wide range of technologies through the critical methods developed by the fields of ethnic and feminist studies. Such methods, we argue, can highlight the ways that technologies often leave out marginalized people by replicating colonial hierarchical structures including race, ethnicity, class, gender, and disability.

The Immersive Pedagogy symposium offered an early—if not first-of-its-kind—opportunity to have productive conversations about what critical approaches to 3D/VR/AR could look like from a multidisciplinary and multi-professional perspective. Additionally, the symposium sought to seed collaborations within and beyond academic institutions and stand as a model for future conversations on these topics. In recounting our experiences with different applications of 3D/VR/AR technology in pedagogical spaces, the group tackled a number of thorny issues, such as accessibility in hardware and bias in asset stores, while acknowledging that we would need to continue the dialogue by reconvening. We sought to develop teaching materials collaboratively with the long-term plan of sharing these resources through a variety of means, including via open-access publications. In the remainder of this essay, the Immersive Pedagogy organizers describe the symposium’s theoretical foundation and methodological approaches as a model for structuring communities around 3D/VR/AR, summarize some of our group’s findings, and invite digital humanities practitioners to help us to continue this work.

Structuring a Symposium on Decolonial Models of Immersive Pedagogy

Because the initiative was organized by CLIR postdoctoral fellows, the symposium emphasized diverse ways that libraries participate in creating, curating, and preserving 3D/VR/AR pedagogical materials. We considered faculty, staff, and students as equal partners in 3D/VR/AR projects, and aimed to include early career researchers at the table. Overarching goals for the symposium included teaching faculty and librarians how to support and enable learning for students using 3D technologies, and to help students to disseminate skills within their own communities. By bringing together scholars from a wide range of disciplines and professions, we addressed problems while identifying new ones. Participants had the opportunity to share links and descriptions to their projects (current and in progress) with each other prior to the symposium via a Slack channel and Google Docs. They also shared information on their work during a lightning talk round as examples of the kinds of humanistic projects 3D/VR/AR could cultivate. The symposium began and ended with keynotes from experienced practitioners whose work modeled creative and responsible uses of the technologies.

Our opening keynote speaker, Angel Nieves (Associate Professor of History and Digital Humanities at San Diego State University), presented “Developing a Social Justice Framework for Immersive Technologies in Digital Humanities.” Nieves’s talk outlined strategies for achieving social justice through digital-supported inquiry, highlighting his own work on Mapping Soweto, a 3D reconstruction of apartheid South Africa. Nieves emphasized the need to ground digital work in women of color theory and argued that fields such as ethnic studies have developed a foundational structure that would benefit the field of digital humanities as a whole:

If we brought the sorts of methodological and practice-based questions about power, privilege, and access from ethnic studies to our work in immersive technologies, we might begin to see new ways of harnessing these tools–that originated as part of the military industrial complex–to serve our social justice needs. (Nieves 2019)

Mapping Soweto draws from Belinda Robtnett’s (1997) work on social movement theory, revealing the often messy, multilayered narratives of social movements by visualizing a map of spatial liberation. This 3D representation shows what Nieves terms an “intersectional cartography,” or a network of social activists—especially networks of women and young girls—across townships “and how those activist networks were embedded into the physical geography and vernacular architecture of individual houses, streets, and neighborhoods” (Nieves 2019). Attention to intersectionality further reveals the ways multiple identities—township, gender, sexual orientation, class, and race—came together to form a cohesive activist movement, whose complexities are often lost in the official retelling of history. In particular, Nieves identified immersive technologies as one way to “re-establish coalition-building potential” (2019) with local communities and reminded us that the important work of recovering marginalized histories for social justice is often messy.

Two image composite. Top image is of Angel Nieves standing behind a podium delivering his keynote speech. Bottom is a slide showing a Unity 3D model of Winnie and Nelson Mandela House, in Soweto South Africa (generated September 2018).
Figure 1. Angel Nieves presents “Developing a Social Justice Framework for Immersive Technologies in Digital Humanities” at the Immersive Pedagogy symposium.

Our closing keynote speaker, Juliette Levy (Associate Professor of History at the University of California, Riverside), presented “How Not to be a Replicant: Working Towards a Useful VR.” Working with a team of women programmers, Levy has developed VR simulations for teaching abstract concepts related to historical thinking, interpretation, and writing. Levy’s keynote presentation focused on the question of gaming and interactivity; and she traced the origin of her experimentations in VR from teaching large lecture classes numbering in the hundreds in hybrid and online courses. Rather than approach VR in the mode of cultural heritage projects, reproducing a historical location, to deal with pedagogical problems commonly experienced in online learning, Levy’s team built Digital Zombies (see Levy 2017), an abstract simulation meant to introduce students to the hierarchy of library information and assessment of primary and secondary resources through game-based learning. Levy envisioned a VR environment for her historical research methods class that not only encouraged students to follow a written outline of research steps, but to extend their library experience in a more immersive, playful way by completing a series of game-like missions related to research that students would be more likely to remember. Levy argued that the cognitive effect of a VR experience has a lasting impact on users: “What matters about doing something in VR isn’t about what happens in VR, but what happens outside of VR, after the VR experience” (Levy 2019). Yet, despite the advantages of VR, Levy warned that a lack of critical conversation and pedagogy around digital literacy can have dire consequences, as increasingly ubiquitous immersive technologies become exploited to misrepresent historical events. The stakes for fomenting critical conversations between technology creators, consumers, and scholars, therefore, are quite high, as they could have lasting effects on how people choose to build and interpret virtual representations of historical events and people.

Juliette Levy stands at the podium while presenting a slide reading “fake news, fake history, alternative facts, virtual reality or fake reality” in front of an image of John Lennon and Che Guevara playing the guitar.
Figure 2. Juliette Levy presents “How Not to be a Replicant: Working Towards a Useful VR” at the Immersive Pedagogy symposium.

The symposium included five workshops that centered on theory, methods, and practices significant to and capable of incubating pedagogy related to US Latinx, Latin American, and Caribbean studies, which we prioritized when considering applicants. The workshop topics were: (a) Decolonial Methodology and Theory; (b) Accessible Immersive Pedagogy; (c) Integrating Immersive Technology in the Classroom; (d) Critical Writing for Immersive Tech; and (e) Collaboratively Designing 3D/VR Experiences. The Immersive Pedagogy organizers, joined by Jasmine Clark (Temple University) and Juliette Levy, led the participants through these interactive workshops (“Program” 2019). Pedagogical content crafted by participants before, during, and after the symposium included a bibliography of 3D/VR/AR-related readings, an archive of workshop slides, video recordings of keynote presentations, adaptable templates for pedagogical activities, and working models of 3D/VR/AR pedagogical applications. For example, Kat Hayes and Samantha Porter submitted a video walkthrough of their IOS app Virtual MISLS that explores historic buildings at Fort Snelling, while Meaghan Moody and Carol Salmon submitted a description of their work with students using a virtual map of historic Paris to better understand life under German occupation during World War II.

Carnegie Mellon University (CMU) Libraries hosts the symposium’s materials on its institutional repository, KiltHub. KiltHub provides stable, long-term global open access storage for 3D/VR/AR assets, and functional applications, as well as pedagogical and technical documentation. Materials in this repository are held for a minimum of ten years, ensuring that what is submitted will remain available past typical terms of software updates. The teaching materials produced during and following the symposium will also be published in the Digital Library Federation’s Pedagogy Working Groups open-access series, the DLF Teach Toolkit. The materials will be revised and tested, including during a pre-conference workshop at DLF’s Annual Forum 2020, pending acceptance.

Group of twenty-one symposium attendees sitting in a circle actively engaged in a workshop discussion session.
Figure 3. Immersive Pedagogy symposium participants in discussion.

The following essay sections explore the key components of the symposium, which outlined the theoretical foundations to decolonizing development and curation of 3D/VR/AR tech, before guiding participants through workshops on decolonial critique and accessible design, on integrating immersive technology into the classroom and beyond, and on collaboratively designing 3D/VR projects.

Decolonial Foundations: Critical Approaches to the Development and Curation of 3D/VR/AR Technologies

To practically introduce the decolonial methodologies and theories crucial to our workshops on developing and curating 3D/VR/AR materials, the Immersive Pedagogy symposium opened with a workshop, led by Gauthereau and Young on the “walkthrough method” (Light, et. al. 2018, 881–900), a critical analysis of technology using the Unity Asset Store as an example. This exercise was contextualized through a theory of decolonial pedagogy and a discussion on the critical analysis of the game platforms that curate content for 3D modeling and representation.

The application of decolonial theory and methods to digital pedagogy allows students to interrogate and resist colonial, hierarchical epistemologies, especially the privileging of Western European and Anglocentric knowledge structures. Such an approach is increasingly necessary as 3D/VR/AR technologies become integral to Western education systems and overwhelmingly applied to cultural heritage projects by and for Western consumers. While colonialism refers to the “political and economic relation in which the sovereignty of a nation or people rests on the power of another nation,” making that nation an “empire,” coloniality “refers to long-standing patterns of power that emerged as a result of colonialism, but that define culture, labor, intersubjective relations, and knowledge production well beyond the strict limits of colonial administration” (Maldonado-Torres 2007, 243). Thus, coloniality denotes the ways in which colonial hierarchies of power continue to structure our everyday lives (i.e. racialized class hierarchies, labor hierarchies, gender hierarchies, the gender binary, racism, etc.). Decolonialism urges us to actively de-link from colonial epistemologies and ontologies in order to avoid re-creating colonial worldviews and hierarchies.

Considering the ways that 3D/VR/AR technologies allow users to create immersive worlds and environments, the symposium sought to stress the need to avoid replicating the colonial gaze. Representing marginalized people through this gaze continues to enforce racialized and gendered hierarchies of power. Colonial epistemologies continue to control knowledge production, not only through institutional archives, but also through academic research, digital projects, and 3D/VR/AR environments. Jacqui Alexander and Chandra Talpede Mohanty argue that decoloniality has a “pedagogical dimension” as it obligates us “to understand, to reflect on, and to transform relations of objectification and dehumanization, and to pass this knowledge along to future generations” (1997, xxviii-xxix). For this reason, the symposium’s first workshop exercise involved guiding participants through a decolonial walkthrough of the Unity Assets Store. The walkthrough method requires researchers to directly engage with “an app’s interface to examine its technological mechanisms and embedded cultural references to understand how it guides users and shapes their experiences” (Light, et. al. 2018, 882). We asked participants to browse and search the Unity Assets Store for 2D, 3D, audio, and animation assets and interrogate them using a decolonial approach, as well as to document their walkthrough by taking notes, taking screenshots, and recording audio-visual content.

To guide the decolonial inquiry, we asked participants to consider a set of questions adapted from Roopika Risam’s discussion of the stakes of postcolonial and decolonial digital humanities (2019, 35–46):

  • What are compulsory activities within the Unity Asset Store?
  • What are the social hierarchies within the menu system?
  • To whom and which types of users is this knowledge accessible?
  • What is considered a “legitimate” asset within the Asset Store?
  • Whose epistemologies, such as histories, languages and memories, are considered important enough to archive in the Asset Store?
  • What knowledge or assets are privileged within the Asset Store?
  • Does the asset avoid the exoticization or fetishization of a people/cultures?

This inquiry resulted in participants recognizing the disproportionate representation of a Eurocentric worldview. For example, they noticed that the search term “Viking” yielded twice as many results as “Native American,” whereas the term “Indigenous” yielded zero. Among results for the search term “Mexican,” participants discovered a Mexican Restaurant Pack that reflected generalized stereotypes of Mexican aesthetics and cuisine, reduced to bottled hot sauce, chips and salsa, and a decorative green parrot. Assets also reduced the multiple and varied cultures, nations, flora, and fauna of the entire African continent to the myopic colonial imaginary of only the Serengeti, populated by wild animals. During group break-out sessions exploring the Asset Store, participants discovered a potential intervention through editing crowdsourced user tags. Like during Wikipedia Edit-a-Thons, users could challenge the authenticity of colonial representations of people, cultures, and nations by tagging or reviewing assets as not authentic, representing stereotypes, reproducing colonial views, etc. Since the symposium, unfortunately, Unity has removed the user tagging option and currently limits metadata generation to the individual uploading the asset.

This workshop stressed that engaging in decolonial work requires a constant questioning of how knowledge (3D/VR/AR environments, research, stories, syllabi, etc.) is being produced, who is producing it, whose stories are being told, and how these stories are being told. Not only should we consider what histories are told in the digital world, but we must also attend to the ways in which they are produced. As a result, the participants learned that generating and interacting with 3D/VR/AR environments they must use decolonial methods to acknowledge their role as world-creators and reflect on the ways that these technologies often replicate colonialism.

In the following workshop, Clark foregrounded the ableism endemic to technological innovation in the West, introducing participants to accessible user design for virtual reality. This involved a tutorial on developing alternate access plans for disabled students in classrooms. Clark’s work with Temple Libraries’ colleagues Jordan Hample and Wermer-Colan has prioritized research into and creation of accessible features for VR during their development of the Virtual Blockson: A Primary Source Teaching Tool for Secondary Education (Clark 2018, np). Clark’s workshop overviewed the standards of the World Wide Web Consortium’s Web Content Accessibility Guidelines in order to showcase the problems with applying standards created for web-based screens to virtual reality environments and experiences. She related an overview of key advancements that can be made to enable universal design for this emerging technology ranging from innovations in haptic feedback to caption legibility. Clark’s talk focused on guiding participants through strategies for accessing resources for disabled students at their universities. She led participants through an exercise with a template she created for developing “alternate access plans” that enable teachers to offer comparable options for students who cannot use the available VR and/or AR hardware and software. This approach to accommodating students with different learning styles provides a realistic way for teachers to work with emerging technologies in academic institutions, most of which still lack sufficient resources to support disabled students in the use of analog technologies.

Virtual Lessons: Integrating Immersive Technology in the Traditional Classroom and Beyond

After the symposium’s opening workshops on decoloniality and anti-ableism in immersive pedagogy, Levy’s workshop put to practice the principles she laid out in her closing keynote address on the idiosyncratic game mechanics for simulating virtually interactive dialogue and exam questions involving classification. VR offers, Levy argued, a unique pedagogical opportunity, functioning as a distraction-free zone where her students were able to recollect experiences at a much higher rate compared to other learning activities. During the workshop, Levy asked symposium participants to select several library books from various library collections and work in groups to think about how to put the texts in conversation with each other based on titles, subject headings, table of contents listing, and a quick skim of their contents. Levy demonstrated how and why she constructed a VR environment that simulated this activity, as her students had to physically place boxes with various titles onto empty shelves in an order that reflected connections. The application of VR to this type of historiographical exercise, Levy maintained, left a lasting impression on the students that they were able to put into practice for essay assignments. Levy’s emphasis throughout her workshop on the pedagogical significance of “what happens before and after” the virtual experience, furthermore, offered a valuable foundation for the subsequent workshop on integrating writing exercises to guide student learning during virtual and augmented reality experiences.

Wermer-Colan’s workshop modeled how to guide undergraduate students across the disciplines through a structured composition exercise for reflection, in particular, by guiding the participants through a reflection on what they hoped to learn and do in the coming school year as they sought to develop their immersive pedagogy projects. To provide a context and model for students before their writing reflections, Wermer-Colan summarized his current projects employing 3D technologies for Temple University Libraries’ Digital Scholarship Center (now the Loretta C. Duckworth Scholars Studio). Temple Libraries has experimented with transforming the purposes of library collections, development, and reference work to enhance its learning and technology outreach, including through its Innovative Teaching with Makerspace Technology Grant and its newly constructed VR Lab in the new Charles Library. Wermer-Colan’s past experiences working in the Medgar Evers College Writing Center in the City University of New York (CUNY) system helped him to think about ways the Digital Scholarship Center can use 3D/AR/VR technology to enhance learning across the disciplines.

As an example of Temple Libraries’ supporting the use of immersive technologies in class-room projects, Wermer-Colan detailed a collaborative project with Ajima Olaghere, Assistant Professor of Criminology working with her ethnography students to do “systematic social observation” of Philadelphia neighborhoods. This project used 360 cameras to record neighborhoods affected by Temple’s gentrification of North Philadelphia. The recordings were later viewed on twenty-dollar Desktek smartphone headsets that allowed students to remotely examine environments to understand what contributes to disorder and crime, while the instructor facilitated ways to maintain a critical understanding of what they were viewing. The accompanying writing exercises guided students to reflect on their mediated experiences of urban space and call into question the “broken windows theory,” common assumptions that visible signs of public disorder exacerbate criminal behaviors. The use of phone-based headsets also invited an opportunity for students to consider the physical processes that enable virtual technology. Instructors were faced with the problem of scaling pedagogical uses of VR; as this project used relatively inexpensive headsets, workshop participants considered how to create immersive experiences similar in quality to those offered by state-of-the-art VR headsets like the HTC VIVE or the Oculus Rift that, as of 2020, cost hundreds of dollars.

To illustrate the role libraries and digital scholarship centers can play in the curation of 3D content for teaching and learning, Wermer-Colan overviewed a complementary use of immersive technologies. His collaboration on the Virtual Blockson project with Digital Scholarship Librarian Jasmine Clark, Academic Technician and Developer, Jordan Hample, and Blockson Archivist Leslie Willis-Lowry aims to recreate Temple’s Charles L. Blockson Afro-American Collection as a virtual reality game for innovating the teaching of primary source literacy in high schools across Philadelphia. The project at its heart allows a small, fixed collection and its reading room to be available to students remotely, lowering the intimidation factor and physical limitations of these spaces, while enabling interactive explorations of historical artefacts. The Virtual Blockson offered an opportunity to discuss how libraries can help curate interactive gaming environments for remediating archival collections and cultural heritage sites to foreground previously marginalized histories. In these contexts, virtual reality offers affordances for lowering the barrier for students to use archival sources and spaces, facilitating access and accessibility, and offering students a novel medium through which to conceptualize analog and digital literacies necessary to navigate the changing new media world today.

3D-rendering of the Charles L. Blockson Afro-American Collection, a few sculptures and a painting on display in the reading room.
Figure 4. Screenshot of the Virtual Blockson designed by Jordan Hample using Unity 3D. For more, see Jasmine Clark’s “Progressing Towards an Accessible VR Experience”: https://sites.temple.edu/tudsc/2018/11/07/progressing-towards-an-accessible-vr-experience/.

Wermer-Colan foregrounded in both these projects the use of writing exercises to help students reflect on their virtual experiences in meaningful ways. The 360 SSO writing exercise encouraged humanistic thinking about the technology by asking students to compare their field work exercises with the virtual experience, as well as writing reflections that asked the students to identify various ways the 360/VR technology mediated said experience. Similarly, humanistic writing exercises were designed to guide students before and after their experience of the Virtual Blockson’s introduction to archival spaces, etiquette, and practices through game-based, interactive experiences. Drawing upon the Society of American Archivists’ Standards for Primary Source Literacy and the Common Core Standards for historical understanding, digital literacy, and critical thinking, these critical writing questions ensure students reflect upon the virtual experience of library collections’ historical artifacts from the African diaspora. After offering these models to the Immersive Pedagogy participants, Wermer-Colan guided the group through a critical writing exercise to reflect on their own plans to implement the 3D/VR/AR technology for various pedagogical purposes. Wermer-Colan encouraged participants to think of resources at their local institutions, pedagogical standards in their disciplines, and affordances in the spatialized medium of VR for enhancing their approaches to teaching. The writing exercise simulated the kind of exercise participants could implement in their own pedagogy, while offering an opportunity for the symposium participants to reflect on what they had learned during the workshops.

Feminist Reconstructions: Collaboratively Designing 3D/VR Experiences

The concluding workshop, run by Linker and Young, offered a sustainable model for including students as partners in the creation of 3D/VR pedagogical materials, through an overview of Linker’s time creating the Bryn Mawr Women in Science project with her various undergraduate partners: Elia Anagnostou, Courtney Dalton, Jocelyn Dunkley, Tanjuma Haque, Arianna Li, and Linda Zhu. From 2017 to 2019, Linker taught undergraduate students how to integrate historical inquiry with 3D technology in order to think about women’s invisible scientific labor, the spaces they occupied, the tools they used, and their everyday lived experiences. The project considers Margaret Rossiter’s “The Matthew Matilda Effect in Science,” which articulates a systematic disparity in affording women scientists credit for sophisticated and important discoveries, which in turn necessitates that historians find ways to tell stories in order to make their labor visible. It likewise adapts aspects of Pamela Smith’s Making and Knowing Project by taking seriously the need to consider scientific processes. However, rather than engaging in physical reenactment, students offered up women’s processes in a modern, digital format, contextualized by a recreation of spaces that were no longer intact or available for historical analysis.

3D rendering of a biology lab created for the Bryn Mawr Women in Science Project. Rendering contains depictions of glassware, scientific artifacts, equipment, and laboratory furniture.
Figure 5. Screenshot of the 3D-rendered Advanced Biology Lab c. 1900, from Bryn Mawr Women in Science.

Linker and her students recreated two laboratory spaces that had once existed at Bryn Mawr College in the late nineteenth and early twentieth centuries, the Major Chemistry Lab and the Advanced Biology Lab.[1] Students learned a variety of 3D skills, including 3D modeling, photogrammetry, various mechanics of the Unity 3D game engine, and the Oculus Rift. Interactive WebGL versions of the project are available online, and a VR demo of Bryn Mawr’s Advanced Biology Lab was available at the conference. The Advanced Biology Lab was the site of early genetic research and a place once utilized by Nettie M. Stevens, the subject of Stephen Brush’s Nettie M. Stevens and the Discovery of Sex Determination by Chromosomes. Years before Margaret Rossiter coined the phrase “the Matilda Effect,” Brush identified that Stevens’ discoveries had been overshadowed by male collaborators or individuals working concurrently on the same subject. Her contributions had likely been diminished because she was a woman. Students researched each space by spending time in Bryn Mawr’s Special Collections. Through building each laboratory, the students became aware of how to put historical materials in conversation, as no resource could tell them everything they needed to know to build and contextualize the 3D models. Pedagogically, the two-year process of building was designed to seed humanistic deployment of 3D technologies by undergraduate collaborators. Afterward these students participated in professional presentations of the digital and historical work, and served as ambassadors to various communities in order to disseminate the skills the project cultivated to a wider audience.

Linker enabled her students to accomplish a lot in a short period of time; no student was an expert in the technology or in historical research prior to their tenure on the project. This was intentional, as she sought to teach rather than to employ experts. Students represented a diverse range of interests and majors, and all students participated in each phase of production (rather than assigning humanities majors to research and STEM majors to coding) so that afterwards, they could create projects similar to this on their own. Part of what facilitated their success is that she treated them as equal partners in the project, making decisions with them throughout the two years they worked together.

To prepare her students to participate as equals, she devised a plan that would serve as an introduction to using 3D technology to address social and pedagogical problems, and would also serve as a diagnostic tool for assessing student strengths and interests. Essentially, students were asked to propose and implement a 35-hour project (which could be run over the course of days or weeks, depending on individual need) that used an aspect of the Unity 3D game engine’s functionality to teach users about something the students cared about. Students drafted plans that identified what they knew, what they needed to learn, and were prompted to think about modularity, such that students could scale the project if they were running out of time. Students who were not familiar with coding at first could use Unity’s GUI interfaces to produce fully functional scenes, allowing for students with varying levels of proficiency with computer science to produce something useful by the end of the exercise. By the end of the 35-hour period, students not only had a small project they could put in professional portfolios, but had become proficient in a particular aspect of Unity, thought about the technology as a means to serve others, and in implementing their projects, had a better sense of what they would need to do going forward. It also convinced them that they were capable of using the technology in a way the Unity tutorials did not engender. Linker and Young guided the participants in thinking through how symposium participants might adapt this exercise for their own project teams.

Conclusion

Through the symposium and the workshops described above, participants engaged in conversations around designing socially-conscious pedagogy for 3D/VR/AR. Building a framework for teaching and learning with 3D/VR/AR technologies founded in decolonial theory and practices resonated with our participants. This enabled the group to evaluate how projects and assignments fit into an ethical model for cultural heritage pedagogies. The symposium closed with a productive discussion about what the participants learned, with a focus on planning for future steps.[2] Several participants suggested the importance of backward design, which would specifically place the learning outcomes as the first step in creating 3D/VR/AR and related assignments.

Conversations among group members brought up multiple questions, such as: how do we anticipate student use? How do we adjust our use of 3D/VR/AR in response to unexpected circumstances? How do we introduce emerging technologies in the classroom while accommodating individuals unable to take advantage of the intended purposes of ready-made hardware and software? How can these technologies enhance hybrid and online learning? Are students (or faculty) distracted by the freedom of immersive environments? Can we create bilingual metadata in a VR environment? If one could, where would you display subtitles or transcriptions in a virtual or augmented environment?

These conversations confirmed that digital humanists would benefit from future cross-institution discussions of 3D/VR/AR, as well as from shared access to teaching materials, which are often siloed within institutions and departments. Students engage differently with course concepts and each other, depending on the application of the technology within that course. Student learning is dependent on the skills and interests of individual instructors; collaboration is necessary for producing robust materials and responsible projects. Perhaps the most challenging task is creating accessible and sustainable materials applicable to multiple modes of disciplinary learning outcomes at a time of rapid technological and institutional transformation.

In an effort to increase the reach of the conversations that arose out of Immersive Pedagogy, the symposium organizers are working to produce an open-access, peer-reviewed publication containing lesson plans and educational material to facilitate disciplinary and interdisciplinary work that engages 3D/VR/AR technologies. This project aims to extend the work of the Digital Library Federation (DLF) Pedagogy Working Group’s Teach Toolkit that provides lesson plans for digital library instruction.[3] To guide educators to adapt immersive technologies to the needs of diverse disciplines, the Immersive Pedagogy teaching materials will introduce a range of 3D hardware and software, including asset or game repositories. The teaching materials will include diverse lesson plans with tailored learning outcomes, introducing a representative sample of available immersive technologies and resources while addressing humanistic pedagogical goals. Because this project was born out of the CLIR postdoctoral fellowship program, it aims to contribute to the growing field of scholarship on the crucial role that academic libraries or research and teaching centers can play in the integration of immersive technologies across the curriculum.

The Immersive Pedagogy symposium’s prioritization of decolonialism, feminism, and accessibility speak to a radical and critical perspective that can apply to a range of 3D/VR/AR applications and instruction methods. Indeed, in starting conversations on how to promote making immersive experiences accessible and inclusive, there is an opportunity to move beyond operational concerns to lasting pedagogical practices. For decades, contingencies have transformed education and cultural heritage, requiring us to rethink the potential of emerging communication technologies through a critical lens. More evident in the midst of the COVID-19 global pandemic, which has spurred the need for digital ways of teaching and learning, is the critical pedagogical use of virtual surrogates. These include 360° museum spaces and objects, 3D virtual meeting spaces, photogrammetry models, and interactive exhibits. By addressing upfront, rather than through remediation, the issues of social justice, accessibility, and decolonial pedagogies in immersive technology, educators can leverage these tools to respond to a transformative period in the education system.

Notes

[1] For a discussion of problems and considerations specifically related to the construction of historical 3D spaces, see Sullivan, Nieves, and Snyder 2017.

[2] For more detail, see the Immersive Pedagogy collaborative notes: “Shared Notes Wrap Up Session.” 2019. https://drive.google.com/drive/folders/1TSv8jrQlOlbPwi-TyvyOfV1_ZvA9I4y8.

[3] See the #DLFteach Toolkit 1.0: Lesson Plans for Digital Library Instruction.

Bibliography

Alexander, Jacqui M. and Chandra Talpede Mohanty, eds. 1997. “Introduction.” In Feminist Geneologies, Colonial Legacies, Democratic Futures: xxvii-xxiv. New York and London: Routledge.

Brush, Stephen G. 1978. “Nettie M. Stevens and the Discovery of Sex Determination by Chromosomes.” Isis 69, no. 2: 163–172. https://www.jstor.org/stable/230427.

Cotera, María. 2018. “Nuestra Autohistoria: Toward a Chicana Digital Praxis.” American Quarterly 70, no. 3: 483–504. https://muse.jhu.edu/article/704334.

Clark, Jasmine. 2018. “Progressing Towards an Accessible VR Experience.” Loretta C. Duckworth Scholars Studio. https://sites.temple.edu/tudsc/2018/11/07/progressing-towards-an-accessible-vr-experience/.

Digital Zombies. n.d. http://digitalzombies.ucr.edu.

“Guidelines for Primary Source Literacy.” 2018. Society of American Archivists. https://www2.archivists.org/standards/guidelines-for-primary-source-literacy.

Grayburn, Jennifer, Zach Lischer-Katz, Kristina Golubiewski-Davis, and Veronica Ikeshoji-Orlati, eds. 2019. 3D/VR in the Academic Library: Emerging Practices and Trends. Arlington, VA: Council on Library and Information Resources.

KiltHub Repository. n.d. Carnegie Mellon University. https://www.library.cmu.edu/kilthub/about.

Levy, Juliette. 2017. “‘Digital Zombies’– A Learner-Centered Game: Social Knowledge Creation at the Intersection of Digital Humanites and Digital Pedagogy.” In Social Knowledge Creation in the Humanities: An Open Anthology, edited by Alyssa Arbuckle, Aaron Mauro, and Daniel Powell. ITER Press. https://ntmrs-skc.itercommunity.org/.

———. 2019. “How Not to be a Replicant: Working Towards a Useful VR.” Keynote address, Carnegie Mellon University Symposium, Pittsburgh, PA, June 28.

Light, Ben, Jean Burgess, and Stefanie Duguay. 2018. “The walkthrough method: An approach to the study of apps.” New Media & Society 20, no. 3: 881–900.

Linker, Jessica, Elia Anagnostou, Courtney Dalton, Jocelyn Dunkley, Stella Fritzell, Tanjuma Haque, Arianna Li, Rachel Starry, and Linda Zhu. 2019. Bryn Mawr Women in Science. https://digitalscholarship.brynmawr.edu/howis/.

Losh, Elizabeth and Jacqueline Wernimont, eds. 2019. Bodies of Information: Intersectional Feminism and Digital Humanities. Minneapolis, MN: University of Minnesota Press.

Maldonado-Torres, Nelson. 2007. “On the Coloniality of Being.” Cultural Studies 21, no. 2: 240–270.

McPherson, Tara. 2012. “Why Are the Digital Humanities So White? or Thinking the Histories of Race and Computation.” In Debates in Digital Humanities, edited by Matthew K. Gold. Minneapolis: University of Minnesota Press.

Nieves, Angel David. 2019. “Developing a Social Justice Framework for Immersive Technologies in Digital Humanities.” Keynote address, Carnegie Mellon University Symposium, Pittsburgh, PA, June 27.

“Program.” 2019. Immersive Pedagogy: A Symposium on Teaching and Learning with 3D, Augmented and Virtual Reality. https://events.library.cmu.edu/immersive-pedagogy/program/.

“Public Relations.” 2019. Unity. https://unity3d.com/public-relations.

Risam, Roopika. 2019. New Digital Worlds: Postcolonial Digital Humanities in Theory, Praxis, and Pedagogy. Evanston, IL: Northwestern University Press.

Robnett, Belinda. 1997. How Long? How Long?: African American Women in the Struggle for Civil Rights. Oxford: Oxford University Press.

Rossiter, Margaret. 1993. “The Matthew Matilda Effect in Science.” Social Studies of Science, 23, no. 2: 325–341. http://www.jstor.org/stable/285482.

Sayers, Jentery, ed. 2017. Making Things and Drawing Boundaries: Experiments in the Digital Humanities. Minneapolis: University of Minnesota Press.

Smith, Pamela. n.d. The Making and Knowing Project: Intersections of Craft Making and Scientific Knowing. https://www.makingandknowing.org/. Accessed Dec. 13, 2019.

Sullivan, Elaine, Angel David Nieves, and Lisa M. Snyder. 2017. “Making the Model: Scholarship and Rhetoric in 3D Historical Reconstructions.” In Making Things and Drawing Boundaries: Experiments in the Digital Humanities, edited by Jentery Sayers, 301–316. Minneapolis: University of Minnesota Press.

Unity Assets Store. 2020. https://assetstore.unity.com/.

Acknowledgments

We would like to acknowledge The Andrew W. Mellon Foundation and the Council on Library and Information Resources (CLIR) for the microgrant that funded the Immersive Pedagogy symposium, as well as Carnegie Mellon University Libraries for hosting the event. Thank you to the entire Immersive Pedagogy team, including Eric Kaltman, Neil Weijer, and Chris Young for making the symposium possible. Last, but certainly not least, thank you to all the Immersive Pedagogy participants and keynote speakers, who created a positive, productive community of practice: Andy Anderson, DB Bauer, Katie Chapman, Elena Foulis, Kat Hayes, Juliette Levy, Juan Llamas-Rodriguez, Meaghan Moody, Angel Nieves, Samantha Porter, Coral Salomón, Julia Troche, Jordan Tynes, and Christa Williford.

About the Authors

Lorena Gauthereau is the Digital Programs Manager for the US Latino Digital Humanities program at the University of Houston’s Recovering the US Hispanic Literary Heritage. She received her Ph.D. in English and her M.A. in Hispanic Studies, both from Rice University. Her research interests include US Latinx studies, digital humanities, and decolonial theory. Orcid ID: orcid.org/0000-0002-7185-8982.

Jessica Linker is an Assistant Professor of History at Northeastern University. She was previously a Postdoctoral Fellow and Program Coordinator at the Consortium for History of Science, Technology and Medicine, a Visiting Assistant Professor at Bryn Mawr College, and the Director of Bryn Mawr Women in Science. She researches women’s scientific practices in early America.

Emma Slayton is the Data Curation, Visualization, and GIS specialist at Carnegie Mellon University Libraries. She obtained an MPhil from the University of Oxford in 2013 and completed her Ph.D. at the Faculty of Archaeology, Leiden University in 2018. Her current work centers around improving and supporting digital literacy efforts. Orcid ID: https://orcid.org/0000-0003-2230-3101.

Alex Wermer-Colan is a postdoctoral fellow in Temple University Libraries’ Loretta C. Duckworth Scholars Studio, where he coordinates research and pedagogical projects in cultural analytics and digital media arts. His editorial and scholarly criticism have appeared in PAJ: A Journal of Performance and Art, Twentieth Century Literature, The Yearbook of Comparative Literature, Lost & Found, Indiana University Press, and The Los Angeles Review of Books. Orcid ID: https://orcid.org/0000-0001-7030-6070.

Images are for demo purposes only and are properties of their respective owners. ROMA by ThunderThemes.net

Skip to toolbar