Augmenting the Education Toolkit
An open-source platform for students to learn biology textbook immersively, using augmented reality based on smartphones.
Harvard MDE, 2019
Team member: Anahide Nahhal, Audrey Haque, Berlynn Bai
Keywords: MOOC, AR, Textbook
Tools: Unity 3D, Vuforia, Keyshot AR
Textbooks remain dominate in schools, but emergent technologies like augmented reality (AR) present opportunities to transform education. BIO-AR, an educational toolkit, introduces a mixed-methods approach that transforms existing learning and teaching experiences, allowing learners to interact with invisible spatial structures and configurations. What once laid flat on a page emerges in full, three-dimensional glory with the click of a button.
As a medium of representing and communicating information that is less visible to human scales, Augmented Reality (AR) has been studied by pioneers in the educational community for a long time, but application has remained limited, partly due to cost. This impacts students by simplifying complex, three-dimensional systems, such as those found in biology, to two-dimensional instruction and conceptualization. BIO-AR is an educational toolkit that addresses this by pairing an AR mobile application with high-school textbooks, allowing students to better understand abstract concepts, construct 3D mental images of the cellular system, and retain information. By complementing the widespread use of textbooks, it aims for large-scale adoption in high-school biology classrooms without hurting school budgets, all to the benefit of students.
Problem
Even in the digital age, physical textbooks dominate K-12 education, and there are practical reasons (cost, content, access) that newer formats like e-books have not displaced physical texts (Greene, 2018). There are also educational reasons, including evidence that students retain more information from printed texts (Laster, 2010), which speaks to the interactive (in this case, tactile) nature of engaging a physical object. Researchers have found similar retention value with augmented-reality (AR) tools, which facilitate longer-term memory retention through physical body movement with the AR system (Iulian Radu, an unpublished paper; N. R. Hedley, 2003). Such tools could be particularly valuable in learning chemical structures, astronomy configurations, or other disciplines that require an understanding of spatial structures and configurations, and researchers have also reported higher levels of student motivation in using AR as educational media (Kaufmann and Dünser, 2007).
AR would bring significant value to science education in particular, where biology is traditionally taught via textbooks of two-dimensional images. This means that students currently build two-dimensional mental images of three-dimensional concepts throughout their primary education. Yet the ability to understand abstract concepts and construct three-dimensional images of biological systems and components is critical to problem-solving and designing solutions in the bioengineering space.
Educational content can be taught and learned in a variety of ways, and some are formats like AR are more suitable to specific fields and information retention than others. Despite this, there is a distinct underutilization of such technology classrooms.
Solutions
AR and textbooks could be leveraged as complementary learning tools for improved educational outcomes, and BIO-AR, an augmented-reality mobile application, focuses on high-school biology. As an educational toolkit, it pairs with high-school textbooks and allows students to better understand abstract concepts, construct 3D mental images of the cellular system, and retain information. It is emergent technology that communicates cell biology in more realistic and interactive formats.
BIO-AR is accessible with a smartphone, and the focus on a mobile application that pairs with textbooks was intentional since many public schools have limited funding. Over the past few decades, AR applications have become more widely used on mobile devices, and therefore more portable and less expensive (Yuen, S.; Yaoyuneyong, G.;& Johnson, E., 2011). BIO-AR requires no additional equipment and enables students to read their textbooks interactively, whenever or wherever they are, without burdening school budgets.
Design Goals
Deliver 3D content, including real data
Make this information mobile and accessible
Promote active learning, engaging students to move physically rather than sitting in front of their textbooks
User Experience
Users scan two-dimensional images in a textbook with their smartphone camera to launch the 3D-model viewer. A green box and BIO-AR logo within the textbook will communicate that a two-dimensional image has a 3D model associated with it. Once the target image is detected, the mobile app shows the mental image (or 3D image) on the phone screen to provide an AR experience. The mental image can be a 3D model generated by real data, a 3D video, or a labeled digital model with virtual buttons. Users can move their phone closer or farther to zoom in or out. By moving around, the user has a 360-degree view. Virtual buttons enable users to interact with the 3D models and learn immersively.
Design Process
BIO-AR is based on AR and developed through Unity 3D with a Vuforia plug-in. The API has high accessibility to mobile platforms, including Android and iOS, which makes the product easy to download, test, and implement. The development of BIO-AR consisted of three steps and focused on a neuron as its design and testing material.
Step 1: Target Image Detection
In order to launch the BIO-AR app, physical textbook pages required an AR tag for recognition. Instead of QR codes, target images were chosen to create more connected experiences. Vuforia provided the detection package that could be programmed and modified in Unity.
Step 2: Display Mental Images
As children of the target image in Unity’s hierarchy menu, 3D models of FBX file type, 3D animations, and images attached to plans can be shown in virtual scenes by scanning the target image. Coding with C++ was needed to fuse the game objects. The relevant size, as well as the location between one target image and the mental images generated, can be adjusted manually for optimal interaction.
Step 3: Virtual Button Control
Virtual buttons were introduced to the interface to provide a method for interactively labeling and learning the structure of a neuron through both mental and physical images. The target image worked as a reference for positioning virtual buttons. Note: this feature was not fully realized due to limited development time.
Testing and Challenges
- Initially, the BIO-AR prototype relied on a user’s physical movement to control rotation and zoom, but based on reviewer feedback, such functionality was updated for multi-touch control on a smartphone’s screen.
- To expand functionality with digital models on mobile devices, more plug-ins (other than Vuforia) would be needed, which would present challenges for managing unified APIs (application programming interfaces).
- iOS application was still in progress at the end of the semester.
- A platform (like GitHub) for sharing and connecting digital models to textbooks would be beneficial but would require much work.
Implementation plan
Industrial implementation would include establishing partnerships with leading textbook publishers and building an open-source, 3D-model platform. Ann West, a domain expert in the textbook industry, cautioned the BIO-AR team that approving content for textbooks can be time and effort consuming for textbook publishers. As such, the team identified an opportunity in offering unique, tailored, AR learning experiences to publishers as a means of improving their market competitiveness. Also, individual 3D-model contributors could utilize an open source platform to contribute work, which would not just contribute to the application, but promote the future development of augmented reality in education. Ultimately, the educational toolkit developed for BIO-AR has potential application across a number of fields that require an understanding of complex, 3D systems.