virtual environment research paper

REVIEW article

Virtual reality and collaborative learning: a systematic literature review.

• 1 Centre for Education and Learning, Delft University of Technology, Delft, Netherlands
• 2 Leiden Institute of Advanced Computer Science, Leiden University, Leiden, Netherlands
• 3 Interactive Intelligence, Delft University of Technology, Delft, Netherlands

Background: While research on Virtual Reality’s potential for education continues to advance, research on its support for Collaborative Learning is small in scope. With remote collaboration and distance learning becoming increasingly relevant for education (especially since the COVID-19 pandemic), an understanding of Virtual Reality’s potential for Collaborative Learning is of importance. To establish how this immersive technology can support and enhance collaboration between learners, this systematic literature review analyses scientific research on Virtual Reality for Collaborative Learning with the intention to identify 1) skills and competences trained, 2) domains and disciplines addressed, 3) systems used and 4) empirical knowledge established.

Method: Two scientific databases—Scopus and Web of Science—were used for this review. Following the PRISMA method, a total of 139 articles were analyzed. Reliability of this selection process was assessed using five additional coders. A taxonomy was used to classify these articles. Another coder was used to assess the reliability of the primary coder before this taxonomy was applied to the selected articles

Results: Based on the literature reviewed, skills and competences developed are divided into five categories. Educational fields and domains seem interested in Virtual Reality for Collaborative Learning because of a need for innovation, communities and remote socialization and collaboration between learners. Systems primarily use monitor-based Virtual Reality and mouse-and-keyboard controls. A general optimism is visible regarding the use of Virtual Reality to support and enhance Collaborative Learning

Conclusion: Five distinct affordances of Virtual Reality for Collaborative Learning are identified: it 1) is an efficient tool to engage and motivate learners, 2) supports distance learning and remote collaboration, 3) provides multi- and interdisciplinary spaces for both learning and collaborating, 4) helps develop social skills and 5) suits Collaborative Learning-related paradigms and approaches. Overall, the reviewed literature suggests Virtual Reality to be an effective tool for the support and enhancement of Collaborative Learning, though further research is necessary to establish pedagogies.

1 Introduction

Beginning in the 1980s, academia has studied how to support and enhance Collaborative Learning (CL) in educational settings using technology. Referred to as Computer-Supported Collaborative Learning (CSCL), this pedagogical approach stems from social learning, an educational theory revolving around the idea that “new behavior can be acquired through the observation of other people’s behaviors” ( Shi et al., 2019 ) and focusing on social interaction between learners. CSCL’s strength appears to lie in its flexibility: by using characteristics of technology, both distant and face-to-face collaboration, as well as synchronous and asynchronous collaboration between learners, can be supported ( Stahl et al., 2006 ). As such, CSCL has been attributed numerous affordances, including joint information processing, sharing resources and co-construction of knowledge ( Shawky et al., 2014 ; Jeong and Hmelo-Silver, 2016 ).

An on-going development in the field of CSCL is the use of Virtual Reality, a technology that ‘[transports] a person to a reality (i.e., a virtual environment) which he or she is not physically present but feels like he or she is there’ ( Rebelo et al., 2012 ). These virtual environments (VEs) are shared, simulated spaces that allow distributed users to communicate with each other, as well as to participate in joint activities, making them an effective tool for remote collaboration ( Daphne et al., 2000 ). VEs tend to be highly customizable; their visual representation can be realistic (i.e., similar to reality or containing recognizable elements from reality) or abstract (e.g., three-dimensional representations of abstract concepts) depending on their purpose, making VEs adaptable for many different fields and disciplines ( Jackson et al., 1999 ; Joyner et al., 2021 ). Virtual Reality (VR), then, functions as a human-computer interface, allowing users to access these VEs through a variety of hardware, including flat-surface monitors and displays connected to desktop computers, room-sized devices called CAVE systems that project the VE onto its walls and Head-Mounted Displays (HMDs), helmets or headpieces that visualize the VE individually for each eye. In some cases, users inhabit avatars, virtual embodiments that represent their place inside the VE, though in other cases (such as the aforementioned CAVE systems, where users do not have to wear HMDs), no avatars are required for users to detect each other. Like VEs, the visual representation of avatars can be diverse: avatars can provide realistic depictions of users’ real-life appearances, but can also be visualized as something abstract, such as geometric objects or animals. Using these avatars to mediate interactions with each other, users progressively construct a shared understanding of the VE together ( Girvan, 2018 ). Of particular interest is VR’s ability to “immerse” users, providing them a sense of being inside the VE despite its non-physical, digital nature ( Freina and Ott, 2015 ). This immersion may lead to a state of presence, wherein users begin to behave inside the VE as they would in the physical world ( Jensen and Konradsen, 2018 ). Affordances of VR in education include enhancement of experiential learning ( Le et al., 2015 ; Kwon, 2019 ), spatial learning ( Dalgarno and Lee, 2010 ; de Back et al., 2020 ) and motivation and engagement among different types of learners ( Merchant et al., 2014 ; Chavez and Bayona, 2018 ). While research on VR has generally revolved around discovering its potential to support and enhance education, academics appear to agree that the field of educational use of VR lacks pedagogical practices or strategies, with little focus on how the technology should be implemented to reap its benefits ( Cook et al., 2019 ; Smith, 2019 ; Scavarelli et al., 2021 ).

VR technology has already shown potential for the field of CSCL, improving the effectiveness of team behavior, enhancing communication between group members and increasing learning outcome gains ( Le et al., 2015 ; Godin and Pridmore, 2019 ; Zheng et al., 2019 ). What makes the use of Virtual Reality for Collaborative Learning (VRCL) even more appealing for education is its diversity in hardware and, as a result, the different forms it can take depending on the setting. Whether learners interact with the VEs via display monitors, CAVE systems or HMDs, they all seem to produce positive effects such as positive learning gains and outcomes, as well as engagement and motivation for CL ( Abdullah et al., 2019 ; Zheng et al., 2019 ; de Back et al., 2020 ; Tovar et al., 2020 ).

To advance the field of VRCL, as well as to establish its benefits and affordances, several literature reviews have examined research on VRCL. For example, Muhammad Nur Affendy and Ajune Wanis (2019) , aiming to provide an overview of the capabilities of CL through the adoption of collaborative system in AR and VR, review how VEs are used for different types of collaboration (e.g., remote and co-located collaboration), with different VR hardware (e.g., eye tracking) and multiple intended uses (e.g., increasing social engagement and supporting awareness of collaboration among learners). In comparison, Zheng et al. (2019) evaluate VRCL technology affordances by conducting a meta-analysis as well as a qualitative analysis of VRCL prototypes to explore potential learning benefits; Scavarelli et al. (2021) explore a more theoretical side with the intention to produce educational frameworks for future VRCL-related research, discussing how several learning theories (e.g., constructivism, social cognitive theory and connectivism) are reflected in prior research on the potential of VR as well as Augmented Reality (AR) for social learning spaces.

Together, the literature reviews of Muhammad Nur Affendy and Ajune Wanis (2019) , Zheng et al. (2019) ; Scavarelli et al. (2021) describe a general optimism towards VR in educational settings to support collaboration. The reviews outline VRCL’s strengths as 1) its ability to enhance learning outcomes, 2) its potential to facilitate learning, 3) its effectiveness in supporting remote collaboration between learners, as well as experts and novices, 4) its support for interpersonal awareness between collaborating learners and 5) its diversity, both in terms of its customizability (allowing VEs to better suit objectives) as well as its technology. Affordances of VRCL are identified as 1) social interaction (strengthened by VR’s affordances of immersion and presence), 2) resource sharing (strengthened by VR’s ability to present imaginary elements) and 3) knowledge construction (supported by the two prior affordances of VRCL). Furthermore, challenges and gaps related to (research on) VRCL are outlined. First, accessibility should be considered a primary concern according to Scavarelli et al.,; this does not just relate to the technical accessibility of VR when used in education, but more so to the accessibility of social engagement between learners sharing these virtual learning spaces. Second, they recommend to explore the interplay and connectivity between VEs and the real world, as doing so could reveal new learning theories that innovate VRCL. Third, Zheng et al., suggest that research focus on pedagogical strategies involving VRCL, including how to apply VR to educational settings involving collaboration. Fourth, they propose a focus on finding a balance between using VRCL to recreate (or simulate) existing (“real”) situations and creating new situations that would normally be impossible, considering that prior work has primarily been centered on the former and as such misses out on VR’s potential to do the latter.

Considering that remote collaboration and distance learning, especially since the COVID-19 pandemic, are becoming increasingly important for learners, an understanding of VR’s potential for CL could prove beneficial for the field of education. While research on the topic is apparent, studies focusing on VR’s ability to support and enhance CL are still small in scale ( Zheng et al., 2019 ; Scavarelli et al., 2021 ), accentuating the scarcity of knowledge on the topic. This systematic review specifically centers on scientific research on VRCL, with a particular focus on the empirical knowledge that such literature has established. The aim of this paper is to examine in what ways VR supports and enhances CL according to prior research on these topics; to achieve this, it reports on what VRCL is used for in different fields of education, discusses what research has stated regarding VRCL in terms of affordances and benefits for education, describes the characteristics of VRCL that allow these benefits to come to fruition and provides an insight into the technology behind VRCL, as well as how this compares to the state-of-the-art of VR. In doing so, this study intends to identify possible gaps in the field of VRCL research for possible future studies, in addition to highlighting VRCL’s strengths to support current research. To the best of the authors’ knowledge, this study is the first systematic review on the topic of VRCL. As a means to provide the relevant information, this review addresses the following four research questions.

1. What skills and competences have been trained with use of VRCL (and what should a VRCL environment provide to train these)?

2. What domains and disciplines have been addressed (and why)?

3. What systems have been developed and/or established?

4. What empirical knowledge has been established (and with what methods and/or study designs)?

This section discusses the process of collecting the relevant studies for this literature review. In particular, the inclusion and exclusion criteria, databases and methods used are described.

2.1 Identification

The systematic review used two databases: Scopus and Web of Science. The search query contained the following key elements: 1) collaborative interaction, 2) VR, 3) education, training and learning, 4) simulations of a three-dimensional nature, 5) empirical data and 6) the use of a system (application or prototype). As such, the following search string was used in both databases:

[collaboration OR cooperation OR collaborative OR cooperative OR collaborate OR cooperate] [AND] ["virtual reality” OR “mixed reality” OR “extended reality"] [AND] ["3D” OR 3d OR 3-D OR 3-d OR threedimension* OR three-dimension* OR “three dimension*" OR CGI OR “computer generated” OR “computer-generated” OR model* OR construct*] [AND] [evaluat* OR data OR result* OR observ* OR empiric* OR trial* OR experiment* OR significan* OR participant* OR subject*] [AND] [education OR training OR learning OR university OR school OR vocational] [AND] [system* OR prototyp* OR application* OR program*]

To be considered suitable, papers had to meet five specific inclusion criteria. Firstly, an article had to discuss collaborative or cooperative interaction between human users of a virtual, three-dimensional simulation. Secondly, the article had to include and discuss Virtual-, Augmented-, Mixed Reality (MR) or Extended Reality (XR) as a three-dimensional simulation of a physical space or object(s). While this review focuses on VR for CL, mediums such as AR, MR and XR were included in this search for two reasons. On the one hand, definitions for these mediums appear to overlap to such an extent (with some even considering them too vague and ambiguous ( Tovar et al., 2020 )) that ‘pedagogical advantages of either technologies are [considered] comparable’ ( Sims et al., 2022 ). On the other hand, the mediums in question do not always get defined as separate ones, but rather as different points on one spectrum, commonly referred to as the virtuality continuum, in which ‘“reality” lies at one end, and “virtuality” […] at the other, with Mixed Reality […] placed between’ ( Scavarelli et al., 2021 ). As such, the decision was made to include these mediums, so as to ensure that no pedagogical advantages of VR would be excluded. The third inclusion criterium required an article to include an empirical study (i.e., containing qualitative or quantitative data) for it to be considered suitable. For the fourth and fifth criteria, an article had to contain an educational objective or goal (for human entities) and discuss a system used for educational purposes (for human entities) in order to be eligible.

Additionally, studies would be disqualified from the literature review if they 1) only described a patent, 2) only contained a summary (review) of a conference, 3) only consisted of a literature review, 4) were not accessible to the authors of this study, 5) were not available in English, 6) were a duplicate or a version, edition or release of an older study that already had been included or 7) did not specifically state the number of participants of any experiment involved in the study.

The search query resulted in 1,058 publications for Scopus and 845 studies for Web of Science, resulting in a total of 1,608 studies after duplicates were removed. Using the inclusion and exclusion criteria to filter out ineligible articles (initially based on title and abstract, then on full text), this review resulted in 139 articles analyzed. Results and details of the process (which followed the guidelines of the PRISMA method ( Moher et al., 2009 )) can be seen in Figure 1 . Appendix A shows the complete list of all 139 articles.

FIGURE 1 . Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram of the screening process.

To examine reliability of the selection process, five additional coders screened a random sample of 50 studies individually (10 per coder) using the inclusion and exclusion criteria. After comparing and discussing results, inter-rater reliability (between the first coder and the five coders) was calculated using a Kappa-metric, resulting in a moderate level of agreement of 0.77 ( McHugh, 2012 ) (results can be found in Supplementary Table B1 ).

A taxonomy ( Figure 2 ) was created to help classify all 139 articles. With this review’s research questions in mind, three vital topics were established to function as main categories for the coding process: education, system and evaluation (illustrated in column C1 in Figure 2 ). For RQ1 and RQ2, the first category, education, was established to extract information from the articles, concentrating on six classes. Similarly, information necessary to answer RQ3 was collected by coding attributes related to the second category, system, which included eight classes. Focusing the coding on elements related to the third category, evaluation (with five classes), allowed for extraction of relevant information required to answer RQ4. After the relevant categories, classes (visible in column C2 in Figure 2 ) and attributes (visible in column C3 in Figure 2 ) were decided upon, the classification hierarchy in Figure 2 was constructed, partially based on scientific literature ( Bloom et al., 1956 ; Schreiber and Asnerly-Self, 2011 ; Motejlek and Alpay, 2019 ), to provide assistance during the coding process. For an in-depth description of the motivation behind this classification hierarchy, please see Supplementary Appendix C . While the required information for some of these attributes could easily be inferred directly from each study, other attributes required the first coder to deduce which attributes were applicable.

FIGURE 2 . Classification hierarchy used for coding, including percent agreement ( p a ) and Cohen’s kappa (K) between first and second coder on the right.

To assess reliability of the first coder, a second coder classified articles with the taxonomy ( Supplementary Table D1, D2, D3 ). Inter-rater reliability between the two coders for 30 randomly selected studies was 0.60 (with a percent agreement of 0.85), considered a moderate level of agreement ( McHugh, 2012 ). Additionally, Figure 2 shows the inter-rater reliability for each individual class.

3 Descriptive results

In this section, discussion of descriptive results is divided into three sections according to the structure of the taxonomy. An overview of all results (according to the taxonomy) can be found in Figure 3 .

FIGURE 3 . Results of coding of data found in the literature, according to the taxonomy.

3.1 Education

As a first dimension, elements related to education were analyzed. A majority of the selected articles focused on VRCL in tertiary education (i.e., university), discussing possible uses for students. Educators providing support (e.g., scaffolding) for learners proved most prominent, though not all studies discussed this topic. While a wide selection of educational domains were discussed, computer sciences and social sciences were the most popular fields. Most studies specifically focused on synchronous collaboration. Prevalent among learning paradigms and educational approaches were problem-based learning (PBL) and constructivism. The specific results related to this dimension are found in Table 1 .

TABLE 1 . Distribution of Education-related attributes.

In contrast to the high number of articles focusing on tertiary education (64.0%), primary education was central in 10.8% while 5.0% discussed VRCL in secondary education. A small percentage of studies (6.5%) focused on types of learners outside of formal education (e.g., on-the-job training). In relation to the educators, a little over half of the studies reported on educators supporting the learners by providing varying degrees of scaffolding (55.4%). For 20.9% of cases, educators provided presentations and lectures inside the VE, providing a more passive learning experience. On a broader scope, the studies showed a wide variety of educational domains and fields of expertise to which VR was applied. While approximately a quarter of studies reviewed (25.9%) reported use of VRCL for education, specific domains that were often discussed included computer science, robotics, ICT and informatics (12.2%), social sciences (11.5%) medical fields (9.4%) and engineering (8.6%).

Also shown in Table 1 is the appearance of different types of social learning: 62.6% of studies reviewed discussed synchronous (collaborative) interaction, while in comparison a much lower 18.0% discussed asynchronous (cooperative) interaction. For a 10th of the studies, an expert-novice type of social learning was apparent (9.4%). On the topic of educational approaches and learning paradigms, 29.5% of articles did not seem to discuss any specific approaches. Among those that did, constructivism and PBL were featured substantially (33.1% and 41.0%, respectively), while paradigms such as experientialism, situated learning and distributed cognition were discussed less frequently. Other educational approaches, discussed in 35.3% of articles, included self-regulation and shared regulation (e.g., Al-Hatem et al., 2018 ) as well as cognitive apprenticeship (e.g., Bouta and Retalis, 2013 ). Looking at the learning goals and outcomes, the cognitive domain proved to be popular (50.4%), whereas affective and psychomotor domains were featured much less (7.9% and 5.0%, respectively). Other goals and outcomes included general student engagement (discussed in 31.7%) and support of collaboration amongst learners (60.4%).

The second dimension took a closer look at systems used in the studies, including aspects related to the hardware used (e.g., devices, types of control) as well as users’ interaction with VEs (e.g., degree of virtuality, virtual embodiment). A majority of the studies reviewed did not use VR technologies such as HMD-based VR (HMD VR), but instead focused on monitors and displays when discussing VRCL. Most studies chose general purpose controls (e.g., mouse and keyboard) over more advanced hardware such as positional tracking. A majority of studies provided their participants with full-body embodiment (e.g., avatars) and the ability to manipulate virtual objects while inside the VEs. Approximately a quarter of studies used systems for edutainment purposes (i.e., learning by having fun), while system use for training or therapeutic purposes was less common. Table 2 shows these results in detail.

TABLE 2 . Distribution of System-related attributes.

Results showed a clear preference for 3D (non-HMD) simulations, i.e., a virtual simulation of a (physical) environment projected on a surface or display that is not a Head-Mounted Display (and, as such, is considered less immersive): this degree of virtuality was far more prominent in the reviewed studies (78.4%) compared to the lesser implemented AR/MR (16.5%) and HMD VR (7.2%). The hardware used in these studies reflected this: a large amount (89.2%) implemented flat-surface monitors and displays to present VRCL environments. These studies commonly used desktop computer set-ups that included a keyboard, mouse and monitor, though in the case of AR and MR, surface-based mobile devices were often used. When using the system in a larger setting (i.e., larger group size), studies utilized projector-based (but still flat) surfaces to display the VE (e.g., Bower et al., 2017 ). In some cases, several types of these flat-surface displays were being used in different phases of a study (e.g., Nuñez et al., 2008 ). Cases that used CAVE systems (3.6%) included ImmersaDesks, CAVE-like devices that derive from the original CAVE systems. Studies that involved HMD VR used devices like the Oculus Rift and HTC Vive, while studies revolving around AR and MR implemented devices like the HoloLens. Some studies involved multiple devices to compare effects based on the difference (e.g., monitor-based vs HMD VR, as discussed in Vallance et al., 2015 ) while others discussed implementation of HMD VR and AR-related devices as possible future directions without using these in their experiments. With regard to user interaction, studies that implemented general purpose controls used simple computer keyboard and mouse, though some cases also involved video game controllers such as the Nintendo Wiimote and Nunchuck ( Li et al., 2012 ). Apart from the more default specialized controls such as 3DoF and 6DoF controllers or mobile device-based touch screens, studies also discussed a wide variety of other tools in this category, including multi-touch tabletops, haptic feedback devices, Xbox Kinect and gesture-sensing data gloves. While scarce, gaze control and positional tracking (15.1% and 11.5%, respectively) was primarily found in studies that used (mobile-based) AR and HMD VR, though some studies also provided these through devices such as the HoloLens or as part of a CAVE system.

Of the studies examined for this review, 55.4% discussed (self-developed) prototypes, while 44.6% used (pre-existing) applications. The most prominently-mentioned engine for prototypes was Unity, with % (of 77 studies) using it. Concerning the ones that used applications (62 of 139), more than half discussed VE application Second Life (%), while open-source VEs OpenSimulator and Open Wonderland were used in smaller numbers (% and %, respectively). In regard to the intended function of systems used, the majority of articles described a strictly educational one (58.3%) and revolved around implementing these systems in educational contexts as well as using them to facilitate collaborative learning. Studies that used systems to both educate and entertain (22.3%) tended to focus on game-based learning and serious games, though some cases also discussed video games originally not intended for educational purposes (e.g., World of Warcraft ( Kong and Kwok, 2013 ), Minecraft ( Mørch et al., 2019 )). When training purposes were mentioned (17.3%), this often indicated the use of VEs to train specific expertises, such as liver surgery or aircraft inspection. Rare cases where a system was used for therapeutic purposes (just 2.2%) included use of VRCL to teach social skills to patients with autism ( Ke and Lee, 2016 ) or to train physical activities amongst elderly ( Arlati et al., 2019 ).

Motivation behind studies’ choices for the size of collaboration differed between experimental reasons (e.g., a limited number of participants), pedagogical reasons (e.g., using pairs to better stimulate personal social interaction between members compared to larger groups) and reasons related to the systems (e.g., limited hardware availability). Small groups proved to be the most used group size, with 37.4% describing groups of between three and nine members. Pairs were used in 22.3% of studies. Motivations behind pairs included focus on expert-novice interaction and system capabilities (e.g., support for two users maximum). Articles that described larger groups (ten or more members) generally had entire classes of learners interact with system (15.1%).

Apart from a small number of studies that did not provide sufficient information on the matter, virtual embodiment of the users was featured prominently. In cases where physical attributes were virtually represented by (imagery of) tools (18.0%), the VRCL environment was often implemented for specific training of certain expertises. In general, partial virtual embodiment appears in first person, HMD VR (for example, when only the user’s hands are made visible); while scarce (3.6%), studies that displayed partial virtual embodiment provided some interesting examples outside of HMD VR. Examples of partial embodiment included a detailed 3D face to focus on emotional and social expressions ( Cheng and Ye, 2010 ) and using controllable, flat-surfaced rectangles in a 3D environment on which users’ real-life faces were projected via webcam ( Nikolic and Nicholls, 2018 ). Full-body embodiment proved to be the most popular, with 67.6% of studies using systems that provide users complete (full-body) virtual representation. To a degree, the relatively high number of studies that present full-body embodiment can be explained by the systems that were implemented; applications such as Open Simulator and Second Life provide users with customizable avatars, making a full-body virtual embodiment a default feature. In some cases, however, studies specifically examined the effects of virtual embodiment, such as Gerhard et al. (2001) examining possible influences of different avatars on users’ sense of presence. On the topic of user influence on VEs, a little more than half the studies (53.2%) used systems that allowed (some degree of) virtual object manipulation, whereas approximately a quarter of the studies (26.6%) also provided users the tools to manipulate actual content of the VRCL environment. In 16.5% of studies, the system only allowed users to be visibly present inside the VE, while only 3.6% did not provide sufficient information on the matter.

3.3 Evaluation

For the third dimension, the selected articles were analyzed on how they evaluated applying VRCL. Articles frequently concentrated on evaluation of the system(s), with a higher number of them using self-report evaluation methods. Study design of the studies shows a similar result: pre-experimental study design (typically used for preliminary testing of systems) was regularly implemented, with surveys being a popular method of collecting data. While the number of participants was diverse, roughly half of studies reviewed used a sample size between 1 and 25 participants. The majority of articles discussed positive outcomes, whereas only a small amount featured negative results. Detailed results are displayed in Table 3 .

TABLE 3 . Distribution of Evaluation-related attributes.

The majority of studies focused on evaluating a system’s effectiveness when using it in educational settings (71.2%). These studies concentrated on the system’s capacity to support collaboration between learners. Other topics of discussion were student interest in the system and how the system can facilitate learning. Whenever studies examined processes (34.5%), evaluation would be centered around attempts to understand how group interaction materializes in these environments. This included how learners resolve social conflicts ( Cheong et al., 2015 ) and examining how co-presence (e.g., Kong and Kwok, 2013 ) and PBL take shape in VRCL environments. 35.3% of articles discussed learning outcomes after participants interacted with the system. The few situations where the above three attributes did not apply (3.6%) included a study that aimed to develop design guidelines ( Economou et al., 2001 ) and a study primarily interested in the teacher’s role when learners interact with VEs ( Lattemann and Stieglitz, 2012 ).

Most studies collected self-reported data from their participants (85.6%), while over half used behavioral methods to obtain tracking and observational data (59.0%). Articles that reported on knowledge- and/or performance-based assessments (20.9% of studies) often used pre- and post-tests to acquire their data, while only one appeared to use physiological data, tracking participants’ heart rate (0.7%). A notable number of articles (79.9%) implemented pre-experimental design in their studies. Some of these were case studies, applying VEs to educational settings (e.g., Terzidou et al., 2012 ), while others performed pilot studies to establish a first impression of the effects of a system on specific pedagogical situations (e.g., examining how VE-based application OpenSimulator influences Transactive Memory Systems amongst learners ( Kleanthous et al., 2016 )). Quasi-experimental- (13.7%) and true experimental designs (5.8%) were used scarcely, while only 2 out of 139 studies (1.4%) performed an experiment with single-subject design. With respect to non-experimental and descriptive designs, 84.9% of studies implemented a survey-based design, whereas a little over half used observational designs to collect data (56.1%). In some cases, comparative and correlation designs were implemented (7.9% and 15.8%, respectively).

Table 3 also reveals that approximately half of the studies sampled between 1 and 25 participants (53.2%), while around a quarter (26.6%) used a sample size between 26 and 50 participants. For 13.7% of articles, between 51 and 100 participants were used, whereas only 6.5% discussed using more than 100 participants for collecting data. In terms of outcomes, around half of the studies concluded that their system(s) seemed positive and promising (53.2%), while 17.3% draw positive conclusions based on significant outcomes from statistical hypothesis testing. Negative outcomes were scarce, with only 2.2% of the studies reporting negative results. Mixed outcomes were reported for 7.2% of the studies, whereas 20.1% discussed results that were inconclusive, showed no effect or reported outcomes on which positive and negative effects are not applicable.

4 Qualitative results

In general, the literature reviewed for this paper shows a positive attitude towards the use of VR to support and enhance CL. However, the results quickly make it apparent that the methods of applying VR to educational fields to support and enhance CL can vary greatly amongst the studies examined here. In order to acquire a general understanding how these studies have attempted to support and enhance CL using VR, this section will discuss qualitative results established. The rest of this section will be divided into sub-sections, each focusing on discussing results related to one of the four research questions of this literature review.

4.1 Skills and competences trained with VRCL

A number of elements can be identified regarding skills and competences trained with VRCL. Based on the skills and competences discussed in the reviewed literature, five categories were established for this study with the intention to provide a concise overview. These categories, including examples of each category, can be viewed in Table 4 .

TABLE 4 . Skills focused on in the reviewed literature.

For the types of skills and competences shown in Table 4 to be trained effectively, a VRCL environment requires a number of features that support the learners in learning these abilities. Based on the information provided by the reviewed literature, nine required features and design parameters of VRCL can be identified. First, virtual embodiment plays an important role in how learners view themselves and each other inside the VE, impacting learning outcomes and collaborative behavior by providing a sense of awareness and belonging ( Edirisingha et al., 2009 ; McArdle and Bertolotto, 2012 ). Second, efficient communicational tools are essential for effective collaboration: verbal (audio) communication is crucial ( Economou et al., 2001 ; De Pace et al., 2019 ), though additional modalities such as haptic technology can further enhance collaboration ( Moll and Pysander, 2013 ). Third, usability and accessibility should be taken into consideration: VRCL systems should be accessible to all levels of technical skills as differences negatively affect group cohesion and learning between group members (Y. Chang et al., 2016 ; Denoyelles and Kyeong-Ju Seo, 2012 ). Fourth, learners’ perceived usefulness of the VE also affects group cohesion; factors such as awareness, presence and social presence appear to significantly influence this perceived usefulness ( Denoyelles and Kyeong-Ju Seo, 2012 ; Yeh et al., 2012 ). Fifth, the ability to interact with elements inside the VE are considered key: to optimize learning outcomes, learners must have the option to manipulate elements inside the VE (e.g., virtual objects or virtual tools) in a seemingly natural and intuitive way ( Vrellis et al., 2010 ; Bower et al., 2017 ). Sixth, academic efficacy can be achieved if tasks inside the VE are designed around its educational, collaborative objectives, especially when designed for equal input from all learners in a group ( Wang et al., 2014 ; Nisiotis and Kleanthous, 2019 ). Seventh, educators should be ready to provide support, motivation and moderation of collaboration while learners interact inside the VE ( Lattemann and Stieglitz, 2012 ; Bower et al., 2017 ). However, the eighth feature, a level of autonomy, is equally important for each individual learner, not just in terms of independence from the educators, but more importantly from each other, as this allows them to provide different points of views as well as to explore multiple representations, thus improving CL ( Hwang and Hu, 2013 ). Ninth, implementation of VRCL should make sure to primarily support socialization inside the VE, as underestimating the importance of socialization might lead to features of VR obstructing rather than facilitating CL ( Chang et al., 2009 ).

Surprisingly, only a small number of the literature reviewed focused on goals related to the affective domain (7.9%). With some calling VR the “ultimate empathy machine” ( Rueda and Lara, 2020 , p.6), the medium’s ability to induce emotions has been prominently discussed and studied. Not only has VR been shown to indeed be capable of enhancing empathy amongst users ( Herrera et al., 2018 ), with some even arguing it to be more effective than traditional empathy-shaping methods ( Liu, 2020 ), studies have also suggested it to be an effective tool to offer a uniquely different level of understanding ( de la Peña et al., 2010 ). This would suggest that VR’s ability to create a better understanding of different group members’ points of view could in turn support collaboration between learners.

Similarly, even less literature reviewed focused on goals related to the psychomotor domain (5.0%). Prior studies have been positive and hopeful regarding VR to expand the possibilities of physical training ( Pastel et al., 2020 ). Interestingly, technical features such as positional tracking even seem to be effective in predicting psychomotor outcomes ( Moore et al., 2021 ), which could prove useful for domains that specifically focus on expert-novice training in primarily physical tasks (e.g., certain types of engineering). However, positional tracking, not unlike psychomotor outcomes, is only discussed sparingly (11.5%) in the literature reviewed.

An interesting observation in relation to the evaluation methods used in the scientific literature is that only 1 out of 139 articles used physiological measures. As suggested by research, physiological synchrony between group members can serve as an effective indicator for the quality of interpersonal interaction between them (with a higher physiological synchrony correlating with a higher interaction level) ( Liu et al., 2021 ). Furthermore, physiological measurements can be used to identify multiple predictors related to education and training, including the quality of collaboration between group members ( Dich et al., 2018 ). Additionally, visualizing physiological results of each member of a group to the others in real-time during collaboration has shown to have a positive effect on the empathy levels and cohesion of the group, further suggesting how collaboration between learners could benefit from physiological measures ( Tan et al., 2014 ). Considering VR’s visual characteristics as well as research arguing that physical signals such as electroencephalogram (EEG) can conveniently and unobtrusively be tracked during use of HMD VR ( Tremmel et al., 2019 ), future research on VRCL could prove fruitful in terms of training collaborative skills and competences via use of physiological-based information.

4.2 Disciplines focused on regarding VRCL

When looking at the most prominently-featured domains in the literature reviewed (as shown in Figure 4 ), examining what motivated researchers to study VRCL in the field of 1) education, 2) computer science, robotics and informatics and 3) social sciences can provide an understanding of VRCL’s role in these different disciplines.

FIGURE 4 . Results of the educational disciplines focused on in the reviewed literature.

For the field of education, some studies focus on the potential behind VRCL, intending to discover what it can mean for the development of cognitive and technical skills ( Franco and De Deus Lopes, 2009 ). Other studies focus on possible learning gains, examining how knowledge gained in VEs transfers to the real world (i.e., how learners apply outcomes in VEs to situations in actual reality) or attempting to facilitate this transfer by implementing elements of both ( Carron et al., 2013 ). In certain cases, articles specifically examine VEs’ effects on collaboration and how VR can be used to reinforce CL (e.g., Tüzün et al., 2019 ), whereas others aim to determine if existing educational paradigms such as constructivism can be applied to VRCL environments and, if so, how that affects group knowledge gain between learners ( Girvan and Savage, 2010 ). Together, these studies present a general motivation to discover what VRCL can mean for education and where its potential may lie.

For computer science, robotics and informatics, use of VRCL can be summarized in two motivations: 1) innovate these domains and 2) create a learning community. In the first case, researchers intend to utilize the affordances VRCL environments have to offer to further advance fields such as computer science, which have been criticized in the past for using two-dimensional learning platforms and oral-based teaching methods ( Pellas, 2014 ). With VEs, educators can provide learners realistic yet illusionary worlds that are flexible, customizable and even allow for detailed statistics on learners’ performance ( Champsas et al., 2012 ). In the second case, reviewed articles vocalize a desire to use VRCL to provide learners purposeful collaborative activities that create a sense of belonging to a learning community, using aspects such as awareness, presence and different methods of communication to motivate learners in these fields to work together closely ( De Lucia et al., 2009 ).

In similar fashion, social studies appears to be interested in how socialization between learners is manifested inside VRCL (e.g., Edirisingha et al., 2009 ). Some articles go further, studying how VRCL can support socialization: Molka-Danielsen and Brask (2014) suggest that presence, awareness and belonging allow for communication, negotiation and trust between learners, elements deemed necessary for completing collaborative tasks. Other studies focus on specific characteristics of socialization, such as how gender could affect social interaction and group cohesion inside VEs ( Denoyelles and Kyeong-Ju Seo, 2012 ). Collectively, these articles show a desire to understand how elements related to socialization transfer to VRCL, as well as how these environments can sustain and even enhance those elements.

4.3 Systems developed and/or established for VRCL

The results related to systems used show that there is quite a disparity between use of HMD VR and that of non-HMD VR. Almost 80% of systems implemented non-HMD VR, with AR/MR and HMD VR implemented far less frequently (16.5% and 7.2%, respectively, as illustrated in Figure 5 ). Almost 90% of studies described the use of flat-surface monitors and displays, which, when compared to the 10.8% of studies that used HMD devices, further highlights the low use of HMD VR in the literature reviewed (see Figure 6 ).

FIGURE 5 . Results of the degree of virtuality of systems discussed in the reviewed literature.

FIGURE 6 . Results of the hardware used in the reviewed literature.

The lack of representation of HMD VR in these articles is somewhat surprising, considering this type of virtuality and hardware is commonly associated with the medium of VR ( Dixon, 2006 ; Bonner and Reinders, 2018 ; Jing et al., 2018 ). The statement that research into application of VR to the field of education lacks a focus on HMD VR, however, is not uncommon ( Sousa Santos et al., 2009 ; Scavarelli et al., 2021 ), thus begging the question: why is it underrepresented in the reviewed literature?

One possible explanation could be that HMD VR is known to be difficult to apply to educational settings because of its high costs ( Olmos et al., 2018 ). Some of the articles analyzed for this review were published in the late 90s; while HMD VR technology was already available in those times, devices were more expensive and less technologically advanced compared to the technology that is available now ( Mehrfard et al., 2019 ; Wang et al., 2022 ). Furthermore, the technical skills necessary to implement VR properly in educational settings can prove challenging ( Jensen and Konradsen, 2018 ). Since collaboration involves multiple people, difficulties related to accessibility could be more severe when applying VR to a larger group of learners. Another possible reason is the health risks associated with the technology: HMD VR is often connected to motion sickness and cybersickness ( Olmos et al., 2018 ; Yoon et al., 2020 ). A third reason refers to the general lack of pedagogy on the topic of HMD VR: while the medium’s potential for education is often discussed, general guidelines as to how it should be applied efficiently to educational settings ( Cook et al., 2019 ; Zheng et al., 2019 ) as well as an understanding of how learning mechanisms operate inside VR environments ( Smith, 2019 ) are missing. Naturally, the small size of research done on VR and CL exacerbates this lack even further when specifically discussing VRCL. A possible fourth reason that is more closely tied to this particular literature review is that, despite its popularity in research, HMD VR appears to still lack empirical evidence of its educational value ( Sousa Santos et al., 2009 ; Makransky et al., 2019 ; Radianti et al., 2020 ), which, considering this review’s focus on empirically-based knowledge, could explain its scarcity.

The low representation of HMD VR and high representation of non-HMD VR could be related to the ongoing discussion about what defines VR and how it differs from VEs, as discussed in-depth by Girvan, (2018) . Girvan argues that some use terms synonymously with VR and/or VEs, while others use these same terms to classify different types of VEs, thus creating a fragmented understanding of what these are (and what they are not). Girvan’s point is reflected in the reviewed literature of this paper: while some studies identify Second Life as a “virtual environment” or “virtual world” (e.g., Terzidou et al., 2012 ), others refer to it as “virtual reality” (e.g., Sulbaran and Jones, 2012 ). To prevent further confusion with technologies with similar technical features, Girvan suggests to conceptualize VEs as ‘shared, simulated spaces which are inhabited and shaped by their inhabitants, who are represented as avatars [that] mediate our experience of this space as (…) we interact with others, with whom we construct a shared understanding of the world at that time’. VR, then, should be defined as ‘a technical system through which a user or multiple users can experience [such] a simulated environment’ ( Girvan, 2018 ).

Apart from causing a fragmented understanding of the terms in the literature, different interpretations of VR and VEs also lead to HMD VR and non-HMD VR being described as one and the same thing under the moniker of “virtual reality”. Though this may seem a trivial dispute about labels, treating these two types as identical will lead to misconceptions regarding both, as HMD and non-HMD VR contain different benefits and limitations when applied to education. While some studies showed no differences between the two in terms of specific learning outcomes (e.g., spatial- ( Srivastava et al., 2019 ) and language learning (J. Y. Jeong et al., 2018 )), other research highlighted several differences between HMD and non-HMD. Compared to non-HMD, HMD VR has shown to provide a much higher sense of embodiment, which in turn is hypothesized to lead to higher performances, in particular in psychomotor skills ( Juliano et al., 2020 ; Saldana et al., 2020 ). Similarly, HMD VR appeared superior to computer screens in terms of arousal, engagement and motivation in learners ( Makransky and Lilleholt, 2018 ). In contrast, however, Makransky et al. (2019) reported overloads and distractions caused by HMD VR, leading to poorer learning outcomes compared to non-HMD, a sentiment shared by Parong and Mayer (2021) , who described HMD VR to cause high affective and cognitive distractions. Amati and McNeill (2012) even argue that the difference between HMD and non-HMD VR (and in particular how the two are interacted with by users) have severe implications for teaching and practice.

With all of the above in mind, the low representation of HMD VR in the literature examined for this review can be interpreted in two ways. On the one hand, the underutilization underlines that HMD VR is not being used to its full potential and could very well hold much more promise for the field of education and CL. On the other hand, the low use of HMD VR could suggest that implementation of HMD VR in education and/or CL is, in fact, not worth the trouble it brings with it. Whether HMD VR is a benefit or a burden, then, arguably depends on three important elements: 1) the goals (i.e., what skills and/or competences are supposed to be trained), 2) the setting (i.e., the disciplines and fields to which it is applied), and 3) the affordances of VRCL (and to what degree these conform to the goals and setting).

4.4 Empirical knowledge established regarding VRCL

When summarizing the outcomes of the 139 articles, 70% of the studies reviewed displayed a positive attitude towards the application of VRCL to education. While a relatively low number (approximately 25%) presented statistically significant outcomes, this does illustrate a strong optimism amongst those studying VRCL environments in different fields of education as described in prior literature reviews on the topic. This could also explain the high number of studies that deployed pre-experimental study designs: with VRCL being a relatively new addition to the world of CSCL, as well as one that continues to rapidly advance because of the technology behind it, many seem enthusiastic and eager to see what promises VRCL holds when used in different fields and with different types of learners.

Regarding affordances discussed in the reviewed literature, several features are identified. First, VRCL appears an efficient tool to engage learners and to motivate them to study and learn. The ability to customize VRCL environments and their content provides learners more personalized experiences that better suit their personalities and attitudes, thereby enhancing the motivation to learn on both an individual and group level ( Arlati et al., 2019 ). Furthermore, VRCL’s immersive qualities tend to make the experiences more engaging for learners, encouraging them to engage in presentations and demonstrations as well as to communicate and collaborate with each other ( Avanzato, 2018 ).

The second affordance identified VRCL as a great tool for distance learning and remote collaboration. VEs provide a method for learners and educators to work together and collaborate despite distances. In comparison to other media, however, VRCL brings with it a high sense of immediacy (i.e., ‘verbal and non-verbal behaviors that give a sense of reduction of physical and psychological distance between the communicators’), which in turn presents an increased perception of learning ( Edirisingha et al., 2009 ). Additionally, VRCL’s immersive qualities and high presence allow for environments capable of simulating training as preparation for real-life experiences ( Al-Hatem et al., 2018 ) that simultaneously promote active participation and social interaction ( Mystakidis et al., 2017 ) in a setting that feels personal despite distances between learners ( Desai et al., 2017 ). In certain cases, such as education for learners with physical disabilities, learners and educators even considered connectivity to be more accessible and easier than real-life equivalents ( Aydogan and Aras, 2019 ), illustrating that VRCL environments can potentially go beyond simply being a replacement. To effectively support the distance learning and remote collaboration, however, design of the VEs should focus on providing learners a sense of 1) presence, 2) awareness and 3) belonging to the group ( Molka-Danielsen and Brask, 2014 ).

Thirdly, the literature reviewed suggests that VRCL environments are effective spaces to support multi- and interdisciplinary learning and collaboration. The ability to customize VEs, adapting to suit users’ needs, prevents them from being restricted to just a single specific subject field. This in turn allows educators to change the environments to accommodate many different subject fields and topics so as to make sure that learners from different backgrounds can collaborate with each other undisturbed ( Bilyatdinova et al., 2016 ). Moreover, it seems that VRCL environments made some of the literature studies reviewed realize the importance of interdisciplinary collaboration in the learning process ( Franco et al., 2006 ; Nadolny et al., 2013 ).

The fourth affordance identified might be an unsurprising but nonetheless important one: VRCL seems to be a tool for the development of social skills. While identity construction and projection through virtual embodiments can be complex for learners (depending on their technical skills), VRCL is found to facilitate social presence and foster socialization ( Edirisingha et al., 2009 ). VRCL’s customizability allows learners to integrate personal preferences and identity expressions into processes inside the environment (e.g., through their virtual embodiments), in turn mediating identity and norm construction for real-life social settings ( Ke and Lee, 2016 ). Vital social skills, such as the ability to identify and manipulate basic emotional states, can be taught and trained using VEs, improving learners’ socialization, communication skills and emotional intelligence ( López-Faican and Jaen, 2020 ). Learners’ prior experience with VEs, however, should not be underestimated, as a difference in familiarity with VRCL environments has been shown to impact collaboration ( Bluemink et al., 2010 ).

Fifth, VEs appear fitting for CL-related learning paradigms and educational approaches. Some studies specifically focus on examining to what degree VRCL environments are applicable to paradigms such as constructivism, socio-constructivism and constructionism (e.g., Girvan and Savage, 2010 ; Pellas et al., 2013 ; Abdullah et al., 2019 ), concluding that these indeed go well together. Other studies, however, focus on theories and methods commonly associated with these paradigms. In particular, experiential learning and PBL seem appropriate for VRCL environments. VEs allow for safe, consequence-free learning for exploring, experiencing and practicing without any real-life risks ( Cheong et al., 2015 ; Le et al., 2015 ), making it suitable for experiential learning. Moreover, VRCL’s immersive qualities seem to support and even elevate experiential learning strategies such as roleplay and improvisation, providing learners close to real-world experiences in a controlled environment ( Jarmon et al., 2008 ; Ashley et al., 2014 ). In the case of PBL, each individual learner can use different tools inside VRCL environments to illustrate and represent ideas and suggestions to the rest of the group. Considering that VEs seem great tools for conceptual learning because of their customizability and visual nature ( Brna and Aspin, 1998 ; Griol et al., 2014 ), learners can use these features to explain their point of view in ways that they otherwise could not. As a result, learners appear to become more active and effective in sharing ideas, joint problem solving and the co-construction of mental models when working in groups inside VRCL environments ( Rogers, 2011 ; Hwang and Hu, 2013 ).

Returning to the topic of disparity between HMD and non-HMD VR represented in the reviewed literature, as well as both being discussed as one and the same “Virtual Reality”, an important question to ask is whether the affordances identified here are transferable between the two. HMD and non-HMD VR differ in several ways: they are interacted with differently, face different obstacles when applied to education and appear to have different learning outcomes based on different educational settings.

With the definitions of VEs and VR as given by Girvan (2018) as a frame of reference, however, an answer can be given regarding the transferability of these affordances between HMD and non-HMD VR. Both HMD and non-HMD VR should be considered tools, technical systems through which users can virtually enter VEs, i.e., shared simulated spaces in which they can interact with the environment as well as each other. As such, the affordances described in this paper do not revolve around the tools used, but that which they provide access to: the VRCL environments. Simultaneously, which tool is used to access these VRCL environments can in turn affect both the interaction and the outcome of users’ experiences with VEs. For example, HMD VR might offer more effective development of social skills compared to non-HMD VR, considering the former provides a higher sense of embodiment and, in extension, more intuitive and expansive methods of expression. If, however, cognitive learning outcomes are the most important educational objective, non-HMD VR could be a better option, considering HMD VR’s tendency to cause affective and cognitive distractions. This, then, reflects the aforementioned statement regarding HMD VR being a benefit or a burden. While affordances of VRCL environments apply to both HMD and non-HMD VR, the effect of these affordances depend on 1) the goals, 2) the setting and 3) which affordances of VRCL are most vital to the first two elements. As such, the choice between non-HMD VR and HMD VR should be made depending on those three elements.

5 Conclusion and future research

With current research on the topic being scarce while the demand for remote collaboration and distance learning keeps increasing, this literature review intends to study how VR has been (and can be) used to support and enhance CL. To achieve this, it attempts to answer four research questions regarding prior research on VRCL: what skills and competences have been trained with VRCL and what does VRCL provide in these scenarios? To what educational domains has VRCL been applied and why? What systems have been used for VRCL? And what empirical knowledge has been established regarding VRCL?

This paper identifies five types of skills and competences commonly trained with the use of VRCL. Furthermore, a number of features and design principles are identified in terms of what these environments should offer for these skills to be developed. Educational fields and domains appear to be interested in VRCL because of a desire to innovate, to form communities, to support remote collaboration and to enhance socialization skills of learners. In terms of technology, systems used for VRCL-related purposes appear to predominantly focus on monitor-based (non-HMD) VR and mouse-and-keyboard controls, contrasting what VR is commonly associated with (e.g., HMD VR, specialized controls involving gaze control and positional tracking). This study perceives a general optimism present in the literature reviewed regarding the use of VR to support and enhance CL in learners. Additionally, a number of affordances of VRCL are described, though it is of importance to note that these affordances could differ in strength depending on which type of VR (i.e., non-HMD or HMD) is used.

While the literature on VRCL reviewed for this paper is diverse, it suggests that Virtual Reality can be an effective tool for supporting and enhancing Collaborative Learning. This diversity, however, also highlights that pedagogies of VRCL are lacking, with studies showing many different and contrasting approaches to applying VR to their respective fields for the support of CL. In order to see VR become more adopted as an educational tool for collaborative purposes, pedagogies should be clearly structured, highlighting similarities and differences in regards to both the technologies used and the domains they are used in. As such, this paper proposes a number of suggestions for future research. First, the difference between hardware used in the literature reviewed and the state-of-the-art of VR suggests that further examination of differences between non-HMD and HMD VRCL, both in terms of affordances as well as challenges and obstacles, could lead to a better understanding of VRCL’s potential. Second, despite the advantages VR has for development in affective and psychomotor skills, the scientific literature on VRCL shows only minor focus on these domains. This study argues that CL would benefit from both these domains being featured more prominently and as such encourages more research into these matters. Third, this paper suggests that research into VRCL focuses on using study designs and evaluation methods that are less frequently (or barely) featured in the reviewed literature. While the repeated and dominant use of pre-experimental study design is understandably meant to identify the potential behind the technology, the domain of VRCL (and, in extension, research on VR in education) would benefit from more true experimental design. Additionally, considering that the use of physiological data for evaluation methods appears to be unexplored terrain, this paper suggests that future research into VRCL implements these types of methods.

Author contributions

NvM: Main author VvW: Co-author and coder W-PB: Co-author and supervisor MS: Co-author and supervisor. All authors contributed to the article and approved the submitted version.

This project has been funded by the Leiden-Delft-Erasmus Centre for Education and Learning (LDE-CEL).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2023.1159905/full#supplementary-material

Supplementary Appendix A | List of all articles included.

Supplementary Appendix B1 | Results of agreement between first author and five additional coders on in- and exclusion criteria.

Supplementary Appendix C | Explanation/motivation behind Taxonomy.

Supplementary Appendix D1 | Results of agreement between first author and coder on use of taxonomy’s first category (Education).

Supplementary Appendix D2 | Results of agreement between first author and coder on use of taxonomy’s second category (System).

Supplementary Appendix D3 | Results of agreement between first author and coder on use of taxonomy’s third category (Evaluation).

Abdullah, J., Mohd-Isa, W. N., and Samsudin, M. A. (2019). Virtual reality to improve group work skill and self-directed learning in problem-based learning narratives. Virtual Real. 23 (4), 461–471. doi:10.1007/s10055-019-00381-1

CrossRef Full Text | Google Scholar

Al-Hatem, A. I., Masood, M., and Al-Samarraie, H. (2018). Fostering student nurses’ self-regulated learning with the Second Life environment: An empirical study. J. Inf. Technol. Educ. Res. 17, 285–307. doi:10.28945/4110

Amati, M., and McNeill, M. (2012). Learning from and through virtual worlds: A pilot study of second life. J. Educ. Built Environ. 7 (1), 39–55. doi:10.11120/jebe.2012.07010039

Arlati, S., Colombo, V., Spoladore, D., Greci, L., Pedroli, E., Serino, S., et al. (2019). A social virtual reality-based application for the physical and cognitive training of the elderly at home. Sensors Switz. 19 (2), 261. doi:10.3390/s19020261

Ashley, C., Kibbe, S., and Thornton, S. (2014). Experiential learning in second life: An application in retail management. Atl. Mark. J. 3 (2), 94–112. http://conf.vwbpe.org/index.php/VWBPE/11/paper/view/132 .

Google Scholar

Avanzato, R. (2018). Virtual technology to support student engagement and collaboration in online engineering courses. Comput. Educ. J. 9 (4). https://www.scopus.com/inward/record.uri?eid=2-s2.0-85058935867&partnerID=40&md5=c38a848de46d17a869b0307c6046d06e .

Aydogan, H., and Aras, F. (2019). Design, simulation and virtual implementation of a novel fundamental programmable logic controllers laboratory in a 3D virtual world. Int. J. Electr. Eng. Educ. 59, 266–281. doi:10.1177/0020720919856249

Bilyatdinova, A., Karsakov, A., Bezgodov, A., and Dukhanov, A. (2016). Virtual environment for creative and collaborative learning. Adv. Intelligent Syst. Comput. 416, 371–381. doi:10.1007/978-3-319-27478-2_26

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., and Krathwohl, D. R. (1956). “The classification of educational goals,” in Taxonomy of educational objectives (New York, NY: David McKay ), 62–197.

Bluemink, J., Hämäläinen, R., Manninen, T., and Järvelä, S. (2010). Group-level analysis on multiplayer game collaboration: How do the individuals shape the group interaction? Interact. Learn. Environ. 18 (4), 365–383. doi:10.1080/10494820802602444

Bonner, E., and Reinders, H. (2018). Augmented and virtual reality in the language classroom: Practical ideas. Teach. Engl. Technol. 18 (3), 33–53. http://www.tewtjournal.org .

Bouta, H., and Retalis, S. (2013). Enhancing primary school children collaborative learning experiences in maths via a 3D virtual environment. Educ. Inf. Technol. 18 (4), 571–596. doi:10.1007/s10639-012-9198-8

Bower, M., Lee, M. J. W., and Dalgarno, B. (2017). Collaborative learning across physical and virtual worlds: Factors supporting and constraining learners in a blended reality environment. Br. J. Educ. Technol. 48 (2), 407–430. doi:10.1111/bjet.12435

Brna, P., and Aspin, R. (1998). Collaboration in a virtual world: Support for conceptual learning? Educ. Inf. Technol. 3 (3–4), 247–259. doi:10.1023/a:1009649631868

Carron, T., Pernelle, P., and Talbot, S. (2013). “Issues of learning games: From virtual to real,” in IADIS international conference on cognition and exploratory learning in digital age, CELDA (Texas, TX: International Assn for Development of the Information Society (IADIS) ), 133–140.

Champsas, I., Leftheris, I., Tsiatsos, T., Terzidou, T., and Mavridis, A. (2012). “Open games: A framework for implementing 3D collaborative educational games in opensim,” in Proceedings of the European conference on games-based learning (Cork, Ireland: European Conference on Games Based Learning (ECGBL) ).

Chang, V., Gütl, C., Kopeinik, S., and Williams, R. (2009). Evaluation of collaborative learning settings in 3D virtual worlds. Int. J. Emerg. Technol. Learn. 4, 6–17. doi:10.3991/ijet.v4s3.1112

Chang, Y., Aziz, E. S. S., Zhang, Z., Zhang, M., and Esche, S. K. (2016). Usability evaluation of a virtual educational laboratory platform. Comput. Educ. J. 16 (1), 24–36. doi:10.18260/p.24973

Chavez, B., and Bayona, S. (2018). “Virtual reality in the learning process,” in Advances in intelligent systems and computing (Midtown Manhattan, New York City: Springer International Publishing ), 746. doi:10.1007/978-3-319-77712-2_129

Cheng, Y., and Ye, J. (2010). Exploring the social competence of students with autism spectrum conditions in a collaborative virtual learning environment - the pilot study. Comput. Educ. 54 (4), 1068–1077. doi:10.1016/j.compedu.2009.10.011

Cheong, Y. G., Khaled, R., HolmgȦrd, C., and Yannakakis, G. N. (2015). “Serious games for teaching conflict resolution: Modeling conflict dynamics,” in Conflict and multimodal communication: Social research and machine intelligence (Cham, Switzerland: Springer ), 449–475. doi:10.1007/978-3-319-14081-0_21

Cook, M., Lischer-Katz, Z., Hall, N., Hardesty, J., Johnson, J., McDonald, R., et al. (2019). Challenges and strategies for educational virtual reality: Results of an expert-led forum on 3D/VR technologies across academic institutions. Inf. Technol. Libr. 38 (4), 25–48. doi:10.6017/ital.v38i4.11075

Dalgarno, B., and Lee, M. J. W. (2010). What are the learning affordances of 3-D virtual environments? Br. J. Educ. Technol. 41 (1), 10–32. doi:10.1111/j.1467-8535.2009.01038.x

Daphne, E., William, M. L., Steve, P. R., and Adrian, W. J. (2000). “CVE technology development based an real world application and user needs,” in Proceedings IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000) , Gaithersburg, MD, USA , 14-16 June 2000 . doi:10.1109/ENABL.2000.883698

de Back, T. T., Tinga, A. M., Nguyen, P., and Louwerse, M. M. (2020). Benefits of immersive collaborative learning in CAVE-based virtual reality. Int. J. Educ. Technol. High. Educ. 17 (1), 51. doi:10.1186/s41239-020-00228-9

de la Peña, N., Weil, P., Llobera, J., Giannopoulos, E., Pomés, A., Spanlang, B., et al. (2010). Immersive journalism: Immersive virtual reality for the first-person experience of news. Presence Teleoperators Virtual Environ. 19 (4), 291–301. doi:10.1162/PRES_a_00005

De Lucia, A., Francese, R., Passero, I., and Tortora, G. (2009). Development and evaluation of a system enhancing second Life to support synchronous role-based collaborative learning. Softw. - Pract. Exp. 39 (12), 1025–1054. doi:10.1002/spe.926

De Pace, F., Manuri, F., Sanna, A., and Zappia, D. (2019). A comparison between two different approaches for a collaborative mixed-virtual environment in industrial maintenance. Front. Robotics AI 6, 18–14. doi:10.3389/frobt.2019.00018

Denoyelles, A., and Kyeong-Ju Seo, K. (2012). Inspiring equal contribution and opportunity in a 3d multi-user virtual environment: Bringing together men gamers and women non-gamers in Second Life®. Comput. Educ. , 58(1), 21–29. doi:10.1016/j.compedu.2011.07.007

Desai, K., Belmonte, U. H. H., Jin, R., Prabhakaran, B., Diehl, P., Ramirez, V. A., et al. (2017). “Experiences with multi-modal collaborative virtual laboratory (MMCVL),” in Proceedings - 2017 IEEE 3rd International Conference on Multimedia Big Data, BigMM , Laguna Hills, CA, USA , 19-21 April 2017 , 376–383. doi:10.1109/BigMM.2017.62

Dich, Y., Reilly, J., and Schneider, B. (2018). “Using physiological synchrony as an indicator of collaboration quality, task performance and learning,” in Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Midtown Manhattan, New York City: Springer International Publishing ). doi:10.1007/978-3-319-93843-1_8

Dixon, S. (2006). A history of virtual reality in performance. Int. J. Perform. Arts Digital Media 2 (1), 23–54. doi:10.1386/padm.2.1.23/1

Economou, D., Mitchell, W., Pettifer, S., Cook, J., and Marsh, J. (2001). “User centred virtual actor technology,” in Proceedings VAST 2001 virtual reality, archeology, and cultural heritage (New York, NY: Association for Computing Machinery ), 323–331. doi:10.1145/584993.585052

Edirisingha, P., Nie, M., Pluciennik, M., and Young, R. (2009). Socialisation for learning at a distance in a 3-D multi-user virtual environment. Br. J. Educ. Technol. 40 (3), 458–479. doi:10.1111/j.1467-8535.2009.00962.x

Franco, J. F., Da Cruz, S. R., and Lopes, R. D. D. (2006). “Computer graphics, interactive technologies and collaborative learning synergy supporting individuals’ skills development,” in ACM SIGGRAPH 2006 educators program, SIGGRAPH ’06 (New York, NY: Association for Computing Machinery ). doi:10.1145/1179295.1179338

Franco, J. F., and De Deus Lopes, R. (2009). “Three-dimensional digital enviroments and computer graphics influencing K-12 individuals’ digital literacy development and interdisciplinary lifelong learning,” in ACM SIGGRAPH ASIA 2009 educators program, SIGGRAPH ASIA ’09 (New York, NY: Association for Computing Machinery ). doi:10.1145/1666611.1666626

Freina, L., and Ott, M. (2015). “A literature review on immersive virtual reality in education: State of the art and perspectives,” in Proceedings of ELearning and Software for Education (ELSE) , Bucharest, Romania , April 23--24, 2015 .

Gerhard, M., Moore, D. J., and Hobbs, D. J. (2001). Continuous presence in collaborative virtual environments: Towards a hybrid avatar-agent model for user representation. Lect. Notes Comput. Sci. 2190, 137–155. doi:10.1007/3-540-44812-8_12

Girvan, C., and Savage, T. (2010). Identifying an appropriate pedagogy for virtual worlds: A communal constructivism case study. Comput. Educ. 55 (1), 342–349. doi:10.1016/j.compedu.2010.01.020

Girvan, C. (2018). What is a virtual world? Definition and classification. Educ. Technol. Res. Dev. 66 (5), 1087–1100. doi:10.1007/s11423-018-9577-y

Godin, J. J., and Pridmore, J. (2019). Using design thinking strategies and virtual reality in global virtual teams. Issues Inf. Syst. 20 (4), 104–106. doi:10.48009/4_iis_2019_104-106

Griol, D., Molina, J. M., and Callejas, Z. (2014). An approach to develop intelligent learning environments by means of immersive virtual worlds. J. Ambient Intell. Smart Environ. 6 (2), 237–255. doi:10.3233/AIS-140255

Herrera, F., Bailenson, J., Weisz, E., Ogle, E., and Zak, J. (2018). Building long-term empathy: A large-scale comparison of traditional and virtual reality perspective-taking. PLoS ONE 13, e0204494. doi:10.1371/journal.pone.0204494

PubMed Abstract | CrossRef Full Text | Google Scholar

Hwang, W. Y., and Hu, S. S. (2013). Analysis of peer learning behaviors using multiple representations in virtual reality and their impacts on geometry problem solving. Comput. Educ. 62, 308–319. doi:10.1016/j.compedu.2012.10.005

Jackson, R. L., Taylor, W., and Winn, W. (1999). “Peer collaboration and virtual environments: A preliminary investigation of multi-participant virtual reality applied in science education,” in Proceedings of the ACM symposium on applied computing (New York, NY: Association for Computing Machinery ), 121–125.

Jarmon, L., Traphagan, T., and Mayrath, M. (2008). Understanding project-based learning in Second Life with a pedagogy, training, and assessment trio. Educ. Media Int. 45 (3), 157–176. doi:10.1080/09523980802283889

Jensen, L., and Konradsen, F. (2018). A review of the use of virtual reality head-mounted displays in education and training. Educ. Inf. Technol. 23 (4), 1515–1529. doi:10.1007/s10639-017-9676-0

Jeong, H., and Hmelo-Silver, C. E. (2016). Seven affordances of computer-supported collaborative learning: How to support collaborative learning? How can technologies help? Educ. Psychol. 51 (2), 247–265. doi:10.1080/00461520.2016.1158654

Jeong, J. Y., Seo, S. J., Han, Y. J., Kim, S. J., and Jeong, H. (2018). “Learning English conversations in an immersive virtual reality environment,” in Icce 2018 - 26th international conference on computers in education, workshop proceedings (Philippines: Asia-Pacific Society for Computers in Education ), 638–643.

Jing, Y., Behzad, M., Jeff, H. P., Sarah Farukhi, A., S Khizer, K., Hinton, R. J., et al. (2018). Vital roles of β-catenin in trans-differentiation of chondrocytes to bone cells. Int. J. Ophthalmol. Clin. Res. 5 (2), 1–9. doi:10.7150/ijbs.23165

Joyner, D. A., Mavilakandy, A., Mittal, I., Kutnick, D., and Macintyre, B. (2021). Content - neutral immersive environments for cultivating scalable camaraderie . New York, NY: Association for Computing Machinery , 47–58.

Juliano, J. M., Spicer, R. P., Vourvopoulos, A., Lefebvre, S., Jann, K., Ard, T., et al. (2020). Embodiment is related to better performance on a brain–computer interface in immersive virtual reality: A pilot study. Sensors Switz. 20 (4), 1204. doi:10.3390/s20041204

Ke, F., and Lee, S. (2016). Virtual reality based collaborative design by children with high-functioning autism: Design-based flexibility, identity, and norm construction. Interact. Learn. Environ. 24 (7), 1511–1533. doi:10.1080/10494820.2015.1040421

Kleanthous, S., Michael, M., Samaras, G., and Christodoulou, E. (2016). “Transactive memory in task-driven 3d virtual world teams,” in ACM international conference proceeding series (New York, NY: Association for Computing Machinery ), 23–27. doi:10.1145/2971485.2996728

Kong, J. S. L., and Kwok, R. C. W. (2013). “Knowledge conversion in massive peers: A preliminary study on mutualistic co-presence,” in Proceedings - pacific asia conference on information systems, PACIS 2013 (Korea: Pacific Asia Conference on Information Systems (PACIS) ). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84928485362&partnerID=40&md5=9a1b86ea4622e938ec393f4b968a7fc6 .

Kwon, C. (2019). Verification of the possibility and effectiveness of experiential learning using HMD-based immersive VR technologies. Virtual Real. 23 (1), 101–118. doi:10.1007/s10055-018-0364-1

Lattemann, C., and Stieglitz, S. (2012). “Challenges for lecturers in virtual worlds,” in Ecis 2012 - proceedings of the 20th European conference on information systems (Barcelona, Spain: European Conference on Information Systems (ECIS) ). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84905717967&partnerID=40&md5=3fda0e424f60827726a346d39465e313 .

Le, Q. T., Pedro, A., and Park, C. S. (2015). A social virtual reality based construction safety education system for experiential learning. J. Intelligent Robotic Syst. Theory Appl. 79 (3–4), 487–506. doi:10.1007/s10846-014-0112-z

Li, H., Chan, G., and Skitmore, M. (2012). Multiuser virtual safety training system for tower crane dismantlement. J. Comput. Civ. Eng. 26 (5), 638–647. doi:10.1061/(asce)cp.1943-5487.0000170(asce)cp.1943-5487.0000170

Liu, Y., Wang, T., Wang, K., and Zhang, Y. (2021). Collaborative learning quality classification through physiological synchrony recorded by wearable biosensors. Front. Psychol. 12, 674369. doi:10.3389/fpsyg.2021.674369

Liu, Y. (2020). The application of virtual reality in empathy establishment: Foresee the future. Proc. Int. Conf. Comput. Intell. Appl. ICCIA 2020, 188–193. doi:10.1109/ICCIA49625.2020.00043

López-Faican, L., and Jaen, J. (2020). EmoFindAR: Evaluation of a mobile multiplayer augmented reality game for primary school children. Comput. Educ. 149, 103814. doi:10.1016/j.compedu.2020.103814

Makransky, G., and Lilleholt, L. (2018). A structural equation modeling investigation of the emotional value of immersive virtual reality in education. Educ. Technol. Res. Dev. 66 (5), 1141–1164. doi:10.1007/s11423-018-9581-2

Makransky, G., Terkildsen, T. S., and Mayer, R. E. (2019). Adding immersive virtual reality to a science lab simulation causes more presence but less learning. Learn. Instr. 60, 225–236. doi:10.1016/j.learninstruc.2017.12.007

McArdle, G., and Bertolotto, M. (2012). Assessing the application of three-dimensional collaborative technologies within an e-learning environment. Interact. Learn. Environ. 20 (1), 57–75. doi:10.1080/10494821003714749

McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochem. Medica 22 (3), 276–282. https://hrcak.srce.hr/89395 . doi:10.11613/bm.2012.031

Mehrfard, A., Fotouhi, J., Taylor, G., Forster, T., Navab, N., and Fuerst, B. (2019). A comparative analysis of virtual reality head-mounted display systems . arXiv . http://arxiv.org/abs/1912.02913 .

Merchant, Z., Goetz, E. T., Cifuentes, L., Keeney-Kennicutt, W., and Davis, T. J. (2014). Effectiveness of virtual reality-based instruction on students’ learning outcomes in K-12 and higher education: A meta-analysis. Comput. Educ. 70, 29–40. doi:10.1016/j.compedu.2013.07.033

Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., and Group, T. P. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 6 (7), e1000097. doi:10.1371/journal.pmed.1000097

Molka-Danielsen, J., and Brask, O. D. (2014). Designing virtual collaboration environment for distance learners: Exploring socialization as a basis for collaboration. Lect. Notes Bus. Inf. Process. 186, 74–89. doi:10.1007/978-3-319-09546-2_6

Moll, J., and Pysander, E. L. S. (2013). A haptic tool for group work on geometrical concepts engaging blind and sighted pupils. ACM Trans. Accessible Comput. 4 (4), 1–37. doi:10.1145/2493171.2493172

Moore, A. G., McMahan, R. P., and Ruozzi, N. (2021). Exploration of feature representations for predicting learning and retention outcomes in a VR training scenario. Big Data Cognitive Comput. 5 (3), 29. doi:10.3390/bdcc5030029

Mørch, A. I., Mifsud, L., and Eie, S. (2019). Developing a model of collaborative learning with minecraft for social studies classrooms using role-play theory and practice. Computer-Supported Collab. Learn. Conf. CSCL 1, 272–279.

Motejlek, J., and Alpay, E. (2019). “A taxonomy for virtual and augmented reality in education,” in Proceedings of the 46th SEFI annual conference 2018: Creativity, innovation and entrepreneurship for engineering education excellence ( arXiv ), 1089–1100.

Muhammad Nur Affendy, N., and Ajune Wanis, I. (2019). “A review on collaborative learning environment across virtual and augmented reality technology,” in IOP Conference Series: Materials Science and Engineering 5(1). doi:10.1088/1757-899X/551/1/012050

Mystakidis, S., Berki, E., and Valtanen, J. (2017). Toward successfully integrating mini learning games into social virtual reality environments – recommendations for improving open and distance learning. Edulearn17 Proc. 1, 968–977. doi:10.21125/edulearn.2017.1203

Nadolny, L., Woolfrey, J., Pierlott, M., and Kahn, S. (2013). SciEthics Interactive: Science and ethics learning in a virtual environment. Educ. Technol. Res. Dev. 61 (6), 979–999. doi:10.1007/s11423-013-9319-0

Nikolic, S., and Nicholls, B. (2018). Exploring student interest of online peer assisted learning using mixed-reality technology . Cham, Switzerland: Springer , 715, 48–54. doi:10.1007/978-3-319-73210-7_6

Nisiotis, L., and Kleanthous, S. (2019). “The relationship between students’ engagement and the development of transactive memory systems in MuvE: An experience report,” in Annual conference on innovation and technology in computer science education, ITiCSE, 71–77 (New York, NY: Association for Computing Machinery ). doi:10.1145/3304221.3319743

Nuñez, M., Quirós, R., Nuñez, I., Carda, J. B., and Camahort, E. (2008). Collaborative augmented reality for inorganic chemistry education. WSEAS Int. Conf. Proc. Math. Comput. Sci. Eng. 5, 271–277.

Olmos, E., Cavalcanti, J. F., Soler, J.-L., Contero, M., and Alcañiz, M. (2018). Mobile virtual reality: A promising technology to change the way we learn and teach (Singapore: Springer ), 95–106. doi:10.1007/978-981-10-6144-8_6

Parong, J., and Mayer, R. E. (2021). Cognitive and affective processes for learning science in immersive virtual reality. J. Comput. Assisted Learn. 37 (1), 226–241. doi:10.1111/jcal.12482

Pastel, S., Chen, C. H., Petri, K., and Witte, K. (2020). Effects of body visualization on performance in head-mounted display virtual reality. PLoS ONE 15, 0239226. doi:10.1371/journal.pone.0239226

Pellas, N. (2014). Harnessing and evaluating open sim for the implementation of an Inquiry-Based Collaborative Learning (Ib[C]L) script in computer science: Preliminary findings from a case study in higher education. Open Source Technol. Concepts, Methodol. Tools, Appl. 3–4, 1223–1246. doi:10.4018/978-1-4666-7230-7.ch059

Pellas, N., Peroutseas, E., and Kazanidis, I. (2013). “Virtual communities of inquiry (VCoI) for learning basic algorithmic structures with open simulator and Scratch4OS: A case study from the secondary education in Greece,” in ACM international conference proceeding series (New York, NY: Association for Computing Machinery ), 187–194. doi:10.1145/2490257.2490278

Radianti, J., Majchrzak, T. A., Fromm, J., and Wohlgenannt, I. (2020). A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Comput. Educ. 147, 103778. doi:10.1016/j.compedu.2019.103778

Rebelo, F., Noriega, P., Duarte, E., and Soares, M. (2012). Using virtual reality to assess user experience. Hum. Factors 54 (6), 964–982. doi:10.1177/0018720812465006

Rogers, L. (2011). Developing simulations in multi-user virtual environments to enhance healthcare education. Br. J. Educ. Technol. 42 (4), 608–615. doi:10.1111/j.1467-8535.2010.01057.x

Rueda, J., and Lara, F. (2020). Virtual reality and empathy enhancement: Ethical aspects. Front. Robotics AI 7, 506984. doi:10.3389/frobt.2020.506984

Saldana, D., Neureither, M., Schmiesing, A., Jahng, E., Kysh, L., Roll, S. C., et al. (2020). Applications of head-mounted displays for virtual reality in adult physical rehabilitation: A scoping review. Am. J. Occup. Ther. 74 (5), 7405205060p1–7405205060p15. doi:10.5014/ajot.2020.041442

Scavarelli, A., Arya, A., and Teather, R. J. (2021). Virtual reality and augmented reality in social learning spaces: A literature review. Virtual Real. 25 (1), 257–277. doi:10.1007/s10055-020-00444-8

Schreiber, J., and Asnerly-Self, K. (2011). Educational research. The interrelationship of questions, sampling, design and analysis . Hoboken, New Jersey: John Wiley & Sons, Inc .

Shawky, D., Badawi, A., Said, T., and Hozayin, R. (2014). “Affordances of computer-supported collaborative learning platforms: A systematic review,” in Proceedings of 2014 international conference on interactive collaborative learning, ICL (New York, NY: Institute of Electrical and Electronics Engineers (IEEE) ), 633–651. doi:10.1109/ICL.2014.7017846

Shi, Y., Dua, J., Ahn, C. R., and Ragan, E. (2019). Impact assessment of reinforced learning methods on construction workers’ fall risk behavior using virtual reality. Automation Constr. 104, 197–214. doi:10.1016/j.autcon.2019.04.015

Sims, R., Chang, B., Bennett, V., Krishnan, A., Aboubakar, A., Coman, G., et al. (2022). “Step into my mind palace: Exploration of a collaborative paragogy tool in VR,” in Proceedings of 2022 8th international conference of the immersive learning research network, iLRN 2022. Immersive learning research network (New York, NY: Institute of Electrical and Electronics Engineers (IEEE) ). doi:10.23919/iLRN55037.2022.9815936

Smith, S. A. (2019). Virtual reality in episodic memory research: A review. Psychonomic Bull. Rev. 26 (4), 1213–1237. doi:10.3758/s13423-019-01605-w

Sousa Santos, B., Dias, P., Pimentel, A., Baggerman, J. W., Ferreira, C., Silva, S., et al. (2009). Head-mounted display versus desktop for 3D navigation in virtual reality: A user study. Multimedia Tools Appl. 41 (1), 161–181. doi:10.1007/s11042-008-0223-2

Srivastava, P., Rimzhim, A., Vijay, P., Singh, S., and Chandra, S. (2019). Desktop VR is better than non-ambulatory HMD VR for spatial learning. Front. Robotics AI 6, 50–15. doi:10.3389/frobt.2019.00050

Stahl, G., Koschmann, T., and Suthers, D. (2006). “Computer-supported collaborative learning: An historical perspective,” in Cambridge handbook of the learning sciences . Editor R. K. Sawyer (Cambridge: Cambridge University Press ), 409–426. http://gerrystahl.net/cscl/CSCL_Chinese_traditional.pdf .

Sulbaran, T., and Jones, L. F. (2012). “Utilizing a collaborative virtual reality environment as a training tool for construction students,” in ASEE annual conference and exposition, conference proceedings (Texas, TX: American Society for Engineering Education (ASEE) ). doi:10.18260/1-2--22208

Tan, C. S. S., Luyten, K., van Den Bergh, J., Schoning, J., and Coninx, K. (2014). The role of physiological cues during remote collaboration. Presence Teleoperators Virtual Environ. 23 (1), 90–107. doi:10.1162/PRES_a_00168

Terzidou, T., Tsiatsos, T., Dae, A., Samaras, O., and Chasanidou, A. (2012). “Utilizing virtual worlds for game based learning: Grafica, a 3D educational game in second life,” in Proceedings of the 12th IEEE international conference on advanced learning technologies, ICALT (New York, NY: Institute of Electrical and Electronics Engineers (IEEE) ), 624–628. doi:10.1109/ICALT.2012.55

Tovar, D. F., Jonker, V., and Hürst, W. (2020). Virtual reality and augmented reality in education: A review . Utrecht, Netherlands: Utrecht University .

Tremmel, C., Herff, C., Sato, T., Rechowicz, K., Yamani, Y., and Krusienski, D. J. (2019). Estimating cognitive workload in an interactive virtual reality environment using EEG. Front. Hum. Neurosci. 13, 401–412. doi:10.3389/fnhum.2019.00401

Tüzün, H., Bilgiç, H. G., and Elçi, S. Y. (2019). The effects of 3D multi-user virtual environments on collaborative learning and social presence. Int. Electron. J. Elem. Educ. 11 (3), 221–231. doi:10.26822/iejee.2019349247

Vallance, M., Ibayashi, K., and Goto, Y. (2015). Engineering active learning in 3D virtual worlds. Commun. Comput. Inf. Sci. 533, 268–282. doi:10.1007/978-3-319-22629-3_22

Vrellis, I., Papachristos, N. M., Bellou, J., Avouris, N., and Mikropoulos, T. A. (2010). Designing a collaborative learning activity in second life: An exploratory study in physics. Proc. - 10th IEEE Int. Conf. Adv. Learn. Technol. ICALT 2010, 210–214. doi:10.1109/ICALT.2010.65

Wang, C. J., Low, A. S. L., and Woo, C. S. (2014). “3D system for learning gains in the lower secondary normal (technical) stream,” in 3D Immersive and Interactive Learning (Singapore: Springer ), 93–103. doi:10.1007/978-981-4021-90-6_7

Wang, A., Thompson, M., Roy, D., Pan, K., Perry, J., Tan, P., et al. (2022). Iterative user and expert feedback in the design of an educational virtual reality biology game. Interact. Learn. Environ. 30 (4), 677–694. doi:10.1080/10494820.2019.1678489

Yeh, S. C., Hwang, W. Y., Wang, J. L., and Chen, Y. R. (2012). “Effects of multi-symbols on enhancing virtual reality based collaborative task,” in Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Berlin, Germany: Springer Berlin Heidelberg ), Vol. 7220, 101–111. doi:10.1007/978-3-642-31439-1_10

Yoon, H. J., Kim, J., Park, S. W., and Heo, H. (2020). Influence of virtual reality on visual parameters: Immersive versus non-immersive mode. BMC Ophthalmol. 20 (1), 200–208. doi:10.1186/s12886-020-01471-4

Zheng, L., Xie, T., and Liu, G. (2019). “Affordances of virtual reality for collaborative learning,” in Proceedings - international joint conference on information, media and engineering, ICIME 2018 (New York, NY: Institute of Electrical and Electronics Engineers (IEEE) ), 6–10. doi:10.1109/ICIME.2018.00011

Keywords: virtual reality, collaborative learning, virtual reality education, collaborative virtual environment, virtual reality and collaborative learning, collaborative virtual reality, collaborative virtual reality systems, educational technologies

Citation: van der Meer N, van der Werf V, Brinkman W-P and Specht M (2023) Virtual reality and collaborative learning: a systematic literature review. Front. Virtual Real. 4:1159905. doi: 10.3389/frvir.2023.1159905

Received: 06 February 2023; Accepted: 02 May 2023; Published: 19 May 2023.

Reviewed by:

Copyright © 2023 van der Meer, van der Werf, Brinkman and Specht. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Nesse van der Meer, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• PubMed/Medline
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Immersive virtual environment technology to supplement environmental perception, preference and behavior research: a review with applications.

1. Introduction

2. ive technology, 3. ive research as a complement to traditional human perception, preference and behavior research, 3.1. traditional lab-based experiments, 3.2. field-based experiments, 4. workflow.

Click here to enlarge figure

4.1. Image Acquisition

4.2. image stitching, 4.3. image manipulation, 4.4. image conversion to virtual environment, 4.5. collection of perception, preference and behavior data, 5. ive applications for environmental preference and behavior research, 5.1. ranking application.

5.2. Point Identification Application

6. conclusions, acknowledgments, conflicts of interest.

• Blascovich, J.; Loomis, J.; Beall, A.C.; Swinth, K.R.; Hoyt, C.L.; Bailenson, J.N. Immersive virtual environment technology as a methodological tool for social psychology. Psychol. Inq. 2002 , 13 , 103–124. [ Google Scholar ] [ CrossRef ]
• Loomis, J.M.; Blascovich, J.; Beall, A.C. Immersive virtual environment technology as a basic research tool in psychology. Behav. Res. Method. Instrum. Comp. 1999 , 31 , 557–564. [ Google Scholar ] [ CrossRef ]
• Witmer, B.G.; Singer, M.J. Measuring presence in virtual environments: A presence questionnaire. Presence-Teleoper. Virtual Env. 1998 , 7 , 225–240. [ Google Scholar ] [ CrossRef ]
• Fox, J.; Arena, D.; Bailenson, J.N. Virtual reality: A survival guide for the social scientist. J. Media Psychol. 2009 , 21 , 95–113. [ Google Scholar ] [ CrossRef ]
• Cruz-Neira, C.; Leigh, J.; Papka, M.; Barnes, C.; Cohen, S.M.; Das, S.; Engelmann, R.; Hudson, R.; Roy, T.; Siegel, L.; Vasilakis, C.; Defanti, T.A.; Sandin, D.J. Scientists in wonderland: A report on visualization applications in the CAVE virtual reality environment. In Proceedings of the IEEE Symposium on Research Frontiers in Virtual Reality, San Jose, CA, USA, 25–26 October 1993; pp. 59–66.
• Lantz, E.J. Digital domes and the future of large-format film. LF Examiner 2006 , 9 , 1–4. [ Google Scholar ]
• Lantz, E.J. A survey of large-scale immersive displays. In Proceedings of Emerging Display Technology, New York, NY, USA, 4 August 2007.
• Law, L.E. Fulldome video: An emerging technology for education. Learn. Lead. Technol. 2006 , 34 , 12–15. [ Google Scholar ]
• Wyatt, R. Science visualization within a planetarium setting. In Proceedings of the IAU/National Observatory of Athens/ESA/ESO Conference, Athens, Greece, 8–11 October 2007.
• Yu, K.C. Digital full-domes: The future of virtual astronomy education. Planetarian 2005 , 34 , 6–11. [ Google Scholar ]
• Sumners, C.; Reiff, P.; Weber, W. Learning in an immersive digital theater. Adv. Space Res. 2008 , 42 , 1848–1854. [ Google Scholar ] [ CrossRef ]
• Loomis, J.M.; Hebert, C.; Cicinelli, J.G. Active localization of virtual sounds. J. Acoust. Soc. Amer. 1990 , 88 , 1757–1764. [ Google Scholar ] [ CrossRef ]
• Loomis, J.M.; Klatzky, R.L.; Golledge, R.G. Auditory distance perception in real, virtual, and mixed environments. In Mixed Reality: Merging Real and Virtual Worlds ; Ohta, Y., Tamura, H., Eds.; Ohmsha: Tokyo, Japan, 1999. [ Google Scholar ]
• Zahorik, P.; Kistler, D.J.; Wightman, F.L. Sound localization in varying virtual acoustic environments. In Proceedings of the 2nd International Conference on Auditory Diplays, Santa Fe, New Mexico, 7–9 November 1994; pp. 179–186.
• Zahorik, P.; Wightman, F.L.; Kistler, D.J. On the discriminability of virtual and real sound sources. In Proceedings of Applications of Signal Processing to Audio and Acoustics, New Paltz, NY, USA, 15–18 October 1995; pp. 76–79.
• Zahorik, P. Scaling perceived distance of virtual sound sources. J. Acoust. Soc. Amer. 1997 , 101 , 3105–3106. [ Google Scholar ] [ CrossRef ]
• Ahn, S.J.; Fox, J.; Dale, K.R.; Avant, J.A. Framing virtual experiences: Effects on environmental efficacy and behavior over time. Commun. Res. 2015 , 42 , 839–863. [ Google Scholar ] [ CrossRef ]
• Dinh, H.Q.; Walker, N.; Hodges, L.F.; Song, C.; Kobayashi, A. Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments. In Proceedings of IEEE Virtual Reality, Houston, TX, USA, 13–17 March 1999; pp. 222–228.
• Hoffman, H.G.; Sharar, S.R.; Coda, B.; Everett, J.J.; Ciol, M.; Richards, T.; Patterson, D.R. Manipulating presence influences the magnitude of virtual reality analgesia. Pain 2004 , 111 , 162–168. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Hoffman, H.G.; Seibel, E.J.; Richards, T.L.; Furness, T.A.; Patterson, D.R.; Sharar, S.R. Virtual reality helmet display quality influences the magnitude of virtual reality analgesia. J. Pain 2006 , 7 , 843–850. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Slater, M. A note on presence terminology. Presence Connect 2003 , 3 , 1–5. [ Google Scholar ]
• Davis, S.; Nesbitt, K.; Nalivaiko, E. A systematic review of cybersickness. In Proceedings of the 2014 Conference on Interactive Entertainment, University of Newcastle, Australia, 2–3 December 2014.
• Rand, D.; Kizony, R.; Feintuch, U.; Katz, N.; Josman, N.; Rizzo, A.S.; Weiss, P.L.T. Comparison of two VR platforms for rehabilitation: Video capture vs. HMD. Presence-Teleoper. Virtual Env. 2005 , 14 , 147–160. [ Google Scholar ] [ CrossRef ]
• Kim, K.; Rosenthal, M.Z.; Zielinski, D.J.; Brady, R. Effects of virtual environment platforms on emotional responses. Comput. Method. Program. Biomed. 2014 , 113 , 882–893. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Palmer, J.F.; Hoffman, R.E. Rating reliability and representation validity in scenic landscape assessments. Landscape Urban Plan. 2001 , 54 , 149–161. [ Google Scholar ] [ CrossRef ]
• Stamps, A.E. Use of photographs to simulate environments: A meta-analysis. Percept. Mot. Skills 1990 , 71 , 907–913. [ Google Scholar ] [ CrossRef ]
• Stamps, A.E. Use of static and dynamic media to simulate environments: A meta-analysis. Percept. Mot. Skills 2010 , 111 , 355–364. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ewing, R.; Cervero, R. Travel and the built environment: A meta-analysis. J. Amer. Plan. Assn. 2010 , 76 , 265–294. [ Google Scholar ] [ CrossRef ]
• Willemsen, P.; Gooch, A. Perceived egocentric distances in real, image-based, and traditional virtual environments. In Proceedings of the IEEE Virtual Reality Conference, Orlando, FL, USA, 24–28 March 2002; pp. 275–276.
• Aronson, E.; Wilson, T.L.; Brewer, M.B. Experimentation in social psychology. In The Handbook of Social Psychology ; Gilbert, D.T., Fiske, S.T., Lindzey, G., Eds.; McGraw-Hill: Boston, MA, USA, 1998; pp. 99–142. [ Google Scholar ]
• Blascovich, J.; Bailenson, J. Infinite Reality: Avatars, Eternal Life, New Worlds, and the Dawn of the Virtual Revolution ; Harper Collins: New York, NY, USA, 2011. [ Google Scholar ]
• de Kort, Y.A.W.; Meijnders, A.L.; Sponselee, A.A.G.; Ijsselsteijn, W.A. What’s wrong with virtual trees? Restoring from stress in a mediated environment. J. Environ. Psychol. 2006 , 26 , 309–320. [ Google Scholar ] [ CrossRef ]
• Persky, S. Application of virtual reallity methods to obesity prevention and management research. J. Diabetes Sci. Technol. 2011 , 5 , 333–339. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Garcia, L.; Kartolo, A.; Méthot-Curtis, E. A discussion of the use of virtual reality in dementia. In Virtual Reality in Psychological, Medical and Pedagogical Applications ; Eichenberg, C., Ed.; InTech: New York, NY, USA, 2012; pp. 123–136. [ Google Scholar ]
• Persky, S.; McBride, C.M. Immersive virtuall environment technology: A promising tool for future social and behavioral genomics research and practice. Health Commun. 2009 , 24 , 677–682. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Persky, S. Employing immersive virtual environments for innovative experiments in health care communication. Patient Educ. Couns. 2011 , 82 , 313–317. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ahn, S.J. Incorporating immersive virtual environments in health promotion campaigns: A construal level theory approach. Health Commun. 2015 , 30 , 545–556. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Coulter, A. Patient engagement—What works? J. Ambul. Care Manag. 2012 , 35 , 80–89. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Persky, S.; Kaphingst, K.A.; Allen, V.C., Jr; Senay, I. Effects of patient-provider race concordance and smoking status on lung cancer risk perception accuracy among African-Americans. Ann. Behav. Med. 2013 , 45 , 308–317. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Sallis, J.F.; Floyd, M.F.; Rodríguez, D.A.; Saelens, B.E. Role of built environments in physical activity, obesity, and cardiovascular disease. Circulation 2012 , 125 , 729–737. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Lick, J.R.; Unger, T.E. The external validity of behavior fear assessment: The problem of generalizing from the laboratory to the natural environment. Behav. Modif. 1977 , 1 , 283–306. [ Google Scholar ] [ CrossRef ]
• Stamps, A.E. Effects of multiple boundaries on perceived spaciousness and enclosure. Environ. Behav. 2013 , 45 , 851–875. [ Google Scholar ] [ CrossRef ]
• Aronson, E.; Wilson, T.D.; Akert, R.M. Social Psychology: The Heart and the Mind ; Harper Collins: New York, NY, USA, 1994. [ Google Scholar ]
• Herzog, T.R.; Miller, E.J. The role of mystery in perceived danger and environmental preference. Environ. Behav. 1998 , 30 , 429–449. [ Google Scholar ] [ CrossRef ]
• Herzog, T.R.; Chernick, K.K. Tranquility and danger in urban and natural settings. J. Environ. Psychol. 2000 , 20 , 29–39. [ Google Scholar ] [ CrossRef ]
• Stamps, A.E. Enclosure and safety in urbanscapes. Environ. Behav. 2005 , 37 , 102–133. [ Google Scholar ] [ CrossRef ]
• McCann, R.A.; Armstrong, C.M.; Skopp, N.A.; Edwards-Stewart, A.; Smolenski, D.J.; June, J.D.; Metzger-Abamukong, M.; Reger, G.M. Virtual reality exposure therapy for the treatment of anxiety disorders: An evaluation of research quality. J. Anxiety Disord. 2014 , 28 , 625–631. [ Google Scholar ] [ CrossRef ]
• Riva, G.; Bacchetta, M.; Baruffi, M.; Molinari, E. Virtual reality-based multidimensional therapy for the treatment of body image disturbances in obesity: Controlled study. Cyberpsychol. Behav. 2001 , 4 , 511–526. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Riva, G. Virtual reality in psychotherapy: Review. Cyberpsychol. Behav. 2005 , 8 , 220–230. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Moore, R.; Cosco, N. Using behavior mapping to investigate healthy outdoor environments for children and families: Conceptual framework, procedures, and applications. In Innovative Approaches to Research Excellence in Landscape and Health ; Ward Thompson, C., Bell, S., Aspinall, P., Eds.; Taylor & Francis: London, UK, 2010. [ Google Scholar ]
• Walden-Schreiner, C.; Leung, Y.-F. Spatially characterizing visitor use and its association with informal trails in Yosemite Valley meadows. Environ. Manag. 2013 , 52 , 163–178. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gibson, J. The Ecological Approach to Visual Perception ; Erlbaum: Hillsdale, NJ, USA, 1986. [ Google Scholar ]
• Floyd, M.F.; Bocarro, J.N.; Smith, W.R.; Baran, P.K.; Moore, R.C.; Cosco, N.G.; Edwards, M.B.; Suau, L.J.; Fang, K. Park-based physical activity among children and adolescents. Am. J. Prev. Med. 2011 , 41 , 258–265. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Floyd, M.F.; Spengler, J.O.; Maddock, J.E.; Gobster, P.H.; Suau, L.J. Park-based physical activity in diverse communities of two U.S. cities: An observational study. Am. J. Prev. Med. 2008 , 34 , 299–305. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Steptoe, W.; Steed, A.; Rovira, A.; Rae, J. Lie tracking: Social presence, truth and deception in avatar-mediated telecommunication. In Proceedings of the 28th International Conference on Human Factors in Computing Systems, Atlanta, GA, USA, 10–15 April 2010; pp. 1039–1048.
• Frisoli, A.; Ruffaldi, E.; Filippeschi, A.; Avizzano, C.; Vanni, F.; Bergamasco, M. Indoor skill training in rowing practice with a VR based simulator. In Proceedings of the 10th European Workshop of Ecological Psychology, Pisa, Italy, 5 November 2008.
• Slater, M.; Spangler, B.; Corominas, D. Simulating virtual environments within virtual environments as the basis for a psychophysics of presence. ACM Trans. Graphic. 2010 , 29 . [ Google Scholar ] [ CrossRef ]
• Pertaub, D.; Slater, M.; Barker, C. An experiment on public speaking anxiety in response to three different types of virual audience. Presence-Teleoper. Virtual Env. 2002 , 11 , 68–78. [ Google Scholar ] [ CrossRef ]
• Sommer, R.; Sommer, B.B. A Practical Guide to Behavioral Research ; Oxford University Press: Oxford, UK, 1992. [ Google Scholar ]
• Lange, E. The limits of realism: perceptions of virtual landscapes. Landscape Urban Plan. 2001 , 54 , 163–182. [ Google Scholar ] [ CrossRef ]
• Fernando, R.; Kilgard, M.J. Environment mapping. In The CG Tutorial: The Definitive Guide to Programmable Real-Time Graphics ; Addison-Wesley: Boston, MA, USA, 2003. [ Google Scholar ]
• Greene, N. Environment mapping and other applications of world projections. IEEE Comput. Graph. Appl. 1986 , 6 , 21–29. [ Google Scholar ] [ CrossRef ]
• Smith lab immersive virtual environment applications. Available online: http://cascades.cnr.ncsu.edu/smith_lab/ive_apps.html (accessed on 11 September 2015).
• Daniel, T.C. Whither scenic beauty? Visual landscape quality assessment in the 21st century. Landscape Urban Plan. 2001 , 54 , 267–281. [ Google Scholar ] [ CrossRef ]
• Kaplan, R.; Kaplan, S. The Experience of Nature: A Psychological Perspective ; Cambridge University Press: Cambridge, UK, 1989. [ Google Scholar ]
• Dupont, L.; Antrop, M.; Van Eetvelde, V. Eye-tracking analysis in landscape perception research: Influence of photograph properties and landscape characteristics. Landscape Res. 2013 , 39 , 417–432. [ Google Scholar ] [ CrossRef ]
• Welch, R.B.; Blackmon, T.T.; Liu, A.; Mellers, B.A.; Stark, L.W. The effects of pictoral realism, delay of visual feedback, and observer interactivity on the subjective sense of presence. Presence-Teleoper. Virtual Environ. 1996 , 5 , 263–273. [ Google Scholar ]

© 2015 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).

Share and Cite

Smith, J.W. Immersive Virtual Environment Technology to Supplement Environmental Perception, Preference and Behavior Research: A Review with Applications. Int. J. Environ. Res. Public Health 2015 , 12 , 11486-11505. https://doi.org/10.3390/ijerph120911486

Smith JW. Immersive Virtual Environment Technology to Supplement Environmental Perception, Preference and Behavior Research: A Review with Applications. International Journal of Environmental Research and Public Health . 2015; 12(9):11486-11505. https://doi.org/10.3390/ijerph120911486

Smith, Jordan W. 2015. "Immersive Virtual Environment Technology to Supplement Environmental Perception, Preference and Behavior Research: A Review with Applications" International Journal of Environmental Research and Public Health 12, no. 9: 11486-11505. https://doi.org/10.3390/ijerph120911486

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Int J Environ Res Public Health

Immersive Virtual Environment Technology to Supplement Environmental Perception, Preference and Behavior Research: A Review with Applications

Immersive virtual environment (IVE) technology offers a wide range of potential benefits to research focused on understanding how individuals perceive and respond to built and natural environments. In an effort to broaden awareness and use of IVE technology in perception, preference and behavior research, this review paper describes how IVE technology can be used to complement more traditional methods commonly applied in public health research. The paper also describes a relatively simple workflow for creating and displaying 360° virtual environments of built and natural settings and presents two freely-available and customizable applications that scientists from a variety of disciplines, including public health, can use to advance their research into human preferences, perceptions and behaviors related to built and natural settings.

1. Introduction

Research on how individuals’ values, beliefs, attitudes and behaviors are influenced by built and natural settings is inherently interdisciplinary and has used a variety of research methodologies. Participant observation, social surveys and laboratory experiments are widely used and indispensible tools. However, the ability of each of these methodologies to accurately represent and acutely control environmental stimuli varies widely, often leaving researchers unable to conclusively reject all possible confounding factors affecting their hypotheses. In response, a number of social scientists have become interested in the use of immersive virtual environment (IVE) technology as a complementary methodological tool given its capabilities of representing and controlling environmental stimuli [ 1 , 2 ]. The objectives of this paper are three fold. The first objective is to describe how IVE technology can be used to complement traditional methodologies used to investigate how individuals perceive and respond to built and natural environments; emphasis is placed on potential applications to public health research. The second objective of this paper is to describe a relatively simple workflow for creating and displaying 360° virtual environments of built and natural settings. The final objective is to present two editable and customizable applications that scientists from a variety of disciplines, including public health, can use to advance their research into human preferences, perceptions and behaviors related to built and natural settings.

2. IVE Technology

Virtual environments are “synthetic sensory information that leads to perceptions of environments and their contents as if they were not synthetic” [ 1 ]. IVEs in turn, are ones that ‘surround’ an individual and create the perception they are enclosed within and interacting with environments that provide a continuous stream of stimuli [ 3 ]. IVEs are created through the integration of various hardware and software systems [ 1 , 4 ]. All systems include a user interface displaying the virtual environment to users, a tracking system recording the users’ movements and a computer that selects appropriate portions of the virtual environment to be displayed within the interface. IVE technologies provide a flexible and low-cost option for both creating and displaying virtual environments to research subjects (the IVE system and workflow described in this paper can be fully implemented for under $3000 USD).

IVEs can be experienced simultaneously by groups or by single individuals. Virtual environments can be projected to groups from behind using translucent screens surrounding the group in a cube-like environment [ 5 ] or via front projection systems. Front projection systems, which are more frequently used, can vary widely from semi-immersive environments, such as curved wall displays, to fully-immersive dome environments [ 6 , 7 , 8 , 9 , 10 ]. Group immersion can also be accomplished through arrays of flat screen displays. Across these different group display types, all users wear 3D glasses while a primary user also wears one or more positioning devices which tracks their body movements; the virtual environment is subsequently rendered according to the primary user’s position. While group IVE systems can be used in psychology and social psychology research, they have primarily been used as a novel medium through which educational material are presented [ 11 ].

As an alternative to group IVEs, virtual environments can be presented to single users, such as study participants, via head-mounted display systems. Head-mounted displays present virtual environments via single or multiple displays placed immediately in front of a research participant’s eyes; the intent is to occupy as much of the participant’s field-of-view as possible with the display. An essential component of presenting IVEs via head-mounted displays is tracking users’ head movements. Today, most head-mounted display devices accomplish this via sophisticated inertial tracking systems. Inertial and video tracking systems are also capable of monitoring the position and orientation of users’ hands or even their whole body by wearing specially designed gloves or suits and using software capable of dynamically rendering corresponding virtual objects (e.g., the hands or body positions of avatars). Head-mounted displays have historically been out of the reach of social scientists because of their high costs plus the installation expenses of additional graphics cards required for the simultaneous presentation of images in multiple displays. Advances in the standard offerings of computer hardware systems as well as the large gaming market have drastically improved access. Head-mounted display systems are now commercially available at costs well within even student budgets.

IVEs are capable of displaying and manipulating visual (sight), auditory (hearing), gustatory (taste), haptic (touch), olfactory (smell) or thermal (temperature) sensory stimuli. The reproduction of recorded three-dimensional sound-fields has been explored for over three decades [ 12 , 13 , 14 , 15 , 16 ]. Haptic stimuli, while being used relatively less than auditory stimuli, is already being applied to questions in environmental psychology such as the study of individuals’ environmentally responsible behavior [ 17 ]. Olfactory and thermal stimuli are infrequently used in research using IVE technology; however, some empirical evidence suggests they can play important roles in perception, cognition and memory [ 18 , 19 , 20 ]. The array of directly controllable stimuli provides virtual environment developers the ability to present immersive environments with varying degrees of realism, or ‘presence’ as it is commonly referred [ 21 ]. This provides the opportunity for researchers to examine the effects of various stimuli’s existence and magnitude on individuals’ perceptions, preferences and behaviors. However, almost all previous research utilizing IVE technology has used fully synthetic virtual environments. The field of psychophysics for example, is focused entirely on understanding psychological responses to virtual stimuli that represent physical objects. While this line of research has led to many nuanced developments in understanding of how visual cues and image quality affect sensory and perceptual responses, the use of fully synthetic virtual environments has limited adoption of the technology by investigators with more applied research agendas. There is a differentiation between IVE research based on wholly synthetic environments and research focused on replicating natural environments; the workflow presented below focuses on the latter.

IVE technology provides scientists with the capacity to collect an array of response data varying in ontological complexity. Low-level reflexive responses such as physiological reactions (e.g., heart rate, skin-conductance, skin-temperature, respiration patterns, etc .) can be easily gauged through commercially available biomonitoring hardware and software. Alternatively, more complex high-level responses (e.g., actions, movements, speech, etc .) can be measured through a variety of different methods such as audio/video recordings of individuals as they experience different virtual environments. Moreover, these response data can be collected simultaneously enabling scientists to develop a more acute understanding of how various controlled environmental stimuli simultaneously affect multiple response systems.

As with all data collection methods, there are limitations to the use of IVEs. The first is cybersickness, which is nausea, dizziness and general discomfort caused by the unique stimuli of a virtual environment [ 22 ]. Cybersickness is experienced by a small fraction of individuals involved in IVE-based research, with almost all negative side effects dissipating after several minutes. Most individuals susceptible to cybersickness are aware they become nauseous when they do not have control of their movements (e.g., from previous experiences getting car- or sea-sick). These individuals tend to self-select out of participating in IVE-based research. However, all IVE-based research must require explicit informed consent about the possibility of becoming nauseous following individual institutions’ institutional review board guidelines. A second limitation is that developing IVE applications capable of collecting preference, perception and behavioral response data does require some familiarity with programming. This is not a major limitation however, as several IVE software packages come with detailed supporting documentation and have active online discussion groups. The third concern is the potential effect of the technology itself on biasing responses. Comparative research [ 23 , 24 ] has found different methods of displaying virtual environments elicits different emotional, perceptual and behavioral responses, raising concerns about the use of IVEs as the sole-method in which hypotheses are tested. This concern is valid; the findings generated from IVE-based research should be corroborated against other methods whenever possible. The next section illustrates how IVE-based research can complement traditional human perception, preference and behavior research focused on understanding how individuals perceive and respond to built and natural environments; an emphasis is placed on potential applications to public health research.

3. IVE Research as a Complement to Traditional Human Perception, Preference and Behavior Research

The suite of data collection methods available to social scientists interested in how humans perceive and respond to built and natural settings has increased over the past several decades. The dominant methods of representing built and natural environments has transitioned from the use of static imagery to the use of basic dynamic media (e.g., videos), to the use of interactive media (e.g., navigable walkthroughs) and finally to the use of fully immersive virtual simulations [ 25 , 26 , 27 , 28 ]. The transition from static to dynamic imagery illustrates a shift in scientists’ desire to move beyond describing social phenomenon (observational designs of individuals in the field) to discerning significant relationships (correlation analysis between specific image attributes and a variety of user characteristics) and eventually testing causal hypotheses by controlling variables of interest (experimental manipulation of image attributes). However, the shift to the use of dynamic imagery has also come at the expense of ecological validity (the realistic representation of an environment) as most early dynamic representations were created with computer hardware and software systems only capable of generating relatively simple and unrealistic virtual environments [ 29 ]. This trade-off between experimental control and ecological validity has been described as a major methodological problem facing psychological and behavioral research [ 30 ]. IVEs coupled with advances in computer technology (particularly processing speeds and graphic display capabilities) and software applications, however, provide the opportunity to both exert strict control over independent variables/treatments and present experimental stimuli in an extremely realistic format [ 1 ], thus providing the highly desirable ability to maximize internal validity while minimizing threats to external and construct validity. In short, IVEs provide scientists with the ability to maximize the benefits of traditional lab-based experiments ( i.e. , control over independent variables and randomization of treatments) and field-based experiments ( i.e. , high psychological realism of the phenomenon under study).

Several social science disciplines have capitalized on the development and recent advancements in IVE systems. The fields of psychology [ 2 ] and social psychology [ 1 ] have been using IVEs to address basic psychological and social phenomenon for several decades [ 31 ]. However, more applied disciplines such as environmental psychology [ 32 ] have been slower to integrate IVEs as a complementary methodological tool. Within the public health domain, substantive advances have been made in several specific areas such as obesity prevention and maintenance [ 33 ], dementia [ 34 ], genomics research [ 35 ] and health communication [ 36 ]. Within genomics research, Persky and McBride [ 35 ] outline how IVE technology could be used to assess and improving genetic test uptake rates amongst potential test-takers; however, the technology has yet to be applied in this context. Persky and McBride [ 35 ] also suggest IVE technology can be used to simulate distal consequences of individuals’ immediate decisions, a concept that was recently examined by Ahn [ 37 ] who found individuals exposed to a distal health outcome (weight gain as a result of soft drink consumption) through an IVE consumed less than individuals who only received information about the likely long-term effects of increased soft drink consumption.

Specific to behavioral research within the public health domain, Persky and McBride [ 35 ] suggest IVE technology provides an opportunity to develop patient-specific behavioral change interventions. Behavioral change appeals are most effective when tailored to patients’ mental states, health needs and sociodemographic characteristics [ 38 ] and IVEs provide an opportunity to rapidly modify select aspects of appeals (e.g., different modes of presenting health information), consequently improving the likelihood of achieving positive behavioral outcomes. Recent research within this area has yielded insightful findings. For example, Persky and her colleagues examined African-American patients’ ability to accurately describe risks associated with lung cancer after an IVE-based intervention in which they interacted with a virtual physician [ 39 ]. Risk perceptions were significantly less accurate when the patient interacted with a Caucasian as opposed to African-American physician; the effect was still significant even after controlling for patients’ perceived level of trust in the physician.

While the work of Persky and her colleagues is a focused example of using IVE technology to study behavioral change interventions, similar insight could be gained in other public health research that more directly involves assessing how individuals perceive and respond to built and natural settings. For example, health facilities around the United States have been rapidly adopting “park prescriptions”, which involve encouraging patients as well as members of the general public to use public park and recreation areas for physical activity in an effort to improve mental health and prevent chronic disease. Michelle Obama’s well known “Let’s Move Outside” campaign, the “Leave No Child Inside” campaign, the Children and Nature Network and the National Park Service’s “Healthy Parks, Healthy People” initiative are all notable examples. Despite the popularity of these campaigns and initiatives, no behavioral change research has examined whether or not they have altered individuals’ likelihood of visiting nearby parks and greenspaces. IVE-based research can be used to assess the efficacy of information disseminated by these programs amongst different populations. A variety of factors known to influence the adoption of preventive behaviors [ 38 ] such as the mode of information delivery (e.g., print vs. radio vs. television), the source of the information (e.g., physicians, celebrities, etc .) and the financial cost of adopting preventive behaviors are all likely to affect intended park use. Additionally, the quality and characteristics of available parks will influence individuals’ willingness to use them [ 40 ]. This diverse combination of factors, as well as others, has made it difficult for behavioral scientists to make definitive and empirically grounded statements about how best to promote the use of public parks and greenspaces amongst diverse publics. IVE technology provides the opportunity to examine and isolate confounding factors in a rigorous, yet flexible way. Different modes and sources of information can be examined across individuals with different sociodemographic backgrounds; and different park and neighborhood characteristics can be manipulated through the construction of different virtual environments. Hundreds of local, state and national campaigns, initiatives and programs have been developed to mitigate rising rates of chronic preventable diseases. However, these efforts almost always lack empirically grounded evaluative research that could improve their efficacy. IVE technology holds the promise of alleviating many of the methodological barriers faced by researchers in this area. In the subsequent sections, a description is provided of how social scientists can use IVEs as a supplement to traditional data collection methodologies. The discussion focuses on lab-based experiments and field-based experiments.

3.1. Traditional Lab-Based Experiments

Laboratory-based experiments are a particularly attractive methodology because they provide precise control over treatment effects. Investigators have direct control over both who is exposed to certain independent variables which allows randomization and control over the range across which those independent variables fluctuate. The ability to control independent (treatment) variables without affecting other confounding effects results in research findings with a high degree of internal validity [ 30 ]. The benefits derived from direct control over the administration of, and variability within, the independent variable, however, comes at the expense of external validity. Results from lab-based experiments are often criticized for their limited generalizability to large populations which is a major barrier to generating actionable policy and management recommendations [ 41 ]. Research attempting to understand preferences and perceptions associated with imagery of built and natural settings has typically been designed using lab-based experiments that present research participants with static color or grayscale images of either photorealistic scenes or computer generated illustrations [ 27 , 42 ]. This type of experimental research has focused primarily on measuring landscapes’ aesthetic appeal and using images portraying natural or semi-natural landscapes [ 27 ].

While the use of static color or grayscale images of landscapes in lab-based experiments has become relatively common, the approach does have several methodological shortcomings. Primarily, the mundane realism of lab-based experiments tends to be low, which means there is little concordance between how an individual experiences a particular treatment in the lab and how (or even if) they would experience it in their everyday lives [ 43 ]. Consequently, there is a concern the psychological processes (e.g., perceptions of safety) differ when a treatment is administered in the lab relative to when it is experienced in individuals’ everyday lives. The extent to which a specific psychological process can be reproduced in the lab setting is referred to as psychological realism [ 43 ]. IVEs can enhance the psychological realism of an experiment measuring human perceptions of, and preferences for, built and natural settings by supplementing existing lab-based methods. Consider for example, the growing body of research on the perceived safety of built and natural environments [ 44 , 45 , 46 ]. This scholarship has been built upon experimental methods whereby individuals are presented with a series of photographs, computer-generated renderings or illustrations, and asked to rate the images on rating scales (from extremely safe to extremely threatening). The images presented to participants are experimentally varied relative to some conceptually important independent variable. Typically, the independent variables include measures of an area’s enclosure or spaciousness. For example, a barrier such as a wall or hedgerow might be manipulated in its height and horizontal area. While this scholarship has made inroads into understanding how elements of the physical environment affect perceived safety, it is unclear whether the psychological process of fear-elicitation that individuals experience in day-to-day life can be comparably induced simply by having them look at a photograph, computer generated rendering or illustration. This could be a substantial threat to the validity of the research and clear analogs could be drawn to other domains of interest. The use of IVE technology can potentially mitigate the issue of low psychological realism given it allows various physical environmental stimuli to be presented in a highly realistic manner.

Applied psychology and social psychology research in other domain areas has already begun to capitalize on the high psychological realism IVE technology enables. For example, psychotherapists have successfully used virtual environments in the treatment of anxiety [ 47 ], eating disorders [ 48 ] and various phobias [ 49 ]. In short, supplementing controlled experimental designs with data generated through exposure to IVEs can increase the external validity of research findings due to IVEs increased capacity to present built and natural settings with an extremely high degree of mundane and psychological realism.

3.2. Field-Based Experiments

Given the criticisms of lab-based experiments’ failure to generate generalizable results, many investigators have opted to use field-based experimental designs. While field-based experiments offer the obvious advantage of high-levels of psychological realism, it is often extremely difficult to exert control over exogenous factors that might influence the relationship between the independent variable being manipulated across field settings and the dependent variable of interest.

As an example of the methodological difficulties inherent with field-based experiments, consider recent work using on-site behavior mapping [ 50 ], which involves the systematic observation and classification of individuals’ behavior relative to their spatial location within a physical environment. The method, which is gaining acceptance as a tool for mapping and monitoring visitor use in a variety of outdoor settings such as parks [ 51 ], allows investigators to discern how different environmental settings (e.g., meadows vs. forest edges) affect individuals’ behavioral patterns. Although research administrators follow strict observational protocols (e.g., times at which observations are recorded, the direction in which an environment is scanned) for recording individuals’ behavior, the primary methodological limitation of this approach is the inability to precisely determine what characteristics of an environment are responsible for eliciting certain responses from individuals; this is because these factors are difficult, if not impossible, to control [ 50 , 52 ]. Because of this threat to internal validity, behavior mapping research is often limited to demonstrating correlational findings, which of course are less persuasive in affecting policy change and enhancing scientific understanding.

Supplementing participant observation research with research using IVE technology offers a solution to the aforementioned methodological problem. IVE technology provides the ability to confirm and refine social and behavioral phenomenon observed in the field because investigators are able to evaluate the extent to which different physical characteristics elicit variations in behavioral responses. IVE-based research provides investigators with the ability to manipulate characteristics of built and natural settings that, in the field, are permanent or very difficult and costly to change. In a relatively short amount of time, an investigator can construct two or more nearly identical virtual environments with only one distinguishable factor. The behaviors of individuals within an immersion system could then be collected to discern more acutely and with a greater degree of internal validity the causal mechanisms driving the variations observed in the field settings.

The flexibility offered by virtual environments extends beyond modifications to the inanimate objects that comprise a built or natural setting. Researchers can also modify the characteristics and number of people present within those settings. This is a particularly appealing option for investigators studying how the use of a physical space affects individual perceptions and behaviors. For example, there is now a large literature devoted to understanding how the sociodemographic characteristics of existing park users affects potential park visitors’ willingness to use the space [ 53 ]. This research, however, has been impeded by an inability to manipulate the sociodemographic characteristics of park users and evaluate the subsequent perceptions and behaviors of potential park users [ 54 ]. IVE technology overcomes this impediment by providing investigators the ability to populate park areas with avatars of different races and ethnicities. IVE technology infinitely expands the ‘field’ in which traditional field-based research has been grounded. Mixed-methods approaches that use both IVEs and traditional field-based approaches allow investigators to explore contingent perceptions and behaviors that can be validated against reality.

Another advantage of IVE technologies in field-based and participant observation studies is its ability to simultaneously collect a variety of human responses that vary in ontological complexity. Eye tracking systems can measure simple or involuntary responses to virtual environments such as individuals’ gaze patterns and physiological attention measures such as blink-rate and pupil size [ 55 ]. Head tracking systems can measure slightly more complex responses, namely what objects individuals choose to orient their heads toward in a simulated environment [ 56 , 57 ]. Hand tracking devices can be used to collect data on even more complex responses such as what individuals choose to interact with in a virtual setting. Finally, and perhaps most interestingly, microphones synched to video recordings of individuals’ exploration of a virtual environment can be used to collect verbal responses from research participants [ 58 ]. Furthermore, IVE technology has the ability to incorporate qualitative methods into research designs to generate more detailed, and perhaps more conceptually insightful, findings to explain why individuals are engaging in a particular behavior, which is a major limitation of field-based research [ 59 ].

In brief, IVE technology has much to offer scientists with an interest in acutely measuring individuals’ perceptions of, preferences for, and behavioral responses to built and natural settings; the benefits are particularly salient for public health research. Specifically, IVEs can be used as a supplemental methodology to reduce or eliminate the limitations associated with lab-based and field-based experimental research designs as they can reproduce ‘real-world’ stimuli, duplicate psychological processes that occur in people’s daily lives, produce generalizable findings and generate more acute and internally valid measures of individuals’ preferences for, perceptions of, and behavioral responses to, built and natural settings.

4. Workflow

The construction of IVEs for research applications is a relatively simple and logical process requiring only a working knowledge of either digital image manipulation and processing or computer programming and application development. The workflow presented in Figure 1 can be used to guide readers’ understanding of IVE and application development. All of the workflow processes described use low-cost and commercially available hardware and software. The workflow follows a flexible five-step process involving: (1) image acquisition; (2) image stitching; (3) image manipulation; (4) image conversion to a virtual environment; and (5) data collection.

Workflow for developing immersive virtual environments of built and natural settings.

4.1. Image Acquisition

The workflow is initiated through the collection of digital imagery. One method for image acquisition involves a DSLR camera fitted within a robotic controller (e.g., GigaPan’s EPIC Pro controller). High quality DSLR cameras can be purchased for under $800 USD and the cost of robotic controllers does not exceed $1000 USD. The controller is mounted atop a tripod ( Figure 1 a) and precisely controls the movement of the camera, enabling it to be rotated around a single point. The controller is programmed with the camera’s field of view, which varies depending upon the camera’s zoom extent. The controller rotates the camera and triggers the shutter to collect an array of images. Each image overlaps adjacent images by approximately 33%. The image acquisition process culminates with the collection of an array of images that can be arranged in a rectangular grid.

4.2. Image Stitching

The next step of the workflow involves stitching the rectangular array of images to create a full 360° equirectangular image with vertical dimensions that include both poles. The development of advanced image matching algorithms designed to detect and match invariant features appearing in image pairs ( Figure 1 b) has led to the creation of fully automated image stitching processes integrated into commercially available software; most software only requires the user to arrange images in their correct relative positions. The author’s lab group uses the Autopano Giga 4 image-stitching software produced by Kolor (cost is approximately $225 USD), however, other alternatives include PTGui Pro and Adobe Photoshop. Photo manipulation software can be used to correct inconsistencies in exposure, color balance and white levels. Once an image array is stitched together, it can be reverse projected and displayed as an IVE (discussed later). Custom curved mirrors attached to the camera lens can also be used to circumvent the need to stitch together image arrays; however, acquiring images in this way requires specialized software to re-project the single image for display via a navigable interface.

4.3. Image Manipulation

This stage in the workflow process is optional; its inclusion will depend on the research questions being explored. The process involves taking an individual stitched equirectangular image and manipulating certain features within the image following an experimental design. While this stage is optional, it eliminates the possibility of confounding effects arising from the lack of complete experimental control. For example, photo-elicitation methods in environmental psychology have involved examining preferences for natural settings with varying biophysical attributes (e.g., preferences for forests that are either clear cut or selectively harvested). The photos used for comparison are often taken at different locations, at different times of day and contain potentially confounding features, such as the presence of rivers and wildlife, which might influence perception and preference data.

Image manipulation can follow any experimental design. The researcher only has to choose which attributes of the environment are of interest (e.g., extent of canopy coverage, presence of built features, etc .) that can be manipulated across a meaningful range. Following an experimental design, a series of manipulated images can be created; these image sets form the experimental treatments across which perception, preference and behavioral response data are collected ( Figure 1 c).

Image manipulation requires the use of photo editing software such as Adobe Photoshop (cost is approximately $300 USD) and should be done with high regard for maintaining realism [ 25 ]. A final image set should be consistent in ecological validity ( i.e ., respondents should not be able to discern which images are of artificially created environments). The author’s lab group pilot tests using online marketplaces (e.g., Amazon.com’s Mechanical Turk) where randomized image sets are presented to study participants who are asked to rank the set relative to the realism of each image. The rank scores are used to gauge degree of realism [ 60 ] which is compared between original and manipulated images using the Friedman non-parametric test. Image sets exhibiting significant variation in realism are rejected from inclusion in the final experimental design.

4.4. Image Conversion to Virtual Environment

Converting stitched 2D equirectangular images into virtual environments involves a process known as cube mapping; this process uses a six-sided cube as a map shape [ 61 , 62 ]. The panoramic image is projected onto the cube’s faces; each face in turn is saved as an individual image file that will later be used to render the virtual environment ( Figure 1 d). Cube mapping can be completed through commercially available software; the author’s lab group uses Pano2VR (Garden Gnome Software).

The final step in displaying IVEs involves dedicated virtual reality software capable of rendering a virtual environment and displaying it in a user interface. The author’s lab group uses a commercially available software toolkit (Vizard; costs range from approximately $80 USD for the ‘Lite’ edition which is suitable for the applications described later to more than $10,000 USD for multi-seat ‘Development’ and ‘Enterprise’ editions) to read-in and map the six cube images and display the virtual environment in a user interface. The Vizard toolkit offers a high degree of flexibility in virtual environment creation, allows for the integration of multiple methods of interacting with virtual environments, and offers a low-cost edition.

4.5. Collection of Perception, Preference and Behavior Data

The final stage in the workflow involves the collection of data from research participants ( Figure 1 e). As noted above, IVE technology provides scientists with the capacity to collect an array of data varying in ontological complexity. Low-level responses such as physiological reactions (e.g., heart rate, skin-conductance, skin-temperature, respiration patterns, etc .) can easily be gauged through biomonitoring hardware and software. Higher level responses such as actions, movements and speech can be measured through various methods such as audio recordings of individuals’ responses to specific prompts. Response data can also be collected simultaneously, enabling scientists to develop a more acute understanding of how different controlled environmental stimuli simultaneously affect multiple behavioral response systems.

Collectively, the workflow is a relatively simple process for designing and developing virtual environments. In the final section of this paper, two freely available and modifiable custom applications created to gain a more empirically robust and valid understanding of how individuals perceive and respond to built and natural settings are presented.

5. IVE Applications for Environmental Preference and Behavior Research

The first application involves generating ranking data from individuals’ preferences among three virtual environments. The second application generates Cartesian coordinates for markers placed by participants while experiencing a virtual environment. The process of implementing research designs using both of these applications is presented in Figure 2 and more detail is available at [ 63 ].

Workflow diagram for both the ranking and place marker applications (HMD = Head Mounted Display).

5.1. Ranking Application

Several well-developed lines of research are devoted to understanding and accurately gauging the “quality” of built and natural settings. Typically experts or members of a specific audience are queried about how the composition of a setting ( i.e ., presence, absence, or composition of biophysical features) affect its aesthetic appeal [ 64 ]. Given pictures elicit greater recollection and cognitive elaboration [ 65 ], two-dimensional images of built and natural settings are commonly used to query individuals’ perceptions on the aesthetic appeal, degree of realism, desirability or some other specific dependent measure related to those images. The first IVE-based application offers a more sophisticated approach, allowing individuals to rank virtual environments while experiencing them via a head mounted display.

As described in the workflow above, investigators first need to collect imagery at a minimum of three locations, stitch, manipulate (if following an experimental design) and convert images into virtual environments using cube-mapping software. Then, cube face imagery can be placed in a hierarchical file structure on a central, local machine. The application will construct three distinct virtual environments using the appropriate cube faces placed within this directory. The three constructed virtual environments comprise the choice-set that individuals are presented with while in a head mounted display.

The data collection process begins by reading a script to research participants detailing the task of ranking the environments (e.g., prompting individuals to rank the environments from “most safe” to “least safe”). Once instructions have been given, participants are placed in the head mounted display and asked to slowly and deliberately explore each of the virtual environments by moving their head. Participants only experience one environment at a time, but can directly compare the attributes of alternative environments through the use of a wireless game controller using the up and down buttons (This could alternatively be accomplished through keyboard command. However, the use of keyboards requires research participants to remain seated throughout the experiment, which is likely to lead to an inability to see and explore portions of the image behind the default viewing position). Once respondents have viewed and explored each virtual environment, they are prompted to select the most appealing environment with the aid of a wireless gaming remote. The application records a respondent’s initial selection in the database (e.g., environment 3 = rank 1 ) and removes that environment from the choice set. The respondent is again prompted to select between the remaining two environments and their subsequent responses are recorded accordingly in the database (e.g., environment 2 = rank 2, environment 1 = rank 3 ). This ranking experiment can be repeated as many or as few times as needed according to the experimental design being employed. The application’s sample code repeats the ranking process four times resulting in the collection of 12 data-points (four locations with three environments each; Table 1 ). These data can subsequently be analyzed using a variety of statistical techniques to discern significant variations across rankings within a single choice-set; they can also be analyzed using multi-level ranked logistic regression models to discern significant relationships across all choice sets.

Example of data collected through the ranking application.

5.2. Point Identification Application

Investigators focused on examining individuals’ perceptions, preferences and behaviors relative to different stimuli have a long-standing interest in discerning which characteristics of a built and natural setting elicit certain responses. For example, emerging work has used ocular tracking technologies, which map the focus of individuals’ gaze across a static image, to discern what characteristics of natural settings they focus on when viewing an image [ 66 ]. The second application presented here allows investigators to collect data on preferences for specific elements within virtual environments of built and natural settings (e.g., the presence, absence, or composition of biophysical features). Specifically, the application allows individuals to select specific points within a virtual environment after being given a prompt. Point locations are recorded by the application, thus providing investigators the ability to generate highly acute maps corresponding to individuals’ preferences. Again, a sample executable and detailed description of the application is available at [ 63 ].

The point identification application uses a singular virtual environment, which can either represent a real or digitally manipulated setting. After receiving specific instructions, participants are asked to explore the virtual environment and locate specific “points” (which correspond to the variables being examined) with a target-like icon originally placed at the center of their field of vision. For example, individuals can be prompted to identify three points within the environment that deter from its use as a place for physical activity. Participants confirm their selections with a wireless gaming controller. The application’s sample code version allows individuals to identify three specific points within the environment. Points can be elicited through prompts that vary relative to the specific research question being investigated. The application records both the world (spherical) and environment (cube-mapped) coordinates of respondents’ point selections in the database. The collection of these coordinates provides the ability to disassemble the cube into its six sides and retain the location of user-defined points. The Cartesian (two-dimensional) coordinates can subsequently be visualized relative to their concentration ( Figure 3 provides an example). Beyond visualization, investigators can generate summary statistics of points located in specific areas of a setting or explore the data with more advanced spatial statistics.

Example of an unfolded virtual environment without ( above ) and with ( below ) a heat map visualization of points generated with the place marker application.

6. Conclusions

IVE technology is quickly becoming a ubiquitous part of our modern technologically integrated lives. The widespread availability of IVE technology has made it an additional methodological tool accessible to investigators studying human perceptions, preferences and responses to built and natural settings. This article has detailed how IVE technology can be used to supplement traditional methods focused on discerning how human perceptions, preferences and behaviors change relative to varying environmental stimuli; particular emphasis has been placed on possible avenues for future research within the public health domain. Robust mixed-methods designs using IVE technology along with more traditional lab- or field-based experiments hold the potential to contribute not only to methodological rigor, but also to the examination of novel and previously unexplored phenomena. Several specific research areas where IVE technology could have an immediate impact have been described. Specifically, perception, preference and behavior research has typically used data collected by presenting static imagery to individuals in a lab environment or via web-based or paper questionnaires. This methodology, while being foundational to our understanding of environmental preferences and perceptions, is limited in its ability to realistically and wholly replicate environments as individuals experience them in their everyday lives. IVE technology holds the promise of bolstering this shortcoming through the generation of virtual environments that can be experienced and explored with nearly as much realism as daily life [ 67 ].

Other field-based methodologies, such as behavior mapping also have potential to be augmented with IVE technologies. Field-based experimental designs can be easily criticized for their lack of internal validity and inability to control for exogenous factors that may confound the relationship between environmental characteristics and perception, preference or behavioral responses. IVE technologies provide these field-based investigations with the opportunity to wholly replicate built and natural settings save for specific features believed to influence the dependent variable of interest.

This article also presented a workflow for creating and displaying 360° virtual environments of built and natural settings. The workflow is not intended to be a step-by-step guide for adopting IVE technology. Rather, it is intended to be sufficiently detailed enough to provide individual investigators and research teams with direction and guidance on how the use of IVE technology can be initiated. Further development and refinement will begin with investigators’ individual needs and capabilities. To further assist scientists in their potential adoption of IVE technology, sample code for two freely-available and editable applications has been developed and presented. The first application collects ranking data from individuals’ preferences among three virtual environments while the second collects coordinate locations of points placed by users as they experience and explore a virtual environment. Both applications have the potential to be used in the near future to address a wide variety of research questions.

In conclusion, the use of IVE technology has become more and more common in applied arenas of certain fields such as medicine, psychotherapy and education. However, the technology’s use to address long-standing research questions related to human perceptions of, and preferences for, built and natural settings has remained limited. Due to increasing consumer demand, immersion systems are well within the reach of nearly all researchers. The description of the technology’s advantages as a research method, its workflow and sample applications for capturing and displaying built and natural settings, will enable more scientists to expand the boundaries of their work and develop a deeper understanding of the complex interactions between individuals and the environments in which they live.

Acknowledgments

This work was supported by NC State University’s College of Natural Resources. The author would like to thank Zac Arcaro, Makiko Shukunobe and Zahra Zamani for their contributions to this research.

Conflicts of Interest

The author declares no conflict of interest.

Exploring Challenges and Impacts: Insights from School Teachers in Virtual Learning Environments

Arab World English Journal, No. Special Issue on CALL Number 10. July 2024 Pp.172-190

19 Pages Posted: 7 Aug 2024

Magdelina Anak Nugak

International Islamic University Malaysia

Siti Fatimah Abd Rahman

National University of Malaysia (UKM) - Faculty of Education

Noorlila Ahmad

International Islamic University of Malaysia (IIUM), Students

Nor Asiah Mohamad Razak

Universiti Pendidikan Sultan Idris

Date Written: July 28, 2024

As Virtual Learning Environments become increasingly integral to educational practices, this study delves into the often-neglected realm of challenges faced by teachers in the implementation of virtual teaching and learning. This study aims to explore the challenges and impacts faced by teachers in implementing virtual learning environments. Next is to identify the best features in addressing the challenges and impact of virtual learning implementation among teachers. Employing a qualitative case study design, the research conducted semi-structured interviews and observations involving primary school teachers from level 1 (years 1 to 3) and level 2 (years 3-6). The findings illuminate multifaceted challenges and constraints encountered by teachers, resonating across the realms of teacher dynamics, school infrastructure, and student engagement. Efforts to surmount these challenges revolve around recognizing teachers as crucial exemplars and elucidating the responsibilities they shoulder. Moreover, the study underscores the pivotal role of the learning environment and atmosphere during virtual teaching and learning implementations. In essence, teachers emerge as driving agents shaping the practical construction of the curriculum, necessitating a supportive environment and comprehensive infrastructure. The implications of this research extend towards fostering effective teaching and learning practices, ensuring a more conducive educational landscape for both educators and students in the era of virtual pedagogy.

Keywords: virtual learning environment, online education, challenges and impact, school teacher

Suggested Citation: Suggested Citation

Magdelina Anak Nugak (Contact Author)

International islamic university malaysia ( email ), national university of malaysia (ukm) - faculty of education ( email ).

Bangi Selangor Malaysia

International Islamic University of Malaysia (IIUM), Students ( email )

Universiti pendidikan sultan idris ( email ), do you have a job opening that you would like to promote on ssrn, paper statistics, related ejournals, pedagogy ejournal.

Subscribe to this fee journal for more curated articles on this topic

Anthropology of Education eJournal

Cognitive psychology ejournal, arab world english journals.

Subscribe to this free journal for more curated articles on this topic

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 02 August 2024

Wireless ear EEG to monitor drowsiness

• Ryan Kaveh   ORCID: orcid.org/0000-0003-4146-7259 1   na1 ,
• Carolyn Schwendeman 1   na1 ,
• Leslie Pu 1 ,
• Ana C. Arias 1 &
• Rikky Muller   ORCID: orcid.org/0000-0003-3791-1847 1  

Nature Communications volume  15 , Article number:  6520 ( 2024 ) Cite this article

1259 Accesses

94 Altmetric

Metrics details

• Biomedical engineering
• Electrical and electronic engineering
• Predictive markers

Neural wearables can enable life-saving drowsiness and health monitoring for pilots and drivers. While existing in-cabin sensors may provide alerts, wearables can enable monitoring across more environments. Current neural wearables are promising but most require wet-electrodes and bulky electronics. This work showcases in-ear, dry-electrode earpieces used to monitor drowsiness with compact hardware. The employed system integrates additive-manufacturing for dry, user-generic earpieces, existing wireless electronics, and offline classification algorithms. Thirty-five hours of electrophysiological data were recorded across nine subjects performing drowsiness-inducing tasks. Three classifier models were trained with user-specific, leave-one-trial-out, and leave-one-user-out splits. The support-vector-machine classifier achieved an accuracy of 93.2% while evaluating users it has seen before and 93.3% when evaluating a never-before-seen user. These results demonstrate wireless, dry, user-generic earpieces used to classify drowsiness with comparable accuracies to existing state-of-the-art, wet electrode in-ear and scalp systems. Further, this work illustrates the feasibility of population-trained classification in future electrophysiological applications.

Similar content being viewed by others

Accurate whole-night sleep monitoring with dry-contact ear-EEG

Driver drowsiness estimation using EEG signals with a dynamical encoder–decoder modeling framework

Flower electrodes for comfortable dry electroencephalography

Introduction.

Drowsiness and fatigue while operating heavy machinery can be life-threatening. It is estimated that over 16.5% of fatal vehicle accidents in the United States include a drowsy driver resulting in over 8000 deaths and $109 billion in damages 1 , 2 , 3 . In addition to private and commercial (trucking) accidents, the National Safety Council has also cited drowsiness as the most critical hazard in construction and mining. While these deaths may be prevented with common risk assessments, fatigued individuals are often unable to recognize the full extent of their impairment before it is too late 4 . Drowsiness monitoring solutions use camera-based eye-tracking, steering trajectory sensors, or electrophysiological recording devices 5 , 6 , 7 . While they can be a good fit in automotive scenarios, eye tracking is obscured by sunglasses and other obstructions while steering sensors can be susceptible to false alarms on rough roads. User-centered recording modalities such as body-worn cameras, photoplethysmography (PPG), electrodermal activity, electrocardiography (ECG), electrooculography (EOG), and electroencephalography (EEG) are becoming increasingly popular because they are highly portable and adaptable to professional work environments 8 , 9 , 10 , 11 . These modalities have been incorporated into multiple form-factors such as eye-tracking glasses 12 , PPG/ExG tracking helmets 7 , and in-ear ExG sensors 13 , 14 . Of these methods, ExG generally achieves the highest drowsiness detection accuracies 15 .

Surface EEG is a safe, non-invasive method of monitoring the brain’s electrical activity from the scalp. Clinically, the most prevalent use of EEG is the monitoring and diagnosis of stereotyped neurological disorders related to sleep and epilepsy. These clinical systems generally use large, scalp-based, gold (Au) and silver/silver chloride (Ag/AgCl) electrode arrays 16 , 17 , 18 . Au forms a capacitive interface due to its inert nature, while Ag/AgCl forms a faradaic interface between Ag and skin. The AgCl is a slightly soluble salt that quickly saturates the skin and forms a stable electrode-skin interface. To maintain a low-impedance electrode-skin interface, contact is improved with skin preparation from an overseeing technician. While suitable for occasional, short-term monitoring, existing wet electrode arrays tend to be large and delicate for everyday use. Additionally, prolonged use of devices that require skin abrasion can result in skin irritation and lesions, further limiting their long-term use 19 , 20 . To promote use outside the lab and simplify clinical measurements, recent wearable EEG monitoring systems have focused on using smaller form-factor wet electrode arrays (e.g., cEEGgrid) 21 and dry electrodes that eliminate the use of hydrogels, integrating electronics and electrodes into a headset form factor, and software packages that allow for use in more everyday applications. The improved wet electrode systems (e.g., the cEEG grid) can provide unobtrusive EEG monitoring for 7+ h, but still requires hydrogel application (limiting day-to-day use). Dry electrode systems for research (e.g., CGX systems and Emotiv), commercial (e.g., Muse headband and Neurosity), and hobbyist (e.g., OpenBCI and Brainbit) have similarly demonstrated impressive EEG recordings of spontaneous and evoked neural signals and enabled disease monitoring, brain-computer interfaces (BCIs) and meditation guidance. As these commercial systems’ popularity increases, more and more wireless EEG systems are being developed and deployed across different environments 22 , 23 , 24 , 25 . The least cumbersome systems employ dry electrodes that minimize set-up time but generally still require skin cleaning and electrode surface treatments. Furthermore, the associated software packages require training to use 23 , 24 . Lastly, headset electronics are better suited for research and clinical environments as opposed to public, everyday use.

Discreet, multi-channel EEG recordings from inside the ear canal have been demonstrated 26 , 27 , 28 with recent advancements focusing on earpiece design, electrode materials, and multi-sensor arrays. The ear canal is an ideal sensor location due to its inherent mechanical stability and wealth of potential recording modalities. In-ear sensors and electrodes are well situated to record temporal lobe activity, blood oxygen saturation, head movement, and masseter muscle activity making it ideal for multi-modal sensing if high spatial coverage is not required 29 , 30 . While some applications may treat muscle activity or ear canal deformation as interference signals, these signals can be useful for other general ExG workloads. It is also important to note that in and around-the-ear EEG is inherently limited in gathering spatially encoded brain-activity relative to broader scalp arrays 27 , 28 , 29 , 30 , 31 . Many successful designs have leveraged hydrogel coated on flex-pcb arrays or user-customized earpieces to record ExG features such as EOG, low-frequency EEG (1–30 Hz), and evoked potentials (40–80 Hz) 26 , 27 , 28 , 32 , 33 . These wet-electrode based, custom earpiece systems established the feasibility of in-ear monitoring for attention monitoring, seizure monitoring, whole night sleep monitoring, and sleep stage classification 34 , 35 , 36 , 37 . Due to their user customized approach, earpieces require a case-by-case integration schemes to minimize earpiece volume resulting in variable electrode positioning. The required skin-preparation and hydrogel also can lead the conductive bridging between electrodes, limit-user-comfort, and reduced electrode lifetime 38 . The next step to more scalable deployment of in-ear ExG recordings would be the utilization of one-size-fits-most (user-generic) earpiece designs, dry electrodes, wireless electronics, and electrode materials that do not require maintenance.

Recent user-generic earpieces equipped with wet electrodes, dry electrodes 39 , 40 , 41 , 42 , PPG, and/or chemical sensors have achieved high degrees of accuracy for brain-state and activity classification 39 , 40 , 43 , 44 , 45 , 46 . Additionally, dry-electrode based in-ear ExG have recorded low frequency neural rhythms, evoked potentials, and EOG comparable to wet-electrode. While potentially more susceptible to noise due to higher electrode-skin impedance (ESI) interfaces 47 , dry electrodes eliminate the use of hydrogel, simplify the earpiece application process, and can improve user comfort. To achieve a middle ground between comfort and low ESI, state-of-the-art dry electrodes employ a wide range of solutions ranging from exotic materials, conductive composites, capacitive interfaces, solid-gels, and high-surface area 3D electrodes (microneedles, fingers, and nanowires) 20 , 40 , 41 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 . PEDOT:PSS and IrO 3 are commonly used in the small-scale production of rigid electrodes due to their superior conductivity and faradaic interfaces 57 , 58 , 59 . Both materials promote charge transfer by leveraging doped surfaces and high effective surface areas. Conductive, flexible composites, such as silvered-glass silicone and carbon-infused silicone, are not as conductive as PEDOT:PSS and IrO 3 but offer significantly greater comfort. Conductive composites are made from polymers or elastomers that can be molded into arbitrary shapes for anatomically fit electrodes and use added conductive particles to achieve a desirable ESI. The more conductive particles that are added will ultimately limit polymer cross-linking and may lead to cracking over time 60 . The clinical and industry standard materials are silver/silver chloride (Ag/AgCl) and gold due to their cost, biocompatibility, and electrical properties. Ag/AgCl can be painted on 3D electrodes to form consistent, faradaic, low-impedance interface through hair and grime. Furthermore, Ag/AgCl is also popular for consumable electrodes since the conductive particles deplete over time 61 . Gold electrodes are more inert, can be repeatedly reused, and form a capacitive interface that is not reliant on added conductive ions. While potentially more susceptible to motion artifacts and interference, gold’s lifetime and chemical properties make it ideal for long-lasting ExG recording systems. Most commercial wearables and existing in-ear ExG systems use Ag/AgCl, Au, or conductive composite electrodes 24 , 62 , 63 , 64 .

Electrodes are just one piece of signal acquisition. Neural recording hardware is required to digitize neural signals and transmit them to a processing unit/base-station for offline processing. Neural recording hardware for more consumer-facing products tend to be tailor-made with low bandwidth, noise, and power specifications 65 , 66 , 67 . These devices usually have bandwidths around 100 Hz and can achieve ultra-lower power operation (<100 μW 67 ). Research focused devices, however, utilizing high resolution and bandwidth hardware enables greater investigation outside the original project description. Such versatile systems generally support higher channel counts (16–64+), commercial wireless protocols (bluetooth or Wi-Fi), higher sampling rates (500–1000 Hz), and can take advantage of different signal modalities (e.g., EMG) at the cost of higher power (>50 mW) 42 , 46 , 68 . Low-noise and high-resolution systems allows for greater flexibility, repeated interpretable signal processing (frequency analysis, time-domain averaging, etc.) and algorithm development to illuminate different feature classes, mitigate interference, and discover new potential applications. Such systems have been used to build brain-machine interfaces with P300 responses and steady-state evoked potentials 27 , 29 , 34 , 69 , 70 . When adapting existing electronics for use with wearable dry electrodes, increased ESI, system noise, and interference susceptibility bear important considerations for power requirements and any downstream machine learning algorithm 71 , 72 . Employing versatile, higher power electronics with more interpretable, light weight classical algorithms (e.g., logistic regression, support vector machines, random forest) is an important first step for future sensor and power optimizations. To this effect, this work uses an existing, high channel count, high bandwidth system to enable studying the relationship between the employed ExG electrode technology and drowsiness detection.

In addition to system optimisation, the choice of machine learning algorithm determines system functionality from the perspective of training, data, and processing requirements. Every-day ExG systems would ideally work out of the box, improve over time, and continue to provide feedback when wireless connectivity is poor and there is unreliable access to large processing power (construction sites, planes, and trucks). Classical algorithms such as logistic regression, SVMs, and random forest have demonstrated impressive success in classifying neural signals with limited datasets 15 , 25 , 73 , 74 . Neural network-based algorithms have also achieved impressive results 75 , 76 , 77 , and are good candidates for further research. Neural network-based algorithms, on average, require more training data than SVMs, logistic regression, and random forest, making them difficult to work with on smaller data sets. Furthermore, interpretable algorithms such as logistic regression and SVMs enable greater visibility into which types of features have sufficient SNR for classification and could potentially be applied to different applications. Lastly, algorithms such as SVMs, logistic regression, and random forest generally require less processing power than similarly performing neural net or perceptron-based architectures, making them ideal for low-power, edge-based deployments on existing microcontrollers. Additionally, while existing in-ear ExG BCIs have achieved high classification accuracies with user-specific training and validation 35 , 43 , 76 , 78 , 79 , ideal in-ear ExG wearables would leverage pre-trained algorithms so never-before-seen users can use these devices without time-consuming training. This user-generic classification has been explored in scalp-based drowsiness monitoring with great success but not yet with in-ear ExG 15 .

This project is the first integration and demonstration of wireless, dry-electrode in-ear ExG sensors used for drowsiness classification. To this effect, a novel in-ear EEG sensor manufacturing method coupled to a pre-existing wireless data acquisition platform is presented and verified with open-source machine learning classification on 9-subjects. A fabrication process for dry, gold-plated electrodes suitable for repeated, comfortable, low-impedance earpieces is introduced and tested over the course of months of electrode use. This electrode technology provides a unique method for the rapid prototyping of reusable, Au electrodes that remain stable over 12 months of use. These electrodes can replace existing solutions that rely on shorter-lifespan Ag/AgCl electrodes or expensive materials such as platinum or IrO3. The earpieces are then coupled with wireless, discreet electronics capable of taking uninterrupted, low-noise neural measurements for over 40 h 46 to form a wearable, in-ear ExG system. The resulting Ear ExG BCI is then demonstrated with a nine-subject drowsiness monitoring study. Low-complexity temporal and spectral features are extracted from the recorded ExG data and used to train multiple, offline machine learning models for automated drowsiness detection. The best-performing model utilizing a support vector machine achieved an average drowsy-event detection accuracy of 93.2% when evaluating on users it has seen before and 93.3% when evaluating never-before-seen users. This system and its use of offline classifiers lay the groundwork for future, discreet, fully wireless, long term, longitudinal brain monitoring (Fig.  1 ).

Envisioned systems could be discreetly worn throughout the day to comfortably record neural signals from inside the ear canal, perform drowsiness detection, and provide feedback.

Results: ear ExG drowsiness monitoring platform

Modular electrode design, fabrication, and assembly, earpiece design.

Easy-to-use neural wearables require a user-generic earpiece and electrode scheme designed for recording across multiple demographics and for comfortable, long-term wear. To achieve these requirements, electrode and earpiece designs were derived from refs. 46 , 80 and resulted in a small, medium, and large size of a single design with modular electrodes. Electrodes are positioned near the ear canal such that they do not pass the isthmus of the ear canal, which tends to develop a corkscrew shape as individuals age. This earpiece is designed to account for these age-related changes. Previous studies 30 , 41 , have highlighted high value electrode locations that minimize channel-to-channel correlation while maximizing mechanical stability. To also maximize electrode surface area across different individuals, small, medium, and large sized earpieces were designed with slightly differing electrode sizes. The final “medium-sized” earpiece is comprised of four 60 mm 2 electrodes inside the ear canal and two 3 cm 2 electrodes on the ear’s concha cymba and concha cavity (Fig.  2a ). The in-ear electrodes are cantilevers that apply gentle outward pressure to achieve lower ESI over previous iterations (370 kΩ to 120 kΩ at 50 Hz 46 ) and improve mechanical stability. The out-ear electrodes act as fiducial guideposts to ensure the electrodes contact the same surface with each wear. Furthermore, electrodes outside the ear are good reference and ground candidates due to their increased distance from the brain or any muscle. To improve the earpiece assembly and further increase comfort over 46 , a soft earpiece body with a manifold in-ear design was 3D printed with a clear methacrylate photopolymer (Fig.  2a ). Each rigid electrode is attached to this soft, elastic substrate and moves independently from the other electrodes to fit in a subject’s ear (Fig.  2b ). This new, modular assembly properly demonstrates the capabilities of the manifold earpiece fabrication process.

a The final earpieces are composed of four in-ear electrodes and two out-ear electrodes. Manifold 3D-printed earpieces are assembled by plugging rigid, gold-plated earpieces into a soft, flexible skeleton. b The out-ear electrodes press against the ear’s concha cymba and concha bowl, while the in-ear electrodes contact the ear canal’s aperture. In-ear electrodes only enter the first 10 mm of the ear canal. c Diagram and photographs of electrode fabrication: i) Electrodes are 3D printed or molded. ii) The bare electrodes are sandblasted and cleaned. iii) The electrodes are electroless copper plated via exposure to surfactant, catalyst, and copper sulfate solutions in sequence. iv) A nickel layer is electroless plated. v) A final gold layer is electroless deposited.

Electrode fabrication

A low-cost, fully electroless plating process was developed to enable rapid prototyping of arbitrary shaped electrophysiological sensors. Electrodes were 3D printed with a clear methacrylate polymer (Fig.  2c ) and sandblasted to increase surface roughness. Samples were then submersed in different catalyst baths to develop copper, nickel, and gold metal layers. Lastly, tinned copper wires are soldered directly to the electrode surface for integration with the neural recording front end. This plating process is expanded on 45 , 81 , with the addition of a nickel layer that limits grain-boundary diffusion of copper and significantly extends electrode lifetime 81 , 82 , 83 . Furthermore, the nickel-plating step removes the need for repeated electroless palladium plating and the overall number of fabrication steps. While other in-ear electrodes use expensive materials like IrO 3 or hydrogels 39 , 40 , this improved layer stack-up (Cu, Ni, Au) is reminiscent of printed-circuit-board fabrication and enables similar levels of scale for electrode prototyping. The final surface contains at least 0.5 µm of copper, 0.5 µm of nickel, and 0.25 µm of gold and is suitable for dry electrode recording.

Plating process characterization

Material acid dip tests and tape tests.

The final electrode surfaces were physically and chemically robust. Kapton tape was applied around the entire electrode surface and then removed. No visible gold, nickel, or copper was removed with the tape indicating strong adhesion to the methacrylate substrate 81 , 84 . Electrode samples were also dipped in nitric acid baths to test the porosity and continuity of the gold surface. While concentrated and dilute nitric acid will readily dissolve copper and nickel, respectively, neither will etch gold. No noticeable differences were observed after dipping gold-plated electrodes into a 1M nitric acid bath. Control samples of copper and nickel, however, were quickly etched down to the bare methacrylate surface. The acid dip tests and subsequent microscope inspections (Fig.  3a ) found no micro or nano cracks that may affect the electrode’s surface or electrical properties.

a Representative light microscopy images of plated surfaces showcasing the roughness resulting from sandblasting. b Stylus Profilometer measurements of a flat sample after each plating step. c Absolute sheet resistance measurements, mean (red circle), and standard deviation (error bars) immediately after plating. d In-ear electrode-skin impedance magnitude, phase, and magnitude fit. Standard deviation of electrode magnitude shown in shaded green region. e Constant phase element electrode model used for fitting.

Surface roughness characterization

Light microscopy photographs and stylus profilometry measurements were used to assess surface roughness between each step of the plating process on a single flat sample. Figure  3b plots the normalized surface topography of the sample during each plating step. The reported Rp values are the standard deviation of the plotted lines. Though surface roughness decreases slightly with each subsequent plating step, the final gold surface is still much rougher than a simple, planar surface. This increases electrode surface area, promotes better film adhesion, and reduces ESI 50 , 81 , 84 , 85 .

Sheet resistance

Sheet resistance was characterized by a 4-point probe immediately after plating. 40 sheet resistance measurements were taken of each copper-, nickel-, and gold-plated samples. As prepared, copper-plated samples, nickel-plated samples, and gold-plated samples exhibited an average sheet resistance of 177.9 ± 109, 95.5 ± 13, and 30.3 ± 3.7 mΩ □ −1 , respectively (Fig.  3c ). With each subsequent metal layer, the sheet resistance stabilized, and the surfaces became more conductive.

Bioimpedance of In-ear electrodes across multiple users

Impedance spectroscopy was used to assess in-ear electrode-skin impedance. Four subjects took impedance measurements (20 total measurements) between the in-ear electrodes and the out-ear cymba electrode. To account for future, real-life conditions with cerumen and oil, no skin preparation was performed before each trial, and measurements were repeated until all four electrodes in the ear canal were measured. Since the ESI measurements include two dry electrodes, the plotted values were divided by two to demonstrate the average ESI of a single dry electrode. All measurements were performed with an LCR meter (E4980 A, Keysight) powered by a wall outlet and arranged as a two-point probe where a single electrode is considered a single probe. The LCR meter was configured with a current limit of 0.5 mA to prevent sensation or injury. While the LCR meter is designed to achieve high accuracy (within 3%) even in the presence of powerline interference, electrode cables were shielded by ground wires to further minimize interference. All impedance results were fitted to an equivalent circuit model (spectra shown in Fig.  3d , circuit model shown in Fig.  3e ) to better understand motion artifact settling times associated with the phase elements of the electrode skin interface and provide reference for future analog front-end designs. At 50 Hz, the interface has an average impedance of 120 kΩ and phase of −33°.

Lightweight ExG recording system

ExG was recorded using an existing compact, wireless recording platform affixed to a headband (Fig.  4a ). The platform, known as WANDmini, is a wireless neural recording frontend built for and already deployed in previous in-ear EEG studies 46 . It is adapted from a system originally designed for electrocorticography and comprises a custom neural recording circuit 68 , 86 , (NMIC 86 , Cortera Neurotechnologies, Inc.), a microcontroller, and a Bluetooth radio for wireless transmission. The NMIC digitizes up to 64, fully differential channels of electrophysiological activity with a sampling rate of 1 kSps. WANDmini arranges the NMIC’s channels in a monopolar montage with a single reference electrode. This arrangement is it suitable for EEG, EOG, and EMG recording and provides enough sampling and channel count headroom to remove any recording electronics related bottlenecks. An onboard microcontroller and radio packetizes and streams digitized neural data to a base station connected to a host machine over Bluetooth Low Energy (BLE) (Fig.  4a ). System power is dominated by the microcontroller and Bluetooth transmission (98.3%) thus making unused channels immaterial from a power perspective. With the NMIC and WANDmini power consumptions, 700 μW and 46 mW, respectively, a 3.7 V 550 mA battery can provide ~44 h of runtime. In summary, the NMIC’s significantly lower power than common commercial neural frontends (e.g., ADS1298/1299), high channel count, and sufficiently low noise floor makes it ideal for use in modular in-ear EEG prototypes. NMIC and WANDmini specifications are listed in Table  1 and further detailed in Supplement section  II.h . The host machine uses a custom graphical user interface (GUI) that plots and saves all incoming data and cues for the trail overseer. This custom GUI is unique to this work and provides the test subject with a reaction time game, auditory cues, and visual alerts during experiments. More information about the GUI is available in section 2h of the supplement.

a Subjects sit beside a laptop displaying a basic reaction time measuring game. A head-worn WANDmini, secured in a 3D-printed enclosure, records and transmits ExG from contralaterally worn earpieces to a base station via BLE while the subject plays the game. All captured ExG can be live plotted for the trial overseer while the game records subject’s reaction times and Likert survey responses. b Recorded ExG, reaction times, and Likert items are used to generate features and labels for a brain-state classifier. Drowsy events, shaded in green, are determined when a subject’s reaction time and Likert response cross a drowsiness threshold that is determined per subject. Using both the reaction time and Likert scores enables robust label creation that is agnostic to temporary user error.

EEG characterization and user-generic drowsiness detection

Drowsiness study.

To characterize the full system performance, 35 h of Ear ExG data was recorded during a nine subject drowsiness study. Subjects wore two earpieces with the electrodes organized in a contralateral monopolar montage. Previous works have demonstrated that electrodes on a single earpiece are sufficiently distant from each other to measure ExG 37 , 41 , but greater signal amplitude can be recorded with electrodes placed across both ears 39 , 45 . To induce drowsiness, subjects played a repetitive reaction time game. Every 60 s, a user was prompted to press a random number between 0 and 9 and their reaction time was recorded (Fig.  4a ). Every 5 min, the user was prompted to enter a Likert item according to the Karolinska Sleepiness Scale (KSS). This scale is frequently used to evaluate subjective sleepiness and ranges from 0 = “extremely alert”, to 10 = “extremely sleepy, fighting to stay awake” 87 . Queue intervals (60 s and 5 min) were selected based on initial experimentation and previous works that demonstrated a balance between minimizing disturbances and frequent datapoints 45 , 88 . All recorded ExG, cue timing, reaction times, and Likert items are saved by a custom GUI for post-processing and machine learning model training (Fig.  4b ). Immediately after each trial, reaction time and Likert items were thresholded per subject to automatically generate alert/drowsy labels for each trial since behavior and response time metrics are heavily correlated with drowsiness 6 , 87 , 88 . By taking both an objective and a subjective drowsiness measurement, high-confidence data labels could be generated in face of user-error and user-bias (memory of previous KSS scores affecting subsequent scores). Both objective and subjective measures must agree to classify an event as drowsy. Furthermore, as noted in previous works, reaction times and likert scores are variable on a subject-to-subject basis. As a result, each trial was thresholded on a per subject basis. Each trial contained at least one drowsy event, and 65 drowsiness events were recorded across 34 trials.

Drowsiness classification pipeline

The training pipeline for ExG data consisted of post-processing, feature extraction, and model training steps (Fig.  5a ). ExG recordings were referenced to maximize spatial covering, band pass filtered, and segmented into 50 s or 10 s windows. If a window of data exhibited an artifact greater than 10 mV (from motion) it would be discarded. This was happened very infrequently as most artifacts were less than 1 mV above the baseline rms voltage. Temporal and spectral features relevant for ExG-based drowsiness detection were implemented to target ocular artifacts and activity in standard EEG frequency bands relevant to drowsiness detection: delta ( δ , 0.05–4 Hz), theta ( θ , 4–8 Hz), alpha ( α , 8–13 Hz), beta ( β , 13–30 Hz), and gamma ( γ , 30–50 Hz). Binary (alert/drowsy) classification was performed with low-complexity logistic regression, support vector machine (SVM), and random forest classifier models.

a Ear ExG experimental recordings are re-referenced, filtered, cleaned of motion-contaminated epochs, and then undergo feature extraction and model training. b Cross-validation is performed similarly, featurized ear ExG epochs are fed to all three classification models. Model outputs are then fed to an event detector that performs a moving average and then thresholds the resulting classifications to estimate alert and drowsy states.

Three cross-validation techniques were used to estimate model performance across varying usage scenarios: user-specific, leave-one-trial-out, and leave-one-user-out. User-specific cross-validation trained models on n  − 1 trials for the subject, tested on their remaining trial, and averaged the results after n independent iterations to determine drowsiness detection accuracy for a single subject. Leave-one-trial-out cross-validation trained models on 33 of the recorded trails, tested on the remaining trial, and averaged results after all 34 independent iterations to determine the study’s overall drowsiness detection accuracy. Leave-one-user-out cross-validation trained on recordings from eight subjects, tested on the remaining subject’s recordings, and averaged results after all nine independent iterations. This evaluated detection accuracy when using population training and deploying on a never-before-seen subject. Due to the inherent imbalance between drowsy and alert classes, each classification model employed a balancing scheme where over-represented classes are given a smaller class weight than under-represented classes. In the case of drowsy vs. alert, alert epochs are given a class weight inversely proportional to the number of epochs. This allows classes to be treated more fairly across all training/cross-validation regimes (since they will all have different class balances). During validation, class probabilities returned from the classifier models were filtered with a 3-tap Hamming window FIR filter and thresholded to achieve final binary outputs (Fig.  5b ).

Drowsiness classification results

Alpha modulation ratio.

Alpha waves (8–12 Hz) are a spontaneous neural signal that can reflect a person’s state of relaxation, which makes them an important spectral feature in ExG-based drowsiness classification 15 . A sample recording from a single user demonstrating alpha wave modulation is presented in Fig.  6a . This modulation is clear in the time–frequency spectrogram (Fig.  6a ). To assess the modulation ratio more quantitatively, Fig.  6a also plots the average power across the entire alpha band while the subject opens and closes their eyes every 30 s. The presented sample data’s modulation ratio was 2.001.

a Spectrogram demonstrating alpha modulation when the subject closes their eyes. Alpha bandpower (8–12 Hz put through a 2 s rolling average filter for clarity) is modulated by 4× in amplitude when eyes are closed. b Logistic regression event detection with 10 s feature windows. c Support vector machine event detection with 10 s feature windows. d Random Forest event detection with 10 s feature windows. ( e , f , g ) Drowsiness event detection using 50 s feature windows. Standard Deviation (Std Dev) shown across results from all nine users.

Classifier comparison across validation schemes

The overall average of the user-specific classification results ranged from 77.9% to 92.2% across all models and feature window sizes. In the user-generic leave-one-trial-out case, average classification accuracy was higher and ranged from 91.4% to 93.2% when cross-validating across the 34 trials. This is most likely due to the increased amount of data available for training. Lastly, the leave-one-user-out validation scheme achieved average classification accuracies from 88.1% − 93.3% across all users, window sizes, and models. Figure  6b–g showcases average model accuracy and standard deviation where appropriate.

10 s vs 50 s windows

Two feature windowing schemes were investigated, 10 s (Figs.  6b–d ) and 50 s (Fig.  6e–g ) windows. All training steps, including feature selection, are performed independently. The 10 s feature windows result in significant performance loss in the user-specific validation scheme. For example, the average user-specific logistic regression-based classifier performance increased from 77.9% to 90.8% when increasing feature window sizes to 50 s. Minimal accuracy loss, however, was observed when using leave-one-trial-out and leave-one-user-out validation schemes with features from 10 s windows. This minimal accuracy loss is most likely due to the increased amount of training data available (~30 trials) to the models relative to the user-specific cases where individual models only train on a 1−4 of trials.

Classifier architecture comparison

Three low-complexity machine learning models were used to promote the scalability and usability of the drowsiness detection platform. All models were implemented in Python 3.8 using scikit-learn packages. Logistic regression models were implemented with a stochastic average gradient descent solver. L1 regularization was used to add a penalty equal to the absolute value of the magnitude of the feature coefficients. Support vector machines were implemented with a radial basis function (RBF) kernel to account for data that may not be lineally separable. The trained models utilized a maximum of 400 support vectors and a regularization parameter, C  = 1. Random forest models were implemented with 100 trees and a maximum depth of five to prevent overfitting. These implementations resulted in memory footprints that were estimated using python’s pympler package. The logistic regression, SVM, and RF models required 2.8 kB, 144.2 kB, and 63.8 kB respectively. These memory requirements are well within the capacity of modern microcontroller’s embedded memories (e.g., 32-bit ARM Cortex-M).

Since all three models achieve high accuracy, it is clear that drowsiness is classifiable with in-ear eeg recording. No model shows markedly greater performance or another. The logistic regression model is more computationally efficient, requires significantly less memory, and can be more easily trained/deployed with smaller datasets. It is important to verify that logistic regression continues to perform as well across larger demographics, a topic for future studies.

We have reported the design and fabrication of in-ear dry electrodes along with the assembly and evaluation of a wireless, wearable, in-ear ExG platform for offline drowsiness detection on never-before-seen users. All aspects of this platform can be adapted to different use-cases. The 3D printed and electroless Au-plated electrodes can be rapidly augmented for any anatomically optimized wearable and used/re-used for long periods of time, WANDmini can support multi-day electrophysiological monitoring, and the presented offline classifiers demonstrate the potential for future dry-electrode based brain-state classification. In contrast to other state-of-the-art in-ear recording platforms (Table  2 ), the electrodes, wireless electronics, and lightweight algorithms presented lay the groundwork for future large-scale deployment of user-generic, wireless ear ExG brain-computer interfaces that use multiple machine learning algorithms.

Our results are promising for the development of the next generation of standalone wearables that can monitor brain and muscle activity in work environments and in everyday, public scenarios. To realize these standalone, wireless systems, future work requires integrating these classifiers on-chip for real-time brain-state classification and miniaturizing all the hardware into a pair of earbuds. Furthermore, the hardware would need to support online classification to allow for full-day, itinerant use. Lastly, it would be important to take this miniaturized hardware and implement a user-study with a wider demographic. By monitoring in-ear EEG across individuals aged 18–65+, further age specific models can be investigated. If a monolithic model is unable to classify drowsiness stereotypes across such a large age range, it would be interesting to provide models with context such as age, gender, known sleep disorders, and previous night’s sleep quality. Furthermore, the feature selection performed in this work suggests that simpler calculations such as bandpower ratios are sufficient for drowsiness classification. If this remains the case across larger demographics, then feature extractors can ignore computationally expensive features such as standard deviation, different entropy measures, etc. to reduce power in embedded classification scenarios. With aforementioned integration, a pair of ear ExG buds would significantly enable long term, daily recording ExG without interrupting a user’s day or stigma. These measurements would enable an entirely new era of research for tracking long-term cognitive changes from disorders such as depression, Alzheimer’s, narcolepsy, or stress.

Study approval and ethical consent

The user study, subject recruitment, and all data analysis was approved by UC Berkeley’s Institutional Review Board (CPHS protocol ID: 2018-09-11395). Informed consent was received by all participants in the study for their results to be included in presented figures/data.

Both the electrodes and earpiece were printed with a stereolithography (SLA) 3D printer (Formlabs Form 3 printer) with a standard, clear methacrylate photopolymer (Fig.  2c ). An SLA printer was used due to its increased precision over standard filament deposition modeling (FDM) based printers. In SLA printers, thin layers of photosensitive polymer are cured by a laser. The resulting printed surfaces must be washed and cured in UV to achieve the final 3D part.

The original 3D printed surface is highly anisotropic due to the structure’s uniformly printed layers. To create a more heterogenous surface, electrode structures were sandblasted with 100 grit white fused aluminum oxide blasting media (Industrial Supply, Twin Falls, ID) to remove the regular surface pattern leftover from the printing process while also increasing the effective surface area. The sandblasted samples were then sonicated in a bath of Alconox cleaning solution for ~10 min and rinsed with DI water. Lastly, the electrode structures were treated in a bath of 1% benzalkonium chloride (Sigma Aldrich 12060-100G) surfactant solution for 10 min. These surface treatment steps ensure a clean plating surface with high surface energy and lead to improved catalyst/metal layer adhesion.

The samples are then submersed in catalyst and plating baths. First, the electrodes are submerged in a beaker of palladium-tin catalyst for 10 min followed by a copper plating solution for a minimum of six hours. This initial plating step results in a thick copper layer that will oxidize if left out in ambient atmosphere. As a result, samples would then be quickly rinsed, dried, and placed in a nickel-plating bath for ~10 min (Sigma Aldrich 901630). Afterwards, the electrodes are placed in an electroless gold plating solution for approximately 15 min. In between plating steps, the samples were rinsed with DI water and dried thoroughly.

WANDmini: ExG recording hardware

The WANDmini board contains a neural recording frontend (NMIC), a SoC FPGA with a 166 MHz Advanced RISC Machine (ARM) Cortex-M3 processor (SmartFusion2 M2S060T from Microsemi), and low-energy radio (nRF51822 from Nordic Semiconductor). The SoC FPGA forms a custom-designed 2Mb/s digital signal and clock interface with a single NMIC, aggregates all data and commands into packets, then streams all the packets to the 2Mb/s 2.4 GHz low-energy radio.

WANDmini also contains a 20 MHz crystal oscillator as a clock source, on-board buck converters (TPS6226x from Texas Instruments), a battery charger circuit (LTC4065 from Linear Technology), and a 6-axis accelerometer and gyroscope (MPU-6050 from InvenSense). While WANDmini can record up to 64 channels of electrophysiological data and motion information from the accelerometer, the drowsiness detection application only uses 11 channels for ExG monitoring. Future applications may integrate real-time motion artifact cancellation and classification directly into the WANDmini’s SoC FPGA.

Subject selection and earpiece application

Nine subjects (7 male, 2 female, ages 18–27) volunteered for this study. Subjects were requested not to exercise or drink caffeine before any trial. Prior to the first experiment, subjects tried out small, medium, and large earpieces and selected the pair they felt were most comfortable and secure in ear. During this onboarding session, subjects also familiarized themselves with the GUI.

At the start of the drowsiness trials, subjects were given their preferred ear EEG earbuds to wear, as well as an electronics headband with a fully charged Li-Po battery and the WANDmini recording hardware. To maintain a realistic daily use scenario, the subjects did not clean or prepare their skin and no hydrogel or saline was applied to the earpiece dry electrodes. The trial hosts also did not help subjects don/doff the headband or earpieces unless explicitly requested. After the experiments, the earpieces were cleaned with 70% isopropyl alcohol since they would be later used by other subjects.

Electrophysiological recording setup

Each earpiece has six electrodes, four inside the ear canal and two outside the ear canal. The default recording arrangement employs two contralaterally worn earpieces to maximize spatial coverage and recorded signal power 27 , 39 . These two earpieces provide up to 11 ExG channels with a common reference. Either of the concha cymba electrodes can be used as a reference (the un-used one can be used as an additional sense electrode). After initial experimentation, it was determined that the right concha cymba electrode was sufficient as a reference electrode across all subjects. As a result, each ExG channel is referenced against the right concha cymba electrode in a monopolar montage (electrode Y in Fig.  2a ). A single wet Ag/AgCl electrode was applied to the subject’s right mastoid and connected to battery ground for interference reduction.

Drowsiness trial overview

Subjects participated in multiple drowsiness trials to enable both user-specific and user-generic training. Subjects were not familiar with the ear EEG work when selected. No more than five trials were recorded per subject to maintain a diverse data pool. Prior to the trials, subjects were informed of the study purpose and requested to have a ‘normal night’s rest’ (subjectively) and not drink caffeine prior to the trial. Trials took place in a quiet, indoor office space between 8 a.m. and 5 p.m. when the lights were on. After donning the ear eeg system, the subject was left alone in the trial space until the end of the recording session. During the trial, the subject would sit at a desk in front of a laptop with a custom GUI. Subjects were instructed to only perform the reaction game task and not look at personal devices for the extent of the trial. Subjects were allowed to move their heads, readjust in their seat, and move their arms, but were asked to stay seated during the entire session (to minimize motion artifacts). Each trial was 40–50 min in length and was self-ended by the subject to prevent the interruption of a drowsy event. At the end of the trial, the subjects removed the headband and earpieces themselves. They were instructed to wait at least 24 h before participating in subsequent drowsiness trials to maximize variation between trials.

Label generation

Recording both objective and subjective drowsiness measures made the label generation process robust to user-error momentary distractions (when an alert user looks away from the laptop). Ear ExG samples were labeled as “drowsy” if the user reported a drowsiness Likert item >5 and if their reaction time was more than double the average from the first 5 min of recording. The labels were then passed through a 3-sample rolling average filter and thresholded to achieve a binary label.

Re-referencing and filtering

ExG re-referencing was used to maximize spatial covering across contralateral earpieces. Each in-ear electrode was re-referenced to the left concha cymba electrode and processed with the 11 EEG channels recorded with the right concha cymba electrode. To remove power-line interference (60 Hz in North America) while maintaining as much EEG activity as possible, both the recorded and re-referenced EEG channels were bandpass filtered from 0.05–50 Hz. Filters were implemented with a 5th order butterworth high pass filter (corner of 0.05 Hz) and a 5th order Butterworth low pass filter (corner of 50 Hz). Both filters were implemented in python but can also be implemented with infinite impulse response (IIR) filters with 16 bit registers for use in FPGA/embedded applications.

Data segmentation

Filtered ExG was segmented to remove ExG artifacts related to decision-making and motor planning in response to GUI cues. Each epoch began 10 s after a reaction time cue and ended when the next reaction time cue was provided. When using the maximum window size, features were calculated for these 50 s epochs. When using a reduced window size, each 50 s epoch and its corresponding label were divided into five 10 s windows. To focus classification on drowsiness onset, epochs were considered “sleep” if a subject’s rection time exceeded 10 s. These epochs were excluded from the study.

Feature extraction and selection

Temporal and spectral features were extracted in Python 3.8 from the segmented ExG data. Low-complexity features were calculated for each window of ExG data and across all the recorded and re-referenced channels. Voltage standard deviation and maximum peak-to-peak voltage amplitude were calculated in the time-domain to target eye blink artifacts and motion. Welch’s method (using a 1000-point Fourier transform, 500 sample overlap, and Hamming window) was used to calculate the power spectral density (PSD) and attain frequency characteristics that relate attention and relaxation. The following spectral features were calculated prior to training: maximum PSD, peak frequency, and PSD variance were calculated for δ, θ, α, β, γ EEG bands. Absolute and relative band powers were also calculated for the following bands and ratios: δ, θ, α, β, γ, α/β, θ/β, (α + θ)/β, and (α + θ)/(α + β). Relative bandpower is the specific band relative to the total PSD from 0.5–50 Hz. Furthermore, features of the previous epoch were included to account for changes in ExG activity, since temporal and spectral features relate to characteristics that changes during the onset of drowsiness such as attention and eye movement. A complete table of features used in offline training (prior to feature selection) is in Table  3 .

All features were scaled by subtracting the median and scaling according to their interquartile range. To reduce input feature count, feature selection using an analysis of variance (scikit-learn Python 3.8) was performed to determine the top 20 features (total) that minimize redundancy and maximize class variation during training. Only these 20 features are included during model training and validation. This feature selection also implicitly selected best performing electrodes across users (most likely due to some electrodes fitting better than others). The same feature type was also selected for multiple channels (e.g., the top 20 features would include alpha band power from channels 1, 5, and 10). Contralateral channels (where sense and reference electrodes are in different ears) were always weighted higher than ipsilateral channels. The most used features (in order of importance) are shown in Table  4 .

Spectral features associated with eye movement, relaxation, and drowsiness were the most important for model training. Furthermore, the previous epoch’s features were also generally important. This is corroborated by results from other works on scalp data in refs. 14 , 15 , 79 . All feature extraction was performed in Python using numpy. For implementation into an embedded/FPGA environment, these features can be calculating using a coarse fast-Fourier transform, look-up-tables, and the CORDIC algorithm.

Statistics and reproducibility

No statistical method was used to predetermine sample size. No data were excluded from the analyses. The experiments were not randomized. The investigators were not blinded to allocation during experiments and outcome assessment.

Reporting summary

Further information on research design is available in the  Nature Portfolio Reporting Summary linked to this article.

Data availability

The Experimental Ear EEG data collected in this study is available at https://github.com/MullerGroup/EarEEG_Drowsiness . Due to IRB restrictions, access may be restricted to any raw EEG data. If there are any issues accessing the repository, please contact [email protected], [email protected] or [email protected]. Example code and a deployable notebook can be found in the GitHub repository. Source data used in all figures are provided with this paper.  Source data are provided with this paper.

Code availability

The source code used for offline model validation and analysis of results is available at https://github.com/MullerGroup/EarEEG_Drowsiness .

Landrigan, C. P. Driving Drowsy Commentary [Online]. Available: www.vtti.vt.edu/PDF/100-Car_Fact-Sheet.pdf (2008).

National Highway Traffic Safety Administration and U. Department of Transportation. “Crash Stats: Drowsy Driving 2015”. [Online]. Available: https://crashstats.nhtsa.dot.gov/Api/Public/ViewPublication/812446 (2017).

Tefft, B. C. Prevalence of motor vehicle crashes involving drowsy drivers, United States, 1999-2008. Accid. Anal. Preval. 45 , 180–186 (2012).

Article   Google Scholar  

National Safety Council, Wearables for Fatigue Monitoring, [Online]. Available: https://www.nsc.org/workplace/safety-topics/work-to-zero/safety-technologies/fatigue-monitoring-and-wearables (2020).

Arakawa, T. Trends and future prospects of the drowsiness detection and estimation technology. Sensors 21 , 7921 (2021).

Article   ADS   PubMed   PubMed Central   Google Scholar  

Stancin, I., Cifrek, M. & Jovic, A. A review of eeg signal features and their application in driver drowsiness detection systems. Sensors 21 , 3786 (2021).

Adão Martins, N. R., Annaheim, S., Spengler, C. M. & Rossi, R. M. Fatigue monitoring through wearables: a state-of-the-art review. Front. Physiol. 12 , 790292 (2021).

Malathi D., Dorathi Jayaseeli J. D., Madhuri S. & Senthilkumar K. Electrodermal activity-based wearable device for drowsy drivers. J. Phys. Conf. Ser. 1000 , 012048 (2018).

Schneider, E. N. et al. Electrodermal responses to driving maneuvers in a motion sickness inducing real-world driving scenario. IEEE Trans. Hum. Mach. Syst. 52 , 994–1003 (2022).

Reddy B., Kim Y., Yun S., Seo C., & Jang J. Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops 121–128 (2017).

Nguyen T., Ahn S., Jang H., Jun, S. C., & Kim J. G. Utilization of a combined EEG/NIRS system to predict driver drowsiness. Sci. Rep. 7 , 1–10 (2017).

Li, Y. et al. A CNN-based wearable system for driver drowsiness detection. Sensors 23 , 3475 (2023).

Hong, S., Kwon, H., Choi, S. H. & Park, K. S. Intelligent system for drowsiness recognition based on ear canal electroencephalography with photoplethysmography and electrocardiography. Inf. Sci. 453 , 302–322 (2018).

Hwang, T., Kim, M., Hong, S., & Park, K. S. Driver drowsiness detection using the in-ear EEG. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS , Institute of Electrical and Electronics Engineers Inc., 4646–4649 (2016).

Barua, S., Ahmed, M. U., Ahlström, C. & Begum, S. Automatic driver sleepiness detection using EEG, EOG and contextual information. Expert Syst. Appl. 115 , 121–135 (2019).

McFarland, D. J. & Wolpaw, J. R. EEG-based brain-computer interfaces. Curr. Opin. Biomed. Eng. 4 , 194–200 (2017).

Olejniczak, P. Neurophysiologic basis of EEG,. J. Clin. Neurophysiol. 23 , 186–189 (2006).

Article   PubMed   Google Scholar  

Jackson, A. & Bolger, D. J. The neurophysiological bases of EEG and EEG measurement: a review for the rest of us. Psychophysiology 51 , 1061–1071 (2014).

Drees, C. et al. Skin Irritation during video-EEG monitoring. Neurodiagn. J. 56 , 139–150 (2016).

Di Flumeri, G. et al. The dry revolution: evaluation of three different eeg dry electrode types in terms of signal spectral features, mental states classification and usability. Sensors 19 , 1365 (2019).

Debener, S., Emkes, R., De Vos, M., & Bleichner, M. Unobtrusive ambulatory EEG using a smartphone and flexible printed electrodes around the ear. Sci. Rep. 5 , 16743 (2015).

Niso, G., Romero, E., Moreau, J. T., Araujo, A. & Krol, L. R. Wireless EEG: a survey of systems and studies. Neuroimage 269 , 119774 (2023).

Mihajlovic, V., Grundlehner, B., Vullers, R. & Penders, J. Wearable, wireless EEG solutions in daily life applications: what are we missing? IEEE J. Biomed. Health Inf. 19 , 6–21 (2015).

Xu, J. & Zhong, B. Review on portable EEG technology in educational research. Comput. Hum. Behav. 81 , 340–349 (2018).

Noachtar, S. & Rémi, J. The role of EEG in epilepsy: a critical review. Epilepsy Behav. 15 , 22–33 (2009).

Kidmose, P., Looney, D., Ungstrup, M., Rank, M. L. & Mandic, D. P. A study of evoked potentials from ear-EEG. IEEE Trans. Biomed. Eng. 60 , 2824–2830 (2013).

Mikkelsen, K. B., Kappel, S. L., Mandic, D. P. & Kidmose, P. EEG recorded from the ear: characterizing the Ear-EEG method. Front. Neurosci. 9 , 438 (2015).

Looney, D. et al. An in-the-ear platform for recording electroencephalogram. In Proc. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS , 6882–6885 (2011).

Looney, D. et al. The in-the-ear recording concept: user-centered and wearable brain monitoring. IEEE Pulse 3 , 32–42 (2012).

Kappel, S. L., Makeig, S. & Kidmose, P. Ear-EEG forward models: improved head-models for Ear-EEG. Front. Neurosci. 13 , 943 (2019).

Meiser, A., Tadel, F., Debener, S. & Bleichner, M. G. The sensitivity of ear-EEG: evaluating the source-sensor relationship using forward modeling. Brain Topogr. 33 , 665–676 (2020).

Article   PubMed   PubMed Central   Google Scholar  

da Silva Souto, C. F. et al. Flex-printed Ear-EEG sensors for adequate sleep staging at home. Front. Digit. Health 3 , 688122 (2021).

Wang, Z. et al. Conformal in-ear bioelectronics for visual and auditory brain-computer interfaces. Nat. Commun . 14 , 4213 (2023).

Mikkelsen, K. B. et al. Accurate whole-night sleep monitoring with dry-contact ear-EEG. Sci. Rep. 9 , 1 (2019).

Article   CAS   Google Scholar  

Mikkelsen, K. B. et al. Self-applied ear-EEG for sleep monitoring at home. In Proc. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS , Institute of Electrical and Electronics Engineers Inc. 3135–3138 (2022).

Zibrandtsen, I. C., Kidmose, P., Christensen, C. B. & Kjaer, T. W. Ear-EEG detects ictal and interictal abnormalities in focal and generalized epilepsy—a comparison with scalp EEG monitoring,. Clin. Neurophysiol. 128 , 2454–2461 (2017).

Article   CAS   PubMed   Google Scholar  

Lee, J. et al. A 0.8V 82.9 μW In-ear BCI controller system with 8.8 PEF EEG instrumentational amplifier and wireless BAN transceiver. VLSI Circuits 123–124 (2018).

Li, G., Wang, S. & Duan, Y. Y. Towards conductive-gel-free electrodes: Understanding the wet electrode, semi-dry electrode and dry electrode-skin interface impedance using electrochemical impedance spectroscopy fitting. Sens. Actuators B Chem. 277 , 250–260 (2018).

Article   ADS   CAS   Google Scholar  

Goverdovsky, V., Looney, D., Kidmose, P. & Mandic, D. P. In-Ear EEG from viscoelastic generic earpieces: robust and unobtrusive 24/7 monitoring. IEEE Sens J. 16 , 271–277 (2016).

Bertelsen, A. R. et al. Generic dry-contact ear-EEG; Generic dry-contact Ear-EEG . 10.0/Linux-x86_64. (2019).

Kaveh, R. et al. A Wireless, multielectrode, user-generic ear EEG recording system, In Proc. IEEE Biomedical Circuits and Systems Conference (BioCAS) (2019).

Paul, A., Lee, M. S., Xu, Y., Deiss, S. R. & Cauwenberghs, G. A versatile in-ear biosensing system and body-area network for unobtrusive continuous health monitoring. IEEE Trans. Biomed. Circuits Syst. 17 , 483–494 (2023).

Paul, A. et al. Attention state classification with In-Ear EEG. In Proc. BioCAS 2021 - IEEE Biomedical Circuits and Systems Conference, Institute of Electrical and Electronics Engineers Inc. (2021).

Xu, Y. et al. In-ear integrated sensor array for the continuous monitoring of brain activity and of lactate in sweat. Nat. Biomed. Eng . 7 , 1307–1320 (2023).

Schwendeman, C., Kaveh, R. & Muller, R. Drowsiness detection with wireless, user-generic, dry electrode ear EEG. In Proc. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS , Institute of Electrical and Electronics Engineers Inc., 9–12 (2022).

Kaveh, R. et al. Wireless user-generic ear EEG. IEEE Trans. Biomed. Circuits Syst. 14 , 727–737 (2020).

Chi, Y. M., Jung, T. P. & Cauwenberghs, G. Dry-contact and noncontact biopotential electrodes: Methodological review. IEEE Rev. Biomed. Eng. 3 , 106–119 (2010).

Dong, H., Matthews, P. M. & Guo, Y. A new soft material based in-the-ear EEG recording technique. In Proc. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS , 5709–5712 (2016).

Lee, J. H. et al. CNT/PDMS-based canal-typed ear electrodes for inconspicuous EEG recording. J. Neural Eng. 11 , 4 (2014).

CAS   Google Scholar  

Yang, L. et al. Materials for dry electrodes for the electroencephalography: advances, challenges, perspectives. Adv. Mater. Technol. 7 , 2100612 (2022).

Lopez-Gordo, M. A., Sanchez Morillo, D. & Pelayo Valle, F. Dry EEG electrodes,. Sensors 14 , 12847–12870 (2014).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Li, G. L., Wu, J. T., Xia, Y. H., He, Q. G. & Jin, H. G. Review of semi-dry electrodes for EEG recording. J. Neural Eng. 17 , 051004 (2020).

Liu, W. et al. Electrical impedance performance of metal dry bioelectrode with different surface coatings. Sens Actuators A Phys. 269 , 515–523 (2018).

Fu, Y., Zhao, J., Dong, Y. & Wang, X. Dry electrodes for human bioelectrical signal monitoring. Sensors 20 , 1–30 (2020). MDPI AG.

Xu, J., Mitra, S., Van Hoof, C., Yazicioglu, R. & Makinwa, K. A. A. Active electrodes for wearable EEG acquisition: review and electronics design methodology. IEEE Rev. Biomed. Eng. 3333 , 1–1 (2017).

Google Scholar  

Guermandi, M., Cossettini, A., Benatti, S. & Benini, L. A wireless system for EEG acquisition and processing in an earbud form factor with 600 hours battery lifetime. In Proc. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS , Institute of Electrical and Electronics Engineers Inc., 3139–3145. (2022).

Negi, S., Bhandari, R., Rieth, L. & Solzbacher, F. In vitro comparison of sputtered iridium oxide and platinum-coated neural implantable microelectrode arrays. Biomed. Mater. 5 , 15007 (2010).

Article   ADS   CAS   PubMed   Google Scholar  

Meyer, R. D., Cogan, S. F., Nguyen, T. H. & Rauh, R. D. Electrodeposited iridium oxide for neural stimulation and recording electrodes. IEEE Trans. Neural Syst. Rehabil. Eng. 9 , 2–11 (2001).

Green, R. A. et al. Substrate-dependent stability of conducting polymer coatings on medical electrodes. Biomaterials 33 , 5875–5886 (2012).

Yu, Y. H. et al. New flexible silicone-based EEG dry sensor material compositions exhibiting improvements in lifespan, conductivity, and reliability. Sensors 16 , 11 (2016).

Suzuki, H. & Taura, T. Thin-film Ag/AgCl structure and operational modes to realize long-term storage. J. Electrochem Soc. 148 , E468 (2001).

Prieto-Avalos, G. et al. Wearable devices for physical monitoring of heart: a review. Biosensors 12 , 292 (2022).

T&W Engineering [Online]. Available: https://www.tweng.com/ (2024).

Naox Technologies [Online]. Available: https://www.naox.tech/science (2024).

Knierim, M. T., Bleichner, M. G. & Reali, P. A systematic comparison of high-end and low-cost EEG amplifiers for concealed, around-the-ear EEG recordings. Sensors 23 , 4559 (2023).

Kim, C. et al. Sub-μ Vrms-noise sub-μ W/channel ADC-direct neural recording with 200-mV/ms transient recovery through predictive digital autoranging. IEEE J. Solid-State Circuits 53 , 3101–3110 (2018).

Article   ADS   Google Scholar  

Lee, J. et al. A 0.8-V 82.9-μ W in-ear BCI controller IC with 8.8 PEF EEG instrumentation amplifier and wireless BAN transceiver. IEEE J. Solid-State Circuits 54 , 1185–1195 (2019).

Zhou, A. et al. A wireless and artefact-free 128-channel neuromodulation device for closed-loop stimulation and recording in non-human primates. Nat. Biomed. Eng. 3 , 15–26 (2019).

Polonenko, M. J. et al. Estimating audiometric thresholds using auditory steady-state responses. In Proc. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS , 1–9, (2016).

Zhu, Y., Li, Y., Lu, J. & Li, P. EEGNet with ensemble learning to improve the cross-session classification of SSVEP-based BCI from Ear-EEG. IEEE Access 9 , 15295–15303 (2021).

Nagarajan, A., Robinson, N. & Guan, C. Investigation on robustness of EEG-based brain-computer interfaces. In Proc. Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS , Institute of Electrical and Electronics Engineers Inc., 6334–6340 (2021).

Burrello, A. et al. Energy-efficient wearable-to-mobile offload of ML inference for PPG-based heart-rate estimation. In Proc. Design, Automation & Test in Europe Conference (2023).

Aggarwal, S. & Chugh, N. Review of machine learning techniques for EEG based brain computer interface. Arch. Comput. Methods Eng. 29 , 3001–3020 (2022). Springer Science and Business Media B.V.,.

Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F. & Arnaldi, B. A review of classification algorithms for EEG-based brain-computer interfaces, J. Neural Eng. 4 (2007).

Subasi, A. & Erçelebi, E. Classification of EEG signals using neural network and logistic regression. Comput. Methods Prog. Biomed. 78 , 87–99 (2005).

Lee, Y.-E., & M. Lee, M. Decoding visual responses based on deep neural networks with ear-EEG signals. In Proc. 8th International Winter Conference on Brain-Computer Interface (BCI) (2020).

Zhu, M. et al. Vehicle driver drowsiness detection method using wearable EEG based on convolution neural network. Neural Comput Appl 33 , 13965–13980 (2021).

Lee, Y. E., Shin, G. H., Lee, M. & Lee, S. W. Mobile BCI dataset of scalp- and ear-EEGs with ERP and SSVEP paradigms while standing, walking, and running. Sci. Data 8 (2021).

Nakamura, T., Goverdovsky, V., Morrell, M. J. & Mandic, D. P. Automatic sleep monitoring using Ear-EEG. IEEE J. Transl. Eng. Health Med. 5 , 1–8 (2017).

Staab, W. J., Sjursen, W., Preves, D. & Squeglia, T. A one-size disposable hearing aid is introduced. Hear. J. 53 , 36 (2000).

Kaveh, R. et al. Rapid and scalable fabrication of low impedance, 3D dry electrodes for physiological sensing. Adv. Mater. Technol. 2200342 (2022).

Deckert, C. A. Electroless copper plating. a review: Part I. Plat. Surf. Finish. 82 , 58–64 (1995).

Pucic, S. P. Diffusion of copper into gold plating. In Proc. IEEE Instrumentation and Measurement Technology Conference . (1993).

Genova, V., Paglia, L., Marra, F., Bartuli, C. & Pulci, G. Pure thick nickel coating obtained by electroless plating: Surface characterization and wetting properties. Surf. Coat. Technol. 357 , 595–603 (2019).

Chung, J., Bieri, N. R., Ko, S., Grigoropoulos, C. P. & Poulikakos, D. In-tandem deposition and sintering of printed gold nanoparticle inks induced by continuous Gaussian laser irradiation. Appl Phys. A Mater. Sci. Process 79 , 1259–1261 (2004).

Johnson, B. C. et al. An implantable 700 uW 64-channel neuromodulation IC for simultaneous recording and stimulation with rapid artifact recovery. In Proc. Symposium on VLSI Circuits (2017).

Geiger Brown, J. et al. Measuring subjective sleepiness at work in hospital nurses: validation of a modified delivery format of the Karolinska Sleepiness Scale. Sleep Breath. 18 , 731–739 (2014).

Kaida, K. et al. Validation of the Karolinska sleepiness scale against performance and EEG variables. Clin. Neurophysiol. 117 , 1574–1581 (2006).

Download references

Acknowledgements

The authors thank the Ford University Research Program, Bakar Spark Award, Cortera Neurotechnologies (acq. Nia Therapeutics), and the Berkeley Wireless Research Center sponsors. The authors would also like to thank Prof. Jan Rabaey and team, Miguel Montalban, Andy Yau, Adelson Chua, Justin Doong, Aviral Pandey, and Natalie Tetreault for technical support.

Author information

These authors contributed equally: Ryan Kaveh, Carolyn Schwendeman.

Authors and Affiliations

University of California Berkeley, Berkeley, CA, 94708, USA

Ryan Kaveh, Carolyn Schwendeman, Leslie Pu, Ana C. Arias & Rikky Muller

You can also search for this author in PubMed   Google Scholar

Contributions

R.K. developed the earpieces and fabrication techniques. R.K., C.S., and L.P. fabricated all test sensors. C.S. developed and implemented the machine learning algorithms. R.K. and C.S. performed the experiments and analysis. R.M. oversaw all aspects of the research project. A.C.A. and R.M. oversaw the fabrication process development. R.K., C.S., A.C.A., and R.M. wrote and edited the manuscript.

Corresponding authors

Correspondence to Ryan Kaveh , Carolyn Schwendeman or Rikky Muller .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Peer review

Peer review information.

Nature Communications thanks Sheng Xu, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, source data, source data, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Kaveh, R., Schwendeman, C., Pu, L. et al. Wireless ear EEG to monitor drowsiness. Nat Commun 15 , 6520 (2024). https://doi.org/10.1038/s41467-024-48682-7

Download citation

Received : 01 October 2023

Accepted : 09 May 2024

Published : 02 August 2024

DOI : https://doi.org/10.1038/s41467-024-48682-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Aug. 8, 2024

Youth athletes in Hall County will benefit from immersive virtual reality injury prevention training, providing them with a realistic training environment similar to that of professional athletes.

Imagine if playing a video game could reduce your risk of injury and make you a better athlete. The Emory Sports Performance And Research Center (SPARC) is turning this dream into reality for youth athletes in Georgia.

As the football season approaches, Emory SPARC is expanding its Georgia Initiative for Virtual Reality in Education and Sports (GIVES) program to Hall County youth football players. The GIVES program uses virtual reality (VR) technology to enhance injury prevention, rehabilitation, and sports performance, providing young athletes with training tools like the pros use.

Funded by a $4.5 million grant from the Arthur M. Blank Family Foundation, the program uses immersive VR technology to create realistic training environments and disseminate Emory research into local communities. These environments support injury prevention, rehabilitation, and in-game performance.

Leveraging virtual reality for injury prevention and performance

Emory SPARC, located at the Atlanta Falcons Flowery Branch headquarters, is a leading research center dedicated to researching and disseminating injury prevention strategies for young athletes.

“Our mission is to help athletes play well and stay well. Emerging research shows that incorporating virtual reality tools into sport-specific training can significantly enhance both injury prevention and performance,” says Greg Myer, PhD , director of the Emory Sports Performance And Research Center. “We are thrilled to partner with Hall County Schools to implement the virtual reality training programs developed in our laboratories. By getting these tools into the hands of coaches, athletes, and school medical staff, we can encourage kids to be active, while reducing the risk of injuries.”

A football player at Flowery Branch High School engages in virtual reality football drills.

This football season, the GIVES program has been deployed to all football players across seven Hall County high schools, with goals to reach over 15,000 students, including those from nine middle schools, over the next academic year. The program’s newly developed VR football drills, created in collaboration with Rezzil, target critical decision-making abilities required during a football game, such as identifying open targets and making accurate directional movements. Additionally, the program includes reaction games designed to test athletes’ sensorimotor capabilities under various conditions, mirroring real-game situations.

Myer says that by engaging athletes in dual-task cognitive and motor training—making them think while they perform a sports skill—they increase their cognitive reserve. This means the athlete has more brain power available to assess and respond to the demands of sports performance and competition. By training both the mind and the body together, they create better performing and safer athletes.

Community support and involvement

The program is distinguished by its school-based nature, with all testing and training conducted on-site at schools. Emory researchers and local high school work-based learning students facilitate the program, ensuring comprehensive support for participants. Teachers, coaching staff, and administrators are also encouraged to engage in the VR testing alongside student athletes, fostering a community of excitement and participation.

“We hope that this training and the research it generates will help make our athletes more agile and prevent injuries,” says Jimmy Lawler, athletic director at Flowery Branch High School. “Our main goal is to keep our athletes healthy, so they can continue to pursue their dreams of playing college sports. We don't want to see a talented player get injured and miss out on the opportunity to play the sport they love at the next level.”

To support the GIVES program, Emory SPARC makes use of hundreds of Meta Quest VR headsets, allowing many students to simultaneously benefit from the immersive training experience.

"The parents are excited for their kids to be part of this program because they recognize the value and reputation of Emory,” Lawler added. “They know this program can make a real difference for their young athletes.”

The rollout of the football VR programming is beginning in Hall County, home of IBM Performance Field, the Atlanta Falcons practice facility, with additional schools in Georgia to follow.

To learn more or apply, visit https://emorysparc.com/gives/ .

• School of Medicine
• Emory University Orthopaedics & Spine Hospital
• Emory Healthcare
• School of Medicine's Department of Medicine
• Woodruff Health Sciences Center
• News Releases

Recent News

Download emory news photo.

By downloading Emory news media, you agree to the following terms of use:

Creative Commons Attribution-NoDerivatives 4.0 International Public License

By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-NoDerivatives 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions.

Section 1 – Definitions.

• Adapted Material   means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image.
• Copyright and Similar Rights   means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section   2(b)(1)-(2)   are not Copyright and Similar Rights.
• Effective Technological Measures   means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements.
• Exceptions and Limitations   means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material.
• Licensed Material   means the artistic or literary work, database, or other material to which the Licensor applied this Public License.
• Licensed Rights   means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license.
• Licensor   means the individual(s) or entity(ies) granting rights under this Public License.
• Share   means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them.
• Sui Generis Database Rights   means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world.
• You   means the individual or entity exercising the Licensed Rights under this Public License.   Your   has a corresponding meaning.

Section 2 – Scope.

• reproduce and Share the Licensed Material, in whole or in part; and
• produce and reproduce, but not Share, Adapted Material.
• Exceptions and Limitations . For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions.
• Term . The term of this Public License is specified in Section   6(a) .
• Media and formats; technical modifications allowed . The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section   2(a)(4)   never produces Adapted Material.
• Offer from the Licensor – Licensed Material . Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License.
• No downstream restrictions . You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.
• No endorsement . Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section   3(a)(1)(A)(i) .

Other rights .

• Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise.
• Patent and trademark rights are not licensed under this Public License.
• To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties.

Section 3 – License Conditions.

Your exercise of the Licensed Rights is expressly made subject to the following conditions.

Attribution .

If You Share the Licensed Material, You must:

• identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated);
• a copyright notice;
• a notice that refers to this Public License;
• a notice that refers to the disclaimer of warranties;
• a URI or hyperlink to the Licensed Material to the extent reasonably practicable;
• indicate if You modified the Licensed Material and retain an indication of any previous modifications; and
• indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License.
• You may satisfy the conditions in Section   3(a)(1)   in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information.
• If requested by the Licensor, You must remove any of the information required by Section   3(a)(1)(A)   to the extent reasonably practicable.

Section 4 – Sui Generis Database Rights.

Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material:

• for the avoidance of doubt, Section   2(a)(1)   grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database, provided You do not Share Adapted Material;
• if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material; and
• You must comply with the conditions in Section   3(a)   if You Share all or a substantial portion of the contents of the database.

Section 5 – Disclaimer of Warranties and Limitation of Liability.

• Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.
• To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.
• The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.

Section 6 – Term and Termination.

• This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically.

Where Your right to use the Licensed Material has terminated under Section   6(a) , it reinstates:

• automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or
• upon express reinstatement by the Licensor.
• For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License.
• Sections   1 ,   5 ,   6 ,   7 , and   8   survive termination of this Public License.

Section 7 – Other Terms and Conditions.

• The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed.
• Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License.

Section 8 – Interpretation.

• For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License.
• To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions.
• No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor.
• Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority.

Creative Commons is not a party to its public licenses. Notwithstanding, Creative Commons may elect to apply one of its public licenses to material it publishes and in those instances will be considered the “Licensor.” The text of the Creative Commons public licenses is dedicated to the public domain under the   CC0 Public Domain Dedication . Except for the limited purpose of indicating that material is shared under a Creative Commons public license or as otherwise permitted by the Creative Commons policies published at   creativecommons.org/policies , Creative Commons does not authorize the use of the trademark “Creative Commons” or any other trademark or logo of Creative Commons without its prior written consent including, without limitation, in connection with any unauthorized modifications to any of its public licenses or any other arrangements, understandings, or agreements concerning use of licensed material. For the avoidance of doubt, this paragraph does not form part of the public licenses.

• Skip to main content
• Skip to search
• Skip to footer

Products and Services

Cisco Security

Master your goals. innovate. we'll tackle threats..

Get powerful security across all your networks, cloud, endpoints, and email to protect everything that matters, from anywhere.

If it's connected, you're protected

Cisco Security “The Hacker”

More connected users and devices creates more complexity. Cisco Security Cloud makes security easier for IT and safer for everyone anywhere security meets the network.

Deliver smarter, stronger security

Protect your organization across a multicloud environment, while simplifying security operations, improving scalability, and driving data-informed outcomes, powered by Cisco Talos.

Unlock better user experiences

Create a seamless experience that frustrates attackers, not users, by granting access from any device, anywhere, and adding more proactive security controls.

Deliver cost-effective defenses

Improve ROI by consolidating vendors, reducing complexity and integrating your security.

Strengthen security resilience

Unified, end-to-end protection maximizes value, minimizes risk, and closes security gaps everywhere to defend against evolving threats. Protect access, apps, and innovation across your network to secure your future.

Cisco Secure Firewall

Better visibility and actionable insights across networks, clouds, endpoints, and email allows users to respond confidently to the most sophisticated threats at machine scale.

Featured security products

Cisco hypershield.

A new groundbreaking security architecture that makes hyperscaler technology accessible to enterprises of all sizes and delivers AI-native security for modern data centers and cloud.

Cisco Secure Access (SSE)

A converged cybersecurity solution, grounded in zero trust, that radically reduces risk and delights both end users and IT staff by safely connecting anything to anywhere.

Detect the most sophisticated threats sooner across all vectors and prioritize by impact for faster responses.

Cisco Multicloud Defense

Gain multidirectional protection across clouds to stop inbound attacks, data exfiltration, and lateral movement.

Secure applications and enable frictionless access with strong MFA and more. Establish user and device trust, gain visibility into devices, and enable secure access to all apps.

Cisco Identity Services Engine (ISE)

Simplify highly secure network access control with software-defined access and automation.

Security Suites delivered by Cisco Security Cloud

Cisco User Protection Suite

Get secure access to any application, on any device, from anywhere. Defend against threats targeting users and deliver seamless access for hybrid work.

Cisco Cloud Protection Suite

Secure your apps and data with a powerful, flexible framework for a hybrid and multicloud world.

Cisco Breach Protection Suite

Secure your business by investigating, prioritizing, and resolving incidents through unified defense and contextual insights from data-backed, AI-powered security.

Customer stories and insights

Global partnerships fight to end child exploitation together.

"Marriott has long championed human rights and human trafficking awareness. Combating CSAM is an important extension of that work. The IWF provided the level of rigor we needed in a URL list, and Cisco's security technology provided the means to easily apply it."

Abbe Horswill, Director, Human Rights and Social Impact

Company: Marriott International

The NFL relies on Cisco

"From securing stadiums, broadcasts, and fans to protecting the largest live sporting event in America, the right tools and the right team are key in making sure things run smoothly, avoiding disruptions to the game, and safeguarding the data and devices that make mission-critical gameday operations possible."

Add value to security solutions

Cisco Security Enterprise Agreement

Instant savings

Experience security software buying flexibility with one easy-to-manage agreement.

Services for security

Let the experts secure your business

Get more from your investments and enable constant vigilance to protect your organization.

Sharpen your security insights

Cisco Cybersecurity Viewpoints

Set your vision to a more secure future with Cisco Cybersecurity Viewpoints. With specialized content from podcasts to industry news, you'll walk away with a deeper understanding of the trends, research, and topics in our rapidly changing world.

--> AGU