In the field of brain-computer interface (BCI) research, electroencephalography (EEG) is the preferred data acquisition method due to its relatively low cost, mobility, non-invasiveness, and high temporal resolution. However, a major concern in BCI research is its susceptibility to noise. To address this issue, multimodality—the combination of multiple information sources or modalities as input—has been proposed. This approach enables a flexible and broader selection of modalities and thus, could potentially increase the robustness of the classification. A strong candidate that is gaining ground is virtual reality (VR).

The integration of VR into EEG experiments not only provides an affordable and safe training environment for exploring cognitive states but also makes the field more dynamic and engaging. While the fusion of VR and BCI holds promise for improving classification accuracy of mental workload detection, it also amplifies the challenges associated with using these two head-mounted technologies.

Beyond the lab: making brain-computer interfaces ready for real-world scenarios

For BCI to evolve from strict laboratory conditions to real-life applications, the technology must be adaptable to more dynamic contexts and resilient to noise. VR is a promising candidate for fusion with EEG as it provides a cost-effective environment for exploring realistic scenarios and allows the integration of sensors. Our research investigates whether combining EEG with data from VR headsets can improve mental workload detection compared to using EEG alone. Previous research has demonstrated the robustness of the workload classifier in different display modalities, such as VR, and in unusual body postures, such as standing [1]. Our current study builds on this by exploring whether the fused data maintains this robustness.

Our approach

In our study, 20 healthy participants performed a workload-inducing cognitive task under four conditions that differed in posture and display modality. The task, also called the ‘sparkles paradigm’ [2], consisted of alternations between an arithmetical task phase and a relaxation phase. Participants performed the task while sitting and standing, using EEG-only and EEG combined with a VR headset. We simultaneously recorded EEG and psychophysiological data from the headset and then imported and synchronized it using the BeMoBIL Matlab toolbox [3]. After pre-processing and classifying the EEG, pupil dilation, and heart rate data independently, we used the principle of Majority Voting to fuse the data. Through this method, we obtained a final prediction based on the predictions of all three modalities, selecting the prediction that occurred most frequently for each trial. The classification accuracy of the final prediction for the test data was used as a measure for the statistical analysis.

‍

EEG alone surpasses multimodal fusion for accurate workload detection

Our findings show that the EEG-only method outperforms the fused method for standing posture. For the seated posture, no significant difference was found between the EEG-only method and the fused method. Importantly, our workload classifier remained robust when using fused data for standing posture. Although our results relate to static postures where multimodal VR-BCI is applicable, they represent a step towards more dynamic research and application contexts. The fusion of the data shows no significant increase in classification accuracy, indicating that EEG alone is sufficiently powerful and that adding multiple inputs does not justify the additional complexity of the analysis.

EEG-only approaches remain the gold standard in BCI research

Although VR-BCI data fusion has great potential to improve classification quality, the method is still at an early stage. Our results show that the workload classifier remains robust for the fused data in a standing posture.  However, EEG-only approaches currently provide better results and remain the gold standard in BCI research. This research marks a step towards integrating more dynamic elements into BCI and paves the way for future advancements in the field.

By further exploring the integration of VR and EEG, we aim to open up new possibilities for BCI applications in real-world, dynamic environments, improving the resilience and practicality of the technology.

‍

Read the thesis>>

References:

[1] Gherman, D. E., & Zander, T. O. (2022). An investigation of a passive BCI’s performance in different user postures and presentation modalities. C Ponference ROGRAMME, 29.

[2] Zhang, X., Krol, L. R., & Zander, T. O. (2018, October). Towards task-independent workload classification: shifting from binary to continuous classification. In 2018 ieee international conference on systems, man, and cybernetics (smc) (pp. 556-561). IEEE.

[3] Klug, M., Jeung, S., Wunderlich, A., Gehrke, L., Protzak, J., Djebbara, Z., ... & Gramann, K. (2022). The BeMoBIL Pipeline for automated analyses of multimodal mobile brain and body imaging data. bioRxiv, 2022

‍