Effects of questionnaires in VR user studies

Questionnaires are among the most common research tools in virtual reality (VR) evaluations and user studies. However, transitioning from virtual worlds to the physical world to respond to VR experience questionnaires can potentially lead to systematic biases.

Administering questionnaires in VR (inVRQs) is becoming more common in contemporary research. This is based on the intuitive notion that inVRQs may ease participation, reduce the Break in Presence (BIP) and avoid biases.

In this paper, we perform a systematic investigation into the effects of interrupting the VR experience through questionnaires using physiological data as a continuous and objective measure of presence. In a user study (n=50), we evaluated question-asking procedures using a VR shooter with two different levels of immersion. The users rated their player experience with a questionnaire either inside or outside of VR.

Our results indicate a reduced BIP for the employed in VRQ without affecting the self-reported player experience.

Takeaways

We investigated how question-asking itself breaks the VR experience and if inVRQs can minimize the break in presence and therefore, reduce uncontrolled biases in performance and self-reports.

We conducted a user study (n=50) where we recorded biosignals of subjects while they played a simple VR shooter either with low or high visual fidelity and filled out questionnaires regarding their PX both in VR and on PC in physical reality. Our results clearly show that both questionnaire modalities produce Break in Presence.

Moreover, the physiological responses in the EDA for inVRQs are significantly lower and shorter than to outVRQs.

These results show evidence that inVRQs are less invasive than outVRQa and provide more reliable self-reports. Our findings suggest an influence of BIPs on performance of subsequent tasks. Moreover, these effects might become even more prominent in VEs high-quality VR experiences, such as AAA games as there is “more immersion to break”. Our findings can help researchers and designers to apply the appropriate instruments for their study design and lay groundwork for the design of inVRQs to provide validated and standardized methods of question-asking in VR.