Project Vison
This study is aimed to make such a direct comparison between the conventional lab-based (as a control group) usability testing, conventional lab-based AR usability testing, and a remote synchronous AR usability testing method to determine whether typical outcome variables (e.g., performance, satisfaction, accuracy) of usability tests would be affected by the reduced experimental control (e.g., presence of experimenter, user environment).
My Role
UX Designer & Researcher (solo project)
I designed the whole study based on current problems and developed the AR prototype to compare the usability in different user environments. I was in charge of all processes, from the initial development of the prototype to conducting usability experiments and data analysis.
This portfolio web page gives a brief overview of the project and focuses on the AR of design. If you would like to learn more about the research process, check out the link below!
The Problem
It has been demonstrated that virtual prototypes can be used productively in usability testing. Virtual prototypes, Augmented Reality (AR) in particular, have received some attention as platforms that support AR such as cell phones and iPads, which combine both the real and virtual worlds and become ubiquitous.
Structured AR usability testing in the product development phase enables obtaining early user feedback, allowing companies to focus on UX from the early design phase. Therefore, many studies have been conducted to investigate how AR can be beneficial in the design and usability testing. Due to Covid-19, there have been attempts to perform AR usability testing with remote settings, but the authentic AR experience has not been provided to the participants due to complicated setup and logistics problems. They revealed many challenges such as different experimental mindsets, communication with participants, a sense of distance from the actual AR experience, and technical issues.
The review of the research literature revealed no studies that directly compared synchronous remote testing and in-lab testing using true AR experience.
The advantages of using Augmented Reality (AR)
AR allows a deep involvement of the user on an emotional and behavioral basis with higher attention to what participants are doing (Verhulst et al. 2021).
AR addresses some major limitations of physical prototypes: cost and time to construct, and the availability in the early stage of product development (Oberdörfer et al. 2021; Razek et al. 2018).
Remote Usability Testing
Previous remote setup usability testing studies have revealed many limitations though they did not utilize the AR prototype:
Direct user observation is missing (Janina Sauer et al. 2020).
The presence of a moderator is low (Alhadreti et al. 2011)
Reduced experimental control (e.g., noise during the task, phone notifications) (Andreasen et al. 2007)
Similarity of lab-based testing to different forms of extra-laboratorial testing
(Juergen Sauer et al. 2019).
The Objective of the Study Thesis
This study is aimed to make such a direct comparison between the conventional lab-based (as a control group) usability testing, conventional lab-based AR usability testing, and a remote synchronous AR usability testing method to determine whether typical outcome variables (e.g., performance, satisfaction, accuracy) of usability tests would be affected by the reduced experimental control (e.g., presence of experimenter, user environment).
Details on Study Prototype
Handheld Display (HHD) such as iPad was utilized for its easy accessibility to investigate usability testing using AR prototypes.
On a remote AR testing method, the assistance to participants from the administrator will be limited. Therefore, many interactions or complex-to-use products were excluded.
In the previous study, issues were highlighted when testing AR prototypes that implemented products that were larger than actual iPads (e.g., TVs, cushions, etc.).
Multimodal feedback such as visual feedback, background sound, and effect sound creates better immersion for Augmented Reality (Ha, Chang, and Woo 2007).
The accuracy of representing the product's natural interaction is high when the digital product is tested (e.g., physical knob).
Therefore, a smartwatch was selected to be represented in AR for the study; using a digital product was considered more accessible in implementing an immersive AR experience than a physical product
AR Experience Development Process
Software used
A smartwatch was 3D modeled for AR experience using the rhinoceros 6 software. Then, using the Vuforia engine package, the image target and the object were linked together in Unity, allowing the iPad camera to recognize the image target and load the smartwatch object.
The interactions required for screen transition were used with a modification of the lean touch code in the C# library. After completing the app development, Xcode, MacOS app made by Apple, was used to deploy to the iPad.
Final Design
A higher fidelity prototype with improvements in the UX was developed after the pilot study; visual and sound feedback is included for every interaction.
Tasks
Launch Spotify on their AR prototype
Swipe left to see the menu
Go to "Download“
Open the "AR Study“ playlist
Press the arrow button to play the song
User Study Design
Research Question:
How do remote AR testing and in-person AR testing differ in usability and user satisfaction among regular users with no disabilities?
Independent Variables:
Type of prototype is used for AR
Dependent Variables:
Time taken to complete the tasks, User environment, Device used for testing
Accommodation for in-person Testing
Accommodation for Remote Testing
Participants
The subject group was regular users with no disabilities, and they were recruited by using university networking/advertisements (flyers, announcements, or email) and word of mouth.
The test result data is expected to be nonparametric, 25 participants for each setup were needed for a reliable result (a total of 75 participants).
Data Collected
Results
Users spent more time on performing the task with the remote synchronous AR setup (Mdn = 41.04) than the time users spent on the lab-based product testing (Mdn=35.81) and lab-based AR testing (Mdn = 37.61). However, The average SUS scores for respective setups are calculated 87.80, 88.70, and 82.19, which are all considered to be acceptable as per SUS usability standards (Sauro 2011).
After the project…
I started this project with curiosity, where I wanted to break down the barrier of the inconvenience of users having to come to the site for usability testing. However, as the project progressed, I realized that many variables and unanswered questions could affect the results in remote testing for Augmented Reality.
This thesis evaluated the method of delivering the actual AR experiences to remote users and the efficiency and accuracy of usability testing with a prototype. I believe this information is fundamental to the overall software design that will enable future researchers and companies to adopt augmented reality for usability testing.
Of course, there is much more work to be done! The next steps in this project would be to test the physical product implemented AR prototype because the results might appear differently due to a lack of the natural interaction
If you would like to see details about the research process, check out the link below!