The beauty of (remote) observation

A couple of months ago, we were working on a validation study for a system that continuously analyses comprehensive medical device data to identify patient condition and assist in reducing the risk of life-threatening events. This system is based on live, streaming data which serves as a friendly reminder to the caregiver who is in charge of the patient. This system is intended to work together with the intended user, rather than cause annoyance or frustration. 

During our study, we worked alongside 15 specialised nurses to validate the system. However, only one half of the team was able to make it to the US to moderate the test session, whilst the Observer had to observe remotely. There was a lot of scepticism behind this as this was territory we weren't familiar with at the time, but as we were setting up and making changes to facilitate the best observation angles, we realised that there were many benefits to this situation. 

One - due to the observer needing a clear view of the system, we screen shared the whole session whilst the participant was using the system, so the observer was able to see everything in real time without any glare or reflection. This also meant that there was one less camera that was visible to the participant, which in accordance with Grubaugh et al. (2005) was better for reducing use errors as they found there were higher error rates in the group that had extra monitoring (usability lab with cameras on tripods and audio recording all around the participant) compared to the least amount of monitoring (no-one way mirrors, windows, or any visible recording devices). 

Two - there was another angle of the camera so that the observer was able to look at the participant in real time alongside the screen share to observe facial expressions and body language alongside their usage of the system. 

Three, during the screen share, the observer was able to screenshot the exact moment a close call, difficulty or use error occurred so that the data, when presented to the client, would be clear and instantly comprehensible. It was also useful when the participant would share any subjective feedback regarding the system as the observer was able to screenshot the exact image to illustrate the feedback, back to the client. 

Four, the Hawthorne Effect (also known as the Observer Effect) is where participants in studies change their behaviour because they are hyperaware that they are being watched. A study conducted by Schulte-Mecklenbeck and Huber (2003) asked 40 German university students to complete certain Web-based tasks in a lab setting and 32 students do the same thing in their own homes. The results showed that the students in the lab took double the amount of time and double the amount of clicks to complete the tasks, compared to the students at home. Therefore, the researchers concluded that the participants in the lab felt more pressure to perform well. In our case, for this study, even though the participants were made aware they were being observed through the cameras, the observer was out of view, therefore, the participants did not feel hyperaware they were being observed, which we believe may have contributed to the performance of this study. 

Overall, this study highlighted the changes we have had to endure due to a heavy reliance on remote testing and highlighted that even though there are some hindrances to remote observation, there are also many benefits that follow suit which we will continue to carry over as 'typical' usability testing resumes.

References

B. Grubaugh, S. Thomas, and J. Weinberg (USA) (2005). The Effects of the Testing Environment on User Performance in Software Usability Testing.

Schulte-Mecklenbeck M, Huber O. Information search in the laboratory and on the Web: With or without an experimenter. Behavior Research Methods, Instruments, & Computers. 2003 May;35(2):227-35.