VR App Evaluation[edit | edit source]
We conducted iterative prototype testing of the VR app, first going through many iterations with the app developers themselves, and then showing early users the platform and gathering feedback. The immediate feedback about the app from end users was that it had "incredible graphics", that it was "intuitive to navigate", and the videos were "easy to understand". For more information about how to use the app and to watch videos about the interactive interface, please click here.
Simulator Evaluation[edit | edit source]
The physical simulator underwent a number of evaluation and prototype testing phases. We initially created a simulator that incorporated a pressure sensor as a way to provide nuanced feedback to users about the amount of pressure required to stop blood flow through the simulator. This simulator required an Arduino to be installed, and required a longer total build time to complete the simulator.
There were concerns raised by both Firefighters in initial user testing (August 2021) and GSTC judges about the reproducibility and ease of use of this model. For pictures and videos of the various simulator phases, please visit this link.
Prototype Testing[edit | edit source]
We took this feedback back to our engineers and worked together to create a simplified version of the simulator that did not require any electronics, was cheaper (~$55 vs ~$90), and was able to be built by end-users in about 1.5 hours vs 20 hours.
In October 2021, we tested building the simple model of the simulator model with Guatemalan Firefighters. They successfully completed the build using only the instruction manual, and the process took 45- 90 mins. For more information on the simplified version of the simulator, please visit this link.
Internal Validation[edit | edit source]
We proceeded to internal validation of our VR app and simulator.
We used three different methods of validation: 1. Subjective validation of the physical model, 2. construct validation of the physical model, and 3. assessment of skill transfer of the training in the app and the physical model.
We tested 2 groups:
- Non-experts: Medical students without experience in tourniquet placement or trauma management
- Experts: MDs with experience in tourniquet placement
For the subjective validation, novice and expert subjects placed tourniquets on the simulator and later filled out a survey with 8 questions about the simulator's realism and quality on a 5-point agreement Likert scale (Strongly agree, agree, neither agree nor disagree, disagree, strongly disagree). The questions asked were as follows:
1. This tool is useful for training in hemorrhage control protocols
2. This tool is useful for training
3. The simulator stimulates learning
4. On a scale of 1-5, what score do you give the simulator?
5. This tool helps with critical thinking and decision making
6. This tool is realistic
7. Blood flow is realistic in the simulator
8. I am satisfied with the simulator
The results of the subjective evaluation (non-experts n=14, experts n=7) were as follows:
We found no difference between the groups in the subjective evaluation of the module.
The objective validation was performed through a construct validation test where novices and experts placed tourniquets and their performance was measured using a checklist to determine if there were differences in skills between the two groups.
Ten non-expert and 10 expert participants applied the tourniquet prior to completing the VR app training modules. Results were as follows:
We can see that experts can place the tourniquet faster than the non-experts at baseline, implying that the simulator reveals differences in ability between experts and non-experts. We postulate that with a bigger test group we could see differences in all the items of the checklist.
The skills transfer validation was carried out by measuring novices before training and after solving the clinical cases in the application and training on the simulator, using the same checklist
This table shows the early results of placing a tourniquet before and after the training using the VR app and the simulator:
We also measured the performance between experts and non-experts post training, finding no differences between the 2 groups (where the pre-test comparison between these two groups showed a significant difference in the speed of appropriate application between the experts and non-experts).
References[edit | edit source]
Weiman, S (2019). Retention of Tourniquet Application Skills Following Participation in a Bleeding Control Course, Journal of Emergency Nursing, Volume 46, Issue 2, 2020,Pages 154-162 (fig 1)