Tibial Fracture Fixation/Evaluation

Evaluation[edit | edit source]

A description of how the original prototype design was iterated and developed with user-centered design methodology (including user testing) is outlined below:

Expert Input[edit | edit source]

Our team created an online survey to help initially guide the design of the module’s frugal simulators, psychomotor skills training, and targeted self-assessment frameworks. With mentorship support from the Royal College of Surgeons of Ireland, this survey received responses from 5 male orthopedic surgeons in two countries with a combined total of 85 years of practice experience. 80% of the survey respondents are orthopedic surgeons based in a LMIC.

• 100% of survey respondents agreed or strongly agreed that training on high fidelity simulators that accurately replicate human cortical bone hardness will translate into improved plunge depths when performing bicortical drilling on patients compared to low fidelity simulators.
• All survey respondents agreed or strongly agreed that training on powered drills reduces the risk of learners developing anti-skills for bicortical drilling on patients compared to manual drills.
• When asked about the importance of powered drills for bicortical drilling skills training, survey respondents stated that it was "[c]ritically important to have powered drills available" and "[i]t provides for a better outcome and improves efficiency."

Based on the survey feedback, we chose to:

• Design high-fidelity, 3D printed bone models that simulate human cortical bone hardness to better train learners to minimize plunge depths
• Train learners to use powered drills designed for resource-constrained settings for the Bicortical Drilling Skills training module

The survey responses also helped guide the inclusion of checklist items on the Bicortical Drilling Skills Checklist for the self-assessment framework.

Simulator Build[edit | edit source]

• Although our team had originally proposed to design 3D printed cylinders (and even other geometric, non-organic shapes) with bicortical anatomy, realistic far cortex thickness values, and cancellous bone porosity and microstructure to train learners on bicortical drilling skills, we opted to provide anatomic bone models for bicortical drilling skills training in preparation for modular external fixation procedures.^[1][2] The rationale for this design choice was based on the desire to accelerate the learning curve for users by providing high-fidelity artificial bone models and to minimize the risk of learners developing anti-skills for bicortical drilling through training on models that do not simulate the external contour or cross-sectional shape of human tibial shafts.
• To optimize bone simulation model aesthetics and enable these models to print support-free, a base was added to the bottom of the bone models using Blender, an open-source 3D design software program.
• During user testing in Nigeria, we reduced the height of the base and removed the fibula component from the Tibial Shaft Transverse Fracture Simulator because the fibula is not required for skills training and this would reduce filament costs for the learner.
• To further reduce material costs in LMICs, we later re-designed the Tibial Shaft Transverse Fracture Simulator to reduce the diameter of the base.
• To help the learner in setting up the simulators, each 3D printed bone simulation model displays the model number translated in the 6 official languages of the United Nations, and the model gender using internationally recognized graphical symbols for males and females. We received written authorization from the International Organization for Standardization to use ISO 7001 graphical symbols for males and females on our gender-specific bone simulation models.
• We added a semi-engraved drilling direction arrow on the bottom of the Tibial Shaft Simulator to help users orient this simulator in the vise clamp.
• The removal of the fibula from the Tibial Shaft Transverse Fracture Simulator had the unintended consequence of eliminating a key anatomic landmark for model orientation. Our user testing in Nigeria showed that our instructions need to emphasize that 3D Printed Adult Tibial Bone Model #2 is inserted into the left vise clamp and 3D Printed Adult Tibial Bone Model #3 is inserted into the right vise clamp to ensure proper orientation of the left Tibial Shaft Transverse Fracture Simulator for skills training. Based on user feedback, we added a semi-engraved drilling direction arrow on the base of 3D Printed Adult Tibial Bone Models #2 and #3 to assist with proper orientation.
• Our user testing in Nigeria showed that all bone models require a vise attachment to allow the user to secure the model inside a standard vise clamp to maximize learner safety during simulation training. We do not recommend having an assistant hold a bone model during drilling training because it exposes the assistant to potential injury.
• A vise attachment was added to the Tibial Shaft Simulator to allow it to be secured in a vise clamp for safe Bicortical Drilling Skills training. We increased the height of the vise attachment of the Tibial Shaft Simulator to 6.0 cm in order to maximize compatibility with different sizes of locally available vise clamps in Nigeria.
• In LIMCs, 3D printer filament is usually imported and not locally produced. To reduce material costs in LMICs, we re-designed the Tibial Shaft Simulator to make the vise attachment with a hollow opening. Our initial user safety testing indicates that this design could compromise the mechanical strength of the Tibial Shaft Simulator for Bicortical Drilling Skills training so this cost-saving feature was not used in order to maximize safety.
• Each 3D printed fracture fragment of the Tibial Shaft Transverse Fracture Simulator has a vise attachment so when it is placed inside a standard vise clamp, the bone model will be properly positioned to simulate a patient in the supine position.
• We added detailed instructions on how to input the customized settings for the 3D printed bone models into a 3D slicing program in case the user was not familiar with using advanced 3D slicer settings.
• Our user testing in Nigeria showed that manually wrapping the Tibial Shaft Transverse Fracture Simulator in cellophane can keep a displaced fracture loosely attached.
• One potential drawback of Ultimaker FDM 3D printers are that they typically require applying a thin layer of glue or adhesion sheet on the glass build plate and/or including a brim on the 3D printed model to ensure adequate adhesion of the 3D printed object to the glass build plate.^[3][4][5] On April 2, 2022, we tested a powder-coated PEI build plate sheet magnetically attached to the glass bed of an Ultimaker 2+ Extended 3D Printer and demonstrated that one of the 3D Printed Adult Male Tibial Bone Models could be successfully printed with PLA without the use of a glue layer or adhesion sheet on the build plate or adding a brim to the 3D printed model.^[6] On May 22, 2022, our Nigerian 3D printing partner confirmed it was possible to print the 3D Printed Adult Female Tibial Bone Models without a brim or the use of any adhesive or glue on the glass build plate of the Ultimaker S3 printer when the layer height setting is 0.3 mm.
• To save on printing time and electricity costs, we originally recommended to print the Adult Tibial Bone Models at the fastest speed (0.4 mm layer height) but this resulted in unacceptable print quality issues including rough overhangs and layer shifts. On May 18, 2022, we revised the recommended print settings to a maximum layer height of 0.3 mm which resulted in outstanding print quality for the 3D Printed Adult Female Tibial Bone Models by AIGE Limited on an Ultimaker S3 3D printer in Port Harcourt, Nigeria. On May 21, 2022, we updated our print setting instructions to advise that the layer height setting should not exceed 0.3 mm to avoid print quality issues.
• On May 3, 2022, we were notified by AIGE Limited that there was a significant increase in the printing time of the 3D Printed Adult Male Tibial Bone Models from January 22, 2022 to April 22, 2022 which would impact pricing and delivery times. On May 17, 2022, a significant print quality issue was noted on the base of a 3D Printed Adult Male Tibial Bone Model. On May 19, 2022, we conducted an analysis and identified these issues were caused by a misunderstanding of the instructions for the top layer setting. On May 21, 2022, we created general and printer-specific step-by-step instructions with screenshots to minimize print setting entry errors for manufacturing the 3D Printed Adult Male Tibial Bone Models on 6 commonly available and affordable desktop 3D printers. On May 23, 2022, we received feedback from an Ugandan 3D printing organization stating, "Thank you very much for the revised instructions. I have taken a look at them, and they were helpful."
• On May 19, 2022, we also uploaded the g-code files for commercially available open-source 3D printers to avoid errors when inputting the customized print settings and to eliminate print file preparation time for re-printing in the future.
• On May 23, 2022, we created step-by-step instructions with screenshots to help local 3D printing entrepreneurs calculate a price quote.
• We observed that the outstanding results of our internal prototyping of the 3D Printed Tibial Bone Models on the open-source Prusa i3MK3S printers in Toronto, Canada were not reproduced when printing the models on an open-source Ultimaker S3 printer in Port Harcourt, Nigeria. To address this, we used the open-source Prusa Slicer program to identify and fix any potential mesh errors in the 3D models before slicing the model in Ultimaker Cura to prepare the file for printing on other printers. On June 25, 2022, we updated our internal checklists to ensure that Prusa Slicer will be used to identify and fix any potential mesh errors before 3D models are made available on Appropedia.
• Based on feedback provided on July 16, 2022 from one female medical officer learner and one male orthopedic surgeon in Cameroon, we clarified on one of our Simulator Build pages that "[a]ny powered orthopedic surgical drill that is compatible with 5.0 mm diameter self-drilling Schanz screws can be used for this skills training module."

Skills Practice[edit | edit source]

Based on user feedback from Uganda, we explained the function of each hardware component, and listed and photographed the Surgical Hardware in order of use by the practitioner in the Modular External Fixation Kit for Simulation Training with 3D Printed Tibial Adult Bone Models knowledge page.

In the procedural steps for Modular External Fixation for an Open Tibial Shaft Transverse Fracture, we identified and highlighted the critical phases of the procedure in which only essential conversations and activities in the operating room should occur to avoid distracting the operator from their performance of his or her duties.^[7]

Our user testing in Nigeria showed that a sponge can be cut out to fit over the medial aspect of each fracture fragment to simulate the soft tissue overlying the periosteum, and the sponge can be wrapped with cellophane to simulate the skin overlying the soft tissue on the Tibial Shaft Transverse Fracture Simulator. This permits the user to practice making a stab incision for the Schanz screw insertion sites.

Our user testing in Nigeria found that the sponge, if not properly retracted, gets entangled in the drill bit. Therefore, the sponge material was removed from the Tibial Shaft Transverse Fracture Simulator, and the cellophane is used to simulate the overlying skin for the stab incision for the subcutaneous medial tibia. We decided it was not necessary to simulate the various soft tissue layers with different materials for the simulator because of the superficial location of the medial tibia. This has the benefit of reducing simulator costs, simplifying the simulator build, and minimizing simulator assembly time for the learner.

Our user testing in Nigeria showed that the drill sleeve assembly should not be placed directly on the 3D printed bone model but should be held 3.0 mm above the bone model while drilling to prevent the plastic strands from getting stuck inside the drill sleeve assembly. Normally, the drill sleeve assembly is placed directly on the bone during real clinical scenarios.

Our user testing in Nigeria indicated that external irrigation would be useful during Bicortical Drilling Skills training but is not required for powered drilling through the near cortex only for the Modular External Fixation for an Open Tibial Shaft Transverse Fracture skills training module.

Our user testing in Nigeria showed that the 3D printed bone simulation models would tolerate bicortical drilling with a 3.2 mm drill bit and 3.5 mm self-drilling Schanz screws but not with 4.5 mm self-drilling Schanz screws.

Our user testing in Nigeria showed that the 3D printed bone simulation models would tolerate near cortex drilling and manual advancement with a 3.5 mm self-drilling Schanz screws without the need for pre-drilled bicortical holes. It is acceptable to use smaller sized hardware for simulation skills training because the fundamental principles for modular external fixation are the same irrespective of hardware size.

We added a section to this module that educates the learner on the differences between the simulation training hardware and the hardware they would use in the operating room. We later conducted additional testing to identify 3D printed materials to permit learners to use the same sized hardware for simulation training and operating room procedures.

Our user testing in Nigeria showed that 3.5 mm diameter Schanz screws and 6.5 mm rods and clamps can be used to reduce and stabilize the simulated fracture of the Tibial Shaft Transverse Fracture Simulator.

Our subsequent user testing in Nigeria showed that:

• the Tibial Shaft Simulator made from Ultimaker PLA (Shore Hardness 83D) was able to mechanically tolerate bicortical drilling with a 4.0 mm drill bit, and
• the Tibial Shaft Transverse Fracture Simulator made from Ultimaker PLA (Shore Hardness 83D) and Ultimaker Tough PLA (Shore Hardness 79D) were able to mechanically tolerate powered drilling of a new (unused) self-drilling 4.5 mm Schanz Screw through the near cortex and manual advancement to anchor the self-drilling 4.5 mm Schanz Screw into the far cortex.^[8][9]

These findings will permit learners to use the same sized hardware for simulation training and real clinical scenarios. We hypothesize that the key differences from our prior user testing is that new (sharp) Schanz screws were used and that fresher filament was likely used.

We also observed that two other filament materials (ABS and CPE+) exhibited mechanical failure during powered drilling of a new self-drilling 4.5 mm Schanz Screw through the near cortex and manual advancement to anchor the self-drilling 4.5 mm Schanz Screw into the far cortex. Although these two filaments are described as stronger materials, their Shore Hardness D values are lower than PLA and Tough PLA.^{[8][9][10][11][12][13]}

Filament Material Testing for 3D Printed Adult Male Tibial Bone Models

Simulator	Model #	Filament Brand	Filament Material	Shore Hardness	Test Observations
Tibial Shaft Simulator	1	Ultimaker	PLA	83D[8]	Model was able to tolerate powered drilling of a 4.0 mm drill bit through both cortices.
Tibial Shaft Transverse Fracture Simulator	2	Ultimaker	PLA	83D	Model was able to tolerate powered drilling of a self-drilling 4.5 mm Schanz Screw through the near cortex and manual advancement of the 4.5 mm Schanz Screw for anchoring into the far cortex.
Tibial Shaft Transverse Fracture Simulator	3	Ultimaker	PLA	83D	Model was able to tolerate powered drilling of a self-drilling 4.5 mm Schanz Screw through the near cortex and manual advancement of the 4.5 mm Schanz Screw for anchoring into the far cortex.
Tibial Shaft Transverse Fracture Simulator	3	Ultimaker	Tough PLA	79D[9]	Model was able to tolerate powered drilling of a self-drilling 4.5 mm Schanz Screw through the near cortex and manual advancement of the 4.5 mm Schanz Screw for anchoring into the far cortex.
Tibial Shaft Transverse Fracture Simulator	3	Ultimaker	ABS	76D[12]	Model was unable to tolerate powered drilling of a self-drilling 4.5 mm Schanz Screw through the near cortex and manual advancement of the 4.5 mm Schanz Screw for anchoring into the far cortex.
Tibial Shaft Transverse Fracture Simulator	3	Ultimaker	CPE+	75D[13]	Model was unable to tolerate powered drilling of a self-drilling 4.5 mm Schanz Screw through the near cortex and manual advancement of the 4.5 mm Schanz Screw for anchoring into the far cortex.

Our user testing in Nigeria showed that unused (sharp) 4.5 mm diameter Schanz screws and 11.0 mm diameter rods and corresponding clamps can be used to reduce and stabilize the simulated fracture of the Tibial Shaft Transverse Fracture Simulator.

On April 20, 2022, we conducted user testing in Canada with unused (sharp) 5.0 mm diameter self-drilling Schanz screws and a 4.5 mm drill sleeve and showed that the 3D Printed Adult Male Tibial Bone Model #2 was able to mechanically tolerate powered drilling of near and far Schanz screws into the near cortex and manual advancement of the Schanz scews into the far cortex without perforation. The learner observed that there is a discernable "give" when the tip of the Schanz screw exited the near cortex and penetrated the cancellous bone when drilling the near and far Schanz screw insertion sites in the 3D Printed Adult Male Tibial Bone Model. Therefore, learners can use 4.5 or 5.0 mm diameter self-drilling Schanz screws on the 3D Printed Adult Tibial Bone Models.

Self-Assessment[edit | edit source]

For plunge depth measurement on the Tibial Shaft Simulator, we switched to using modelling clay for the backstop instead of foam material because clay is reusable and can be locally obtained in Nigeria and Uganda.^[14][15]

In response to the feedback we received at the August 18, 2021 Prototype Showcase, our team added a Far Cortex Breakthrough Detector to the Tibial Shaft Simulator for Bicortical Drilling Skills training that uses an electric circuit and powered orthopedic surgical drill designed for resource-constrained settings.

To maximize safety and minimize the risk of the learner touching any conductive material in a live electrical circuit during simulation training:

• we added an extra insulated alligator clip to minimize the length of non-insulated metal wire for the Far Cortex Breakthrough Detector
• we revised the simulator build and use instructions to re-emphasize and explain the safety rationale for the mandatory use of gloves during training.

Our user testing in Nigeria showed that wrapping the entire clay backstop in cellophane keeps the clay from covering the aluminum foil during plunge which interrupts the breakthrough detection circuit. This helps to prolong the duration of the auditory signal for far cortex breakthrough detection.

Our user testing in Nigeria showed that a depth gauge can be used to measure plunge in a clay backstop wrapped in cellophane.

Our user testing gave us significant concerns that augmented feedback for plunge detection can foster anti-skills for bicortical drilling by training the self-assessed learner to depend on extrinsic (augmented) feedback instead of intrinsic feedback (drilling by sound and feel) to adjust the applied drilling force to minimize plunge depth.

Our user testing has observed the unintended consequence of adding augmented feedback for plunge detection during bicortical drilling skills training is that it distracts the learner from paying attention to drilling by sound and feel of the drill bit to minimize plunge. This was observed in user testing sessions on two continents where users had accidentally missed the step to connect the augmented feedback circuit and exhibited significant plunge because they were dependent on augmented feedback.

In the video below, you can observe how the practitioner inadvertently missed the step to connect the plunge detector circuit and exhibited significant plunge because he was dependent on the augmented feedback circuit for plunge detection. In the second segment, the practitioner is using the augmented feedback circuit for plunge detection. Although this reduces plunge during the simulation training, this could lead to overconfidence in the learner and result in worse plunge depths in real clinical procedures where plunge detection is not available.

A literature review supports our concerns and raises questions on the utility of augmented feedback for bicortical drilling skills and other orthopedic surgical simulation skills training.

• Studies have found the use of augmented feedback increases performance during surgical simulation skills training but has either no effect, or a detrimental effect during retention tests assessing long-term learning.^[16][17][18] These findings can be attributed to learner attendance to easily available extrinsic feedback over intrinsic cues (attentional processing hypothesis) and learner dependence on augmented feedback (guidance hypothesis).^[18]
• A 2009 study found that although augmented feedback for plunge detection was beneficial to initial outcomes during the preliminary learning stages for bicortical drilling, the effect was transient and did not lead to significant long-term learning benefits.^[16] Learning retention tests showed that all novice groups learned equally well on how to prevent plunging, regardless of if they received augmented feedback (in the form of an auditory tone when plunging occurred) during bicortical drilling skills training. For bicortical drilling skills training, focusing on intrinsic cues (drilling by sound and feel of the drill bit) is essential for long-term learning outcomes to minimize plunge, and should not be overridden or distracted by augmented feedback.
• A 2007 study on computer-assisted total hip replacement simulation training found that augmented feedback improved performance during training but had no effect on long-term learning outcomes.^[18] These results are consistent with prior literature that indicated that the learning benefits ascribed to extrinsic feedback are often transitory, and do not translate into better performance outcomes when evaluated on retention tests.^[19]
• A 2005 dental study that used visual augmented feedback for basic cavity preparation tasks observed that augmented feedback was beneficial to performance during training but detrimental to performance on retention tests.^[17] This was attributed to feedback dependence, which can occur when feedback is provided too frequently causing the learner to perform poorly in its absence. A subsequent study found that the frequency of augmented feedback in training had no effect on learning retention.^[20]
• A 2005 dental surgical simulation training study added expert tutorial input to promote proper processing of feedback and decrease augmented feedback dependence.^[21] However, this strategy requires expert mentors and thus, cannot be used in our self-assessed surgical simulation training module.
• A 2008 study found that distracting noise impairs bicortical drilling performance and has been shown to adversely impact the plunge depths of intermediate residents and orthopedic surgeons.^[22]

Based on these published scientific findings, the likely best course of action is to remove augmented feedback for plunge detection entirely from our self-assessed Bicortical Drilling Skills training module to promote learner processing of intrinsic cues to minimize plunge, prevent learner dependence on augmented feedback, and optimize long-term learning and clinical outcomes for minimizing plunge.

On February 22, 2022, and April 6, 2022, we conducted further user testing with a total of 8 learners to evaluate the utility of the augmented feedback feature of the Tibial Shaft Simulator. We used a paired t-test to analyze our user testing data (n=8), we found no statistical difference (p = 0.42) in learner plunge depth values with or without augmented feedback.

Based on these user tester findings, we removed the augmented feedback feature from the Tibial Shaft Simulator toː

• reduce simulator costs by up to $51.47 USD, simplify the simulator build, and minimize simulator assembly time for the learner
• increase learner safety with the removal of the live electric circuit
• permit the learner to use any locally available powered surgical drill for bicortical drilling skills training
• simplify the bicortical drilling skills training steps of our self-assessed module
• prevent the development of learner dependence on augmented feedback to minimize plunge ("anti-skills"), and
• improve long-term learning and clinical outcomes for bicortical drilling.

We will still be including a clay backstop to permit the learner to measure plunge depth values which will provide targeted feedback to the learner during bicortical drilling skills training.

Our user testing in Nigeria showed that the transparent cellophane will permit the learner to visually inspect and confirm that the self-drilling Schanz screws did not perforate the far cortex of the Tibial Shaft Transverse Fracture Simulator for the self-assessment framework.

Based on our user testing in Nigeria during the Discovery and Finalist Award phases, we have decided not to use Far Cortex Breakthrough Detection and not to include the Plunge Depth Measurement for the Tibial Shaft Transverse Fracture Simulator. The rationale for these design choices is because:

• the likelihood of exiting the far cortex and plunging is very low with the manual advancement of the Schanz screw into the far cortex
• the transparent cellophane allows the learner to visually inspect and confirm that the self-drilling Schanz screws did not perforate the far cortex
• plunge detection has not been shown to confer long-term learning benefits in reducing plunge
• we want to avoid fostering learner dependence on augmented feedback ("anti-skills") since the learner will not have augmented feedback during the real procedure, and
• this increases simulator fidelity, reduces simulator costs, simplifies the simulator build, and minimizes simulator assembly time for the learner.^[16][17]

On October 18, 2021, we tested the NASA Task Load Index for Nigerian users of the module but observed that these survey's generic questions did not provide useful, targeted feedback to the learner above and beyond the module's existing self-assessment frameworks, and that learner compliance with completing the forms was poor. We removed the NASA Task Load Index from our self-assessment frameworks because it did not add any additional training value to the learner and this would reduce the administrative burden on the learner.

Acknowledgements[edit | edit source]

This work is funded by a grant from the Intuitive Foundation. Any research, findings, conclusions, or recommendations expressed in this work are those of the author(s), and not of the Intuitive Foundation.

References[edit | edit source]