A successful Human Factors Validation, also known as a Human Factors Validation Study, is an essential step for most medical devices on the way to approval according to the guidelines of the FDA (U.S. Food and Drug Administration).
To ensure that this process runs smoothly, familiarize yourself with all the necessary components at an early stage.
In the first part of this series of articles, we discussed the FDA’s guidance document “Applying Human Factors and Usability Engineering to Medical Devices” in detail and provided valuable tips.
You are now in the second part of our article series, in which we summarize the key aspects of human factors validation in detail.
If you still need to read the first part, we recommend that you do so to gain a comprehensive understanding of the topic. You can find the article following this link.
We wish you an informative read!
And how do you implement everything? Assistance: Applying Human Factors
To implement the FDA requirements in a targeted manner, the FDA repeatedly provides guidance documents that provide essential assistance and create a better understanding of how implementation can succeed.
One particularly important aid is „Applying Human Factors and Usability Engineering to Medical Devices”. In Chapter 8 you will find all the notes on Human Factors Validation Testing.
We have summarized the other contents below. The contents of chapters 8.1 to 8.1.4 can be found in the first article in this series. Here we start with chapter 8.1.5 of the guidance document: Data collection.
Data collection:
In the human factors validation test, observable data and knowledge and subjective experiences of the test participants should be recorded.
Observable data: This includes participants’ performance on critical tasks and scenarios, any areas of use and errors that occur, and “near misses” where participants almost made mistakes but were able to correct them in time. This data is recorded during the test and discussed later in participant interviews.
Knowledge task data: Since not all critical tasks can be assessed by observation (for example, understanding contraindications or warnings), these should be collected by interviewing participants. The questions should be open and neutrally formulated.
Interview data: Post-test interviews should be conducted with participants to capture their subjective assessments and perspectives. These can provide information about usage difficulties, comprehension problems, and other relevant aspects that could have been more obvious during the test. The questions should be open and neutral, covering both general impressions and specific experiences of the critical tasks or scenarios.
In summary, data collection in the Human Factors Validation Test should include observations of participant performance as well as direct questioning and interviews to obtain a comprehensive picture of the usability and safety of the device.
(Source: Applying Human Factors and Usability Engineering to Medical Devices, chapter 8.1.5)
Evaluation of the data:
The analysis of the results from the human factors validation tests should be qualitative to evaluate whether adjustments to the device design, labeling, or user training are necessary to reduce the risks associated with using the device to an acceptable level.
To do this, it is necessary to bundle the observation and knowledge data collected during the test with the interview data and evaluate it thoroughly. The aim is to identify the roots of any usage errors or difficulties.
The identified causes must then be analyzed about the associated risks to determine the potential for possible damage and the urgency of further risk management measures.
Depending on the extent of the risk management strategies implemented, it may be necessary to repeat the tests. Areas of the user interface that have led to usage errors or critical problems should be addressed through targeted risk management measures.
It may be helpful to conduct additional preliminary analysis and evaluation to refine and finalize the modifications.
A further human factors validation test should then be performed on the modified user interface elements to verify the success of the risk management measures in reducing risk and to ensure that no new unacceptable risks have arisen.
(Source: Applying Human Factors and Usability Engineering to Medical Devices, chapter 8.1.6)
Assessment of the residual risk:
A residual risk remains with every device, even with an optimally designed user interface. After the human factors validation, all remaining risks must be analyzed in detail to check whether they can be minimized or eliminated.
Real residual risks can no longer be practicably reduced or eliminated. Results that show serious usage errors are only acceptable in pre-market submissions if it is demonstrated that further reduction is not possible. The benefits of the device outweigh the remaining risks.
The analysis should clarify how the errors or problems related to the user interface arose and whether design changes would be possible and practical.
Serious usage errors and problems with high residual risk must be described in the human factors validation report.
Plans to fix known design flaws in future device versions are not acceptable. The discovery of serious errors could also indicate inadequate analysis and modification of the user interface during the development phase.
(Source: Applying Human Factors and Usability Engineering to Medical Devices, chapter 8.1.7)
Human Factors Validation Testing for medical devices that are already on the market but have been adapted
When reviewing human factors for modified devices already on the market, the risk analysis must include all modified aspects of the device and all affected elements, including user interactions that the modifications have directly or indirectly influenced.
Depending on the results of the risk analysis, additional human factors validation may be necessary. Testing should focus on hazardous use scenarios and critical tasks but may be limited to the areas affected by the design changes.
Suppose a device is modified in response to use-related problems. In that case, human factors validation should evaluate the revised design and ask users how they compare the new design to the old one. This serves to assess the effectiveness of the changes better and to identify further potential difficulties or necessary modifications.
However, the evaluation is more effective if it includes the user’s comparison between the modified and the previous design. The test leaders should explain the known problems and then show the participant the previous and the new or modified version of the user interface.
Participants should then answer questions such as the following:
- “Do you think the new design is better than the old one? Please tell me how the new one is [better/worse] than the old one.”
- “How effective do you think these changes will be in preventing the user error from occurring? Please tell me why you think it will [work/not work].”
- “Could these changes cause any other usage difficulties? What types of difficulties?”
- “Are these changes sufficient or is further modification needed? How should it be modified?”
(Source: Applying Human Factors and Usability Engineering to Medical Devices, chapter 8.2)
Testing through real use:
Real-use tests are necessary for devices whose use or environment is particularly complex and which cannot be fully reproduced in simulated tests. Examples are prostheses or programming devices for hearing aids.
Such tests should follow prior simulated use tests to ensure basic safety and correct design of the device. They follow the same guidelines as simulated human factors tests, but may require an Investigational Device Exemption (IDE) if they are needed for the safety and efficacy evaluation of the device. You can find out what this looks like in detail on the FDA’s own website.
Again, test participants and environments should be representative, and testing should be conducted in a way that minimizes user interaction with the device.
Real-use testing can also be part of clinical studies, but the conditions and monitoring levels in such studies may differ from those in real-world use.
User self-reports should, where possible, be supplemented by observational data to provide a complete picture of device use. Consultation with internal ethics committees is necessary to establish specific security and privacy measures for trial participants.
(Source: Applying Human Factors and Usability Engineering to Medical Devices, chapter 8.3)
The correct documentation of your human factors validation:
Find out what belongs in complete documentation, how it is checked by the FDA and how you can efficiently meet the documentation requirements for your human factors engineering in our article „Deep Dive: Human Factors Report.“
Conclusion
We summarize:
Careful collection of observable data, knowledge task data, and interview data provides an in-depth understanding of user interaction and associated risks.
Qualitative analysis of this data is crucial to make necessary adjustments in design, labeling, or user training and reduce risks to an acceptable level.
For devices already on the market but have been modified, a new risk analysis and possibly further human factors validation is required, focusing on the areas affected by the design changes.
Real-use tests are necessary for complex application areas and should follow previous simulated tests. Proper and comprehensive documentation of all tests and analyses is essential as it is central to FDA review.
Do you need help with the planning, implementation, and documentation of your human factors validation? Please get in touch using our contact form. We look forward to hearing from you.
