The successful summative evaluation of medical devices is crucial for approval. However, inadequate planning, unclear execution and poor follow-up can often lead to costly delays.
In this article, we present 10 best practices to help you successfully and efficiently plan, conduct and complete a summative evaluation for medical devices. By applying these best practices, you can not only save time and resources that would be incurred if approval is not obtained, but also ensure that your products meet the highest standards and ensure patient safety.
Enjoy reading!
1. Choose the right process
We strongly recommend using Usability Engineering according to the IEC 62366-1 standard or the Human Factors Engineering process as the right process for all steps before, during and after the summative evaluation of medical devices. These standards provide a comprehensive framework for the systematic integration of usability principles throughout the medical device development process. Compliance with this standard establishes best practices and methods for analyzing, designing and evaluating the usability of medical devices.
A structured approach according to IEC 62366-1 ensures compliance with regulatory requirements and contributes to the successful approval and market launch of medical devices and saves you some discussions with regulatory bodies.
2. Plan thoroughly
Thorough planning is critical to the success of your summative evaluation. Important considerations include creating a realistic timeline that allows sufficient time for participant recruitment, data collection, analysis and reporting. It is important to note that recruiting selected user groups can be time consuming.
Budget constraints must also be considered to ensure that all aspects of the study are financially feasible.
A well-thought-out plan considers potential barriers and includes strategies to overcome them.
3. Prepare yourself properly: Iterative tests and analyses
Performing iterative (“formative”) testing and analysis makes it possible to use findings from previous test phases to continuously improve designs and concepts. After each round of testing, data is carefully analyzed and interpreted to identify areas for improvement.
Adjustments are then made to the product or system and tested again (until no more critical operating errors occur).
Although these formative tests are not mandatory, we have yet to see a medical device pass a final usability test without preparation through iterative testing.
4. Recruit the right target group
The selection of participants is a crucial step to ensure that they are an accurate reflection of your target audience for the product or system being tested. This is a critical step for approval. If you do not have an accurate reflection of the target group, you will not get approval.
It is important that your user groups are precisely defined in their essential characteristics. For broad user groups such as lay users, it is advisable to include a balanced mix of characteristics such as age, gender, experience level and other demographic factors. This ensures that the results of the study are representative and can be applied to the population as a whole.
In addition, it is important to consider factors such as the availability and willingness of participants to participate.
5. Respect ethical principles
Respecting ethical principles and adhering to informed consent is crucial and contributes to moral responsibility, especially when conducting summative evaluations.
It is of paramount importance to protect the privacy of participants, ensure their voluntariness and obtain informed consent.
Participants should be fully informed about the purpose of the study, their role in it, possible risks and the use of their data.
In addition, the confidentiality of their personal information should be guaranteed and the possibility of withdrawing from the study should be offered without negative consequences.
Compliance with these ethical standards is not only required by law, but also promotes the trust and cooperation of the participants.
6. Make sure you use the right scenarios
In the summative evaluation of your product, you should consider the following: The selection of scenarios to be tested can either include all hazard-related scenarios or make a reasoned selection based on the severity of potential harm and other circumstances specific to the medical device and the manufacturer (see IEC 62366-1:2021-08, Chapter 5.5).
Also remember to properly document not only the selection of scenarios for summative evaluation, but also the selection scheme chosen and a sufficient justification for the choice of the respective scheme. If you have not properly justified your choice, the approval authority will refuse you approval.
7. Pay attention to realistic test environments
The use of realistic test environments helps to increase the validity and reliability of the results by conducting the study in an environment that is as close as possible to the actual usage situation.
For medical device tests in particular, it is advisable to conduct them in a realistic environment to simulate the real working conditions of medical personnel. By using realistic environments, authentic user behavior and interactions can be captured, leading to more relevant and applicable findings.
It is also important to realistically represent background noise and other potential sources of interference. In some cases, such as simulating an operating room, it may be necessary for a well-equipped medical usability and UX agency to provide the necessary equipment and space within your budget.
8. Ensure the quality of the data
Ensuring data quality is the basis for reliable results in studies. Therefore, it is crucial to take measures to minimize bias and errors during data collection.
This can be done by using standardized test procedures, training the researchers and carefully selecting the measurement instruments. In addition, it is important that data is systematically collected, stored and analyzed to ensure consistent and reproducible results. This step is also critical for approval.
9. Pay attention to the correct documentation
Comprehensive and detailed documentation of the research process and results is crucial to ensure transparency and traceability.
This includes recording the study methodology, participant data, testing procedures used and data collected.
Solid documentation enables others to review, replicate or build on the study. It contributes significantly to the transparency and credibility of the study and is an essential component for the publication and presentation of results, especially to regulatory authorities such as Notified Bodies or the FDA who are responsible for your approval.
10. Perform a root cause analysis
An essential best practice for summative usability evaluations of medical devices is the implementation of a Root Cause Analysis (RCA). This method makes it possible to determine the causes of identified problems or shortcomings during the evaluation. By looking beyond the superficial symptoms, the underlying causes that led to the problems can be identified.
Conclusion
By integrating these best practices, summative evaluations for medical devices can be carried out even more effectively, drastically increasing your chances of a successful approval.
You now know how to best prepare for a summative evaluation. If you are faced with the question of whether you should conduct the summative evaluation internally or with an external partner, we recommend our article “Summative evaluation for medical devices – what are the advantages and disadvantages of conducting it internally or externally?“.
Do you need help with the planning, implementation and documentation of your summative evaluation? Please get in touch using our contact form. We look forward to hearing from you.
