What exactly is a summative evaluation or summative usability evaluation? How does it differ from a formative evaluation? How does the summative evaluation proceed and what do you have to consider? What exactly is evaluated?
If you have asked yourself one or more of these questions, then you will find the appropriate answers here. In this article, we will explain what is behind the term “Summative Evaluation”. You will also learn why it is so incredibly important for your medical device.
What is meant by the term “Summative Evaluation”?
IEC 62366-1 defines summative evaluation as follows: “The purpose of a summative evaluation is to evaluate the Usability of the User Interface as it relates to the successful completion of the tasks associated with the hazard-related use scenarios”.
The goal is a final evaluation of whether the medical device can be used by the users without unacceptable risks. In other words, you want to prove that your medical device is safe to use and will not cause harm to users, patients or bystanders. At the end of the summative evaluation, you have shown for which safety-related aspects of your product the overall risk has been reduced to an acceptable level and for which it has not. The determination of the level at which the overall risk is acceptable is determined by your risk management, e.g., by formulating acceptance criteria.
As a follow-up to the summative evaluation, you look at the results and check whether and which use errors or usage difficulties have occurred. If a use error or user difficulty can lead to a hazardous situation (i.e., it is a safety-critical use error), a root cause analysis must be performed. Root cause analysis is the analysis of the causes of an error.
Formative vs. Summative Evaluation: What is the difference?
Formative evaluations are development-accompanying evaluations of the usability of your product. The objective here is to iteratively improve the product by exploring its use and adding product requirements. Thus, through formative evaluations you test and improve the safety of use, the freedom from errors and the efficiency of the use of your product.
Summative evaluations are conclusive and evaluative. The objective here is to demonstrate that your product is safe to use and meets the acceptance criteria. A summative evaluation is mandatory for the fulfillment of the Usability Engineering process according to IEC 62366-1 and therefore important for your clearance according to MDR.
Why is the summative evaluation so important for your product?
Quite simply: because it is mandatory. Without a final usability evaluation, there will be no clearance for your medical device. Only if you have passed the summative evaluation, you have successfully proven that your product is safe for users, patients and third parties.
But regardless of this, at this point, as a manufacturer, you are once again ensuring that you are bringing a safe product to market and not putting users, patients and third parties at unnecessary risk. This proof can also be important in liability issues.
How does the summative evaluation proceed?
Normally, a final evaluation is a usability test with the product equivalent device. This means that all materials and device components must be available in pilot series. Testing is done with real users of the target group. The test must be carried out and documented by usability experts.
In a summative evaluation, a user is observed performing a work task with an interface or product. The work tasks correspond to the hazard-related use scenarios that were identified in advance and selected for the summative evaluation. In the execution, the focus here is on safe and efficient operation.
Since the use should take place in an environment that is as realistic as possible, this environment must be simulated. The acquisition of appropriate equipment should therefore be included in the planning in advance. It is also essential to simulate the use environment with all possible disruptive factors.
Use under realistic conditions also means that the users participating in the summative evaluation were trained to use the product in the same way that real users will be trained in the future.
Now follows the root cause analysis already mentioned. Here, all security-critical use errors or usage difficulties that have occurred are reviewed for their root causes. A subsequent evaluation in collaboration with risk management clarifies whether there are any unacceptable risks. If there are none, there is nothing to stop approval from the usability point of view.
What do you need to consider?
- Recruiting: the users you (get) recruited for your summative evaluation must be representative of the target group.
- Exclusion of hazards: Since the product you are testing is not yet approved, there must be no danger to users during the summative evaluation! If users are about to cause harm to themselves by using the product incorrectly, intervention must be provided at an early stage.
- Changes excluded: After the summative evaluation, nothing else may be changed in your product. Any change then means a new summative evaluation or a very good justification.
- The right preparation: Incorrect implementation results in the product not being cleared. This is followed by a new summative evaluation, which is time-consuming and costly. A compilation of the most important tips can be found in our “17 Dos & Don’ts for the execution of the summative usability evaluation“.
- Know the requirements of specific markets or particular products: While the IEC 62366 is globally valid, there are market-specific and product-specific requirements that must be considered in advance.
Conclusion
The summative evaluation is the final evaluation of a medical device. It is mandatory for the clearance of the product. Proper preparation and complete documentation are essential for the success of the undertaking. It is best to get a pragmatic and professional partner on board for the planning and implementation in order to get through the final usability evaluation of your medical device unerringly.
Learn how we help our clients successfully plan, execute, and document summative evaluations here.
Are you close to planning the summative evaluation? What are your specific obstacles? Feel free to comment on this post or get in touch via our contact form.
