With the summative evaluationwhich is the final evaluation, you prove that the interface of your medical device can be used without critical risks. Everything has to be right on this point. You have already completed several formative evaluations and now feel ready to consider the summative evaluation? Or perhaps you are simply planning ahead and want to gather all the information for the summative evaluation, so that you can get started immediately when your product is ready?
Immediately, you think of a complete list of questions for conducting the summative evaluation. After all, you want to get the best possible test result and successfully hold your clearance in your hands.
In practical use, a lot of additional questions arise regarding the interpretation of IEC 62366-1 and its “translation” to one’s own product.
Incorrect execution of the summative evaluation can result in the product not being approved. This quickly results in costs due to delays and a renewed execution of the evaluation. We provide you with our expertise so that you can avoid these costly mistakes.
In this article, you will find 17 do’s and don’ts that will be of great help to you in conducting the summative evaluation properly and help you to avoid mistakes. This way you can make sure to achieve a better result and secure the clearance.
First of all: your mindset.
Your mindset for summative evaluations should be quite different from that for formative evaluations. With formative evaluations, you pursue the goal of making your product safer and better. They accompany the development. With the summative evaluation you show that your product is safe and ready for the market. This happens at the end of development. The summative evaluation is conclusive and evaluative. Accordingly, you should prepare sufficiently before carrying it out.
Our Do’s and Dont’s give you clear instructions on what to do and what to avoid. It also clarifies what you may and may not do according to IEC 62366-1.
So let’s start with the Do’s and Dont’s.
1: Start to plan as soon as possible.
It is important that you approach summative evaluation with the right plan. You should determine the following things. Remember to document everything well from the beginning.
- Specify the methods you will use to test the safety of your product. You must also document the proper rationale for these methods directly.
- Special focus should be placed on evaluating usage errors.
- What is a realistic usage environment, and how can it be maintained for testing?
The IEC requires complete documentation of your procedure. The User Interface Evaluation Plan is the document in which you record all relevant things. It is the summary of how and what you want to test to prove that your product is really safe to use. Once you have created the evaluation plan, you should gradually ensure that your product delivers the desired results in the tests.
If the results are still not satisfactory, you should eliminate the errors before the summative evaluation. You must also make sure that the correct scenarios that can lead to critical error sources, i.e., the hazard-related-use scenarios, are included in the evaluation plan. This is where experts can help you, should you be unsure.
If you deliver a strong concept with sensible test methods and test cases, then this creates the basis for clearance. A weak concept, on the other hand, guarantees you lengthy discussions with the Notified Body.
So, with the right evaluation plan, you can ensure in advance that you can enter the summative evaluation as confidently as possible.
2: Have the laws of your specific market in mind.
Although IEC 62366-1 is globally harmonized to the respective standards of different markets, it can happen that the American market, for example, has completely different requirements for the clearance of a medical device at certain points than the European market. Here, you need to know in advance where the product needs to be approved and what there may be to consider. The help of an agency that is familiar with all the standards can help to keep track and meet all the requirements.
3: Document properly and carefully from the start.
Everything that is not documented, did not happen. This may sound harsh at first, but it makes sense. From the very beginning of your planning, remember to document your methods, your formative evaluations, and errors found. Not only will this make it easier for you to analyze errors and root causes, it is also essential for the evidence you will need to provide during the summative evaluation.
4: Have a clear understanding, of what data is really needed.
For the summative evaluation, objective but also subjective data counts. Thus, data is collected on the participants’ subjective assessment of the product, but also objective data on the question “Is the product also safe and easy to use?”.
What is a better way to collect this data than a usability test? In most cases, summative evaluation is exactly that.
5: Define hazard-related-use scenarios based on risk management.
Hazard-Related-Use Scenarios are the usage scenarios that can, directly or indirectly, lead to a critical situation.
Your Hazard-Related-Use scenarios determine the cases for which your product will be tested in the summative evaluation. Thus, they take up a very important part of the planning.
They should be defined in close coordination with risk management. And this is best done right from the start! In this way, you avoid discovering critical errors that could cost you clearance too late in the process.
Keep in mind that you have to integrate the scenarios as a single document into your usability engineering file. Here, too, experts can help you.
6: Keep in mind that product training must also be tested.
If your product requires training, this must also be included in the summative evaluation. Therefore, test runs should also take place for the training before it is included in the summative evaluation.
For the execution of the summative evaluation, this means that you have to invite the target group and train them in the same way, as it would happen after the clearance of your product.
Also, be sure to demonstrate a realistic amount of time between training and performing the activity. The summative evaluation should take place under realistic conditions. Accordingly, it would be unrealistic to train the user every time immediately before use.
7: Instructions for use must also be tested.
All Instructions for Use are considered part of the user interface and must therefore be tested. Instructions for Use are the information that you must provide, so that the user can use your product in a goal-oriented and safe manner.
Instruction manuals, package inserts, and safety warnings are thus also evaluated. These should also be ready to stand up to the requirements of the summative evaluation.
8: Recruit the right target group.
To get valid test results, you need to test with the right target group. There is no way around this. The trick is to distinguish the target groups according to relevant and essential characteristics.
The more specific your product is, the more specific the target group may be. The users must always match the defined purpose of your product. If you test with the wrong target group, then it can jeopardize your clearance. This is not only true for summative evaluation. The fit of the target group is already particularly important for the formative evaluations, since these form the basis for the product development and for your argumentation towards the Notified Body.
9: Recruit a sufficient number of participants.
If you test with a too small number of people, you will not get significant results. The results become more volatile as the test group becomes smaller and smaller. This weakens your argumentation basis towards the Notified Body.
To have a solid basis here, you should test with at least 15 people per target group. (source)
10: Test with the Production Equivalent Device.
In the summative evaluation, the Production Equivalent Device is tested. This means that all materials and device components must be available in pilot series. If your Production Equivalent Device does not pass the test, this can lead to major delays and high costs.
Therefore, if possible, you should run a formative evaluation with the Production Equivalent Device in advance to be sure that it will also withstand the test conditions of the summative evaluation.
You should also be aware that after the summative evaluation, nothing more can be changed in your product. Every change then means a new summative evaluation.
11: Test with the right equipment.
The summative evaluation is concerned with the handling and use of the product. An unapproved medical device must not be used on patients. Accordingly, you must test with the right equipment to simulate use on the patient.
Invasive products, for example, must be tested on dummies. This and other equipment can quickly become very price-intensive. In terms of cost, it can be more attractive to look for a specialized agency, that can provide the equipment you need for your tests. Search for an agency that already owns, maintains and cares for the equipment that is necessary to conduct your tests.
12: Provide a real environment with real sources of interference.
For the summative evaluation, your product should be tested under conditions that are as close to reality as possible. Sources of interference and interference with other devices must also be taken into account here. Take these into consideration during the execution and prepare your product and test setup accordingly.
13: Have the summative evaluation conducted by appropriate individuals.
The summative evaluation may only be carried out by suitable persons. Persons involved in the development are excluded here.
The more inexperienced with the product and the less in-house the suitable person is, the easier it will be for you to argue your case to the Notified Body.
What criteria must appropriate people meet?
- You are part of the appropriate user group.
- In other words, they have the right qualifications, the right profession, the right age, and all the skills and abilities that the target group must have.
- In the case of patients, a specific clinical picture may also be a prerequisite for being considered appropriate person.
14: Do not ask questions during the summative evaluation.
The summative evaluation should take place under conditions that are as realistic as possible. Accordingly, you are not allowed to intervene, ask questions or even help during the test.
Even if you are not allowed to intervene during the summative evaluation, follow-up questions about possible errors are very important.
15: Avoid disturbing the realistic test conditions.
In formative evaluations, it can be helpful to ask the user to speak their thoughts aloud while using the product. This Thinking Aloud method is not permitted in summative evaluations. Again, it is important not to disturb the most realistic conditions possible. So what brings to light important sources of error during formative evaluations as a common UX method, is not allowed in the final evaluation.
16: Consider the follow-up surveys.
At the end of the evaluation, there is a root cause analysis of the mistakes made. Follow-up surveys are very important here. Participants are asked about their use of the product. The aim is to find out whether their expectations were met and how satisfactory the use was.
117: In the end, the results decide whether clearance is granted. If necessary, you will have to adapt your product further.
Whether a product is safe is judged based on the results of the summative evaluation.
Depending on the assessment of the results, you may have to adapt your product further until it is safe for use. So you should ensure up front that no more critical errors occur to avoid the cost of repeated summative evaluation.
Professionally conducted formative evaluations can make your life much easier in this regard. They are the best way to uncover critical errors at a stage of your product when it is relatively inexpensive to fix them. Formative evaluations should have a correspondingly high priority for you.
Are you missing the acceptance criteria?
Acceptance criteria clarify the question of when an use error is okay and when it is no longer acceptable. According to the current status of IEC 62366-1, the proof of acceptance criteria is no longer necessary. We will discuss what exactly has changed in the standard in a separate article.
Conclusion
Now you have an overview of the do’s and don’ts for summative evaluations. You know, how you can improve your results in a targeted way and be prepared for the requirements for clearance in the best way possible. Of course, your specific case may need a few more special aspects to consider.
We are happy to help you prepare your product for summative evaluation and relieve you from worrying about compliance with the requirements.
To have a look at your specific case and meet your individual challenges, we offer a free strategy meeting.
We look forward to hearing from you. Simply use our contact form to get in touch with us.
