Summative Usability Evaluation according to IVDR – Your Checklist

You are already planning your Summative Usability Evaluation according to IVDR and want to make sure you get it done without missing any important steps? Do you want an overview of the steps you need to successfully execute the Summative Evaluation for your IVD according to IVDR?

You may also be facing new challenges regarding the manufacture of your IVD as a result of the IVDR coming into force and want to complete the now indirectly required Usability Engineering with the Summative Usability Evaluation for the first time. Then this article is the right place for you, because here you will not only get a downloadable checklist, but also important explanations of the individual sub-items in this text.

But this article also offers value added for manufacturers with experience in the final Usability Evaluation: Here you get all important actions in the course of the Summative Usability Evaluation according to IVDR at a glance. In this way, you can also ensure a pragmatic procedure for future Summative Evaluations.

In this checklist you will find everything you need to consider before, during and after the Summative Usability Evaluation according to IVDR. We have clearly laid out the individual points for you and explained each one. At the end of this article you will find your “cheatsheet”, i.e. the checklist in pure form, without explanations for download. This way you can make sure that nothing is forgotten.

1: You are prepared properly?

Are you planning the right process?

If you are developing a new product for the market, you must go through the complete Usability Engineering process according to IEC 62366-1. For an existing product (legacy product), you can go through a much shortened process that only relates to the usability risk assessment of the product. The simplified process is called User Interface of Unknown Provenance. However, in this article we focus on the case that requires the whole process.

Have you found the right partner for the execution?

The Summative Usability Evaluation of your IVD is best approached with a partner who will help you reach your goal pragmatically and with whom the collaboration will be professional and pleasant.

Make sure that your partner can provide the necessary equipment and premises. It is also particularly important that he specializes in the medical market. Case studies and testimonials can give you information about the specialization of the service provider and the quality of the service.

A first meeting to get to know each other can give you a good impression of how the respective partner operates and whether it also fits interpersonally.

Have you defined all important factors around your IVD?

At this point, also check whether all risk-related aspects of your IVD have been clearly defined and named. If you have already gone through the entire Usability Engineering process according to IEC 62366-1, you have already defined the following factors of your IVD. Especially important for the Summative Evaluation according to IVDR are:

• The Use Specification including the correct user groups
• All characteristics of the user interface that are related to its safety
• The list of all potential usage errors
• You have defined the critical tasks
• We strongly recommend performing a use-related risk analysis. Here, it can help to have gone through formative usability test runs with your IVD. These in fact help to identify further Use Errors and thus Hazard-Related Use Scenarios!

Are there specific MDCG guidance documents for your IVD?

MDCG Guidance documents provide you with very clear implementation requirements for your specific IVD. All Summative Evaluation activities should therefore be preceded by a search to see if there is a Guidance document applicable to your case. A list of all published MDCG Guidance documents can be found here.

Do you need samples from real people for your Summative Evaluation, or can a simulated use take place?

IVDs that have not yet been cleared may not be used on real people. However, if a Summative Usability Evaluation is not possible without sampling real users, then you will need:

• Federal agency approval.
• The approval of an ethics committee.
• Patient insurance for each participating user.

All of this means a relatively large expense for you. Accordingly, it makes sense to consider whether a realistic evaluation of the usability of your product is possible without sampling, and if so, how.

2. Is your product ready for the Summative Evaluation?

Is your product availabe in a pilot series or as a Product Equivalent Device?

In the Summative Evaluation, you test with the Product Equivalent Device or the pilot series. Your product must therefore be available accordingly.

Are your IFUs and instructions available in final form?

The Instructions for Use, or IFUs, or instructions and training that your users need to use your IVD are also part of the Summative Evaluation. Accordingly, these are also tested. The IFUs, instructions, training, but also packaging and labeling are seen as part of your product.

Changes to these are not allowed after the product has been cleared until the new versions of the instructions also pass a summative study.

So, you should make sure that IFUs, instructions and training are completed and can also be understood by the user without errors. This is especially important for lay users of an IVD. This is where formative Usability Testing of the Instructions can help you.

3. Your product is prepared for the requirements of the Summative Evaluation?

Have you demonstrated that the user interface of your IVD is safe to use?

According to IEC 62366-1, the user interface includes the following elements:

“The USER INTERFACE includes all elements of the MEDICAL PRODUCT with which the USER interacts, including physical aspects of the MEDICAL PRODUCT, as well as visual, auditory, and tactile displays, and is not limited to software interfaces.” (source: IEC 62366-1:2021-08)

To know for sure that all these elements work without unacceptable risks, you should have already gone through formative usability testing.

Are the performance requirements set by the IVDR met?

Only when your IVD has no or only acceptable use-related risks have you met the performance requirements set out in Annex I of the IVDR. Again, formative testing can give you clarity on whether your IVD already meets all requirements.

You have demonstrated that your IFUs and instructions accompanying the product are easy to understand?

All information and instructions accompanying the product must be easily understood by the intended users and interpreted without error.

You have demonstrated safe and error-free use across all phases of use?

Your formative testing has shown that your user group can operate your product safely and without errors. With IVDs, this naturally includes interpreting the results in addition to the actual use.

You have mininized all use-related risks of your IVD?

The best way to do this is in iterative formative usability testing. Use-related risks include:

1. The IVDR requires that risks be reduced by ergonomic features:

“they reduce, as far as possible, the risks due to ergonomic characteristics of the device and the environment in which the device is intended to be used (patient safety oriented device design)” (source: IVDR:2017 – Annex I, Chapter 1, 5 a)

2. Reducing risks by taking into account the capabilities of the intended users:

“they take into account the technical knowledge, experience, education and training, and, where appropriate, the environment of use, as well as the health and physical condition of the intended users (product design oriented towards lay, professional, disabled or other users)” (source: IVDR:2017 – Annex I, Chapter 1, 5 b)

4. You have recruited the right users and enough of them?

The recruited users match your Intended User Group?

The users for your summative study must correspond to the Intended User Group, i.e. the user group you defined in advance. A guideline can help here, which screens the potential participants according to the right criteria during recruitment.

The recruited users are a representative sample of the respective group?

Here you should look for a realistic age representation (or representation of other characteristics). To be able to argue reasonably, there should be at least 15 people per user group. In some cases, however, more participants may be useful to be able to argue better.

You have considered to recruit more users than needed?

Plan for sufficient over-recruitment to compensate for potential participant dropouts. Otherwise, lengthy re-recruitment may occur, delaying completion of the summative study.

5. You have prepared the Usability-labs properly?

The right equipment is available and prepared?

Check in advance whether the equipment for the Summative Evaluation is available. If sampling can be simulated, you may need a suitable test dummy for your specific case.

Are the right premises available for the tests?

Have you already found the right Usability labs? Or have you already rented the right premises, if applicable? When selecting the premises, make sure that they match the later conditions as closely as possible.

The premises must allow you to map the later use environment of your IVD with all important external factors. Is the place noisy or hot or particularly stressful? Then you must pay particular attention to this.

You have checked the completeness of the equipment for the recording and streaming?

The purpose of a Summative Evaluation is to provide objective evidence that there are no unacceptable risks associated with the use of your product. The easiest way to provide this objective proof is by observing the actual execution of actions.

For the proof, the records of the tests are very important for your Summative Usability Evaluation. Therefore, check that you have enough cameras and SD cards. Spare batteries as well as spare storage media are also very important. You want to stream the test live? Then you should also check the equipment for this and buy any missing equipment.

You have chosen the right location for the right authority?

The FDA’s Human Factors Engineering process requires that testing for the Summative Usability Evaluation takes place in the USA.

If you want your IVD to be cleared not only under IVDR, but also under FDA, then you should test in the USA. Your Summative Evaluation can then also be cleared in Europe. Unfortunately, this is not possible the other way around.

6. If necessary: Have you properly trained your users?

You have trained according to your IFUs and trainings?

For the training of your users, you may only use the material that will also be available after clearance of your IVD.

You have thought of a realistic time interval between training and use?

After training, users start to forget content. It is required that you maintain a realistic time gap between training and use. Realistic in this context means: As the gap would probably be in real life. 

7. Do you have the appropriate personnel to perform the Summative Usability Evaluation?

The people conducting the evaluation are not involved in the development of your product?

The tests must not be performed by anyone who has been involved in the development.

You have sufficient usability researchers for your study?

It is important that the study is conducted by trained usability researchers to avoid bias in the results. Look for proof of qualification in the medical field for Usability Engineering in the researchers. You have enough of them for your study? Very good.

You have designated a „person responsible for regulatory compliance”?

Article 15 of the IVDR sets criteria for „at least one person responsible for regulatory compliance who possesses the requisite expertise in the field of in vitro diagnostic medical devices. The requisite expertise shall be demonstrated by either of the following qualifications:“. (IVDR-2017: Article 15 a))

8. The execution of the Summative Evaluation is in accordance with the IVDR?

You have created realistic conditions of use?

As mentioned in subpoint 5, you need to create a realistic use environment in which to test.

You avoid user questions during the implementation?

Realistic terms of use in this case also means use without personal instructions and assistance.

The safety-critical use errors (Hazard Related Use Scenarios) are tested?

Remember the safety-critical usage errors from subpoint 1? These will be tested in the Summative Evaluation. Make sure to map them completely or according to your chosen rationale in your Usability Engineering File for the tests.

For self-use IVDs: you have demonstrated that users can interpret all the results of your IVD without error?

For this, you can prepare cards that show test results of your IVD. The users can then interpret these in the Summative Evaluation. If the interpretations of the lay users coincide with those of the experts, you have proven that lay users can also interpret the results of your IVD without error.

You consider that Knowledge Tasks have to be queried as well?

During your Summative Evaluation, you will not only test users’ use of your IVD. You must also test whether the users have understood the information and instructions for action in the IFUs. Is your IVD only allowed to be used up to a certain room temperature? Then you could have the users answer a question about this. This must then be answered with the help of the IFUs.

You think about the post-hoc interviews, i.e. the follow-up surveys after use?

These follow-up surveys on Use Errors are particularly important because they form the basis for Root Cause Analysis, which becomes especially important in your follow-up to the study.

9. You have done the follow-up and data evaluation?

You have collected the data of your Summative Usability Evaluation and would like to analyze it?

Your goal is to show that using your IVD only comes with an acceptable level of risk. So you look at errors and use difficulties made during the study. You are looking at close calls, use difficulties, use errors, and misuses.

You go through the data participant by participant and document the events that occurred. At the end, you evaluate across all participants how many participants were able to complete a particular usage step without an error.

After your evaluation, you have clear percentages that show which proportion of users were able to perform which usage steps correctly! 

You have done a Root Cause Analysis?

This is where the data from your post-hoc interviews mentioned in point 8 come into play. There must be no misuse and no wrong usage step that can lead to a Hazardous Situation, the reasons for which you cannot explain with the help of Root Cause Analysis.

You have created a Summative Report?

The summative report contains the results of your study. It also lists the use errors and the severity of their consequences.

You have checked your evaluation for completeness?

You are sure that you have not forgotten any data?

You submit the report on your Summative Usability Evaluation together with the Usability Engineering File describing the complete Usability Engineering process.

You have demonstrated the safe use of your IVD?

Your product has clearly demonstrated in the Summative Evaluation that the criteria for safe use of your IVD addressed in point 1 could be met? Congratulations, you have proven that the usability of your product does not pose any unacceptable risks for the intended user group!

The Checklist for download

Here you can download our checklist again without the explanations. As a practical PDF.

Conclusion

With the right planning and approach, the Summative Evaluation according to the requirements of the IVDR can be conquered. Keep this checklist handy at all times and move towards your goal as efficiently as possible, step by step.

Do you still have questions? We will be happy to help you prepare your IVD for the Summative Evaluation and then execute it together. We’ll be happy to take the worry about compliance off your shoulders. Here you can find out how a Summative Usability Evaluation works with us.

To look at your specific case and meet your individual challenges, we offer a free strategy meeting.

We look forward to hearing from you. Simply use our contact form to get in touch with us.