Summative usability evaluation – your checklist for preparation

Are you responsible for the approval of a medical device? Then you cannot avoid the summative evaluation. With this article you are well prepared.

In this article and the attached checklist, you will find everything you need to consider before, during and after the summative usability evaluation. We have explained the individual points for you. At the end of this article you will find your “cheat sheet”, i.e. the checklist in its pure form, without explanations for download. This way you can ensure that nothing is forgotten.

Have fun reading!

The correct preparation

Are you planning the right process?  

If you are developing a new product for the EU market, you must go through the complete usability engineering process in accordance with IEC 62366-1. For an existing product (legacy device), you can go through the much shorter process, which only relates to the risk assessment of the usability of the product, the User Interface of Unknown Provenance process. The right partner for the implementation should already have a fixed process in place to guide you through the summative evaluation.

Have you found the right partner for the implementation?

For a comprehensive evaluation of the user-friendliness of your medical device, we recommend working with a partner. A partner who supports you effectively and ensures a professional and pleasant collaboration is crucial and saves time and costs. Make sure that the partner has the necessary equipment and appropriate premises. Specialization in the medical field is of particular importance. Case studies and testimonials can provide information about the expertise and quality of the service.

An initial meeting allows you to get to know the partner better and determine whether the working methods and interpersonal chemistry are a good fit.

The right partner will also help you to tick off most of the points on this checklist, either partially or completely!

Have you defined all the important factors relating to your medical device?

At this point, also check whether all risk-related aspects of your product have been clearly defined and named. If your product has gone through the entire usability engineering process in accordance with IEC 62366-1, you have already defined the following factors.  

The following are particularly important for summative evaluation:

• The use specification including the correct user groups
• All characteristics of the user interface related to its security
• The list of all potential usage errors
• You have defined the critical tasks.
• We strongly recommend carrying out a use-related risk analysis. It can help to have carried out formative usability test runs with your medical device. These help to identify further use errors and therefore hazard-related use scenarios! If you do not identify these and they then occur for the first time in the summative evaluation itself, your approval is seriously at risk.

Are there specific MDCG Guidance documents for your product?

MDCG Guidance Documents provide you with very clear implementation requirements for your specific medical device or IVD. All summative evaluation activities should therefore be preceded by a search to see whether there is a guidance document that applies to your case. A list of all published MDCG Guidance Documents can be found here.

Is your product prepared for the requirements of summative evaluation?

Have you ensured that the user interface is secure?

To be sure that all these elements work without unacceptable risks, you should have already run formative usability tests. From our experience, you should run at least 1 to 3 formative tests with your product. We have never seen a product pass the summative evaluation without previous formative testing.  

Are the performance requirements defined by the MDR fulfilled?

Compliance with the performance requirements according to Annex I of the MDR is only guaranteed if your medical device has no or only acceptable use-related risks. Formative testing can help you determine whether your device already meets all the necessary requirements.

Have you ensured that your IFUs and instructions accompanying the product are easy to understand?

All information and instructions supplied with the product must be easy for the intended users to understand and interpret without errors.  

Have you ensured safe and error-free use across all operating phases?

Your formative testing has shown that your user group can operate your product safely and without errors.  

Have you minimized all use-related risks of your medical device?

Again, the best way to do this is through iterative formative usability tests. Usability-related risks include:

1. Reducing risks through ergonomic features
2. Reducing risks by considering the capabilities of the intended users

Have you recruited the right users (and enough of them)?

Do the selected users correspond to your intended user group?

The users for your summative study must correspond to the intended user group you have defined in advance. A guide can help you screen potential participants based on the right criteria.

Are the selected users representative of their respective group?

It is important to ensure a realistic age distribution (or distribution of other relevant characteristics). At least 15 people per user group should be included to enable sound reasoning. In some cases, however, more participants may be useful to create a stronger argument.

Have you considered over-recruitment?

Plan for sufficient over-recruitment to compensate for possible participant absences. Otherwise, delays in completing the summative study could occur due to subsequent recruitment.

Have you prepared the usability labs correctly?

Is the necessary equipment available and ready for use?

Before starting the summative evaluation, it is important to check whether the required equipment is available. It may be necessary to procure special devices or provide suitable test dummies for your specific use case.

Are suitable premises available for the tests?

Are suitable UX labs available or have suitable premises been rented? When selecting the test environment, it is important to ensure that it corresponds as closely as possible to the subsequent conditions of use. The premises should make it possible to reproduce all relevant external influences of the later usage environment of your medical device or IVD, such as noise, heat or stress.

Has the completeness of the equipment for recording and streaming been checked?

The summative evaluation aims to objectively demonstrate that the use of your product does not pose any unacceptable risks. This evidence is best provided by observing actual actions. Recordings of the tests are more than helpful for a successful evaluation. Sufficient cameras, SD cards, spare batteries and storage media should therefore be available. If live streaming is planned, the appropriate equipment must also be checked and supplemented if necessary.

Has the right location been selected for the relevant authority?

For the FDA’s human factors engineering process, the tests for human factors validation, i.e. the equivalent of the summative usability evaluation, must be carried out in the USA. If approval according to both MDR and FDA is sought, the tests should be carried out in the USA, as a summative evaluation carried out in the USA can also be recognized in Europe. Unfortunately, the reverse is not possible.

If necessary: Have you trained your users properly?

Have you trained according to your IFUs and training courses?

For the training of your users, you may only use the material that will also be available after approval of your medical device or IVD. In addition, you may only present the IFUs and training courses as they will be used later.

Have you considered a realistic time interval between training and use?

After a training course, users tend to forget content. It is recommended to keep a realistic time gap between the training and the actual use, which corresponds to the probable time gap in practical use.

Do you have the appropriate personnel to carry out the summative usability evaluation?

The persons carrying out the tests are not involved in the development of your product?

The tests must not be carried out by persons who were involved in the development. Failure to do so will result in lengthy discussions with the Notified Body, which will jeopardize or delay approval.

Do you have enough qualified usability researchers for your study?

Conducting the study by qualified usability researchers is crucial to avoid bias in the results. Make sure that the study investigators can demonstrate that they have the necessary qualifications in medical usability engineering.  

Have you appointed a “person responsible for regulatory compliance”?  

Article 15 of the MDR sets out criteria for a “person responsible for regulatory compliance”. You must appoint such a person in your company and ensure that the requirements of the MDR are complied with here. Among other tasks, this person is responsible for “ensuring that the conformity of devices is adequately checked in accordance with the quality management system under which the devices are manufactured before a device is released”. (MDR-2017: Article 15 a))

Is your product ready for summative evaluation?

Is your product available in pilot series or as a product equivalent device?

In the summative evaluation, you test with the Product Equivalent Device or the pilot series. Your product must therefore be available accordingly.

Are your IFUs and instructions available in final form?

The Instructions for Use as well as instructions and training for the use of your IVD are essential components of the summative evaluation. They are therefore also tested. IFUs, instructions, training, packaging and labeling are considered integral parts of your product.

It is important to ensure that IFUs, instructions and training are complete and understandable, especially for non-experts using the product. Formative usability testing of the instructions can be helpful to ensure that they can be understood without errors.

Is the summative evaluation carried out in accordance with the MDR?

Have you created appropriate conditions of use?

As mentioned above, it is important to create a realistic use environment for the tests.

Do you avoid asking your own questions during the implementation?

In this case, realistic conditions of use mean that use takes place without personal instructions or assistance.

Are safety-critical usage errors (hazard-related usage scenarios) tested?

Think about the safety-critical usage errors mentioned in a previous sub-item. These should be tested in the summative evaluation. Make sure that these are considered for testing in full or according to your documentation in the Usability Engineering File.

Have you also considered knowledge tasks?

During the summative evaluation, not only the use of the product by the users should be tested, but also their understanding of the information and instructions in the IFUs. For example, if your product can only be used at certain room temperatures, you could ask users about this and evaluate their answers based on the IFUs.

Are you remembering the post-hoc interviews, i.e. the follow-up surveys after use?

These interviews on use errors are of particular importance as they form the basis for the root cause analysis, which is important during the follow-up of the study.

Have you completed the follow-up and data analysis?

You have the data collected from your summative usability evaluation and would like to analyze it?

Your aim is to demonstrate that your product can be used without unacceptable risk. You therefore focus on the errors and usage difficulties that occurred during the study, including close calls, use difficulties, use errors and misuses.

You go through the data participant by participant and document the events that occurred. You then evaluate across all participants how many of them were able to perform a specific usage step without errors. Based on this evaluation, you receive clear percentages that show what proportion of users were able to perform which usage steps correctly.

Have you carried out a root cause analysis?

Here you use the data from your post-hoc interviews. There should be no misuse or incorrect usage step that could lead to a dangerous situation without you being able to explain the causes with the help of root cause analysis.

Have you created a summative report?

The Summative Report summarizes the results of your study, including the identified usage errors and their severity.

Have you checked your analysis for completeness?

Are you sure that no data has been forgotten? You submit the report on your summative usability evaluation together with the Usability Engineering File, which describes the complete usability engineering process.

Have you demonstrated the safe use of your medical device?

Has your product clearly shown in the summative evaluation that the criteria for safe use have been met? Congratulations, you have proven that the usability of your product does not pose any unacceptable risks for the intended user group!

The checklist for download

Here you can download our checklist again without the explanations. As a practical PDF.

Conclusion

With the right planning and approach, summative evaluation is conquerable. Keep this checklist to hand at all times and move towards your goal as efficiently as possible, step by step.

There are a few pitfalls that you should avoid when conducting a summative evaluation. You can find these in our article: “Summative usability evaluation – you should avoid these pitfalls at all costs”.

Do you still have questions or need support? We will be happy to help you prepare your medical device for the summative evaluation and then carry it out together. We are happy to relieve you of the worry about compliance with the requirements. Here you can find out how we conduct a summative usability evaluation.

In order to look at your specific case and meet your individual challenges, we offer you a free strategy discussion. Simply use our contact form to get in touch with us.