Are you responsible for the approval of a medical device? Then you must prove that your product can be used without unacceptable risks. In many cases, this proof is provided by a summative evaluation (EU) or a human factors validation (USA).
But how can you best prepare for this? That’s exactly what this article is about. We explain the individual points of preparation in detail. In addition, you will find a compact checklist for download at the end of the article. This will help you ensure that you don’t forget anything in your day-to-day work.
Before we get started, however, we would like to answer one of the most important questions.
Are you even ready for a summative evaluation/human factors validation?
We see it time and time again in practice: the schedule calls for summative evaluation or human factors validation for usability validation, but the product isn’t ready. The result is stress for everyone involved and often the decision to go ahead with the study anyway. This usually fails, resulting in unplanned costs and even more delays.
So what needs to happen before the summative evaluation/human factors validation can be carried out? Here are the most important points:
- You should have carried out at least one, preferably several, formative user studies. The following applies especially to new developments: In practice, validation studies repeatedly fail due to a poor user interface. Users make mistakes that the development team would not have thought possible. Therefore, conduct formative testing with the intended user groups.
- You should also have tested the instructions for use (IFU), the labels, and the training formatively with users. Here, too, problems arise time and again in practice. We have seen countless times that users despair of the IFU or do not understand the training. Make sure that you also test and improve these aspects formatively with the user groups.
- You have tested formatively with the right user groups. We have already integrated this in the two previous points, but here it is again explicitly stated. The capabilities and limitations of your target groups often rise usage-related risks in practice. These often become apparent in a formative study and can thus be controlled.
- You should have based all formative studies on the Use-Related Risk Analysis (URRA). The URRA is required in the US (FDA) anyway. However, we also recommend it for the EU. If you have already used the URRA for the formative studies, then you have prepared your product in the best possible way for the summative evaluation/human factors validation.
As you can see, we consider formative user studies to be extremely important. To put it another way: we have yet to see a product that has successfully completed the validation study without prior formative research.
Have you checked off all the points above? Perfect! Then you are now ready for the actual preparation.
Are you planning the validation study for the right market?
The processes for summative evaluation (EU) and human factors validation (US) are very similar. However, there are also differences. One example is that you must provide evidence of a formative evaluation for the FDA (this does not have to be a user study). This does not apply to the EU.
There are also different rules for conducting the usability validation study. It is therefore important to find out about the planned approval strategy and plan the right study based on this:
- If you only want to get your product approved in the US, the FDA requires you to do human factors validation with participants who reside in the US. In theory, there are exceptions, but in practice, a study in the US is often the best solution.
- If you only want to get approval in the EU, there’s no such requirement. Our recommendation is to test in the EU, as the costs are often significantly lower than in the US. Testing in other countries is also theoretically permissible, but this may lead to discussions regarding comparability.
- If you are seeking dual approval under MDR and FDA, it is advisable to conduct the validation study directly in the US and use the study for the EU. The reverse is usually not possible.
Are there specific guidance documents for your product?
For products to be approved under MDR, MDCG guidance documents may contain additional information for planning and conducting the validation study. These guidelines specify certain requirements of the regulation, such as product types or risk classes. A list of all published MDCG guidance documents can be found here.
For products subject to FDA regulations, the Human Factors Guidance and, if applicable, other specific FDA documents are authoritative. A good starting point for your research can be found here. Depending on the product, the requirements for the human factors validation study can be significantly more extensive or, conversely, no validation study may be necessary.
Therefore, check early on which guidelines apply to your case and integrate their requirements into your preparation.
Have you defined all risk-related factors for your product?
Before you start the summative evaluation or human factors validation, all safety-related aspects of your product should be documented and reviewed.
These include in particular:
- the complete use specification, including the intended user groups
- all user interface features that are relevant to safety
- a structured list of potential use errors
- the definition of hazard-related use scenarios (EU) / critical tasks (USA)
The hazard-related use scenarios or critical tasks precisely define the safety-critical situations that you must test in the usability validation study. In the US, the identification of critical tasks is based on the mandatory Use-Related Risk Analysis (URRA). In the EU, an URRA is not mandatory, but we strongly recommend it.
For MDR approval (EU), you should keep in mind that the summative evaluation is also an opportunity to validate additional risk mitigations that you have identified in the IFU or labeling as risk-minimizing measures. These must also be integrated into the validation study accordingly.
Is your product, including IFU, labeling, and training, available in its final form?
For the summative evaluation/human factors validation, the product, including the IFU and labels, must be available in its final form. Risk-relevant changes are then no longer possible without repeating the validation. This is referred to as the “product equivalent device” or the zero series.
The training, if available, should also be in its final form. However, you may omit the training for the validation study, but do not make them more detailed than they would be in later use.
Is your test environment suitable and ready?
A realistic test environment is crucial for a robust summative evaluation or human factors validation. What this looks like in concrete terms depends heavily on your product and the intended conditions of use.
Keep the following points in mind in particular:
- Equipment: In addition to your product equivalent device, including packaging, instructions, training, and labeling, all other (consumable) materials, devices, and, if necessary, test dummies required for realistic simulation must also be available and ready for use.
- Realistic spatial environment: Typical conditions of real-world use must be replicated, such as lighting conditions, noise levels, stress factors, and ergonomic requirements. The exact environmental conditions you need to replicate also depend on your risk assessment, so careful consideration is important here.
- Recording technology: Cameras, microphones, storage media, and, if necessary, streaming infrastructure must be available, tested, and well positioned to fully document the test process. Recording is not a requirement, but it makes the test much easier for the test moderators and makes it easier to trace problems later on.
- Additional rooms for observers/training, etc.: Ideally, there should be separate rooms or areas for observation, test preparation, or training. However, whether additional rooms are needed depends on whether spectators want to be present during the tests and whether training is necessary.
Recruitment Part 1: Do the participants correspond to your intended target groups?
The participants in your summative evaluation or human factors validation must match the intended user groups that you defined in your use specification. Only then will the results of the validation study be methodologically robust and usable.
A detailed guide (“screener”) helps you select potential participants in advance – e.g., based on profession, experience, clinical setting, or individual skills. Make sure that the participants are representative in terms of age, gender, experience, or other relevant characteristics.
Recruitment Part 2: Are you planning to carry out the study with enough participants?
The number of participants is a key factor in the regulatory acceptance of your summative evaluation or human factors validation. It ensures that the results are meaningful, reliable, and comprehensible.
For a summative evaluation within the framework of the MDR, the following rule of thumb often applies in practice: At least 15 participants in total – divided among all relevant user groups.
This number can be adjusted up or down depending on the complexity of the product, the number of user groups, or the risk class. In individual cases, a smaller group may therefore be justified – provided that the reasoning is documented in a sufficiently convincing manner.
The requirements for human factors validation according to FDA guidance are stricter: at least 15 participants per user group. This means that if you have two user groups, you must test at least with 30 people. What constitutes a user group depends largely on the characteristics of the individuals and their tasks. For example, products intended for lay users may require a very granular breakdown according to different age groups or pre-existing conditions. If in doubt, we recommend that you carry out an FDA pre-submission, during which you discuss the planned testing, including user groups, in advance.
Please also note that you should plan for over-recruitment to compensate for dropouts, e.g., due to illness. We expect a dropout rate of approximately 20-30% and recruit more people accordingly.
Have you prepared the procedure, including the test protocol and moderation guide?
A summative evaluation/human factors validation should follow a fixed procedure. Typically, it is specified who does what and when, and what materials are provided for this purpose and in what form. The goal is clear: everyone should go through the process in the same way. All these points are usually found in the test protocol.
In addition, so-called knowledge tasks should be integrated. These are targeted comprehension questions about safety-related information from the IFU, the label, or warning notices. The FDA expects these tasks to be an integral part of human factors validation. This is not expressly required by the MDR, but can be used to validate risk mitigations, identify gaps in understanding, and support the subsequent classification and root cause analysis of use errors.
In order to achieve the highest possible level of standardization, we also prepare a moderation guide. This contains detailed instructions for the moderator. For example, a fully formulated task that is read aloud to the users. In this process, we typically also document other points that are important for the correct procedure. For example, the signing of a confidentiality and data protection agreement.
Have you properly trained the users, if necessary?
If your product requires training or instruction, this step should also be realistically reflected in the summative evaluation or human factors validation.
In concrete terms, this means that you use the same materials that will be available after approval and that users are prepared in the same way. As already described, you may provide less training, but never more.
The time between training and application is equally important. Experience shows that users forget some of what they have learned after training – this effect must be taken into account in the study design.
There are no fixed guidelines for MDR-compliant studies, but an appropriate interval is recommended. The FDA, on the other hand, explicitly expects a so-called “decay period” – i.e., a realistic period of time between training and application that is based on the actual context of use. Depending on the product, this interval can be several hours, one day, or even several weeks.
If training and application are carried out immediately after each other in the study, even though there is a longer period between them in everyday life, this is not permitted from a regulatory perspective and jeopardizes the validity of the entire usability validation.
Do you have the right personnel to carry out the study?
For a valid and regulatory-approved summative evaluation or human factors validation, it is crucial that the personnel involved are both independent and qualified.
The study must not be conducted by persons who were involved in the development of the product, either directly or indirectly. A lack of separation between development and evaluation can compromise the neutrality of the results and lead to queries from notified bodies or the FDA.
In addition, the moderation team must be technically competent. The study director should have proven experience in usability engineering for medical devices and be familiar with the requirements of IEC 62366-1 (EU) or the FDA’s Human Factors Guidance (USA). The documentation and analysis of the results also requires methodological expertise.
For MDR approvals, a person responsible for regulatory compliance must also be appointed in accordance with Article 15 MDR. This person bears overall responsibility for compliance with regulatory requirements, including with regard to usability evaluation.
Have you found the right partner to carry out the study?
For a comprehensive assessment of the usability of your medical device, we recommend working with a specialized service provider. A competent partner will support you in planning, conducting, and evaluating the study, has the necessary equipment and suitable premises, and is familiar with both the requirements of IEC 62366-1 (EU) and the FDA guidelines for human factors validation (USA).
Make sure that your partner has experience with the target markets. Case studies and testimonials can help you make your choice. An initial meeting will give you an insight into the partner’s working methods, communication style, and how well they fit in with your corporate culture. A good partner can also help you implement many of the points on this checklist, either in part or in full.
Have you planned and conducted a test run and adjusted the process as necessary?
Are you ready to go? Then it’s time for the final rehearsal. This involves running through the entire study with a small number of participants (usually 1–2 people).
This test run serves to identify procedural errors, comprehension problems, or technical difficulties at an early stage – and to make any necessary adjustments before the actual test series. The following are particularly critical:
- Unclear tasks or missing materials
- Logistical errors (e.g., incorrect order, forgotten forms)
- Problems with training, consent forms, or recording
In practice, we conduct test runs one to two days before the start of the study to allow enough time for any corrections. If the test runs smoothly, these first participants can be transferred directly to the regular study.
However, if discrepancies arise, there is enough leeway to make adjustments.
A planned and documented test run is not a mandatory part of validation, but we strongly recommend it as a final safeguard before the start.
Would you like to know when you need to complete which step?
In this article, you have learned what is important when preparing a summative evaluation or human factors validation. In addition to the “what,” the “when” is often a major challenge in practice:
- How do you plan the timeline – from the initial document check to submission?
- How much time do the various steps require – including test preparation, recruiting, implementation, analysis, and documentation?
- What are typical bottlenecks in practice and how can they be avoided?
You will find answers to precisely these questions in our compact webinar: Your Perfect Timeline for Summative Evaluations & Human Factors Validation Testings
Conclusion & download the checklist
A successful summative evaluation or human factors validation begins long before the first test – with clear processes, complete documentation, realistic framework conditions, and a professional team.
The points in this article will help you systematically go through all the crucial preparations – regardless of whether you are seeking approval in the EU or the US. For everyday use, we have converted the article into a compact checklist. You can download it here.
To get started right away, schedule a strategy meeting with our team. We are happy to support you on your way to successful approval.
