Short answer:
Summative evaluation is conducted at the end of user interface development to obtain objective evidence that the user interface can be used safely by the intended users under the intended conditions of use. The focus is on safety-related use scenarios and the assessment of potential use errors.
That’s the short answer. In practice, however, a few more questions quickly come up:
- How exactly does IEC 62366-1 define a summative evaluation?
- Why is it important?
- How does it differ from a formative evaluation?
- And how does it differ from the FDA’s Human Factors Validation?
We address these questions in this article.
1. What is a summative Evaluation according to IEC 62366-1?
Section 3.13 of IEC 62366-1 defines summative Evaluation as follows:
“USER INTERFACE EVALUATION conducted at the end of the USER INTERFACE development with the intent to obtain OBJECTIVE EVIDENCE that the USER INTERFACE can be used safely.”
(IEC 62366-1:2015, including Amendment 1:2020, Section 3.13)
The key phrase is “objective evidence.” It is not enough for the development team to consider the design mature. Safe use must be assessed on the basis of verifiable results. To do so, the summative Evaluation examines the safety-related use scenarios selected in advance.
The evaluation is performed using the final or production-equivalent user interface. “Production-equivalent” means that the version used in the evaluation adequately represents the finished product in all characteristics relevant to use and assessment. This could be, for example, a device from a pilot production run. A prototype, however, is not production-equivalent simply because it is already functional.
Another important point: The user interface includes more than a display or software screen. Depending on the medical device and how it is used, it may include, for example:
- physical controls,
- displays, alarms, and signals,
- packaging and labeling, and
- instructions for use (IFU) and, where applicable, intended training materials.
The basis for selecting what to evaluate is the hazard-related use scenarios identified during the usability engineering process. These are use scenarios that may lead to a hazardous situation or harm and are often associated with potential use errors.
Under Section 5.5 of IEC 62366-1, the manufacturer can select all identified scenarios or a justified subset. Such a selection is based on the severity of potential harm resulting from use errors and, where relevant, other product- or manufacturer-specific circumstances. The selection approach, its rationale, and the results must be clearly documented in the Usability Engineering File.
The goal, then, is not to test as many functions as possible. It is to derive a systematic, risk-based selection of safety-related use scenarios.
2. Why is summative Evaluation important?
As previously described, summative Evaluation is an essential part of the usability engineering process under IEC 62366-1. It provides the evidence manufacturers need to assess use-related risks at the end of user interface development. For the technical documentation, the test results alone are not enough: the reasoning and process leading to them must also be traceable.
For manufacturers, this means, in particular:
- justifying the selection of safety-related scenarios and the study design,
- systematically capturing use errors and use difficulties,
- assessing observed problems with respect to their causes and potential harm, and
- linking conclusions, required actions, and residual risks to risk management and the Usability Engineering File.
One important point when evaluating the results: A summative Evaluation does not automatically “fail” simply because use errors occur. Under Section 5.9 of IEC 62366-1, observed use errors and use difficulties must be identified. If they could lead to a hazardous situation, their root causes must be investigated. Depending on the findings, additional risk control measures and a reassessment of residual risks may be necessary.
This also has implications for project planning. If fundamental problems only become apparent during the summative Evaluation, changes to the user interface, instructions for use, or training may be needed, along with further evaluation and documentation activities. Good preparation reduces this risk, but it does not replace the final evaluation.
In practice, a robust summative Evaluation begins long before participant recruitment. It builds on a consistent Use Specification, which describes, among other things, the intended use, user groups, and usage environments, as well as the identification of usage-related risks, a well-reasoned selection of scenarios, and an appropriate evaluation plan.
In summary: Summative Evaluation is a key element at the end of user interface development, as it provides objective evidence of whether a medical device can be used safely in selected safety-related use scenarios. If unexpected use errors or use difficulties occur, the findings help identify their root causes and assess whether the remaining residual risks are acceptable or require additional measures. The results are systematically documented and linked to the risk management process and the Usability Engineering File, supporting the regulatory assessment of use-related residual risks. In this way, summative Evaluation makes an essential contribution to patient safety and the technical documentation required to demonstrate regulatory compliance.
3. What is the difference between formative and summative Evaluation?
Both types of evaluation examine how users interact with the user interface. However, they serve different purposes and take place at different stages of development.
Formative Evaluations support the development process. They help teams understand which elements already work well, where users encounter difficulties, and what changes could improve the design. Early concepts and prototypes can also be evaluated.
Findings from formative Evaluations feed into further user interface development, requirements, and, where applicable, the assessment of use-related risks.
In short: Formative Evaluation = understand and improve.
Summative Evaluations
Summative Evaluation takes place at the end of user interface development. Rather than comparing new design concepts, its focus is the final assessment of safe use based on the selected hazard-related use scenarios. For this purpose, the user interface must be final or production-equivalent.
In short: Summative evaluation = conduct the final assessment and obtain evidence.
In practice, formative Evaluations should be used to identify and address relevant use problems as early as possible. A summative Evaluation cannot replace this iterative learning process. Conversely, a successful formative test does not replace the final evaluation under IEC 62366-1.
4. What is the difference between summative Evaluation and Human Factors Validation?
The terms are often used interchangeably in MedTech because both refer to a final evaluation of the user interface. However, summative Evaluation under IEC 62366-1 and Human Factors Validation as described by the FDA should not automatically be treated as identical.
IEC 62366-1 is an international standard and an important basis for the usability engineering process for manufacturers seeking access to the European market. A summative Evaluation focuses on the hazard-related use scenarios selected in accordance with the standard.
In the U.S. context, the FDA describes human factors validation in its guidance, “Applying Human Factors and Usability Engineering to Medical Devices.” A key focus is on critical tasks: user tasks whose incorrect performance or omission could cause serious harm to patients or users. The study design should also appropriately reflect the intended user groups, realistic use conditions, and final user interface. Data collection and analysis should support a sound assessment of safe and effective use.
In practice, the differences mainly concern:
- how scenarios or critical tasks are identified and selected,
- specific expectations for user groups, study design, and test execution, and
- the required rationale, evaluation, and regulatory documentation.
The FDA itself notes that Human Factors Validation is sometimes called “summative testing,” but that this term does not always encompass all the elements described in its guidance. This is another reason to assess the requirements of each target market separately.
Under suitable conditions, a single study can provide evidence for both markets. However, the relevant requirements should be considered from the start of study design. Simply relabeling a study initially planned for only one regulatory framework is not sufficient.
Conclusion
Summative Evaluation is more than a final usability test. It provides a traceable basis for assessing whether the final or production-equivalent user interface can be used safely. Key elements include a justified selection of hazard-related use scenarios, an appropriate evaluation plan, and careful analysis and documentation of the results.
Teams that use formative evaluations early and consistently address use-related risks throughout development create a much stronger foundation for the final evaluation.
Are you wondering whether your product requires a summative Evaluation or Human Factors Validation, and whether one study could address the requirements of different target markets? In our on-demand webinar, “Do You Really Need a Summative Evaluation or Human Factors Validation?”, we explore these questions in more depth and offer guidance for planning your next evaluation.
Register for the webinar and learn more about when a summative Evaluation or Human Factors Validation is needed and what to consider when planning your study.
