Short answer: In usability engineering for medical devices, you will sooner or later encounter the term “formative evaluation” during the development process. But what exactly does it mean?
A formative evaluation is the assessment of an existing user interface during development. The goal is to identify strengths, weaknesses, and unexpected user errors and to use the insights gained to inform further design. Unlike summative evaluation, the focus is not yet on final validation, but rather on learning, understanding, and improving.
The definition is relatively clear. In practice, however, further questions quickly arise: What exactly constitutes the user interface? Why should formative evaluations be conducted early and iteratively? How do they differ from user research and from summative evaluation? We address precisely these questions in this article.
1. What is a formative evaluation according to IEC 62366-1 and the FDA?
The FDA defines the term in its guidance document “Applying Human Factors and Usability Engineering to Medical Devices” as: “User interface evaluation conducted with the intent to explore user interface design strengths, weaknesses, and unanticipated use errors.”
Furthermore, the FDA explains the purpose of formative evaluations during development: “Formative evaluations are used to inform device user interface design while it is in development.”
It adds that formative evaluations should focus in particular on issues where prior analyses have identified potential safety-related usability difficulties or where design decisions for the user interface are not yet final.
The IEC 62366-1 also describes formative evaluation as “USER INTERFACE evaluation conducted with the intent to explore USER INTERFACE design strengths, weaknesses, and unanticipated USE ERRORS (DIN EN IEC 62366-1:2020, Section 3.7).” This definition thus corresponds to the FDA’s definition.
The standard adds that teams typically conduct formative evaluations iteratively during the design and development process and before the summative evaluation to support user interface design.
Both definitions thus focus on the same fundamental concept: A formative evaluation is intended to provide insights that can be used to improve the user interface during development. For example, the following can be examined:
- whether users understand a workflow,
- whether controls and information are unambiguous,
- whether warnings are interpreted correctly,
- whether individual steps are understandable,
- whether the user interface contributes to potential use errors.
The user interface encompasses more than just a display or a software interface. This also includes – if present for the product – physical controls, signals, packaging, labeling, instructions for use, and training.
Depending on the research question, a formative evaluation can be conducted, for example, as a user study, expert assessment, task analysis, or test using mock-ups and prototypes.
2. Why are formative evaluations important?
A medical device may function technically as intended yet still be difficult to understand or prone to errors during use.
Formative evaluations therefore help manufacturers to:
- identify potential use errors early on,
- improve the user interface in a targeted manner,
- review risk control measures,
- base design decisions on findings rather than just assumptions,
- and avoid costly changes at the end of development.
The FDA explicitly cites formative evaluations as a suitable means of uncovering previously unrecognized usage-related hazards and use errors and identifying additional critical tasks. The earlier a fundamental problem is identified, the easier it is generally to resolve.
3. What is the difference between formative evaluation and user research?
User research is the umbrella term for the systematic study of users – typically conducted empirically in the context of actual use. This involves, for example, understanding goals, tasks, needs, problems, and contextual factors.
From a UX and product development perspective, a formative evaluation can be classified as a form of user research.
For regulatory classification, however, one point is crucial: In a formative evaluation, a user interface – or interaction with it – is examined. For this, at least part of the user interface must already exist – for example, as a concept, mockup, or prototype. User research, on the other hand, can begin earlier, before a specific user interface exists.
This allows users to be surveyed about existing workflows or observed as they use the current solution to a problem even before the actual development work begins. These insights help identify fundamental requirements for the product. Based on this, an initial prototype can be created, which is then subjected to formative evaluation and iteratively refined.
A simple example:
- You observe nursing staff during their current workflow before a specific solution concept exists: user research using other methods.
- You test a paper prototype with nursing staff and observe where difficulties arise: user research in the form of a formative evaluation.
In short: From a UX and product development perspective, a formative evaluation is a form of user research. Conversely, however, not every user research activity is a formative evaluation.
4. What is the difference between formative and summative evaluation?
Both formative and summative evaluations assess the user interface, but they have different objectives.
Formative evaluation takes place during development. Its results are intended to help understand, review, and further develop the user interface.
Summative evaluation is typically conducted at the end of user interface development. It is intended to demonstrate that the final or production-ready user interface can be used safely and effectively by the intended users under the intended conditions of use.
Simply put: Formative evaluation serves the purpose of learning and improvement. Summative evaluation serves as the final verification that the user interface can be used safely by the intended users under the intended conditions of use.
Conclusion
Formative evaluation is a learning and improvement tool that accompanies the development process.
It helps developers evaluate the user interface early on, identify potential use errors, and implement necessary changes before the design is finalized.
The definition is thus relatively clear. In practice, however, further questions often arise: When should a formative evaluation be conducted? Which approach is appropriate for the current stage of development? And what insights do you need for the next development steps?
In our webinar “Formative Evaluation Done Right: When to Test and What to Learn,” we delve deeper into the topic and provide guidance for practical application.
Register for the webinar and learn more about the effective use of formative evaluations for medical devices.
