What exactly is a formative usability evaluation? Is it mandatory? How is it different from a summative evaluation? How can your product benefit from one (or more) formative evaluation(s)? What is the best possible approach for formative evaluations and which methods are suitable? Have you ever asked yourself one or more of the above questions? In this basic term article, you will find clear answers. Here we want to create an understanding of why formative usability evaluations represent a great opportunity for your product development.
Definition: formative usability evaluation
IEC 62366-1:2020 defines formative evaluation as follows: ” USER INTERFACE EVALUATION conducted with the intent to explore USER INTERFACE design strengths, weaknesses, and unanticipated USE ERRORS”. It also adds, ” FORMATIVE EVALUATION is generally performed iteratively throughout the design and development PROCESS, but prior to SUMMATIVE EVALUATION, to guide USER INTERFACE design as necessary.” (IEC 62366-1:2020; Chapter 3.7) FDA provides a very similar description, “Preliminary analyses and assessments are performed to identify user tasks, user interface components, and usage problems early in the design process. These analyses help focus the HFE/UE processes on the design of the user interface as it is being developed so that it can be optimized for safe and effective use. One of the key outcomes of these analyses is the comprehensive identification and categorization of user tasks, resulting in a list of critical tasks.” (FDA “Applying Human Factors and Usability Engineering to Medical Devices”)
Do you need to perform formative testing?
Formative evaluations of your product are not required by FDA or IEC 62366. However, formative evaluations greatly help you meet the Usability requirements of your product and provide the evidence you require. Most importantly, formative testing helps you meet the following requirement of IEC 62366-1: “The MANUFACTURER shall prepare a list of foreseeable HAZARDS and HAZARD SITUATIONS associated with the use of the MEDICAL PRODUCT as part of the RISK MANAGEMENT PROCESS in accordance with ISO 14971.” (IEC 62366-1:2020; subsection 5.3)
Caution, a common pitfall: if you have mentioned in your own QMS that you do formative testing, then of course you need to demonstrate this for QMS certification, but not necessarily for clearance of your product. The FDA mentions formative evaluation as a very suitable method for the necessary identification of previously unknown use-related hazards and use errors: “For example, formative evaluation can reveal previously unrecognized use-related hazards and use errors and help identify new critical tasks.” (source: FDA “Applying Human Factors and Usability Engineering to Medical Devices”) Again, formative testing is not mandatory, but it is mentioned as an important part of the process that prepares you for summative evaluation!
What is the goal of a formative evaluation?
Formative evaluations identify the strengths and weaknesses of an interface:
- The strengths you want to identify because you don’t want to cut something in any iteration that worked well and delighted users.
- The weaknesses because you want to discover use errors and find out where there are security gaps and potential for improvement. So here you are completing your risk analysis with errors that you hadn’t even thought of before.
In doing so, formative testing allows you to try out many possible solutions for your product. You will find out step by step which approach is the best to meet your goal! In addition to improving the usability of your product, formative evaluations can help you to improve the user experience of your product in a targeted manner.
What is the difference between a summative and a formative evaluation?
A formative evaluation accompanies development and (as defined above) is designed to generate potential for improvement in terms of usability and user experience. A summative evaluation is a final product evaluation. IEC 62366-1 defines it as follows: ” USER INTERFACE EVALUATION conducted at the end of the USER INTERFACE development with the intent to obtain OBJECTIVE EVIDENCE that the USER INTERFACE can be used safely” (IEC 62366-1:2020). Important: There are no formal requirements for a formative evaluation, unlike for a summative evaluation. You can do anything that helps you develop a better product! But it should always be preparatory to the summative evaluation.
Why is formative usability testing so important for your product?
It is highly unlikely that a medical device will pass the summative evaluation without first being usability tested with real users. To put it clearly once again: We have never seen it happen in practice that a summative evaluation has passed if that was the product’s first contact with real users. Many use errors cannot be predicted or guessed. If they happen in a formative evaluation, you can adjust your product design. However, if a critical error occurs in the summative evaluation, it will delay or prevent your clearance. In addition, formative testing helps you meet some of the requirements placed on your product development by IEC 62366-1 and the FDA. You can check off and prove these points required by IEC 62366-1 in the course of a formative evaluation:
- Identify USER INTERFACE characteristics related to SAFETY and potential USE ERRORS (IEC 62366-1:2020; 5.2)
- Identify known or foreseeable HAZARDS and HAZARDOUS SITUATIONS (IEC 62366-1:2020; 5.3)
- Identify and describe HAZARD-RELATED USE SCENARIOS (IEC 62366-1:2020; 5.4)
- Select the HAZARD-RELATED USE SCENARIOS for SUMMATIVE EVALUATION (IEC 62366-1:2020; 5.5)
The FDA also requires some items that you can accomplish through formative testing:
- Formative testing helps you identify hazards and use errors.
- Risk minimizing actions can be defined when the problem is understood through formative evaluations.
- It provides important information that you need for your validation plan.
Formative evaluations are perfect as a learning and decision-making tool: they are about making good design decisions and developing good products. Another important reason: they are the only point in the process where you check how close the product is to the original goals. Formative testing is your opportunity to systematically learn and discover potential for improvement and make decisions based on data instead of gut feelings.
How do you achieve the best possible formative testing for your product?
There are two types of formative evaluation: An expert evaluation checks the interface together with a UX expert. This expert uses guidelines to ensure a consistent evaluation scheme. Many standards can be used here, for example ISO 9241-11:2018-11, which deals with the “Ergonomics of human-system interaction”. This contains, for example, sections of standards on interface design and dialog criteria. In a user evaluation, users are confronted with the current state of the design and then give feedback on it. One wants to find out how well the interface is understandable, whether it fits the user’s tasks and how safe it is to use. Also important is whether it triggers excitement in the user. This feedback can be qualitative or quantitative. It can also be objective or subjective. What data is collected depends on the state of your prototype. Objective and qualitative are, for example, UX tests, while subjective and qualitative correspond to an intensive and little standardized interview. To guarantee the most pragmatic and effective implementation of usability and user experience testing, we rely on our Data Driven UX Design process. Here, formative evaluations take place as follows: We derive user requirements from a context of use analysis, on the basis of which design solutions are developed. We then present these to real users in the form of prototypes. They solve the task specified by you with this prototype. In this way, design solutions can also be evaluated early in the development process, but also at a later point in time. The data is evaluated and incorporated into the design solution. Then, iterative improvements and evaluations are made until no more critical use errors occur. The necessary design decisions are always made on the basis of real user data. You should consider the following points:
- Test in an iterative way! Each iteration you adapt your prototype and thus improve your product step by step.
- A mix of methods is often recommended! You can use a variety of methods. Usability and user experience tests, click paths, online surveys, focus groups or qualitative interviews are conceivable.
- Make sure that your partner can provide the right usability labs and, if necessary, the right dummies for your specific case.
At the end of the formative evaluation, a decision is made whether a further iteration makes sense or whether it is time to move on to practical implementation. If your formative testing has come to an end, you can use this to prepare for the Summative Evaluation: “17 Dos and Dont’s for the summative usability evaluation“.
Conclusion
Formative usability testing is not mandatory (unless you have defined it as such in your own process instructions), but it is perfect for identifying the potential for improvement of your product and for drawing up the required list of hazards and hazardous situations associated with the use of your medical device. You need this for the risk management process according to ISO 14971. In the course of this, the user experience of your product can be improved here in a targeted manner. What is your specific case? Do you already have experience with formative usability testing? Feel free to comment on this post or get in touch via our contact form. We look forward to your feedback!
