Formative Evaluations Series: Early Usability Evaluations to Optimize Medical Product Design

Have you ever wondered how you can optimize the usability of your medical device or combination product without draining your budget? In this post we will discuss the importance of conducting Formative Evaluations throughout product development and provide an overview of different methods of Formative Evaluation. Many types of Formative Evaluation can be performed even before fully functional hardware or software prototypes are available. This allows design changes to be implemented early and will minimize the number of formal prototypes that are needed.

Let’s start by addressing the frequently asked question:

What is a Formative Evaluation?
The international standard on usability, IEC 62366-1, defines a Formative Evaluation as a “User interface evaluation conducted with the intent to explore user interface design strengths, weaknesses, and unanticipated use errors.” Additionally, it clarifies that Formative Evaluations are “generally performed iteratively throughout the design and development process, but prior to Summative Evaluation, to guide user interface design as necessary.”

When submitting most medical devices or combination products for regulatory approval or clearance, it is beneficial for the user interface to have undergone Formative Evaluation. Formative Evaluations are an essential component of the human factors and usability engineering iterative design process (see Figure 1) described in both IEC 62366-1 and the FDA Guidance on Human Factors and Usability Engineering for Medical Devices (2016).

Figure 1: The FDA-Recommended Human Factors Process
Formative Evaluations can be performed throughout the medical device and combination product design process. Early evaluations can be used to help understand the full context of use, including who will use the device and in what environments the device will be used. Additionally, early-stage Formative Evaluations can include interviewing users to identify potential use scenarios and use errors that could occur with the device based on similar product use.

Once design concepts have started to be developed, drawings, product simulations, and low-fidelity mock-ups and prototypes can be used to evaluate the usability of the device. These initial mockups may be as simple as a sketch of the user interface, a flow chart of tasks, or components made of cardboard; they do not need to perfectly replicate the final device. Using much less expensive low-fidelity mockups and prototypes early in the design process saves companies a significant amount of money in both the short and long term.

Formative evaluations can focus on one component of the user interface (e.g., hardware user interface, instructions for use, training) or can focus on all components of the user interfaces collectively.

Later in the design process, more complex, high-fidelity prototypes can be utilized. A formative usability study that covers full product use is recommended just prior to the Summative Evaluation (also referred to as the human factors validation study). This allows all components of the device user interfaces to be used together to ensure you do not have unforeseen use errors going into your Summative Evaluation.

Formative Study Planning
The IEC standard stipulates that regardless of the type of Formative Evaluation to be performed, you must have a formative user interface evaluation plan. This plan should address:

• The evaluation method(s) to be used.

• Which part of the user interface is being evaluated.

• When the formative user interface evaluations will be performed in the usability engineering process.

The standard also lists additional requirements for Formative Evaluations that are usability tests.

Formative Evaluation Results
Formative evaluation results can include:

• Learnings about the user groups, use environments, and context of use

• Identification of new use errors, hazards, hazard-related use scenarios, and shortcomings of the user interface

• Learnings about the study design and simulation for summative testing

The results of Formative Evaluations should be documented and used to determine whether design modifications to the user interface are needed and what form they should take. Any use errors, hazards, and hazard-related use scenario learnings should be added to your use-related risk analysis. Learnings about simulated use should be considered to optimize the summative study design and simulated use environment.

If you are submitting your medical device or combination product to the FDA, they will expect a narrative describing the learnings from the formative evaluations and how the user interface and summative study design has evolved over time as part of the human factors and usability engineering (HF/UE) submission report.

Why Invest in Formative Evaluations?
Formative Evaluations gather important usability information about the medical product early in the design process, when it is easier to rapidly modify the product’s design and before such changes become more complex and costly (e.g., after fixtures and tooling must be modified). When no Formative Evaluations are performed and serious use errors are identified later in the product development process, during Summative Evaluation, the costs for changes become quite significant. The Summative Evaluation becomes a very expensive formative since summative testing will need to be repeated after user interface changes have been implemented to address identified issues.

We can think of Formative Evaluations as performing a series of practice tests before we must take the final test, the Summative Evaluation.

Types of Formative Evaluations

1. Cognitive Walk-Through
Moderators guide test participants through using the medical product or device component. Participants may be asked questions about what they would do at each stage or encouraged to think aloud during the walk-through. Low-fidelity prototypes may be used in the early stages of the design process or high-fidelity prototypes may be used in the later stages. Learn more about cognitive walk-throughs.

2. Simulated-Use Testing
To mimic actual use, the user uses a medical product prototype independently in a simulated realistic use environment while being observed by study personnel. The Moderator takes a more hands-off approach to see how a user will naturally interact with the medical product. Learn more about usability testing.

3. Heuristic Analysis
An assessment of a user interface against widely accepted usability design principles. For example, these include displaying data clearly, using recognition rather than recall memory, and only presenting the information the user needs. Learn more about heuristic analyses.

4. Contextual Inquiry
A human factors expert observes prospective users in the use environment to get a better idea of work practices, social influences, use context, and the participant’s other tools. These evaluations typically take place early in the design and development process. User may be interviewed as part of this process as a follow up to observations or to ask standard questions around context of use. Learn more about contextual inquiry.

5. Usability Expert Review
A human factors expert evaluates the device based on their professional experience, education, and applicable regulations. Additionally, they may also use a heuristic analysis as part of this process. Learn more about expert reviews.

6. User Feedback
User feedback can be collected through interviews, surveys, and focus groups early in the design and development process. Learn more about methods for collecting user feedback.

While we have touched on the most common types of Formative Evaluations used in the medical device industry, this is not an exhaustive list.

References
FDA Guidance on Human Factors and Usability Engineering for Medical Devices (2016)
IEC 62366-1:2015+AMD1:2020