Human Factors Validation Study Guidelines for Medical Devices

Based on the 2016 FDA Guidance: “Applying Human Factors and Usability Engineering to Medical Devices”

Human Factors Engineering is critical to validate the safety and efficacy of a product for FDA submission. The need for validation testing is driven by 21 CFR Part 820.30(g) which states “Design validation shall ensure that devices conform to defined user needs and intended users and shall include testing of production units under actual or simulated use conditions.” Furthermore, the FDA has had an increased enforcement of Human Factors since publishing its 2016 guidance, as well as the industry’s realization that Human Factors can dramatically affect the success of a medical device.

The end goal of the 2016 FDA guidance is to minimize or eliminate use errors which could result in harm. It is thus important to examine all possible use errors and determine their potential severity using a risk analysis. The FDA recommends using the Perception, Cognition, Action (PCA) analysis framework which provides insight into the interaction between a user and the device while considering the environment in which the device is used. Additionally, all components of the user interface including packaging, labeling, instructions, displays and controls should be considered. Critical tasks which could result in serious harm should be evaluated during usability validation. For validation testing, simulated use should be as similar to actual use as possible. This means that Intended users, device use, and the use environment should be sufficiently realistic.

To understand what validation study requirements are expected of manufacturers, the following guidelines and accompanying excerpts have been compiled from the 2016 FDA guidance.

Simulated Use

Simulated use should be sufficiently realistic and generalizable to actual use:

• “The conditions under which simulated-use testing is conducted should be sufficiently realistic so that the results of the testing are generalizable to actual use.”

The simulated use environment should be sufficiently realistic, incorporating factors such as lighting, sounds, distractions, and/or multitasking:

• “To the extent that environmental factors might affect users’ interactions with elements of the device user interface, they should be incorporated into the simulated use environment (e.g., dim lighting, multiple alarm conditions, distractions, and multi-tasking).”

During simulated use, test participants should be allowed to use the device independently and naturally without interference or influence from study personnel:

• “During simulated-use human factors validation testing, test participants should be given an opportunity to use the device as independently and naturally as possible, without interference or influence from the test facilitator or moderator.”

Labeling information should be available for the test participants if it normally would be, however, participants should not be instructed to use it:

• “If users would have access to the labeling in actual use, it should be available in the test; however, the participants should be allowed to use it as they choose and should not be instructed to use it.”

Simulation of customer service phone calls should be realistic:

• “If the users would have access to a telephone help line, it may be provided in the test but it should be as realistic as possible...”

Training

Training should approximate the actual training that users will receive:

• “...training provided to the human factors validation test participants should approximate the training that actual users would receive.”

• “To the extent practicable, the content, format, and method of delivery of training given to test participants should be comparable to the training that actual users would receive. A human factors validation test conducted after participants have been trained differently than they would be in actual use is not valid.”

Training Decay

Training decay should be simulated during validation testing:

• “Because retention of training decays over time, testing should not occur immediately following training; some period of time should elapse.”

Training decay time depends on actual use and should last between 1 hour to several days:

• “In some cases, giving the participants a break of an hour (e.g., a ‘lunch break’) is acceptable; in other cases, a gap of one or more days would be appropriate, particularly if it is necessary to evaluate training decay as a source of use-related risk.”

Participant Performance

Participant Performance of all critical use scenarios should be observed for and reported:

• “The human factors validation testing should include observations of participants’ performance of all the critical use scenarios (which include all the critical tasks).”

All near misses (close calls) should be observed for and reported:

• “‘Close calls’ are instances in which a user has difficulty or makes a use error that could result in harm, but the user takes an action to ‘recover’ and prevents the harm from occurring. Close calls should be recorded when they are observed and discussed with the test participants after they have completed all the use scenarios.”

Apparent Confusion should be observed for and reported:

• “In addition, repeated attempts to complete a task and apparent confusion could indicate potential use error and therefore should also be collected as observational data and discussed during the interviews with test participants.”

Debriefing and Subjective Feedback

Participants should be debriefed following testing:

• “The observation of participant performance of the test tasks and assessment of their understanding of essential information (if applicable) should be followed by a debriefing interview.”

Subjective Feedback of difficulties experienced should be probed for:

• “In the interview, the participant should provide a subjective assessment of any use difficulties experienced during the test...”

Subjective Feedback questions should be neutrally worded, start by asking questions about the device overall and then probe individual use scenarios:

• “The interview should be composed of open-ended and neutrally-worded questions that start by considering the device overall and then focus on each critical task or use scenario.”