FDA Training Decay Pilot Study - Results & Discussion

As of January 2023, UserWise has been acquired by ClariMed, a leader in human-centered development and regulatory practices for medical products. Read more about the acquisition here.

In 2018, the FDA formally accepted a grant proposal from UserWise, a ClariMed company, titled "Training Decay Selection for Medical Product Usability Validation Testing," as a 2-year project. UserWise, in collaboration with San Jose State University (SJSU), successfully completed the initial pilot study in January 2020. The results of this pilot study will be used to inform changes prior to the execution of the main, statistically powered human factors study anticipated to start mid-2020.

The pilot study was conducted with thirteen (13) participants. The pilot study aimed to test a minimum of twelve (12) participants for purposes of informing the study design of the main human factors study. Participants were recruited from the general population to represent non-professional caregivers and assigned randomly into one of four cohorts with various lengths of training decay. The four cohorts represented one-hour, one-day, one-week, and no training, which represented an infinite training decay. See Table 1 for the distribution of participants assigned to each cohort.

Table 1: Training Decay Cohorts

An insulin pump was selected to be fairly difficult to allow for a range of success/failures and therefore allow differences in training decay to be seen. Participants assigned to a trained cohort were provided with a training video of the insulin pump and a hands-on demonstration. Participants had the opportunity to ask clarifying questions during training. 

Immediately after the training session, participants in trained cohorts underwent a baseline assessment to assess their ability to perform the tasks with no training decay. Each participant was asked to use the insulin pump on a manikin in a simulated home environment as if they were a caregiver. Task success, defined as the user’s ability to complete the task safely and effectively, was collected for each task. This baseline assessment was not applicable to participants in the untrained cohort, as those participants only participated in one study session without a training session.

Each task was categorized as either cognitive, psychomotor/motor, or perceptual in order to examine training decay curves for different task types. For example, remembering to check the expiration date was categorized as a cognitive task. These task types will be further refined based on the empirical data from the main, statistically powered study.

Table 2 below lists each task tested and notes the assigned task type, the percentage of task success per each training decay length, and the associated training decay curve. Though the results are not statistically powered, the training decay curve provides a rough indication of which tasks the “ceiling” or “floor” effect were observed for.

1 Task success percentage includes tasks rated as Successful and Difficulty.

While the “ceiling effect” (100% success) was observed for some individual tasks, there was a range of task difficulty such that the majority of tasks show some decay. Each task type shows decreased task success rate as training decay length increases (Figure 1). Cognitive tasks have decreased success rates compared to psychomotor/motor. There were not enough perceptual tasks to assess the individual decay curve (only 2 perceptual tasks).

Figure 1: Training Decay Curves by Task Type

Overall, for all tasks, a reduction in task success was observed for increased training decay (Figure 2). The “ceiling” or “floor” effect was not observed for the overall training decay curve, indicating the insulin pump is an acceptable device for the main human factors study.

Figure 2: Overall Training Decay Curve

The main human factors study for this research project will include a minimum of 125 participants to provide statistically significant data in differences in training decay length across task types. Based on the results of the initial pilot study, a few minor changes will be made to the study methodology and documentation before execution of the main study as outlined in the presentation above.

Please contact us for any feedback on our proposed changes or overall methodology for our FDA-sponsored research project on training decay.

As part of the grant, we have formed a consortium of industry thought leaders to act as volunteer expert reviewers of the protocol and report and advise on the project. We are continuing to expand our consortium. If you are interested in joining the consortium either as an official member or observer, contact us with your job title, company affiliation, company size (if a manufacturer/sponsor), and educational background.