Formative and Summative Evaluation Explained

Formative and Summative Evaluation: Key Differences, Methods, and Best Practices

Under the usability engineering process in IEC 62366-1, medical device manufacturers must show that intended users can operate a device without serious use errors. This is done through two kinds of evaluation. Formative evaluations are run during development to find and fix user interface problems. A summative evaluation is run at the end to confirm the final design is safe and effective to use. This guide explains how each works, how they differ, and how to run them well.

Formative Evaluation

Formative evaluation is carried out while the user interface is still being developed. It lets the manufacturer identify risks related to use errors and usability, verify that mitigations for those risks work, and spot any new risks the design introduces.

Because it happens early, formative evaluation is usually iterative and run on early prototypes or mock-up devices. Product teams get timely input on the interface, make design changes, and test again until the device is ready for the summative study.

Key characteristics:

• Timing: Throughout development, on prototypes and mock-ups
• Participants: Internal team members or actual users, depending on what feedback is needed
• Sample size: No specific sample size is defined
• Focus: All foreseeable use errors

Summative Evaluation

Summative evaluation is conducted once the user interface is developed, at the end of development. It assesses the usability of the final design and confirms that critical use-related risks have been identified and that the applied mitigations work. It is the main evidence that a medical device can be used without causing harm to users or patients.

It should be done with actual intended users in an actual, or realistically simulated, use environment. Participants should operate the device independently, without assistance or guidance. In the US, FDA guidance recommends at least 15 participants from each distinct user group in human factors validation testing.

Key characteristics:

• Timing: After the interface is finalized
• Participants: Representative end users
• Sample size: At least 15 per distinct user group, per FDA guidance
• Focus: Critical risks related to use errors

Formative vs. Summative Evaluation: Key Differences

Methods Used in Formative and Summative Evaluations

Each evaluation type has methods that are most common. In practice they overlap: observation and interviews are also widely used in formative studies, and a manufacturer may combine several methods in one study.

Cognitive Walkthrough

Used in formative evaluation. Participants are guided step by step through the process of using the device. To collect data, they are asked questions and encouraged to talk about their thought processes and any concerns. This shows where the interface is unclear before the design is fixed.

Simulated Use Testing

Used in formative evaluation. Users try the device in a simulated setting. Data comes from interviewing them or observing how they interact with the device, and observations can include any instances of hesitation or confusion.

By Observation

Used in summative evaluation. Test participants are observed while they interact with the device. Repeated attempts to finish a task and visible confusion suggest a potential use error, and any use error that could lead to harm is recorded.

Knowledge Task Data

Used in summative evaluation. Participants are questioned after operating the device to test their understanding of its use. The questions check whether they understand critical warning information and can interpret the user manual, quick start guide, and device labelling.

Interview

Used in summative evaluation. Every use error is investigated by interviewing the participant after the test. Questions cover the difficulties they experienced, such as confusion in interaction or awkward manual manipulations. Interviews should use open-ended, neutrally worded questions so they don't lead the participant, and the root cause of each error should be captured.

When to Use Formative vs. Summative Evaluation

• Use formative evaluation at concept, prototype, and design-refinement stages, whenever a design decision or mitigation needs to be checked, and after any change that could affect the interface.
• Use summative evaluation once the design is frozen and the device, labelling, and training are representative of what will be released. It is the final check before market submission.

A typical sequence:

1. Define intended users, use environments, and critical tasks.
2. Run repeated formative evaluations and revise the design.
3. Freeze the user interface.
4. Conduct the summative evaluation.
5. Analyze use errors, and if critical errors appear, fix the design or mitigations and retest as needed.

Real-World Examples of Formative and Summative Evaluation

The examples below are illustrative, not drawn from specific company cases.

• Insulin pen: In formative testing, an early prototype is tried by nurses and a few patients. Several struggle to read the dose window, so the team enlarges the numbers and retests. In the summative study, at least 15 people from each user group (for example, patients and caregivers) dose using the final pen and instructions without help, and any dosing errors are analyzed.
• Infusion pump: A cognitive walkthrough on a screen mock-up shows that clinicians misread a unit label. The team redesigns the screen before any hardware is built. The summative evaluation later checks that clinicians in a simulated ward can program the pump correctly, including responding to critical alarms.
• Home-use autoinjector: Simulated use testing shows users hesitating about when to remove the cap. The team revises the instructions and device cues. Summative testing with lay users then confirms they can complete the injection safely and correctly answer knowledge questions about the warnings.

Common Challenges in Formative and Summative Evaluation

• Unclear user groups: Missing a distinct user group, such as caregivers, leaves a gap in the evidence.
• Unrepresentative participants or environments: Using colleagues in place of real users can hide problems that intended users would hit.
• Leading or assisting participants: Coaching or hints during summative tests make the results unreliable.
• Poor task selection: Not testing critical tasks, or tasks linked to the risk analysis, weakens the study.
• Late testing: Starting formative work too late leaves little room to change the design.
• Weak root-cause analysis: Recording that an error happened without finding out why makes it hard to fix.
• Unrepresentative training and materials: The instructions, labelling, and training used in testing should match what real users will get.
• Documentation gaps: Results not traced back to the risk management file and usability engineering file can cause problems in regulatory review.

Best Practices for Conducting Formative Summative Assessment

• Plan early. Define intended users, use environments, and use scenarios at the start of development.
• Link to risk management. Choose critical tasks from the use-related risk analysis and trace results back to it.
• Iterate on the design. Run several small formative rounds rather than one large one.
• Recruit the right people. Recruit each distinct user group, and include users with a range of abilities and experience.
• Test the full user interface. Include the device, labelling, instructions for use, and training.
• Keep summative testing unassisted. Let participants work independently, and do not step in unless there is a safety concern.
• Use neutral questions. Ask open-ended, non-leading questions in interviews and knowledge tasks.
• Investigate every use error. Find the root cause and decide whether the design, labelling, or training needs to change.
• Avoid reusing participants. People who took part in earlier rounds have prior exposure and may bias summative results.
• Document everything. Keep the protocol, results, analysis, and design changes in the usability engineering file.

FAQs about formative summative evaluation

FAQs about formative summative evaluation

1. What are the methods used in formative and summative evaluations?

Both evaluation types are part of usability engineering, and each has typical methods. They overlap in practice, so many teams use observation and interviews in both.

• Formative evaluation methods:
  • Cognitive walkthrough: Participants are guided through using the device, then asked to explain their thinking and concerns.
  • Simulated use testing: Users try early prototypes or mock-ups while the team observes and interviews them, noting hesitation and confusion.
  • Other approaches include expert reviews and small usability tests on software screens. Software user interfaces can be evaluated before the code is final, which fits IEC 62304 development.
• Summative evaluation methods:
  • Observation: Participants use the final device while observers record use errors, repeated attempts, and close calls.
  • Knowledge task data: Questions after use test whether participants understand critical warnings, the instructions for use, and the labeling.
  • Interviews: Every use error is investigated afterward with open-ended, neutral questions to find root causes.

2. When should formative evaluation be conducted?

Throughout development, starting as soon as you have a concept, sketch, or early prototype and continuing until the user interface is ready to freeze. It is iterative: test, find problems, change the design, and test again. IEC 62366-1 places formative evaluation in the design, implementation, and formative evaluation step (clause 5.8), and it also involves planning it in clause 5.7.

Formative evaluation is also worth repeating after any design change that touches the user interface. There is no fixed sample size, and participants can be internal team members or actual users. Findings should feed your ISO 14971 risk analysis and be recorded in your design and development file under ISO 13485 clause 7.3.

3. When is summative evaluation performed?

After the user interface is final, as part of design validation. In IEC 62366-1 it is clause 5.9, with the hazard-related use scenarios chosen for testing selected earlier in clause 5.5. It uses the final or production-equivalent device, representative users, and a realistic use environment. Participants operate the device without coaching. The FDA generally recommends at least 15 participants per distinct user group.

Complete summative evaluation before you file an FDA 510(k), premarket approval (PMA) application, or De Novo request, so the human factors report can be included. How much the FDA expects depends on use-related risk more than on medical device classification alone, but higher-risk devices typically need the most complete evidence. If you change the interface afterward, reassess whether the results still apply.

4. Why are formative and summative evaluations important in medical device usability engineering?

They are how you prove people can use the device safely.

• Patient and user safety: They catch use errors before they cause harm, and confirm that risk controls for the interface actually work.
• Regulatory expectations: IEC 62366-1 is an FDA-recognized consensus standard, and the FDA's human factors guidance expects usability evidence for devices where use errors could cause serious harm. In the EU, usability evidence also supports the clinical evaluation report (CER) under the MDR.
• Quality system fit: Under ISO 13485 (and 21 CFR Part 820, now the QMSR), usability results are design validation evidence. They also connect to ISO 14971, because use errors become hazardous situations and risk controls in the risk file.
• Consistent device identity: The tested interface, labeling, and instructions for use should match the released device version. Keep this consistent with the version or model in your GUDID record and on your label.
• Record integrity: If you keep test protocols, data, and reports electronically, 21 CFR Part 11 controls such as audit trails and electronic signature rules apply.

5. How does formative summative assessment help meet IEC 62366 requirements?

It produces the evidence the standard asks for, laid out in a way an auditor can follow. The main links are:

For software with a user interface, findings also feed software requirements, which links usability engineering to IEC 62304 development. The result is a traceable chain from hazard to use scenario, requirement, formative testing, and summative validation.