Formative and Summative Evaluation: Key Differences, Methods, and Best Practices
As part of the usability process (IEC 62366), the manufacturers are required to conduct formative and summative evaluation of the device.
The Usability Engineering process identifies and reduces use errors by identifying and lowering use-related risks associated with common device operation. The purpose of usability engineering is to ensure that the equipment is safe and effective when utilised by users.
As part of the usability process (IEC 62366), the manufacturers are required to conduct formative and summative evaluation of the device. These evaluations are conducted to demonstrate that the intended users can use the device without serious use errors. Formative evaluation and summative evaluation shall be done to ensure any risk related to the usability of the product is identified and controlled.
Formative vs. Summative Evaluation: Key Differences
| Basis |
Formative Evaluation |
Summative Evaluation |
| Definition |
Formative Evaluation is to identify all the risks related to the usage of the user interface during the development phase. This may be an iterative process. |
Summative Evaluation is used to identify critical risks related to the user interface of the device. Usually conducted once the interface is developed (end of development). |
| User Group |
Manufacturer can use its own internal team or may even opt to take help from actual users to get early feedback. |
It should be done with the help of actual or end users of the device in the actual environment. |
| Methods |
Formative Evaluation can be done by using methods given below: Cognitive Walkthrough, Simulated use testing |
Summative Evaluation can be done by using methods given below: Knowledge Task Data, Interview, By Observation |
| Data to be Evaluated |
All the foreseeable use errors. |
Critical risk related to use errors.
|
FORMATIVE EVALUATION
Formative evaluation allows the manufacturer to verify the mitigation identified for risks related to use errors or usability of the user interface. It also helps to identify any new risks that may be related to the user interface. Formative evaluation is an effective method in which the product tech teams get timely inputs related to the user interface and allow them to ensure the design related to the interface is safe. As the evaluation is conducted during the development stage, it is usually an iterative process that is conducted on early prototypes or mock-up devices. The participants of the formative evaluation can also be internal team members or actual users. There is no specific sample size defined at this stage. Throughout this stage of the device development process, design changes should be made and then iterated until the device is prepared for the Summative Study.
Methods Used in Formative Evaluations
The methods used for Formative evaluation are:
- Cognitive walkthrough
- Simulated use testing
Cognitive Walkthrough
|
- In this method, Test participants are guided through the process of using the device.
- Then to collect data, Participants are asked questions and encouraged to talk about their thought processes and concerns they may have.
|
| Simulated Use Testing |
- Through simulated use testing, users may use the device.
- Data can be obtained by interviewing them or observing their interaction with the device.
- Observation data can include any instances of hesitation or confusion.
|
SUMMATIVE EVALUATION
The summative evaluation is conducted to evaluate the usability of the user interface and determine what aspects of a design are functional or useful once the final product is developed. It is carried out to ensure that any critical risks related to usability are identified and to validate the applied mitigation. Summative evaluation is an effective method to ensure that a medical device is safe to use and will not cause any harm to the users and patients.
It should be done with actual intended users in an actual use environment. The participants of the summative evaluation should be encouraged to operate the device independently without any assistance or guidance. As per US FDA guidance, at least 15 individuals from each user group should be included in the validation testing if the device has many separate user groups.
Methods Used in Summative Evaluations
The methods used for Summative evaluation are:
- By Observation
- Knowledge Task Data
- Interview
| By Observation |
- It includes the observation of test participants while interacting with the device.
- Repeated attempts to complete the task and apparent confusion indicate potential use error.
- Observation of use errors that could result in harm.
|
| Knowledge task data |
- Understanding of the use of the device can be tested by questioning the test participants after operating the device.
- Whether the user is able to understand the critical warning information or not.
- This also involves questioning regarding the user manual, quick start guide, and labelling of the device.
|
| Interview |
- All the use errors should be investigated by interviewing the test participant after the test.
- Questions can be asked related to use difficulties experienced during the test (e.g. Confusion in interaction, awkward manual manipulations).
- The interview should be composed of open-ended and neutrally worded questions.
|
In summary, formative and summative evaluation together form the backbone of the usability engineering process under IEC 62366. While formative evaluation identifies and mitigates foreseeable use errors early in development, summative evaluation validates that critical use-related risks have been controlled before the device reaches the market. Conducting both formative evaluation and summative evaluation ensures that the device is safe, effective, and ready for use by its intended users.