Written by Angela Siebeneck, Director of Regulatory Strategy and Policy from Certara based on an online seminar presentation, made in collaboration with TOPRA and Certera, given on 15 November 2022.
March 2023 | Volume 20 | No.3
When I started writing clinical evaluation reports for medical devices in 2014, there was no prescribed schedule for updates, and it was accepted that anyone who had the time could both write or update them. Since then, clinical evaluation has developed into a specialty. More than a simple report, clinical evaluation now needs to be part of the medical device design process to ensure sufficient clinical evidence, avoid nonconformance and questions from the notified bodies (NB).
Those of us who perform clinical evaluations need to know the applicable guidance and regulations required – and the documentation needed – including clinical evaluation plans and reports, and literature protocols and reports. We need to understand how to perform methodologically sound literature reviews, what constitutes ‘state of the art,’ and how to demonstrate equivalence. It is also critical to know how to gather sufficient clinical data and how to conduct post-market clinical follow-up.
More than a simple report, clinical evaluation now needs to be part of the medical device design process to ensure sufficient clinical evidence.
In 2014, the MED DEV 2.7/1 Revision 3 guidance was to be followed, which was approximately 30-pages long and – frankly – vague. That year, I authored three clinical evaluation reports (CER). Between the beginning of 2015 and June 2016, I authored another nine. However, release of the MED DEV 2.7/1 Revision 4 that month changed the meaning of clinical evaluation and turned the regulatory world upside down.
My nine-reports-a-year schedule instantly became 23-per-year, and my reports almost doubled in length. The Rev 4 update expanded the definition of clinical evaluation to include an ongoing procedure to collect, appraise, and analyse clinical data pertaining to a medical device, as well as evaluate whether there is sufficient clinical evidence to confirm compliance with relevant essential requirements for safety and performance when using the device according to manufacturer’s Instructions for Use.
By the end of 2016, we added four CER members to our team to incorporate the MED DEV 2.7/1 Rev 4 updates. In April 2017, Regulation (EU) 2017/745 for medical devices (MDR) was published and contained sections specific to clinical evaluation (Article 61 and Annex XIV). These sections emphasised the need for a systematic process to continuously generate, collect, analyse, and assess the clinical data pertaining to a device to verify its safety and performance when used as intended by the manufacturer.
These changes have made CER writers’ tasks exponentially more complex. Now, they will need to be familiar with more than 80 Medical Device Coordination Group (MDCG) guidance documents, the International Medical Device Regulators Forum (IMDRF) guidance, and MED DEV 2.7/1 Rev 4 guidance document, as well as EU MDR requirements.
In this article, I’ll explain the stages of clinical evaluation and required documentation, the literature protocol and review, key components of the CER, and post-market surveillance (PMS).
Stages of clinical evaluation: Required documentation
MED DEV Rev 4 identifies five stages for clinical evaluation – up from four in the previous revision. The first stage is now Stage 0 — scoping the clinical evaluation, while Stage 1 is identifying the clinical data. In Stage 2, the amassed data is appraised using a systematic approach, and in Stage 3, the data is analysed. In Stage 4, the clinical evaluation is documented in the report with conclusions regarding device safety and performance; this is laid out in Section 11 and Appendix 9 and 10 of Med Dev 2.7/1 Rev 4.
The stages of clinical evaluation equate to deliverables. What used to be a report has now evolved into a clinical evaluation plan (CEP) – a literature search protocol and report for both clinical data and state of the art which then feeds into the CER. Clinical evaluation is not the only area that has had additional deliverables. PMS now requires a plan, the data and the report, and post market clinical follow-up also now requires a plan, the data, and the report.
While the CEP seemed initially ambiguous, I now treat it as an outline – or map – for conducting the CER, literature protocol, and literature review. In terms of the guidance documents, MDCG 2020-13 gives the NB a list of what they should expect in a clinical evaluation plan. This is a great place to start if you haven’t written a CEP before.
The CEP should meet the requirements of Annex XIV Part A Section 1 and should:
- Identify the general safety and performance requirements (GSPR) which require clinical data support.
- State the intended purpose of the device.
- Identify the intended population. n Include the indications and contraindications.
- Provide a detailed description of intended clinical benefits to patients.
- Identify clinical safety data as well as residual risks and side-effects.
- Describe state of the art for the device.
- State the type of clinical evaluation, whether it is based on clinical data or equivalence.
Section 7 and Appendix 3 of the MED DEV 2.7/1 Rev 4 contains additional details.
Literature protocol review
In the CEP, we need to include how we will perform a methodologically sound literature review, starting with a literature search protocol to identify additional clinical data pertaining to our product.
As with the CEP, this may be a standalone document which would be included as an appendix. Alternatively, some clients include it in the clinical evaluation report prior to the literature summary. The guidance calls for a methodologically sound and repeatable search of the literature, which must be included in the protocol for the literature search. It should include the background of the device name/model, why the review is being conducted, any prior literature reviews and those findings, and how it will feed into the risk management process. It should include equivalent devices for the clinical data search and similar devices for the state-of-the-art search.
The protocol should also show how it is objective and consistent with the scope of the clinical evaluation and should include the device description and intended performance of the device, including any claims on clinical performance and clinical safety. Additionally, it should describe the methods for the literature review including:
- Dates for the search.
- Source or databases that will be searched and justification for selection.
- Databases may include EMBASE, PUBMED, Cochrane, Science Direct, Google scholar; MED DEV 2.7/1 Rev 4 recommends two databases.
- For state-of-the-art literature searches, include clinical practice guidelines, applicable standards, and consensus statements.
- Databases may include EMBASE, PUBMED, Cochrane, Science Direct, Google scholar; MED DEV 2.7/1 Rev 4 recommends two databases.
- Search strategy including filters applied.
- These depend on the database, but some of the more expansive allow you to include types of studies, types of devices, and manufacturers.
- Inclusion/exclusion criteria.
- Criteria used to appraise the data. This is vitally important. NB repeating our search often want justification for protocol development.
As you construct a literature protocol, you need to determine your search terms and filters. Our literature search then needs to be objective, non biased, systematic and use appropriate review methods. The MED DEV pg. 38 provides a list of examples of methods to employ, which include:
- PICO (patient characteristics, type of intervention, control, and outcome queries).
- The Cochrane Handbook of Systematic Reviews of Interventions.
- PRISMA (The Preferred Reporting Items for Systematic and Meta-analyses) Statement.
- MOOSE Proposal (Meta-analysis of Observational Studies in Epidemiology).
Once the protocol and literature search Once the protocol and literature search methodology are established, we need to perform a systematic literature review. Many writers run into issues later by conducting a traditional literature review rather than a systematic review. While a traditional review (Table 1) may be used for developing parts of the state-of-the-art section, supporting the specific device or equivalent device with clinical data requires a systematic search.
|Table 1: Traditional vs systematic literature reviews|
|The review question||Broad scope
Looking for supporting evidence
|Searching for studies|| Ad hoc searches
Non-exhaustive or not comprehensive
| Find all published and unpublished literature
|Study selection||Lack clear inclusion/exclusion criteria||Explicit and informed inclusion exclusion criteria|
|Assessing the quality of included studies||May not consider study quality or bias||Assess quality and bias as part of inclusion exclusion|
|Synthesis of existing research||Conclusions are qualitative and not based on study quality||Conclusions based on study quality|
* Brown University Library, 2021
After the search question and terms have been identified in the protocol and the plan has been established for the state of the art and the clinical data, the next step is implementing the search and ensuring it is repeatable and documented. One way to perform searches is to:
- Input the terms and use Boolean logic, then add filters including dates, types of studies, English and Human at a minimum, and perform the search.
- Export the search and save it prior to narrowing the search. Once you have done this, you can begin going through the literature.
- Review each article for inclusion and exclusion criteria based on your protocol criteria.
You can narrow the search initially by correct device or correct indication. If you have too many articles, you can update your protocol by adding additional filters or inclusion/exclusion criteria. For example, when searching for a Peripherally Inserted Central Catheter, I get information on arterial or urethral catheters. I then exclude those and look at the abstracts for additional information. Lastly, I pull the full text of each article for safety and performance data, determining if it should be included for in-depth review.
Once we have our literature, we need to assess the suitability of the data. Table 2 is an example from the IMDRF on clinical evaluation from October 2019.
To assess suitability, we need see if data was generated from the device in question, ask if the device was used for the same intended purpose, determine if the patient group is representative of the intended population and condition, and ask if the data is sufficient for an objective assessment.
|Table 2: Appraisal criteria for data suitability|
|Appropriate device||Was the data generated from the device in question?||
|Appropriate device application||Was the device used for the same intended purpose?||
|Appropriate patient group||Was the data generated from a patient group that is representative of the intended population and clinical condition?||
|Aceeptable report/data collection||Does the data contain sufficient information for an objective assessment||
Clinical evaluation report: key components
Once the literature is summarised, the information gathered should be used to compile the CER. Med Dev 2.7/1 Rev 4 Section 11 and Appendices 9 and 10 outline the content required for the report. This should include the executive summary, the scope of the evaluation, device description, and sections on background and state of the art. It should also include clinical data from the manufacturer and the literature, an appraisal and analysis of the clinical data, and a review of any complaints in the PMS and PMCF, followed by conclusions. These sections should be clearly identified and cross-referenced to ensure that the NB can find what they need.
If you want to claim clinical equivalence, it is important to remember that equivalence is not the same as a US predicate device. You must have access to the technical information for the device, and the same indication, intended purpose, and body site with a similar population, performance, and technical design. Conditions of use should be the same and device specifications and/or physiochemical properties (eg, flow rates, pressure settings, tensile strength, or shelf life) should be similar.
If you want to claim clinical equivalence, it is important to remember that equivalence is not the same as a US predicate device.
In biological terms, the device should use the same materials and contact the same tissue or bodily fluids. Applicants must also identify, evaluate, and disclose any differences; explain why the differences don’t significantly affect clinical performance or clinical safety; and provide justification for the device’s use, as well as the data related to the European population.
A large component of the clinical evaluation is establishing state of the art, to show why your device exists and that it is at least as good as the alternatives. To establish your device as state of the art, review the historical knowledge, state how your device originated, review relevant medical conditions to show how you can help with that disease or condition, and discuss similar devices and alternatives. You will also need to review the benefit-risk of similar devices, or giving no treatment at all, and compare that to your device.
Finally, it is challenging to understand whether you have sufficient clinical evidence. MDCG 2020-6, (Figure 1) which reviewed clinical data requirements for legacy devices, provides a glimpse of what to expect. It states a clinical investigation shall be performed for Class III and implantable devices, but distinct exemptions from this requirement are identified in articles 61(4), 61(5), and 61(6). Sufficient clinical evidence is understood as the result of a qualified assessment, which reaches the conclusion that the device is safe and achieves the intended benefits. It suggests a hierarchy of clinical evidence, presented in a pyramid form, the top being the most robust:
Tier one is high-quality clinical investigations covering all device variants, indications, and patient populations. Tier two covers high-quality clinical investigations with gaps, and tier three is high quality clinical data coming from a registry.
Tier four is studies with flaws, but data can still be justified for inclusion here. Class III legacy devices and implantable legacy devices that are not well-established technologies should have sufficient clinical data as a minimum at level four.
Technologies that are well-established may be able to have lower levels of data if they can confirm compliance with the relevant GSPR and provide evidence from additional sources. However, it is insufficient to rely solely on complaints and vigilance.
We cannot rely on one data source. Every device should have multiple sources, and the clinical evaluation team or person should assess each source, add them together, and determine whether they are sufficient. Any gap or high-risk devices will then need post-market clinical follow up (PMCF).
PMS should include summaries of the post market report, including vigilance reports and trend reports. These should include performance and safety complaints, corrective and preventative actions, public database searches, and recalls. The PMS public databases to search will depend on where you are selling your product but may include: Maude (FDA), EUDAMED (Europe) (once it is up and running), TGA (Australia), database of Adverse Event Notifications, Swiss Medic database on medical devices (Switzerland), BfArM database information medical devices notification (Germany), Health Canada (Canada), and Ministry of Health (Australia). If there are similar or equivalent products, it is important to search those products as well.
PMCF is part of PMS and is expected unless you can justify why it’s not needed. PMCF is a continuous process that updates the clinical evaluation. It is conducted with a PMCF plan and is an element of the overall PMS plan. It can include clinical experience and feedback through surveys, proactive scientific literature screening, evaluation of information from registries, or by conducting PMCF studies.
PMCF is part of PMS and is expected unless you can justify why it’s not needed.
Post-market performance follow up (PMPF) requires you to proactively collect and evaluate data on clinical performance, confirm its scientific validity, ensure the continued suitability and acceptability of identified risks, and the detection of any new risks based on the evidence collected.
Successful clinical evaluation depends on knowledge of guidance documents including MED DEV 2.7/1 REV 4, MDCG 202-5, 6 and 13 and MDR article 61 and annex XIV, as well as qualified authors and evaluators. You need to ensure that you have completed all stages of clinical evaluation, including scoping, identification of the clinical data, appraisal, analysis of that data, and summarising that analysis in the required deliverables.
You must also establish your device as state of the art through a solid background and disease review, identifying similar devices and alternative therapies, and showing that your device is relevant and can treat the disease. And finally, if you are relying on the equivalence that you have one device that is CE marked and you have access to the technical information, you must show that any differences do not affect the safety or performance of your submitted device, ensure that you have looked at all the forms of clinical data, and summarised those to show that it is sufficien