Updated:
When the AI model changes: what to ask the vendor (CDSCO)
CDSCO’s guidance describes an Algorithm Change Protocol for software that changes. A checklist of what it can contain, for a hospital reading a proposal.
An AI model that works well in a pilot may not be the same model a year later. Vendors retrain, update and replace. CDSCO’s Guidance Document on Medical Device Software addresses this with an Algorithm Change Protocol, which it says “may be devised, wherever applicable”.
What the protocol is
The guidance says the protocol shall include an overview of all the procedures to be followed so that changes to the software do not compromise its safety and its intended use. It may be submitted as part of the risk management file. Because it uses the word “may”, a vendor can lawfully have none; the guidance does not make it universal.
What it may contain
According to the guidance, the protocol may contain:
- a data management plan, covering data management, risk assessment, new data collection and quality assurance;
- a performance evaluation and monitoring plan, with metrics, a statistical analysis plan, how often performance is assessed, performance targets and a post-market monitoring overview;
- an algorithm retraining plan, where applicable;
- a software update plan, with version tracking, verification and validation, update triggers, update procedures and ways to communicate updates transparently to end users; and
- a rollback plan, with triggers, backup and recovery procedures and communication to users.
The guidance also asks applicants to document risks including cybersecurity and AI bias, and treats unintended bias in clinical decision-making caused by a software output as a possible indirect harm.
Questions for a vendor
- Do you have an algorithm change protocol for this product, and which changes does it cover?
- How will we be told before a change reaches our wards, and can we delay it?
- What performance figures do you monitor after go-live, and will you share them?
- If an update causes a problem, how is it rolled back and who tells our staff?
Key takeaways
- CDSCO describes an Algorithm Change Protocol as something that “may be devised, wherever applicable”; it is not stated as mandatory for every product.
- It can include data, performance monitoring, retraining, update and rollback plans.
- Ask how updates are communicated and rolled back before the contract is signed.
Sources: CDSCO, Guidance Document on Medical Device Software, Algorithm Change Protocol