By Geoff Bloom, Partner, Bonnie Liu, Associate, and Anna Massasso, Paralegal
How to use AI-generated evidence in TGA submissions
|
Healthcare is generally considered a high-risk setting for AI applications, as flawed AI decision-making can, depending on the application, have serious consequences for patient health, safety and clinical outcomes.
A key question for sponsors seeking to deploy AI applications to assist with regulatory decision-making related to therapeutic goods is how to ensure that these AI models produce high-quality outputs that are credible enough to meet the TGA’s stringent evidentiary requirements. To this end, Australian sponsors interested in incorporating AI to assist with the preparation of regulatory applications and dossiers to the TGA may find it helpful to consider the FDA Guidance. Published in January 2025, the draft guidance reflects the current thinking of the FDA towards the use of AI to produce information or data intended to support regulatory decision-making regarding safety, effectiveness, or quality for medicines and biologicals, but is equally applicable to medical devices.
The FDA’s draft guidance, Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products – Guidance for Industry and Other Interested Parties (January 2025) (FDA Guidance) expressly states that it “does not establish any rights for any person and is not binding on FDA or the public,” which might lead one to doubt the usefulness of the document. In fact, the value of the document is not connected to the extent to which it is binding.
Its value is in providing a well articulated description and assessment of the extent to which information generated with the involvement of AI can be relied on by therapeutic goods regulators, assisting regulators and industry to engage with each other in practical and concrete ways when AI-generated information is used in regulatory submissions. These considerations apply across all jurisdictions. We think that they are wholly applicable in an Australian context.
The guidance establishes a seven-step, risk-based credibility assessment framework. This framework encourages sponsors to consider the types of questions necessary to establish and assess the credibility of AI model outputs for a specific context of use.
| 1. | Defining the question of interest to be addressed by the AI model
The question of interest to be addressed by the AI model is the specific question, decision, or concern being addressed by the AI model. |
| 2. | Defining the Context of Use for the AI model
The “Context of Use” defines the specific role and scope of the AI model used to address a question of interest. Defining the Context of Use involves detailing what the AI will model do and how its output will be used. |
| 3. | Defining AI model risk
The AI model risk is the possibility that the AI model output may lead to an incorrect decision that could result in an adverse outcome, and not risk intrinsic to the model. Model risk is determined by combining “Model Influence” (being the contribution of evidence derived from the AI model relative to other contributing evidence – e.g. compare the influence of a sole black box AI being responsible for an output versus an AI intended to be used in an advisory capacity only) with “Decision Consequence” (being the significance of an adverse outcome resulting from an incorrect decision concerning the question of interest). A higher Model Influence or higher Decision Consequence generally results in increased model risk. |
| 4. | Developing a Credibility Assessment Plan to establish the credibility of AI model output within the defined COU
This step is the centrepiece of the FDA Guidance and contains considerable technical discussion. It requires developing a plan to establish the credibility of AI model output within the established context of use (“Credibility Assessment Plan”). Broadly, this involves taking the following steps to describe the model and evaluate its effectiveness: a. Describing the model – including, amongst other things, model inputs, outputs, architecture, feature selection process and parameters. b. Describing the data used to develop the model – including developmental datasets – i.e. the datasets used for training of the model, developmental data collection, storage and processing, the rationale for the use of the developmental datasets and the appropriateness of the developmental datasets to the context of use. c. Describing the model training – including details of how the model was trained, such as the learning methodology, performance metrics, techniques to prevent over or underfitting, training hyperparameters and calibration, whether the model was pre-trained, and quality assurance and controlling procedures of software methods and how version changes were tracked. d. Model evaluation – the evaluation of the fully trained model to assess the adequacy of model performance for the intended context of use on test data (which is data used to characterize the performance of the model that is independent to development data). |
| 5. | Executing the Credibility Assessment Plan
This simply involves implementing a plan to mitigate the risks of model usage. |
| 6. | Document the results of the Credibility Assessment Plan and discuss deviations from the plan
The results of the Credibility Assessment Plan, and any deviations form the plan, should be included in a report (referred to in the guidance as a “Credibility Assessment Report”). This report should provide information that establishes the credibility of the model for the context of use. |
| 7. | Determine the adequacy of the AI model for the context of use
Based on the results documented in a Credibility Assessment Report, a sponsor can then determine whether a model may or may not be appropriate for the proposed context of use. If the model credibility is not sufficiently established relative to the model risk, sponsors may wish to consider several actions to reduce the risk posed by the AI model. These options can include: a. downgrading the Model Influence by incorporating other types of evidence in conjunction with the AI model to answer the question of interest; b. increasing the rigor of the credibility assessment activities in the AI model’s output by adding additional development data; c. establish appropriate controls to mitigate risk; d. change the modelling approach; or e. consider the credibility of the AI model’s output inadequate for the context of use and revise the model’s context of use. |
Although the FDA Guidance is only a draft, relates to the US, not Australia, and is not even binding in the US, we think that it provides great guidance to both regulators and the regulated on how to assess information to be used to make decisions about decisions on therapeutic goods in Australia.
The guidance is not just about the decision making, but about the whole design of an AI model to be used to generate that information, as well as providing suggestions for how a sponsor or other regulated person may wish to structure their engagement with the TGA in developing the AI model. For example, it suggests that a sponsor contact the regulator early in the model design process, and at regular stages in the development process, to ensure that the regulator, at least in theory, does not have objections to / is likely to accept the results of the AI model.
The TGA makes decisions about science subject to legislation. We are expert at assisting clients to engage with the TGA to give their case the greatest likelihood of success on ultimate legislative questions such as whether the TGA should register a therapeutic good, or should institute a mandatory recall.
If you would like further information or have any queries regarding other matters, please do not hesitate to contact:
