Validating AI and Machine Learning Features Under GxP
A test script asserts an expected result, and a generative model does not have one. Validating AI under GxP therefore moves the assurance to the system around the model: a precise intended use statement, bounded risk, baseline performance, drift monitoring, evidenced human review, and a defined withdrawal trigger.
The question gets asked defensively, usually as an argument for not deploying anything. It is a fair question with a workable answer, and the answer is not that regulated organizations cannot use these features. It is that the evidence looks different from the evidence a deterministic system produces.
This guide covers why scripts break, what an intended use statement has to contain, how to bound risk so a wrong output is caught, and the four controls that stand in for the test script. It sits under what you own and what the vendor owns in eClinical validation.
After reading this you will be able to:
- Explain to a quality colleague why scripted testing does not apply
- Write an intended use statement precise enough to validate against
- Bound a feature so that a wrong output is caught before it matters
- Define monitoring and a withdrawal trigger rather than deploying and hoping
Why Scripts Break
The distinction matters because a lot of what gets called AI sits in the middle column and is more testable than the conversation assumes. A risk score produced by a stable method can be tested at population level: given this dataset, the distribution of scores should look like this, and these known cases should rank above those.
The right-hand column is where the scripted approach genuinely has nothing to attach to. Two runs over the same verbatim term can produce two differently worded but equally reasonable outputs, and no expected-result field accommodates that.
Intended Use, Written Precisely
Everything else hangs from this document, and most of the difficulty dissolves once it is written properly.
The second element does most of the work. "Suggests only, with a human accepting, editing or rejecting every suggestion" describes a bounded feature whose failure mode is a wasted few seconds. "Populates the field directly" describes something entirely different with the same underlying model.
The fourth element is the honest test. If you cannot name the step that catches a wrong output, the risk is not bounded, and no amount of model performance documentation compensates for that.
Bounding the Risk
Bounding is a design activity rather than a documentation one. The question is what the output can reach without a human in between.
A suggestion written to a draft field that cannot be submitted until someone accepts it is bounded by construction. A score that routes a record into a different review queue is bounded if the queue still ends in human review, and unbounded if it routes records away from review entirely. The same model can be either depending on where you place it.
This is the same risk determination the assurance approach applies to conventional software: what happens if it fails, not how sophisticated it is. The general method is in what the assurance approach changes in practice.
Four Controls in Place of the Script
Baseline measurement is the step most often skipped, and without it the other three have no reference point. Measuring performance before release, on data representative of actual use rather than on the development set, is what makes drift detectable later.
The fourth control is the one that distinguishes a validated deployment from an optimistic one. A defined trigger states what change in performance prompts retraining, re-review or withdrawal, and who decides. Without it, a degrading model stays in production because no one owns the decision to remove it.
Evidencing Human Review
Claiming a human reviews the output is easy. Showing it is a records question.
Acceptance, edit and rejection rates are the practical evidence, and they carry a signal worth watching. A feature whose suggestions are accepted without exception, every time, is not demonstrating excellence so much as suggesting that review has become a formality. Reviewers who never disagree are a finding in waiting, and the rate is visible long before anyone asks.
The audit trail has to keep the two contributions separable. If a model proposes a value and a person edits it, the record should show the proposal, the edit and who made it, rather than one entry attributed to the user. Collapsing them removes the ability to answer what the model actually did, which is the first question anyone will ask. The general requirements for trail design and review are in writing an audit trail review procedure.
Models Change Even When You Do Not Change Them
Conventional change control assumes a system stays as it was until someone modifies it. That assumption fails twice with AI features.
A model can degrade through drift without anyone touching it. And where the model is supplied by a vendor, it may be updated on their schedule, which means your validated state changes without a change request on your side. Both need to be handled in the agreement and in your own assessment process: what notice you receive, whether you can evaluate a new version before it reaches you, and what your criteria are for accepting it.
That is the SaaS upgrade question with sharper edges, and the general version is covered in what the vendor validates and what you still own.
Where These Features Actually Earn Their Place
The validation path above is proportionate for a bounded, reviewed feature. It becomes disproportionate quickly as the output moves closer to an unreviewed record, which is a useful filter when deciding where to deploy at all.
The strongest cases tend to be high-volume, low-stakes-per-instance work where a human remains in the loop: coding suggestions, first-pass document classification, surfacing anomalies for review. Our work across clinical and R&D, medical affairs, commercial excellence and post-market surveillance applies the same validation approach across all four, because the question does not change with the domain.
References
- Computer Software Assurance for Production and Quality System Software. US Food and Drug Administration guidance for industry. www.fda.gov
- Good Machine Learning Practice for Medical Device Development, Guiding Principles. US Food and Drug Administration, Health Canada and MHRA. www.fda.gov
- Annex 11, Computerised Systems. EudraLex Volume 4, Good Manufacturing Practice guidelines. health.ec.europa.eu
- ICH E6(R3) Good Clinical Practice. International Council for Harmonisation, adopted 6 January 2025. www.ich.org
This guide describes process and regulatory expectations in general terms and is not legal or regulatory advice. Confirm the current version and applicability of any standard or guidance for your study and region.
Frequently Asked Questions
How do you validate something that gives a different answer each time?
You validate the system around the model rather than asserting a single expected output. That means defining intended use precisely, bounding what the feature is allowed to affect, measuring baseline performance before release, monitoring for drift afterward, and evidencing that a human reviews the output. The assurance comes from the controls and the monitoring record, not from a test script that cannot be written.
Does an AI feature have to be high risk?
No, and the risk depends entirely on what the output reaches. A model that suggests a coding term for a human coder to accept or reject is bounded: a wrong suggestion is caught at the next step. A model whose output flows into a submitted record without review is not bounded, and carries the risk of whatever that record affects. Intended use and bounded risk are the same conversation.
What is model drift and why does it matter for validation?
Drift is performance changing over time as the data the model sees diverges from the data it was built on. It matters because traditional validation assumes a system that behaves consistently until someone changes it, and a model can degrade without anyone changing anything. That is why a baseline measurement and ongoing monitoring take the place of the one-time test, and why a defined trigger for retraining or withdrawal is part of the validation rather than an operational afterthought.
What does evidencing human review actually mean?
Keeping a record that shows review happened and had substance. Acceptance, edit and rejection rates are the practical measure: a feature whose suggestions are accepted without exception every time is evidence that review has become a formality, which is itself worth investigating. The audit trail should distinguish what the model proposed from what the human recorded, so the two are separable after the fact.
Do these features need their own audit trail treatment?
Yes, and the requirement is that the model contribution and the human action are distinguishable in the record. If an output is written into a field and a person subsequently edits it, the trail should show the proposal, the edit and who made it. Collapsing both into a single entry attributed to the user removes the ability to answer what the model did, which is a question that will be asked.
Can Nirmitee Healthtech help validate an AI feature we are deploying?
Yes, and the work starts with the intended use statement, because most of the difficulty resolves once that is written precisely. From there the path is bounding the risk, establishing a baseline, defining the monitoring and the withdrawal trigger, and designing the audit trail so model and human contributions stay separable. Our capability work spans clinical and R&D, medical affairs, commercial and post-market, and the validation approach is common across all four.
