A validation lead at a mid-size sponsor put the problem to us more precisely than any vendor deck has: "I know how to write an OQ script for software that returns the same answer twice. What do I write for something that does not?"
That question is now sitting in front of every life sciences organization piloting AI document authoring, and it is not rhetorical. Retrieval-augmented generation has moved from experiment to production in regulatory and medical writing faster than the validation playbook has adapted. Platforms like AlphaLife Sciences' AuroraPrime generate first drafts of protocols, clinical study reports, safety narratives, and CMC documents by synthesizing upstream source documents and data, and the company announced a Veeva RIM integration in March 2026 that pushes that content directly into the vault where submissions are assembled. The technology arrived. The question of what "validated" means for it did not arrive with it.
Our answer, and the argument of this post, is that the question contains a category error. You do not validate the model. You validate the gate. And the gate, unlike the model, is perfectly ordinary deterministic software that can be regression-tested with the tools you already have, which makes it strange that almost nobody is testing it.
The category error, drawn
Three concentric layers, three different epistemologies:
- The model. Non-deterministic by construction. Sampling, model version changes, and provider-side updates mean identical input does not guarantee identical output. You cannot freeze it and you should stop trying.
- The retrieval pipeline. Measurable in aggregate but not per-run. Chunking, embedding, top-k, reranking, and the index itself all shift retrieval behavior. You can characterize this statistically against a fixed evaluation set. You cannot assert on a single result.
- The control surface. The application around the model: the review interface, the source traceability pane, the citation display, the accept and reject affordances, the audit trail, the permission model, the export path into RIM. This is ordinary web software. It is fully deterministic. It either renders the source link or it does not.
Regulators have been converging on roughly this framing. The FDA's January 2025 draft guidance on AI to support regulatory decision-making organizes everything around context of use and a risk-based credibility assessment rather than around proving a model correct in the abstract. The EMA's reflection paper on AI in the medicinal product lifecycle takes a similarly lifecycle-oriented view. And in January 2026 the two agencies jointly published guiding principles for good AI practice in drug development, whose themes read almost as a specification for what to test: human-centric design, a risk-based approach, a clear context of use, and life-cycle management.
Read those principles as a validation lead and the conclusion is direct. If your control is "a qualified medical writer reviews every AI-generated claim against its source before acceptance," then the object of validation is not the model's accuracy. It is whether that review can actually happen, reliably, every time, in the interface where it is supposed to happen.
What the eval harness tells you, and what it does not
The first instinct of a technically competent team is to build an evaluation harness, and they should. A golden set of representative queries with known-good source documents, scored on faithfulness, groundedness, context recall, and answer relevancy, with thresholds that must hold before a change ships. This is genuinely necessary work and it catches the class of degradation that matters most in RAG: silent quality drift when an embedding model is upgraded, a corpus is re-chunked, a prompt is tweaked, or an LLM is swapped.
What an eval harness cannot tell you is anything about the layer where your regulated control lives. Consider a handful of failures that a perfect eval score is entirely compatible with:
- The traceability pane renders a source citation whose link target 404s. Every generated claim is correctly grounded; no reviewer can verify any of them.
- A release changes the review pane's default state to collapsed. Reviewers now accept drafts without expanding the sources, and the audit trail records approval either way.
- The accept-all control acquires a keyboard shortcut adjacent to the one reviewers use to scroll. Bulk acceptance goes up. Nothing in the model changed.
- A generated section renders correctly in the authoring view and drops a table when exported to the submission format. The document that reaches RIM is not the document that was approved.
- The confidence indicator that reviewers have learned to triage by changes its color scale in an upgrade. Low-confidence output now reads as acceptable at a glance.
Every one of those is a control failure with regulatory consequences, and every one of them is invisible to metrics computed on model output. They are also, without exception, the sort of thing a stored baseline of the rendered interface catches on the next run. That is the gap this post is about.
The control surface is where validation actually lives
Once you accept that framing, the validation plan writes itself, because each documented control maps to a testable rendered state. Here is the mapping we would build for a RAG authoring system in a regulated environment.
| Documented control | What has to be true in the interface | How it is tested |
|---|---|---|
| Every generated claim is traceable to a source | The traceability pane renders, is expanded by default, and every citation resolves to the cited document at the cited location | Baseline the pane; drive each citation link and compare the destination against its baseline |
| A qualified human reviews before acceptance | Accept is unavailable until the source pane has been opened; the reviewer identity is captured and rendered | Replay the review flow in both orders and baseline both states, including the disabled control |
| Edits and rationales are recorded | The rationale field is present, required where policy says required, and the recorded entry renders in the audit view | Seed an edit with a known rationale, baseline the audit view, enforce the text layer on it |
| Generated content is distinguishable from human-authored content | The provenance marker renders on every generated block, in export as well as in the editor | Baseline editor and exported rendering of the same seeded document |
| The approved document is the exported document | Export fidelity: no dropped tables, figures, footnotes, or numbering | Compare the exported rendering against the approved rendering as paired baselines |
| Access is role-appropriate | Each role sees exactly the affordances its SOP grants | Replay the same seeded document as each role, baseline each view |
Nothing in that table requires the model to be deterministic. Every row is a claim about the application, and every claim is checkable by replaying a seeded scenario and comparing the rendered result against a state a qualified human previously approved. That is exactly the epistemology a regulated organization already runs on: a human confirmed this was acceptable, and here is the versioned evidence, and here is the proof it has not moved since.
Seeding a system that will not repeat itself
The obvious objection: if generation is non-deterministic, how do you baseline a screen whose content comes from the model?
You separate the two things that are being conflated. There are screens whose content is generated and screens whose structure is the control. You do not baseline the former as pixels, and you do not need to.
Concretely, three moves:
- Pin the generation. Capture one real generation run against a fixed synthetic corpus, store the output as a fixture, and replay the application against that stored output rather than calling the model live. The screen becomes fully deterministic. You are now testing the interface, which is what you meant to test.
- Mask what must vary. Where live generation is genuinely part of the scenario, mask the prose region and baseline the surrounding chrome: the citation count, the provenance markers, the control states, the pane layout. The structure is the control; the words are the model's business and the golden set's problem.
- Keep the golden set separate and running. The eval harness continues to do its job on model and retrieval quality, on its own cadence. Two systems, two questions, neither pretending to answer the other's.
This split is not a compromise. It is the correct decomposition, and it is close to the argument we made in testing AI features with non-deterministic UI, applied to a setting where the stakes justify the rigor.
Where platforms like AuroraPrime already help, and where the gap remains
It is worth being fair about the state of the tooling, because the better platforms in this space have made deliberate architectural choices that make validation easier rather than harder.
AuroraPrime's public positioning is built on grounding output in verified source documents through RAG rather than open generation, keeping medical writers in the loop rather than automating them out, and recording AI-generated content along with user feedback and edit rationales so that claims remain traceable back to source. The company describes operating within 21 CFR Part 11, HIPAA, GDPR, SOC 2 Type II, ISO 9001, and ISO 27001 frameworks. Those are the right primitives. A system built that way is validatable in principle, which is not something you could say about a chat window pointed at a corpus.
The gap is not architectural, it is operational, and it is the same gap in every enterprise application that ships three or four times a year. Traceability panes are software. Review gates are software. Audit views are software. They change in releases, they change when a customer's configuration changes, and they change when an integration on either side of them changes. A control that was verified once during initial qualification and never checked again is a control whose current state nobody knows.
The honest framing for a buyer is this: your vendor's compliance posture tells you the controls were designed correctly. It does not tell you they rendered correctly in your tenant this morning. Those are different claims and only one of them is continuously verifiable by you.
A concrete first month
If this is a program you are starting rather than auditing, the sequencing that works is small and unglamorous.
- Week one. Write down the controls. Not the features, the controls: the specific statements in your SOP or validation plan that a human relies on. Most teams find they have between six and twelve. This list is the whole scope.
- Week two. Build the synthetic corpus and pin one generation run per document type. Nothing real, nothing patient-identifiable, deterministic identifiers throughout.
- Week three. Baseline the control surface: one scenario per control per role. Ten to twenty baselines total, not hundreds. Get it green and stable before expanding, because a suite that is red for reasons nobody trusts gets muted in week four.
- Week four. Wire it into whatever triggers a change: vendor release notes, configuration deployments, integration updates. Run it unattended and route only unexplained diffs to a human.
Deployment matters here more than in most applications, because the screenshots contain unpublished clinical and regulatory content. Self-hosting means those images never leave your network, which removes the objection that otherwise ends this conversation in a security review.
Frequently asked
Does this mean model evaluation is unnecessary? No. It means model evaluation answers a different question than validation does. Evals tell you whether generation quality has drifted. Control-surface testing tells you whether the human safeguards you documented still function. A program with only one of the two has a large blind spot, and most programs currently have only the first.
Is a screenshot comparison really a validation artifact? It is when it carries the chain of custody: a named human rendered a verdict, the baseline is versioned with a reason, and the run is reproducible from a link. That combination is what turns a passing check into evidence, and it is the same standard we apply in the enterprise compliance guide.
Our vendor is SOC 2 and Part 11 aligned. Is that not enough? It is necessary and it is not sufficient. Those attestations describe how the vendor builds and operates. They say nothing about whether the specific configuration in your tenant renders your specific controls correctly after this quarter's release. Only a test in your tenant answers that.
What if the vendor changes the interface between releases? Then your suite goes red, which is the correct and desired outcome. A vendor-side interface change to a control surface is exactly the event your validation program should be notified about, and today most organizations find out when a reviewer mentions something looks different.
Start here
The uncomfortable summary is that the industry has spent two years arguing about whether AI-generated regulatory content can be trusted, while the layer that actually determines whether a human can check it has gone almost entirely untested. That layer is deterministic, cheap to baseline, and free to replay.
Start with the release-testing playbook for the cross-platform version of this pattern, or read the companion piece on citation drift, which covers the specific failure where generated prose stays plausible while its sourcing quietly moves. If your generated content lands in Veeva, the Vault RIM testing recipe covers the downstream half of the same pipeline. Lastest is open source at github.com/las-team/lastest, self-hosted for free, and every replay after the first costs nothing in tokens, which is the property that makes running this on every vendor release actually realistic.