Data Traceability for AI in Pharma: the 5 Questions from QA teams

A practical guide for Quality, CMC and Regulatory Operations teams turning documents into structured, analysable data at scale, in a way that stands up in front of an auditor: what every value must carry, and five questions to ask before you trust it.

Life sciences teams are moving to AI-assisted work on Product Quality Reviews, CMC analyses and regulatory dossier drafting. One of the biggest challenges is the quality of the underlying data. With parallel drafting, where writing starts before a clinical trial has ended, or simply with the sheer mass of data (a typical phase III trial now generates close to 6 million data points), knowing which data is the latest trusted version is key.

This is already a known issue: data integrity concerns were cited in 15% of the FDA warning letters posted in 2025, according to an analysis by Pazhayattil, Ingram and Bhimavarapu in Pharmaceutical Online. The pressure will only grow as GenAI supports drafting. It is only a matter of time until the FDA finds made-up data in an eCTD.

In a GxP environment, an AI output you cannot trace is an output you cannot use. Traceability is what makes effective human and computerised checks possible in the first place. This guide defines what a traceable AI-extracted value must carry, works through the five questions experienced QA teams ask before they trust a system, and ends with a test you can run on your own documents. Teams that get this right are the ones who move AI beyond the pilot, because every value they publish is defensible.

In this article

• 1. What a traceable AI value carries: the Five-Point Traceability Record
• 2. "Does it flag uncertain values instead of guessing?"
• 3. "Can I see where every value came from?"
• 4. "Is the model locked and versioned like any validated system?"
• 5. "Can we show an auditor who changed what?"
• 6. "Does it know life-science documents, or is it generic AI?"
• 7. The Silent Error Test: checking traceability on your own documents
• Conclusion
• FAQ

1. What a traceable AI value carries: the Five-Point Traceability Record

Data traceability for AI-extracted values means that every value can be followed back to the exact place it was read in a specific, controlled version of the source document, and that its history is recorded: which model read it, which checks it passed, and who confirmed or changed it. If any link in that chain is missing, the value has to be re-verified by hand, and most of the benefit of automation disappears.

The table below is our checklist. It is not a list any regulator publishes; it is our synthesis of what strong traceability looks like, based on our implementation work on batch records, stability reports and toxicology studies.

This may sound theoretical, but in practice it means each value keeps a live link to its source document. Every data point becomes auditable in one click, with a person seeing it in its original context.

Traceability is not the same as lineage

Data lineage describes how data moves and is transformed between systems: from the extraction platform to LIMS, to the data warehouse, to the PQR. Traceability is the wider GxP requirement to reconstruct a value's full history. For AI-extracted data, traceability has to start one step earlier than most lineage tools do: at the exact spot on the source page. Lineage without source-level provenance tells you which pipeline a wrong number travelled through, not whether it was ever on the page.

2. Does it flag uncertain values instead of guessing?

LLMs will guess, often with very good accuracy. Machine learning (ML) extraction models work differently: they look for a known pattern. If no pattern is found, there is no prediction; if the pattern is unclear, the confidence score is low. The question is not whether the model makes mistakes, but whether you know which values to look at.

Setting the confidence threshold

For ML models, uncertainty materialises as a confidence score. Which score should trigger a review is a decision per use case:

• Content feeding regulatory drafting: review anything below 98%.
• Content feeding SOP search: a 95% threshold gives a better ratio of effort to impact.

The confidence score is not the only safety net:

• Review all critical values. Critical process parameters, for instance, can and should all be confirmed by a subject-matter expert.
• Add automatic checks. A temperature expected between 20 and 40 °C should trigger a warning for any value outside that range.

What the regulation says

The most detailed requirements come from the EMA's draft Annex 22, which frames this directly: models should have "an appropriate threshold setting", and where confidence is very low, "it should be considered whether the model should flag the outcome as 'undecided'" (§9.2). In practice:

• Require a confidence score per value, not per document. A document-level score hides the one critical field that was guessed.
• Define thresholds per field, set from test data and documented in validation. Below the threshold, the value goes to a person and never passes silently.
• Add validation rules that do not depend on the model: format, range, specification limits, cross-field consistency (a batch number that matches the header, a result consistent with the stated unit). These catch values that are confidently wrong.
• Route known-hard content by default. Handwriting, overwritten entries and poor scans should go to review as a rule. Tools such as Acodis, for example, send handwritten annotations on batch records to an issue list for human review.
• Monitor the flag rate over time. A rising share of flagged values is often the first sign that inputs have drifted, such as a new supplier template or a new scanner: exactly what the draft asks you to watch (§10.3–10.4).

3. Can I see where every value came from?

Verification is the reviewer's real workload. When expert verification is required, the check itself should take seconds, not minutes.

• Require a direct link from every value to its location in the source, shown side by side with the extracted data, with no searching through the PDF.
• Check that the link points to a specific document version, and that this version is the approved one. This is what catches a correct extraction from the wrong document.
• Test it on hard cases: tables spanning pages, footnotes, values corrected by hand.
• Make sure the reference survives export. When the value lands in LIMS or a PQR, the link back to the source must still resolve.

What traceability does to review time

Take a mid-size site reviewing 400 batch records a year, with 150 critical values each: 60,000 values a year. The assumptions below are illustrative; replace them with your own.

The order of magnitude matters more than the exact figures. Confidence flagging decides how many values a person checks; source traceability decides how long each check takes. You need both to move from full manual verification to genuine review by exception.

4. Is the model locked and versioned like any validated system?

With hosted LLMs, you can pin a model version, but the provider decides when it is retired. Full control means running an open-weight model locally and storing each version, and even then its probabilistic outputs remain hard to explain, and it takes serious computing power to run.

ML models work differently. You can train small, dedicated models on as few as 20 to 30 of your own documents and fully control the version. The model in production is a controlled, stored version: given the same inputs, it produces the same outputs, and it only changes when you decide to update it. For a GxP environment, a deterministic, versioned model is the most defensible setup.

Two common misreadings

• Deterministic does not mean "no confidence score". A model that returns the same value with the same confidence every time it sees the same page is deterministic. Beyond tracking, confidence scores also show where to improve the next model, by pointing to the zones of uncertainty that need more training examples.
• Locked does not mean "never improves". Reviewer corrections can be collected as training data for a next version. That version is retrained, tested against a fixed reference set, and released under change control.

What to ask

• Ask how a new version is trained, tested and released, and who decides when it goes live. The answer should be you.
• Check the model version is stored with every value, not only in a release log.
• Confirm older versions remain available, so a past result can be reproduced on request.
• Ask for the reference set used for regression testing: a fixed sample of known-correct outputs every new version must match before release.

5. Can we show an auditor who changed what?

Every confirmation, correction and rejection of an AI-extracted value is a GMP-relevant action. Annex 11 already requires an audit trail for changes to GMP-relevant data; 21 CFR Part 11 requires secure, computer-generated, time-stamped audit trails (§11.10(e)). The draft revision of Chapter 4 goes a step further, extending ALCOA+ to ALCOA++ with a tenth principle: traceable, the ability to trace the history, modification or location of data.

• Confirmation and override should be traceable and auditable, capturing at least user and timestamp, potentially a justification for particularly sensitive data.
• Organise a validation step with electronic signatures in the worklflow, when the review is an approval
• Run the 2-minute test. Pick a value from a final report and reconstruct its full journey: source page, document version, model version, confidence, checks, reviewer, changes. If you cannot do it in under 2 minutes during the evaluation, you will not do it during an inspection.

6. Does it know life-science documents, or is it generic AI?

In one evaluation we observed, a pharma company tested several vendors and a tool built by its own IT team on the same set of documents. The internal tool read text well but lacked life-science context: how a CoA, a batch record or a stability report is structured, what each component is, and what the values inside it mean. Generic accuracy on generic documents tells you little. What counts is accuracy on your critical fields, on your hardest pages and most complex tables.

• Prefer models built per document type (batch record, CoA, stability report), often combining several specialised models for titles, tables and footnotes.
• Test on your own difficult pages, not on the vendor's demo set.
• Use traceability to make comparisons checkable. With XML output and a clear export structure, you can automate checks and comparisons.
• Discount generic accuracy claims. Accuracy is only a reliable number for a given set of documents and a given output. Ask for precision, recall and F1 scores on your data.

7. The Silent Error Test: checking traceability on your own documents

You can run a meaningful vendor comparison in a few weeks. The number that matters most is not overall accuracy, but the share of wrong values that went through without a flag. A system that is 96% accurate and flags every one of its errors is safer than one that is 98% accurate and flags none.

• Step 1 – Build one test set for every vendor: 20 to 30 documents per type, including your hardest pages (handwriting, overwrites, poor scans, multi-page tables). With tools such as Acodis, that is also enough to configure a first model adapted to the document type, so you test a tuned system rather than a demo.
• Step 2 – Fix the scope and the ground truth first: 10 to 30 critical fields per document type, with correct values agreed by your subject-matter experts before any results come back.
• Step 3 – Run the benchmark on every tool, and use LLMs to automate the comparison of outputs against the ground truth.
• Step 4 – Classify every value into the four outcomes below.
• Step 5 – Ask for the evidence: model version, confidence log, rule results and audit trail of the test run.

Conclusion

AI can take a large share of the reading and manual data entry off Quality and CMC teams. But it only earns a place in GMP records if every value it produces can be traced, checked and defended. Traceability is not a feature, it is the enabler that allows automation and AI to function accurately and experts to check what they need quickly.

The problem is solvable, and it is usually underestimated. Start with one document type and its critical fields, test with your own hardest pages, keep subject-matter experts in the loop, and scale from there. Teams that build this foundation now compound the benefit: every new document type, and every downstream use of the data, inherits values that an auditor can trust.

FAQ