The #1 challenge for AI applications in Pharma – Data – and How to fix it

1. Definition of AI-Ready Data

The diverse views from industry experts

No ISO standard, no FDA guidance, no EMA framework has settled the question. What exists instead is a fragmented landscape of vendor definitions and analyst frameworks — which itself says something about the maturity of the field.

One helpful starting point is the established concept of FAIR data^[6], which refers to research data that adheres to four principles — Findable, Accessible, Interoperable, and Reusable — designed to optimise the reuse of data by both humans and machines. The framework was built to increase scientific efficiency, transparency, and reproducibility. For AI applications, however, it is incomplete.

IBM offers one of the more accessible definitions: AI-ready data is "high-quality, accessible and trusted information that organizations can confidently use for artificial intelligence training and initiatives"^[7], with key attributes including unified, governed, secure, and supported by human oversight.

Gartner, probably the most credible non-vendor voice, provides a more comprehensive framework^[8] organised around seven dimensions:

• Quantification — sufficient data volume
• Semantics and labeling — proper annotation and labeling
• Quality — quality standards specific to the AI use case
• Trust — reliable data sources and pipelines
• Diversity — diverse sources to avoid bias
• Lineage — transparency about data origins and transformations
• AI technique specificity — different AI techniques have unique data requirements

Gartner further adds that AI-readiness is a project-specific topic. We agree — but would nuance that permanent data quality infrastructure is a major enabler that fast-forwards project-specific work. That distinction shapes the two-part structure below: the always-on layer first, then project specifics.

The World Bank^[9] and Qlik^[10] also provide relevant perspectives on governance and the organisational conditions needed to sustain data quality over time.

The Acodis Framework: FAIR-GRACE

FAIR data provides the clearest starting point — but it was designed for scientific data sharing, not AI deployment in regulated environments. Our definition extends it accordingly.

2. The Always-On Infrastructure Layer

Every AI use-case requires its own data preparation — but how fast and successfully that happens depends entirely on the quality of the foundation beneath it. Getting this layer right is the difference between deploying a new AI use-case in a few months versus a year, and the primary reason so many AI projects fail before reaching production. This does not remove the need for project-specific data preparation, but it creates a much higher state of ongoing readiness and significantly faster preparation for project-specific pipelines. Four areas to cover:

Establish a data governance framework

• Cover both principles (how data should be managed) and decision-makers (who decides for what)
• Segment clearly: which data is team-level only, and which is available company-wide — this single distinction prevents most access and compliance conflicts downstream
• Keep it short — a few pages that people can actually read and remember; the most effective frameworks are the concise ones, iterated over time rather than perfected upfront 

Recommended reading on data governance frameworks: see references [11][12][13].

Define data segments and map your systems

• Define key data types at team and company level — what data matters, to whom, and for what purpose
• Map all data systems and assign a source of truth — know where authoritative data lives before any AI pipeline is built
• Agree on a shared nomenclature — standard formats for dates, abbreviations, and naming conventions for the most frequently used items

Digitalise and structure the archive

• Ensure all key documents are in accessible, digital form — this already enables simple "search and retrieve" AI applications
• Define minimum required metadata for the key document types — for SOPs, Batch Records, CoAs, Clinical Studies, agree the top 10 minimum required data points per document type
• Convert documents to structured data at scale — a mass extraction approach with low specificity requirements and reasonable noise tolerance; the goal is coverage, not perfection. Use automation tools such as Acodis to do this at scale with full traceability.

Maintain security and compliance throughout

• Encryption: all data encrypted at rest and in transit. For encryption during model usage (inference), consider homomorphic technology, however it slows down processing
• Access control: access limited to authorised personnel with strict logs maintained
• Access reviews: the biggest risks lie in dormant accounts with full access 

This approach works well to build a general state of readiness or broad applications on top of a data lake. Most AI projects — especially agentic use-cases — need more specific, refined and controlled data to perform.

Let's dive into that !

3. Project-Specific Data Preparation, Step by Step

With a solid infrastructure layer in place, each new AI project builds on a much higher baseline. The following six steps apply to every project — the infrastructure layer compresses the time each one takes.

Step 1 — Define the use case and the data you need

The data type determines the pipeline architecture.

• Distinguish different types of data:
  • Semantic / unstructured content (documents, contracts, reports, emails): requires extraction, classification, and meaning-preserving transformations
  • Structured / time series data (sales figures, sensor readings, lab results): requires numerical consistency, timestamp integrity, and clear definitions
• What is the minimum data inputs required for the models to produce the desired output (a useful test: what would a newly hired expert need to do this job?
• Identify implicit knowledge held by subject-matter experts required for the tasks and how to translate that into machine understandable data: "36 (34 - 38)" means desired temperature is 36 with +/- 2 degree tolerance.

Step 2 — Map and consolidate your sources

Understand the house before you clean it.

• Identify all relevant sources — databases, PDFs, CRMs, data warehouses
• Extract content buried in complex formats (text, tables, and images from PDFs)
• If possible, consolidate data locations to have less integration / sources to maintain 

Step 3 — Clean and transform into AI-ready formats

The most time-consuming step — and the one most directly correlated with model quality.

• Clean: fix errors, remove duplicates, handle missing values, ensure consistency
• Fix and interpret layout: rotated pages, document structure (core vs annex), structure of pages and elements (tables, footers, etc.)
• Structure content: extract text, tables and images with content hierarchy
• Normalise: standardise units and naming conventions
• Capture metadata: source, date, document type, author — this context makes data significantly more useful for downstream AI models
• Adapt the depth of refinement to the use-case: RAG-based use-cases work well with structured content and clean metadata; deeper use-cases like transformation in FHIR standard need deeper refinement
• Adopt a consistent format: whether JSON, XML, or a specific XML dialect, uniformity is what matters

Step 4 — Build your labelled baseline for data preparation

This is the quality foundation everything else depends on. Whilst data preparation can be automated, grappling with the reality on the data and manually checking the content initially is the best way to anchor accuracy at scale. 

• Manually label or verify a representative sample of 30–100 documents / data samples
• Keep that work within to 1–3 people to maintain consistency — variation in labelling is the fastest way to get low performing models
• Ensure the sample captures the full diversity of the underlying data and cases to be processed
• If the scope is too wide, train separate data models rather than force one to cover everything

Use this step to define your data quality standards based on the demands of the use-case:

• Define cleaning standards: how to handle errors, duplicates, missing values
• Define constent structure: text, tables, images with content hierarchy
• Normalise: standardise units and naming conventions
• Define required metadata: source, date, document type, author, context 
• Adapt the depth of refinement to the use-case: RAG-based use-cases work well with structured content and clean metadata; deeper use-cases like transformation in FHIR standard need deeper refinement
• Adopt a consistent format: whether JSON, XML, or a specific XML dialect, uniformity is what matters
• To make all of this easier, use a visual tool with automation features, such as Acodis

Step 5 — Scale up the data pipeline

With a solid base model, begin compounding.

• Process a batch roughly 2× the size of your labelled sample, review and correct
• Repeat — each cycle the model improves and batch sizes can grow significantly
• Use F1 score to track improvements across cycles
• Use confidence scores to route borderline predictions to human review
• Stop scaling when marginal quality gains no longer justify the labeling cost
• To do this at the scale of 100,000s or millions of documents, use cost effective tool such as Acodis 

Step 6 — Integrate your data layer and keep it current

This is never a one-time task.

• Merge all sources into a single, unified data layer
• Preserve metadata and lineage: record where data came from, how it was transformed, and when — essential for auditability and debugging at scale
• Keep adding new data on an ongoing basis, according to the same principles
• Maintain a "known truth" data  set — a fixed sample of known-correct outputs to test against