How PayPal raised internal AI accuracy from 43% to 95% without changing the model
On 10 November 2026, VentureBeat reported that PayPal had raised the question-and-answer accuracy of its internal data science AI assistant from 43% to 95%. PayPal did not replace the underlying large language model (LLM) and did not fine-tune it. The improvement came from an external, model-agnostic “enterprise knowledge layer” that holds the company’s codified business definitions, vetted “golden queries” and analytical playbooks, kept separate from the model itself.
The result is useful because it contradicts a common assumption in enterprise AI: that wrong answers are mainly a model capability problem, solved by waiting for or paying for a newer frontier model. In PayPal’s case, the assistant was getting things wrong because it did not know what the business meant by its own terms. A model can be fluent in SQL and statistics and still misread what counts as an “active” or “new” account inside a specific organisation. Once those meanings were written down and supplied from outside the model, accuracy more than doubled.
For Australian environmental consultants, lawyers, developers and council officers now trialling AI for data interpretation, report drafting and document review, the same failure mode applies. Contaminated land practice depends on precise, context-specific definitions and decision rules. The PayPal case suggests that the firms which codify that knowledge will get more reliable output than those relying on general-purpose models alone. The 43% to 95% figure is PayPal’s internal result as reported by VentureBeat, not an independent benchmark, and should be read that way.
What sits in an enterprise knowledge layer: definitions, golden queries and playbooks
PayPal’s knowledge layer has three components, according to VentureBeat. The first is codified business definitions, which fix the meaning of terms the business uses every day so the model does not infer them. The second is “golden queries”: deterministic, vetted query templates that specify how enterprise data must be extracted to answer a given business question. Rather than improvising a fresh query each time, the assistant calls the approved pattern. The third is analytical playbooks, which set out explicitly how PayPal’s human analysts examine segmentations, handle edge cases and interpret metrics. This is the tacit procedural knowledge that experienced staff carry but that rarely appears in a prompt.
The defining design choice is a clean boundary: all business logic sits outside the model. Ren Zhang, VP and Global Head of AI Product at PayPal, described the intent directly: “The LLM is still the same… If the model is getting better, I’ll just plug in and play. I should not change my knowledge.” When domain logic is embedded in fine-tuned model weights or in prompts written for one specific model, every model upgrade forces the organisation to re-engineer that logic. Keeping it in a separate, portable layer means the knowledge survives model changes intact. A separate VentureBeat Intelligence report found that swapping AI models takes most companies weeks, which illustrates the cost of tying institutional knowledge to a particular model.
The article also addressed a gap between pre-deployment testing and live performance. In VentureBeat Pulse’s October reliability survey, 42% of organisations reported that an agent or LLM passed pre-deployment evaluations but then failed in live production. The cause identified was a shift in real-world distribution: the questions users actually ask differ from those in the evaluation set. Zhang summarised it as: “Your model is still the same. Your process is the same, but the distribution is completely different.”
The practical lesson PayPal draws from this is that high scores on a static test set do not guarantee reliable answers once real users arrive with messier, less predictable questions. Explicit semantic definitions and step-by-step playbooks give the model deterministic grounding that holds up better when the question mix changes. In other words, the knowledge layer is doing work that a better benchmark score cannot.

Australian context
For Australian professional services firms, the PayPal approach maps closely onto how technical consultancies already work, even if that knowledge is not yet written down in a form an AI tool can use. Environmental consultancies, engineering firms and law practices all run on firm-specific definitions, standard methods and senior staff judgement. Most of that sits in the heads of principals, in old reports and in internal templates. A general-purpose LLM has none of it, so it fills the gaps with plausible but generic assumptions.
Contaminated land assessment is a clear example. Ask a general model whether a soil result “exceeds” and it cannot know whether Health Investigation Level (HIL) A for residential land with garden or accessible soil applies, or HIL D for commercial and industrial land, under the National Environment Protection (Assessment of Site Contamination) Measure 1999 as amended in 2013 (the ASC NEPM). It will not know which Health Screening Level (HSL) soil texture class (sand, silt or clay) and depth band (0 m to less than 1 m, 1 m to less than 2 m, 2 m to less than 4 m, or 4 m and deeper) is relevant to a vapour intrusion question. It will not know which ANZG (2018) default guideline values and species protection level were adopted for a receiving water, or how the PFAS National Environmental Management Plan (PFAS NEMP) criteria were applied on a given project. Critically, it may not understand that a Tier 1 screening level exceedance under the ASC NEPM triggers further assessment, not a conclusion that the site poses an unacceptable risk. That distinction carries legal and commercial weight in property transactions and regulator correspondence.
Governance expectations point in the same direction. The Australian Government’s Voluntary AI Safety Standard, released in September 2024, is not mandatory, but its ten guardrails include data governance measures to manage data quality and provenance, testing AI systems before deployment and monitoring them once in use, human oversight, and record keeping. PayPal’s experience with assistants that passed evaluation and then failed in production shows why testing alone is not enough without ongoing monitoring. For a consultancy, writing down the definitions, criteria and decision rules an AI tool relies on, and keeping them separate from the model, also creates a record that a senior reviewer, client or regulator can check.
References and related sources
- Primary source: venturebeat.com
- venturebeat.com
- youtube.com
- NEPM Assessment of Site Contamination
- ANZG Water Quality Guidelines
How iEnvi can help
iEnvi integrates technology and data-driven approaches into environmental consulting. We monitor AI and technology developments that affect how environmental professionals deliver services to clients.
This is an iEnvi Machete news summary. Prepared by iEnvi to summarise the source article for environmental professionals tracking AI, data, and technology developments that affect consulting and project delivery.
Published: 14 Nov 2026
Need advice on this topic? Speak to an iEnvi expert at info@ienvi.com.au or 1300 043 684, or contact us online.
Need advice on this issue? iEnvi provides practical, senior-led environmental consulting across contaminated land, remediation, ecology and environmental risk.
Environmental management plans Contaminated land advice Environmental management Talk to iEnvi