On one 45,004-document corpus, a generative AI review system achieved higher recall than one managed-review implementation. The labor totals compare different endpoints, and the paper’s population-level extrapolation does not fully reconcile.
TL;DR
- GenAI review won on recall: 88% versus 64%. On a 1,000-document blinded validation sample, aiR for Review found 64 of 73 responsive documents; the managed-review TAR workflow found 47. The confidence intervals don’t overlap, and a paired test on the same documents puts the difference at p ≈ 0.0005.
- Observed precision favored managed review: 39% versus 29%. The paper reports only marginal confidence intervals, whose overlap does not test the paired difference. The underlying disagreement counts nevertheless show that aiR’s higher recall came with substantially more false positives. The low absolute precision on both sides reflects a 7% richness rate, not a broken classifier.
- The effort gap was 62 to 1. One attorney, 18 hours of prompt work. Twenty-four people, 1,123 hours across seven business days. But the two workflows stopped at different endpoints — one produced final human-reviewed labels, the other produced predictions that still need review — so this is not a total-effort comparison.
- Model rationales changed the expert’s mind ten times. In a secondary re-review, the subject-matter expert reversed 10 of his own “not responsive” calls after seeing the model’s reasoning. The re-review was one-directional and unblinded, so it shows the rationales were persuasive — not necessarily that they were right.
- Run the numbers yourself. The eDiscovery Cost Calculator models what the effort and accuracy differences mean in dollars on your matter parameters.
Redgrave LLP’s working paper, released in June 2026, is a head-to-head comparison of two eDiscovery review workflows on the same 45,004-document corpus. In one corner: Relativity’s aiR for Review, an LLM-based system where a single attorney writes and iterates a classification prompt. In the other: a managed-review team of 24 running Relativity Active Learning, a conventional commercial continuous-active-learning workflow. Same review protocol. Same blinded reference labels. Same scorecard: recall, precision, elusion, effort.
The Study Design#
The study population: 45,004 documents from the Mallinckrodt collection in the UCSF Opioid Industry Documents Archive — a publicly disclosed set of emails, Word documents, spreadsheets, and presentations from the opioid litigation. The review topic was deliberately hard: a document was responsive only if it contained evidence of compliance with, or violation of, federal pharmaceutical marketing and controlled-substance regulations. Merely discussing sales activities or drug promotion didn’t qualify. The reviewer — human or model — had to apply governing legal standards to the document’s content.
This matters because many TAR benchmarks use simple topical relevance. “Any document related to X” is a less legally nuanced classification task. “Documents evidencing compliance with or violation of specific federal regulatory frameworks” requires the kind of judgment call that separates experienced reviewers from contract attorneys on day one.
Two workflows ran independently against the same corpus, driven by the same review protocol:
aiR for Review (GenAI). One attorney — Ray Mangum, a partner at Redgrave LLP — developed and iterated the prompt over approximately 18 hours across multiple sessions. After 43 prompt iterations, the finalized prompt was applied to the full population in a single processing run. Two scoring details matter for reading the results: 150 documents the system scored as borderline were counted as responsive predictions, as were 12 documents that errored out, because both categories entered the review queue. Neither is treated as a miss when computing recall. This follows the GenAI TAR 1 process described in the Sedona Conference TAR 1 Reference Model: an attorney writes and refines a natural-language prompt instructing the LLM to classify each document, with performance measured against a random control set.
RAL (Active Learning). A first-pass managed review staffed by 21 contract-attorney reviewers, a review manager, a team lead, and dedicated QC reviewers, run through Relativity Active Learning over seven business days under a continuous active learning workflow. The vendor, Cimplifi, trained and calibrated the team against the review protocol.
These two things are not the same kind of object, and the comparison is looser than “GenAI versus active learning” suggests. aiR’s score was converted directly into a prediction. RAL’s result is the combined output of a classifier, 21 contract reviewers, calibration, and QC — so its misses include human coding errors, not only algorithmic ones. The defensible reading is narrower than the headline: the aiR screening rule achieved higher recall than this particular managed-review implementation’s final labels.
Ground truth came from a single subject-matter expert — Kevin A. Reiss, Counsel at Redgrave LLP and a co-author of the paper — who reviewed a 1,000-document simple random sample in a blinded assessment, without access to either workflow’s predictions. His labels: 73 responsive, 927 not responsive. A separate 250-document random control set — 16 of 250 responsive, an initial richness estimate of 6% — was used to refine the aiR prompt, keeping the final validation sample independent of prompt development.
Worth naming plainly: these are blinded single-expert reference labels, not ground truth in the stronger sense. The expert was blinded to both workflows’ predictions, but he is a co-author, he participated in reviewer calibration, and he was the sole adjudicator. There is no second reviewer, no inter-rater reliability measure, and no conflict-resolution process. Every accuracy figure below inherits whatever idiosyncrasy one experienced reviewer brings to a genuinely hard responsiveness standard.
The Headline Numbers#
| Metric | aiR for Review (GenAI) | RAL (Active Learning) |
|---|---|---|
| Recall | 88% (95% CI: 78.2–93.4%) | 64% (95% CI: 52.9–74.4%) |
| Precision | 29% (95% CI: 23.1–34.8%) | 39% (95% CI: 30.9–48.1%) |
| Elusion | 1.2% | 3.0% |
| Missed responsive | 9 of 73 | 26 of 73 |
| Effort | 1 attorney, ~18 hours | 24 people, ~1,123 hours |
Source: Redgrave LLP Working Paper 2026-01. Wilson 95% confidence intervals.
The 95% confidence intervals on recall do not overlap. Among the 73 responsive documents, aiR alone found 20 that RAL missed; RAL alone found 3 that aiR missed.
Those marginal intervals are actually the weaker way to make the point, because both workflows classified the same documents — an unpaired comparison throws that structure away. The paired test is McNemar’s, run on the 20 aiR-only and 3 RAL-only discordant documents: exact two-sided p ≈ 0.0005. The recall conclusion survives the better test comfortably.
The agreement analysis tells the rest of the story. Of the 73 responsive documents, 44 were found by both workflows. Twenty were found only by aiR for Review. Three were found only by RAL. Six were missed by both.
When the two workflows disagreed, aiR for Review overwhelmingly erred toward over-inclusion — flagging non-responsive documents as responsive. RAL’s disagreements ran the opposite direction — its combined workflow (classifier, reviewer coding, calibration, and QC) missed documents that turned out to be responsive.
The Effort Comparison#
The numbers in the effort column deserve their own section because they reframe the accuracy comparison.
aiR for Review: one attorney, approximately 18 hours of prompt development and iteration. After the prompt was finalized, the model processed all 45,004 documents in a single run lasting about one hour of elapsed time (not attorney labor).
RAL: 24 people — 21 contract-attorney reviewers, a review manager, a team lead, and QC reviewers — working approximately 1,123 hours across seven business days. The contract reviewers alone logged 974 hours, including an estimated 42 hours of onboarding and training.
That’s a 62:1 ratio in recorded pre-validation labor. It is the single most quotable number in the study and the one most likely to be misused, because the two totals do not stop in the same place.
The 1,123 hours produced human-reviewed responsiveness labels. The 18-hour figure covers prompt development and automated classification, but excludes human confirmation of the resulting positive set — roughly 22% of the corpus, including the 150 borderline documents and 12 processing errors counted as responsive. That confirmation work was not skipped: the paper notes the full corpus “was also subjected to manual review as part of the broader experiment,” with analysis of it reserved for future work. The labor exists; it simply isn’t in this number.
So the endpoints are materially different — but neither figure represents total production effort. Both exclude shared protocol drafting, validation review, privilege review, redaction, production, and technology charges.
So the honest phrasing is narrower: the recorded pre-validation labor differed by 62:1, but the workflows stopped at different practical endpoints, which makes this neither a total-effort nor a cost comparison. The eDiscovery Cost Calculator (source on GitHub) exists precisely to price the downstream review volume this ratio leaves out.
Precision in Context#
Both workflows posted precision figures that look low in isolation: 29% for aiR for Review, 39% for RAL. A 29% precision rate means roughly seven out of ten documents flagged as responsive weren’t — which sounds like a quality problem until you understand the math.
Precision depends on richness — the fraction of documents in the collection that are actually responsive. When richness is 7%, even a classifier with a modest false-positive rate will flag many more non-responsive documents than responsive ones, simply because non-responsive documents outnumber responsive ones 13-to-1. That base-rate effect is a mathematical property of binary classification.
But richness is not the whole explanation. aiR for Review’s observed false-positive rate was 17%, compared with 8% for RAL. The GenAI workflow deliberately cast a broader net, producing higher recall and lower precision. That is a genuine performance tradeoff, not merely an optical effect caused by prevalence.
The paired numbers make the tradeoff concrete, and they deserve the same treatment as the recall comparison. Of the 927 documents the expert coded not responsive, 35 were flagged by both workflows, 125 by aiR alone, and 38 by RAL alone. That discordance runs heavily one direction — exact McNemar p ≈ 5 × 10⁻¹². The paper publishes only marginal confidence intervals for precision, and those overlap, but interval overlap does not test a paired difference and should not be read as evidence that the workflows performed the same. On the false-positive side, they clearly did not.
The Redgrave paper computes projected precision at varying richness levels, holding each workflow’s observed recall and false positive rate constant:
| Richness | aiR for Review Precision | RAL Precision |
|---|---|---|
| 7% (observed) | 29% | 39% |
| 10% | 36% | 48% |
| 20% | 56% | 67% |
| 30% | 69% | 78% |
At 20% richness — the low end of the 20–25% band the paper describes as typical of many e-discovery reviews — projected precision rises to 56% for aiR for Review and 67% for RAL. These are projections holding each workflow’s recall and false-positive rate constant, not observed results at higher richness.
The Informed Re-Review#
The study’s most striking finding isn’t in the primary metrics. After the blinded assessment was complete, the expert re-examined 151 documents where he had coded “not responsive” but aiR for Review had predicted “responsive.” This time, he could see the model’s rationale, considerations, and document citations.
He reversed 10 of his own calls. A 7% overturn rate — on documents where an experienced attorney with subject-matter expertise had independently concluded the document wasn’t responsive, and then changed his mind after seeing the AI’s reasoning.
The study is careful about this finding. The 10 overturned documents were not folded back into the primary ground truth. The recall and precision figures throughout the paper use only the original blinded labels. The re-review is reported as a secondary analysis.
The re-review was one-directional — only documents where the expert disagreed with aiR’s positive prediction were re-examined. Documents where aiR predicted “not responsive” were not subjected to equivalent scrutiny. The expert also saw the model’s rationale while reconsidering, which is the whole point of the exercise and also its central weakness.
That combination is what limits the claim. With no counterbalanced re-review, no independent adjudicator, and no blinded second pass, the ten reversals are consistent with genuine error correction — and equally consistent with anchoring, automation bias, or the ordinary label instability you get when anyone reviews a hard document twice. The finding suggests model rationales may help surface overlooked responsive material, but this design cannot distinguish correction from persuasion.
What This Means for the Economics#
The Redgrave study provides accuracy data that economic models can use. The eDiscovery Cost Calculator can turn those measurements into scenarios.
The paper also scales its sample results to the full corpus, and those figures do not reproduce. Appendix A states that applying RAL’s observed prediction rate of 12% to the 45,004-document population “yields approximately 5,019” documents flagged. But 12% of 45,004 is 5,400. The sentence contradicts its own arithmetic.
Nor does the rest of the appendix resolve into a consistent calculation. Scaling the validation sample directly — 224 and 120 predicted-responsive documents per 1,000, multiplied by 45.004 — gives roughly 10,100 and 5,400 flagged, against the paper’s 9,971 and 5,019. Either those are undisclosed full-corpus counts, in which case the appendix describes them incorrectly as sample extrapolations, or they are extrapolations, in which case they don’t reproduce.
The missed-document estimates split more cleanly, and the split is instructive. Applying each workflow’s exact observed miss rate to the appendix’s own estimated 3,305 responsive documents reproduces RAL’s figure precisely — 3,305 × 26/73 = 1,177. The same arithmetic for aiR gives about 407, not the 376 the paper reports. One of the two numbers checks out against the paper’s own stated population; the other does not.
None of this touches the validation-sample results, which are internally consistent and independently checkable. But the population-level numbers need clarification from the authors before anyone relies on them in a budget or a meet-and-confer. What survives is the direction: aiR flags roughly twice as many documents for downstream review and misses roughly a third as many responsive ones.
For in-house counsel evaluating an AI-enhanced eDiscovery proposal, the question isn’t just “is the AI accurate?” It’s “what’s the cost of the additional review volume, and what’s the cost of the missed documents?” The calculator lets you model both sides — adjusting the AI efficiency gain and managed-review shift parameters to reflect the Redgrave study’s findings rather than vendor marketing.
The eDiscovery economics breakdown argued that the real savings in AI-enhanced eDiscovery come from restructuring leverage — shifting volume QC from $750/hr associates to $50/hr contract attorneys, because AI pre-screening makes that work routine enough for managed review. The Redgrave result strengthens the case for testing that model: on this task, aiR found more responsive documents than this particular managed-review implementation. Whether it can replace that workflow economically depends on the cost of validating the prompt, reviewing the broader positive set, handling errors and privilege, and operating at the scale and risk profile of the actual matter.
The Defensibility Question#
The Sedona Conference’s TAR 1 Reference Model treats GenAI review and traditional TAR as instances of the same defensibility framework: Scope, Label Control Set, Iterate Model, Classify, Validate. Redgrave shows what one implementation of that framework can look like.
Courts have consistently held that producing parties need not achieve perfection. Rio Tinto PLC v. Vale S.A. and Hyles v. City of New York emphasize reasonable process and the producing party’s role in choosing methodology; they do not create a judicial safe harbor at 78% or 88% recall. A blinded random-sample validation result is evidence supporting reasonableness, not a self-executing finding of defensibility.
The tar-to-transformers post traced how TAR adoption followed a pattern of empirical testing, judicial acceptance, practitioner resistance, and gradual normalization. The Redgrave paper is one useful empirical data point for GenAI review. Replication across corpora, review protocols, teams, and model versions would turn it into a stronger foundation.
The practical question for litigators is therefore not whether 88% automatically makes GenAI review defensible. It is whether the proposed workflow can document its scope, prompt development, validation sample, uncertainty, error handling, privilege process, and human review burden well enough to show that the overall process is reasonable and proportional on the actual matter.
Reproducibility#
This is one study on one corpus with one ground-truth reviewer. The source collection is public — the UCSF Opioid Industry Documents Archive is freely accessible — which distinguishes this paper from vendor benchmarks run on proprietary data. The paper does not, however, identify the precise 45,004-document population or the validation sample at the document level, so an independent researcher cannot reconstruct the exact study set from the archive alone.
The study design is well described, but the replication artifacts are missing. The paper does not publish the aiR for Review prompt, the expert’s detailed review instructions beyond the responsiveness definition, or the code used to compute the reported metrics. It also never identifies the underlying model — no model name, version, product build, or scoring configuration appears anywhere in the paper, which for a study of a continuously updated commercial LLM product means the tested system cannot be pinned down even in principle. There is no funding, product-access, or conflicts-of-interest statement. Without those artifacts, a reader can follow the paper’s logic but cannot independently reproduce the numbers. Publishing them on GitHub — the prompt iterations, the statistical code, the sample identifiers, the model version — would move this from a credible working paper to a replicable study.
There is also a generalization question the design leaves open. The prompt went through 43 iterations against a 250-document control set containing just 16 responsive documents. The final validation sample was properly held out, so the headline recall figure is not contaminated. But 43 rounds of tuning against 16 positive examples is within-corpus development, and nothing here establishes that the resulting prompt transfers to a different custodian population, a different review protocol, a different matter, or a later version of the model. The Grossman & Cormack work became foundational in part because other researchers could test its claims. The Redgrave study would benefit from the same treatment.
Further Reading#
- Redgrave LLP Working Paper 2026-01: Generative AI for Complex Document Review. The full study.
- Relativity, “Defensible Validation of Relativity aiR for Review: Metrics, Methods, and Considerations”. Relativity’s description of the product’s develop-validate-apply workflow, confidence intervals, model-version changes, and revalidation considerations.
- eDiscovery Cost Calculator. Interactive cost model for comparing review workflows. Source code on GitHub.
- eDiscovery Economics: What Your Law Firm’s AI Pitch Is Actually Selling. The cost-calculator breakdown on LegalRealist AI.
- From TAR to Transformers. The TAR adoption timeline and what it teaches about GenAI review.
- Sedona Conference TAR 1 Reference Model. The framework unifying traditional TAR and GenAI approaches.
- Grossman & Cormack, “Technology-Assisted Review Can Be More Effective and More Efficient Than Exhaustive Manual Review” (2011). The foundational TAR study.
- Cormack & Grossman, “Evaluation of Machine-Learning Protocols for TAR” (2014). Validation of continuous active learning.
- Yang et al., “Beyond the Bar: Generative AI as a Transformative Component in Legal Document Review” (IEEE Big Data 2024). Earlier empirical work on GenAI review.
- Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125 (S.D.N.Y. 2015). Judicial endorsement of TAR 2.0 and producing-party methodology choice.
- UCSF Opioid Industry Documents Archive. The source corpus for the Redgrave study.
- Wilson Score Interval (JSTOR). The confidence interval method used in the study.
- EDRM Framework. The Electronic Discovery Reference Model.
- ABA Formal Opinion 512. ABA guidance on lawyers’ duties when using AI tools.
- Fed. R. Civ. P. 26(b)(1). The proportionality standard for discovery.
This post is part of The Client Side series on LegalRealist AI. It is intended for informational and educational purposes only and does not constitute legal advice. The Redgrave LLP working paper has not been peer-reviewed. Study results reflect a single corpus and review topic; performance on other document sets and classification tasks may differ. E-discovery obligations, technology-assisted review standards, and AI review capabilities vary by jurisdiction and are subject to change.
