The FDA just funded a jury of AI judges for radiology reports.
Not one AI. A panel of them. Cross-examining each other.
On September 16, 2026, the FDA awarded Cognita Imaging a $1.29 million research contract to build what the company calls an "LLMs-as-a-jury" evaluation framework for autonomous, generative AI radiology reports. The 18-month contract was funded under the FDA's Broad Agency Announcement program for advanced research and development in regulatory science, effective June 22, 2026.
This is not a product clearance. It is the FDA funding the infrastructure to evaluate a whole new category of AI before that category overwhelms the agency's ability to review it. That distinction matters enormously to anyone building, buying, or deploying radiology AI.
1. The Validation Gap That Makes This Necessary
Traditional AI reader studies have a fundamental scale problem.
When a company submits a radiology AI product to the FDA, the typical validation dataset is a few hundred cases. Sometimes a few thousand, if the team is unusually thorough. These datasets are curated, often from a single institution, and designed to show that the model performs well on the cases it was designed to handle.
The real world looks nothing like this.
A deployed radiology AI encounters patients from dozens of different demographic groups, scanned on equipment from six different manufacturers, in hospitals with wildly different scan protocols and calibration standards, including rare pathologies that show up once every few months but carry life-or-death stakes when they do. A model that aced a 300-case test set at a quaternary academic center may fail catastrophically at a rural critical access hospital where 40% of patients have comorbidities the training set never included.
ð According to Dr. Akshay Chaudhari, principal investigator for the project and associate professor of radiology and biomedical data science at Stanford University: "A few hundred cases may tell you whether a model works in a narrow setting. They do not show every way it can fail in practice."
Cognita's LLM jury framework is designed to fix exactly this. It will stress-test generative AI radiology outputs across approximately 1 million patient imaging exams drawn from the Radiology Partners network - 4,000 radiologists interpreting 55 million imaging studies annually across diverse U.S. care settings.
This is not a theoretical upgrade. It is a 3,000-fold increase in validation scale.
2. What Generative AI in Radiology Actually Does - and Why It Breaks Old Validation Models
To understand why the FDA needed to fund this research, you need to understand how generative AI in radiology differs from the AI that came before it.
The first wave of radiology AI - the 1,357+ FDA-cleared devices that now dominate the market - are detection models. They take an image, analyze it, and flag something: a nodule, a hemorrhage, a fracture. The radiologist sees the flag, reviews the area, and writes the report. The AI narrows the search space. The human does the cognitive synthesis.
Generative AI does something fundamentally different. It takes the full image, processes it across multiple anatomical regions, and drafts an entire radiology report. The radiologist no longer starts from a blank page. They start from a full draft that they review, correct, and approve.
This is faster. Early evidence suggests it can reduce reporting time by 25 to 40 percent per scan. That matters enormously in a system where radiologist burnout is epidemic and backlogs stretch into days.
But it also changes what validation means.
When a detection model fails, it usually produces a clear false positive or false negative: the flag appears or doesn't. When a generative model fails, it can produce plausible-sounding text that is subtly wrong - a correct anatomical location but a mistaken severity grade, a correct pathology but an omitted clinical context, a technically accurate statement that creates a dangerously false clinical impression.
These failures are harder to catch and harder to measure. They require language understanding, clinical reasoning, and an ability to assess what was not said as much as what was. A binary detection model can be evaluated with sensitivity and specificity. A generative report requires something closer to what a senior radiologist does when reviewing a resident's work.
The FDA's challenge is that its existing clearance frameworks were built for the first wave. They are not built for this.
3. How the LLM Jury Framework Works
The architecture Cognita is building for the FDA is worth understanding in technical detail, because it is not just a larger test set. It is a fundamentally different approach to evaluation.
Traditional human reader studies use a panel of expert radiologists to evaluate AI outputs against a ground truth. Each radiologist reads each case. You average the scores. You get a performance number.
The LLM jury replaces most of that human reviewer time with an ensemble of large language models, each acting as an independent evaluator. Each model reads the AI-generated report and the ground-truth radiologist report. Each produces an assessment of whether they match on clinically meaningful dimensions. The jury then looks for disagreements among its own members.
When the LLMs agree, the case is resolved without human review. When they disagree - when one model says the reports match and another flags a critical discrepancy - the case is escalated to an expert human radiologist.
This has three major benefits.
First, it scales. You can process a million cases in the time it would take human reviewers to get through a few thousand. Second, it surfaces the cases that genuinely need human attention rather than having radiologists spend time on clear matches. Third, it provides attribution: the framework can tell you whether an error came from the generative drafting model, from disagreement among the LLM evaluators, or from ambiguity in the original human radiologist's own ground-truth report.
That attribution is critical for regulatory purposes. The FDA needs to know not just that a model fails, but why it fails, and whether the failure is systematic or random.
ð The project will deliver to the FDA: software code for building LLM juries, guidance on implementation across care settings, direct comparisons between large and small validation cohorts, radiologist-reviewed discrepancy studies, and final reports on findings and limitations.
4. The Cognita Story: From Stanford Lab to FDA Partner
Cognita Imaging's path to this contract is worth tracing, because it reflects a broader pattern in how serious radiology AI companies are being built.
Cognita was founded by Dr. Akshay Chaudhari and Dr. Louis Blankemeier, both from Stanford's radiology and biomedical data science programs. Chaudhari built his reputation on vision-language models applied to medical imaging, and the GREEN benchmark - Generative Radiologist Evaluation and Error Notation - emerged from that research.
GREEN is an open-source framework designed to do something that sounds simple but is surprisingly hard: compare what a ground-truth radiologist wrote to what an AI generated, and identify differences that are clinically meaningful rather than just stylistically different. A human radiologist might write "small bilateral pleural effusions" and an AI might write "trace bilateral pleural effusions." Are those the same finding? It depends on the clinical context. GREEN was built to make that judgment systematically.
In late 2025, Radiology Partners acquired Cognita. This gave Cognita something most AI startups desperately lack: a production deployment network. Radiology Partners operates 4,000 radiologists across 55 million imaging studies annually. That is not a test environment. That is the real world.
In March 2026, Cognita received an FDA Breakthrough Device Designation for its generative AI radiology platform - the first such designation in the category. Three months later, in June 2026, Aidoc received its own Breakthrough Device Designation for "First Read," its generative AI tool for detecting four life-threatening findings. The FDA was signaling that this category was moving from research to clinical deployment.
The contract announced in September 2026 is the infrastructure layer that makes responsible deployment possible.
5. The Equity Dimension Nobody Is Talking About
Here is what most coverage of this contract misses.
The 1 million patient exam dataset is not just large. It is diverse. The Cognita announcement specifically calls out analysis across "patient demographics, care settings, imaging equipment manufacturers, and disease types," with particular attention to "uncommon findings rarely captured in smaller datasets."
This is the equity problem in radiology AI made explicit.
Most training datasets for radiology AI are drawn from large academic medical centers. These are institutions with high scan volumes, sophisticated equipment, and patient populations that skew toward urban, insured, and demographically concentrated. A model trained on Stanford or UCSF data has never seen what a chest CT looks like when it comes off a 15-year-old Siemens scanner at a rural FQHC in rural Georgia.
The performance gap between large academic centers and community hospitals for existing radiology AI is not well-documented, because nobody has built the validation infrastructure to measure it. The LLM jury framework is designed to change that.
ð Of particular interest is the framework's explicit focus on "low-prevalence pathologies" - findings that are clinically critical but appear rarely enough that small validation datasets simply don't include them. A lung cancer screening AI trained on 500 cases may never encounter the 11 presentation variants that account for 8% of malignancies in high-risk populations.
For FQHCs and community hospitals deploying AI for their patient populations, this matters directly. The FDA's investment in diverse, at-scale validation is the first systematic attempt to give community health systems evidence that the AI tools they are deploying will actually perform for their patients - not just for academic medical center populations.
6. The Regulatory Signal Behind the Science
The FDA does not spend $1.29 million to solve an abstract research problem. It spends money on research when it anticipates a regulatory challenge it cannot meet with existing tools.
The Broad Agency Announcement program - the mechanism that funded this contract - is how the FDA develops regulatory science infrastructure in advance of the products it expects to be reviewing. The agency is telling the industry something important: generative AI in radiology is coming to the clearance queue at scale, and the FDA needs a new evaluation framework before that happens.
This follows a pattern. The FDA funded reader study methods research before the first wave of detection AI products arrived. It funded AI/ML predetermination change protocol (PCCP) research before companies started requesting pathway flexibility for continuously learning models. The infrastructure always comes before the clearance wave.
The timeline matters here. The contract runs 18 months from June 2026, which means deliverables to the FDA arrive around December 2027. If the FDA issues updated guidance on generative AI radiology clearance standards in 2028, this research will be the foundation.
Companies building in this category now have two years to understand what the new standard will look like - and to start building their validation datasets accordingly.
Deep Dive: What This Means for Radiology AI Procurement
If you are a health system CMO or radiology department chair, the Cognita-FDA contract changes how you should evaluate the generative AI tools you are currently being pitched.
Every radiology AI vendor will tell you their product performed well in validation. The question is now: validated on what? How many cases? From which institutions? Across which scanner manufacturers? For which patient demographics? And against what evaluation framework?
The FDA is establishing that the acceptable answer is approximately 1 million diverse patient exams evaluated by an ensemble LLM framework with expert radiologist review of disagreements. That is the new benchmark. Products that were validated on 300 cases at three academic medical centers do not meet it.
This creates a tiered procurement landscape:
Tier 1: Products with FDA Breakthrough Designation and Active Regulatory Science Partnership
Cognita and Aidoc are currently the only generative AI radiology platforms in this category. They are in active dialogue with the FDA about what their clearance evidence base needs to look like. Buyers of these products are buying into a validation rigor that the market will eventually require of everyone.
Tier 2: Products with Substantial Clinical Deployment Data
Several radiology AI vendors have real-world deployment data from large networks. This data is not structured as formal FDA validation, but it represents scale that exceeds traditional reader studies. Buyers should ask for specific numbers: How many cases? At how many sites? Against which reference standard?
Tier 3: Products with Traditional Reader Study Validation Only
These products have the highest risk exposure as FDA generative AI standards evolve. They may perform well at the sites where they were validated and fail unpredictably elsewhere.
For FQHCs and community health systems specifically, the equity dimension is the most important procurement criterion. You should be asking vendors: does your validation dataset include rural sites? Low-income patient populations? Older imaging equipment? If the answer is no, you are deploying a product that has never been tested on patients like yours.
What This Means For You
FQHC executives and health center leaders: The FDA's focus on diverse patient populations in the LLM jury framework is a direct signal that community health settings will be included in generative AI validation standards. Procurement decisions made today should prioritize vendors who can document performance across your specific patient demographics, not just academic medical center benchmarks.
Health system administrators and CMOs: Budget for validation evidence as a procurement requirement. The $1.29M FDA contract sets a precedent: scale and diversity in validation are not nice-to-haves. Build a vendor qualification checklist that includes sample size, site diversity, scanner manufacturer coverage, and demographic representation in training and test sets.
Radiologists and radiology department chairs: The LLM jury framework is designed to identify the cases where generative AI fails - specifically the rare, ambiguous, and edge-case presentations. Ask vendors to show you their discrepancy analysis on those cases, not just their headline sensitivity and specificity numbers. The outliers are where malpractice risk lives.
Healthcare investors and founders: The FDA's investment in evaluation infrastructure has a two-year payoff horizon. Companies that build their validation pipelines around the LLM jury methodology before it becomes required will have a significant competitive advantage at clearance. Conversely, companies that cut corners on validation today will face expensive retroactive remediation.
Policy advocates: The equity focus in this contract is an opportunity. The FDA is collecting systematic data on how radiology AI performs across diverse patient populations. Advocates should push for public reporting requirements tied to any generative radiology AI clearance, so health systems can make procurement decisions with real equity evidence rather than marketing claims.
The FDA is funding the infrastructure that will determine which radiology AI products are safe to deploy at scale. The companies building toward that infrastructure today are positioning themselves for a regulatory environment that will be more rigorous, more evidence-based, and more equity-focused than anything the field has seen.
The companies that treat validation as a box-checking exercise are building for a market that is about to change.
About the Author
Jonathan Govette is the Co-Founder and CEO of Oatmeal Health, an AI lung cancer diagnostic company catching cancers earlier in the communities that need it most. Oatmeal uses AI to identify unscreened high-risk patients, navigate them to care, and score every lung CT for malignancy risk - billed under CPT 0721T. Stage I survival is 77%. Stage IV is 9%. We work in FQHCs because that gap is largest there.
Jonathan writes daily about radiology, pulmonology, AI diagnostics, health policy, hospital operations, and healthcare startups.
Subscribe to stay ahead of healthcare's most important shifts.
Weekly deep-dives on AI, radiology, health policy, FQHCs, and the business of healthcare - written for operators, clinicians, and investors who want the signal, not the noise.
Subscribe at https://news.oatmealhealth.com
Key References
[FDA Awards Cognita Imaging $1.29M Research Contract - HIT Consultant (September 16, 2026)](https://hitconsultant.net/2026/09/16/fda-awards-cognita-imaging-1-29m-research-contract-llm-jury-radiology-report-evaluation/)
[Cognita Imaging Receives FDA Contract - BusinessWire (September 16, 2026)](https://www.businesswire.com/news/home/20260916329852/en/Cognita-Imaging-Receives-$1.29-Million-FDA-Contract-to-Test-New-Approach-to-Evaluating-Generative-AI-in-Radiology)
[FDA Gives Generative AI in Radiology Two Breakthrough Designations - STAT News (June 25, 2026)](https://www.statnews.com/2026/06/25/radiology-generative-ai-cognita-aidoc-fda-breakthrough-designation/)
[Cognita Imaging Wins FDA Contract to Test LLMs in Evaluating Radiology AI - MedTech Dive](https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/)
[FDA Considerations for Regulation of Generative AI-Enabled Medical Devices - FDA Discussion Paper](https://www.fda.gov/medical-devices/digital-health-center-excellence/considerations-regulation-generative-ai-enabled-medical-devices-discussion-paper-and-request)