2 against 1
Better
+33 points of cases fully correct
95% interval +17 to +50 points. Exact McNemar test on the 10 cases where only one run was right: p = 0.002. No case got worse.
3 against 1
Better
+38 points of cases fully correct
95% interval +15 to +58 points. Exact McNemar test on the 12 cases where only one run was right: p = 0.006. Got worse on 1 case: NCT05576480.
A verdict says better or worse only when the paired bootstrap interval and the exact McNemar test agree. Otherwise it says no clear difference, even if one number is higher.
Every run
| Run | Model and prompt | Cases fully correct | Hallucination rate | Each field | Median latency, tokens |
|---|---|---|---|---|---|
| 1 | groq/qwen/qwen3.8-27bprompt v1, 30 cases scored | 47% | 38% | study type27/30 phase19/30 enrollment27/30 minimum age22/30 |
6453 ms 25,812 in, 1,211 out |
| 2 | groq/qwen/qwen3.8-27bprompt v2, 30 cases scored | 80% | 6% | study type27/30 phase29/30 enrollment27/30 minimum age29/30 |
8483 ms 26,832 in, 1,247 out |
| 3 | groq/openai/gpt-oss-20bprompt v2, 26 cases scored, 4 not run yet | 81% | 2% | study type24/26 phase26/26 enrollment23/26 minimum age26/26 |
6099 ms 24,273 in, 4,225 out |
- correct
- wrong value
- missed, although stated
- invented, although not stated
Where each run lands
Why hallucination is counted on its own
A model that misreads a number and a model that invents one are different risks. A value counts as stated only if it appears in the text the model saw; otherwise the right answer is null, and any value is counted as invented.
The hallucination rate is the invented answers divided by the fields whose right answer is null.