Why I Prefer an AI That Knows When to Say “I Don’t Know”
![]() |
| Both fictional assistants answer 60% correctly, but B makes fewer wrong claims. Original chart illustrating the AA-Omniscience scoring principle; these are not real model results. |
An AI assistant can make a wrong answer look remarkably finished. A tidy comparison table, a precise-looking number and a reassuring explanation can give me the feeling that the research is done before I have checked any of it.
For product and travel decisions, that is exactly the feeling I want to slow down. I value an assistant that identifies what it cannot verify and helps me resolve it. A confident guess about regional compatibility or a ticket condition can cost more than the time saved by getting an instant answer.
That is why AA-Omniscience interests me. It makes a useful distinction between knowing an answer, attempting an answer and getting it wrong.
Getting things right is only part of the picture
Artificial Analysis’s AA-Omniscience evaluation rewards correct answers, penalizes incorrect ones and leaves abstentions neutral. An abstention means the model declines to supply an answer rather than guesses.
I like the incentive behind that approach. A system should benefit from knowing when its knowledge is insufficient, while still being rewarded for answering correctly. Saying nothing to everything is not the goal either: that produces no useful correct answers.
The current methodology describes version 1.1 as 3,000 questions across 42 topics. It also warns that the revised question set is not directly comparable with the original version. A higher or lower number only means something when I know which test produced it.
The same accuracy can hide very different mistakes
Here is a fictional example with 100 questions. These are made-up assistants, not results for any commercial model. For simplicity, there are no partially correct answers.
| Outcome | Assistant A | Assistant B |
|---|---|---|
| Correct answers | 60 | 60 |
| Wrong answers | 40 | 10 |
| Abstentions | 0 | 30 |
| Correct answers ÷ all questions | 60% | 60% |
| Questions attempted | 100% | 70% |
| Net score: correct minus wrong, per 100 questions | +20 | +50 |
Both get the same number right. But Assistant B leaves me with 30 explicit gaps to investigate instead of filling those gaps with wrong answers. In a factual research task, I prefer knowing where the missing information is.
That does not mean B knows more, or that its score of 50 means “50% reliable.” The net score is a balance of rewarded and penalized outcomes. I need the underlying counts to understand it.
The research paper behind the benchmark also defines its hallucination rate in a particular way: wrong answers as a share of the questions not answered fully correctly, including partial answers and abstentions. It is not simply the percentage of every answer that is wrong. Whenever I see a reliability percentage, I want to know what was counted and what it was divided by.
“I cannot verify that” can save a bad purchase
Imagine I ask whether a particular imported phone supports a feature on my mobile network. A useful assistant identifies the exact model, the relevant market and the operator’s documentation. If one of those is missing, I want it to say which one.
An unhelpful answer fills the gap with specifications from a similarly named model. The prose might be excellent; the purchase decision is still built on the wrong device.
I apply the same thinking to travel. A transport timetable, luggage allowance or interpretation service can depend on the date, route and location. I want the current operator’s page and the relevant conditions. A generic statement about what “usually” happens is useful context, but it is not confirmation of my booking.
These are examples of how I judge an answer, not tasks whose outcomes this benchmark guarantees.
A knowledge test is not my complete workflow
AA-Omniscience evaluates factual recall without giving the model web tools or source documents. My real research often involves both. A model’s stored knowledge, its ability to find a current page and its ability to interpret that page are separate things.
I therefore use the benchmark as one signal. It cannot certify that an assistant will read the correct regional specification, notice an outdated price or check a booking condition properly.
Nor does cautious language prove that uncertainty is well judged. Adding “probably” to an invented specification does not make it useful. I want the assistant to stop making the unsupported claim and explain what evidence would settle it.
The instructions I find useful
For a purchase or travel question, my preferred brief is specific:
Separate confirmed facts from assumptions. For facts that can change, link the current official source and identify the market or version. If you cannot verify something, leave it unresolved and tell me the next check needed. Do not fill missing details with a similar product or service.
That prompt is a working habit, not a guarantee. I still open the sources and check the details that could change the decision. A link earns its place by supporting the claim, not merely by existing.
This is the same principle behind checking an AI-built dashboard before trusting its charts. Presentation makes an answer easier to read. Verification makes it useful.
I want AI to reduce the work of researching a decision. An honest gap, accompanied by a sensible next step, helps me do that. A complete-looking answer that quietly invents the missing pieces gives me more work to undo.
.png)
Comments
Post a Comment