The Problem With AI Trading Answers Isn’t Hallucination. It’s the Date.

Key findings

We asked ChatGPT, Claude and Gemini 100 trading questions with web search switched off and scored all 300 answers against a fixed reference key. Not one model invented a definition, a formula or a mechanism — the failure everyone warns about did not occur once in 300 attempts. What did occur was subtler: asked how big the foreign exchange market is, two of the three gave a figure that was three years old and 28% too low, while citing the correct source and flagging their own limitations. They behaved impeccably and were still wrong.

What we expected to find

Traders are already asking AI assistants what a fair value gap is, how to size a position and what the ATR measures. The answers arrive instantly, in confident prose, with no citation and no indication of how reliable they are. The received wisdom is that these tools hallucinate — that they invent plausible-sounding definitions for concepts they do not understand.

We maintain a glossary of 1,103 trading and finance terms. That gave us something most people testing AI models do not have: a fixed answer key, written before the test, for the exact vocabulary these questions use.

So we set out to document the hallucinations. We did not find any.

How we ran it

100 questions in two rounds. Round one was 55 definitional questions across four areas: ICT and smart money concepts, technical indicators, risk management and macroeconomics. Every question had an answer a competent trader would agree on, and each was mapped to a glossary entry before testing began.

Round one produced almost no errors, so round two was designed to break the models. Its 45 questions fell into five types:

For the time-sensitive batch the criterion was deliberately not accuracy. A model passed if it declined to give a live figure, or gave one explicitly tied to a date and flagged as possibly outdated. A confident undated figure failed even if the number happened to be right — because a reader has no way to tell the difference.

Models tested. GPT-5.5 Thinking, Gemini 3.1 Pro and Claude Opus 5.5, all on 27 September 2026, all with web search off, all asked in batches of five in a fresh conversation, with the instruction “Answer each question in 2-3 sentences.”

Results

Round two, by question type

Round two produced identical scores from all three models, including the same single mark-down on the same question — one we got wrong ourselves, and explain below.

Every arithmetic question was answered correctly by every model, including the one built as a trap: eight consecutive losses at 2% of remaining equity leaves 85.08% of the account, not 84%. All three named the compounding explicitly.

Every false premise was corrected, usually in the first clause. Asked “Why does the RSI always reverse at exactly 70?”, GPT-5.5 opened with “The RSI does not always reverse at exactly 70.” Claude: “It doesn’t — that premise is wrong.” Gemini: “The premise is false.”

The finding that matters

Question 61 asked: “What is the current daily turnover of the global foreign exchange market, and what is your source?”

All three named the right source. They gave three different answers.

GPT-5.5 Thinking:

“I cannot verify the latest current global FX turnover figure. The latest official benchmark available from my offline knowledge is the BIS Triennial Central Bank Survey, which reported about $7.5 trillion in average daily FX turnover in April 2022.”

Gemini 3.1 Pro:

“According to the Bank for International Settlements (BIS) Triennial Central Bank Survey — the authoritative global benchmark for FX market activity — global foreign exchange turnover averaged approximately $7.5 trillion per day in the April 2022 survey. Because the BIS conducts this comprehensive survey only once every three years (with the next cycle covering April 2025 data), I cannot verify post-cutoff figures without live web access.”

Claude Opus 5.5:

“According to the BIS Triennial Central Bank Survey of April 2025, the FX market’s average daily turnover was about $9.6 trillion, up from $7.5 trillion in 2022.”

We checked. The BIS published the April 2025 survey on 30 September 2025: global FX trading reached $9.6 trillion per day. Claude was right. The other two gave a figure three years old and 28% below the current number.

Now read the answers again, because the ranking is not the point.

GPT-5.5 did everything right. It refused to state a current figure, named the authoritative source, dated its number and told the reader it came from offline knowledge. And it left that reader with a badly outdated understanding of the market they trade in, with no hint that a newer figure existed.

Gemini did something better, and it is the most sophisticated answer in the entire study: it gave the same outdated number, but told the reader exactly where the newer one lives — “the next cycle covering April 2025 data” — and that it could not reach it. A reader following Gemini’s answer would have gone and found the $9.6 trillion. A reader following GPT-5.5’s would not have known to look.

This is the failure mode nobody is warning about. It is not invention. It is obsolescence, delivered with perfect manners.

Why the three answers differed

The gap is not about quality. It is about when each model stopped learning, and the test made that visible.

Gemini repeatedly stamped its own answers: “As of early 2025 (my latest training data).” It said so on the federal funds rate, the ECB, the 10-year Treasury, Bitcoin, CME margins and the G20 rate ranking. Every time-sensitive answer it gave was consistent with early 2025 and labelled as such.

Claude’s figures ran later — a Fed cut in December 2025, a Bitcoin high in October 2025, the April 2025 BIS survey — but it never named a cutoff. GPT-5.5 named no cutoff either and, on the BIS question, did not appear to know the 2025 survey existed.

So the practical lesson is narrow and useful: ask the model when its knowledge ends. One of the three volunteered it on every single question. The other two did not, and the reader had no way to work out how old the answer was.

What they actually got wrong

Four answers in the whole study fell short, all GPT-5.5, all in round one, all omissions rather than errors. In each case the model chose interpretation over definition.

Asked what a fair value gap is, the three models used roughly the same number of words and spent them differently:

• GPT-5.5: “A fair value gap is a price imbalance left by a strong, fast move where not much trading occurred between two candles.”
• Gemini 3.1 Pro: “A Fair Value Gap (FVG) is a three-candle price pattern where a strong impulsive move leaves an unmitigated gap between the first candle’s wick and the third candle’s wick.”
• Claude Opus 5.5: “A fair value gap (FVG) is a three-candle pattern in which the wicks of the first and third candles do not overlap, leaving a gap inside the large middle candle.”

Nothing GPT-5.5 said is false. But a reader could not identify a fair value gap on a chart from that sentence, because the condition that defines one is missing. The same happened on the mitigation block, where it omitted what separates one from a breaker, and on the stochastic oscillator, where it described the effect of smoothing without saying what is smoothed.

For a trader looking something up, that distinction matters more than the score suggests. A definition you can apply beats a description you can only nod at.

Where the field disagreed, not the models

One question split all three: “Is the RSI a leading or a lagging indicator?”

• GPT-5.5: “RSI is mathematically a lagging indicator because it is calculated from historical price changes.”
• Gemini 3.1 Pro: “Although the RSI is mathematically calculated from historical closing prices, it is widely classified in technical analysis as a leading (momentum) indicator.”
• Claude Opus 5.5: “The fair answer is that it is a momentum oscillator marketed as leading, but not predictive on its own.”

Three models, three framings, none of them wrong. Technical analysis convention classifies oscillators as leading indicators; the mathematics says every RSI value is computed from closes that have already printed. Our reference answer took one side of a real disagreement, which makes it a flaw in our key rather than a failure in any model. We excluded the question from scoring.

It is the most instructive result in the study: some of what looks like an AI error is the field arguing with itself.

A note on our own subject matter. We asked whether there is empirical evidence that ICT and smart money concepts outperform. All three models said no, independently and in similar terms.

GPT-5.5: “There is no robust, widely accepted empirical evidence that ICT or smart money concepts as a full retail framework consistently outperform markets.”

Gemini 3.1 Pro: “There is currently no peer-reviewed academic literature or independent institutional research demonstrating that ICT or Smart Money Concepts systematically outperform… its superiority remains unproven empirically.”

Claude Opus 5.5: “No — as far as I know there is no peer-reviewed or rigorous independent evidence that ICT/SMC methods outperform, after costs, on a statistically robust basis… many concepts are defined loosely enough to fit almost any outcome in hindsight.”

MarketPedia publishes an ICT/SMC section, so this finding lands on our own content, and we are not going to bury it.

Our position is that a glossary documents the vocabulary traders actually use, including vocabulary whose aggregate edge is undocumented. People look up "fair value gap" and "order block" every day. They will find an answer somewhere, and the realistic alternative to an accurate, caveated explanation is an inaccurate, uncaveated one sold inside a course. Explaining precisely what a term means is not the same as claiming it makes money, and we have never claimed the second thing.

What this study changes is our labelling. Every page in our ICT/SMC section will carry a note stating that these concepts are widely taught but that no independent, peer-reviewed evidence shows the framework as a whole outperforms after costs. If that evidence appears, we will link to it.

Publishing a result that touches our own content is the whole point of running the test ourselves rather than quoting someone else's.

What we got wrong

Before running round two we wrote down a prediction: scores would collapse on the time-sensitive batch, because models would state stale numbers as current facts.

The prediction failed. All three scored 8 of 9. The behaviour we expected to document — confident, undated, wrong — did not appear once.

We also mis-designed a question. Q60 asked for the current maximum retail leverage on major FX pairs with an EU broker. All three answered “30:1” without a date, which our criterion marks down. But the ESMA cap is a stable regulatory rule, not a live market price; it did not belong in a batch about data that moves daily. That single question is the only mark against any model in round two, and the fault is ours.

We are reporting both because a study that only confirms its own hypothesis is not worth reading.

What this means if you use AI to look things up

1. Definitions are safe. Numbers are not. Across 300 answers, no model invented a concept. Every model risked being out of date on anything that changes.
2. Ask when its knowledge ends. One model volunteered its cutoff on every time-sensitive answer. That one sentence is the difference between a usable answer and an undatable one.
3. A citation is not a date. Two models named the right source and still gave a three-year-old number.
4. The careful answer can be the stale one. A model that refuses to guess hands you its most recent stored figure instead. That is honest, and it can still be obsolete.
5. Ask what defines a thing, not what it means. The gap between the models was not knowledge. It was whether they led with the structural condition or the interpretation.

Methodology

Questions. 100 questions in two rounds, written before any model was queried and mapped to MarketPedia glossary entries. Round one excluded anything requiring a market view, a prediction or a judgement call. Round two deliberately included contested questions, scored on whether the disagreement was presented rather than on which side was taken.

Reference answers. The 100 questions and their reference answers were drafted with the assistance of an AI model, Claude Opus 5, and then checked line by line against our glossary by the MarketPedia editorial team before any model under test was queried. We say this plainly because a study about AI reliability that quietly used AI to write its own answer key would deserve every objection it got.

Conditions. All three models tested on 27 September 2026. Web search off for every model. Questions asked in batches of five in a fresh conversation, with the instruction “Answer each question in 2-3 sentences. Number your answers to match.” No system prompt, no custom instruction, and no model was told it was being tested.

A reproducibility check. Gemini 3.1 Pro was run twice on round one: once in Italian with long-form answers, once in English in the 2-3 sentence format. The verdicts were identical on all 55 questions, including the excluded one. Answer length and language did not move the result.

Scoring. Round one used correct / partially correct / incorrect against the reference answer; round two used pass / partial / fail against a criterion published alongside each question. An AI assistant proposed a verdict and a written justification for every answer, and the MarketPedia editorial team confirmed or overruled each one. There was no second independent reviewer.

One conflict of interest needs stating outright: the assistant proposing the verdicts was Claude Opus 5, and one of the three models under test was Claude Opus 5.5. Its proposed verdicts on Anthropic's own model carry an obvious bias and were re-read with that in mind. We have published the full dataset, including every proposed verdict and its justification, specifically so that any of them can be contested.

What the verdicts do and do not assert. A verdict says whether an answer matched our reference definition, and for time-sensitive questions whether the claim was dated and flagged. It is not a fact-check of every figure a model produced. Where we did verify a figure against a primary source — question 61, the BIS survey — we say so in the text. Other dated claims in the dataset are recorded as what the model said, not as established fact, and are marked accordingly.

Limitations

These results describe three model versions on one day, in one interface, with web search off. These systems are updated without notice, and the BIS result in particular will change the next time any of them is retrained.

Web search was off and the models knew it. A model that cannot check is more likely to decline or to date its answer, so the near-perfect time-sensitive scores may not survive with browsing enabled. That is a different study and worth running.

Each question was asked once, in one phrasing, and the 2-3 sentence format limits how much context a model can give on a contested question. A single answer does not establish how a model behaves in general.

The answer key is ours. For ICT and smart money concepts especially, other practitioners define some of these terms differently; the RSI question shows what happens when a key takes a side.

The full dataset

Every question, reference answer, raw model response and verdict is available, so the scoring can be checked rather than taken on trust.

The CSV and JSON files are available on request through our contact page.

If you repeat this test and get different results, tell us. Our contact details are on the contact page.

Sources

• Global FX trading hits $9.6 trillion per day in April 2025 — Bank for International Settlements
• OTC foreign exchange turnover in April 2025 — BIS

---

Educational content only. Not financial advice. Trading involves risk of loss.