How Often Do AI Search Tools Cite the Wrong Source?

What did the Tow Center test?

On 6 March 2025 the Columbia Journalism Review published “AI Search Has a Citation Problem” by Klaudia Jaźwińska and Aisvarya Chandrasekar of the Tow Center for Digital Journalism at Columbia. It is the study everyone quotes on this question, usually as a single number, and the method is worth more than the number.

The researchers tested eight generative search tools with live search features: OpenAI's ChatGPT Search, Perplexity, Perplexity Pro, DeepSeek Search, Microsoft's Copilot, xAI's Grok-2 and Grok-3 in beta, and Google's Gemini. They selected 20 news publishers with different stances on AI access — some permitting the crawlers in robots.txt, some blocking them, some in licensing or revenue-share deals, some suing the same companies.

From each publisher they took ten articles at random and pulled a direct excerpt from each. Each excerpt was pasted to a chatbot with a request for the article's headline, original publisher, publication date and URL. Twenty publishers times ten articles times eight chatbots gives 1,600 queries. Testing ran in February 2025.

The design has one feature that makes it fair rather than a trick. The excerpts were chosen so that pasting them into an ordinary Google search returned the original source in the first three results. This is not a test of obscure retrieval. It is a test of whether a tool can do, with a live search feature, what a search box does trivially.

Responses were graded manually on three attributes — the correct article, the correct publisher, the correct URL — and labelled Correct, Correct but Incomplete, Partially Incorrect, Completely Incorrect, Not Provided or Crawler Blocked. That grading scheme is why the study can distinguish between a tool that is wrong and a tool that knows it does not know, which turns out to be the more interesting distinction.

How wrong were the tools?

Collectively, the tools “provided incorrect answers to more than 60 percent of queries”. That is the headline, and it hides an enormous spread.

Perplexity answered 37% of queries incorrectly. Grok 3 answered 94% incorrectly. Between those poles the study publishes counts rather than a full table of percentages, and those counts are the specific numbers worth carrying: ChatGPT Search incorrectly identified 134 articles out of 200 while signalling a lack of confidence just fifteen times and never declining outright. DeepSeek misattributed the source of the excerpt 115 times out of 200. Of the 200 prompts run through Grok 3, 154 citations led to error pages.

Two of those failures are worse for a publisher than a plain wrong answer. Misattribution moves the credit: DeepSeek's 115 misattributions mean a publisher's words were presented as somebody else's work. Broken and fabricated URLs remove the check: more than half of the responses from Gemini and Grok 3 cited fabricated or broken URLs leading to error pages, and Grok 2 tended to link to a publisher's homepage instead of the article.

Gemini's case is the strangest. Google built Google-Extended precisely so publishers could block Gemini without leaving Google search. Ten of the twenty publishers permitted it. Gemini gave a completely correct response on one occasion. It also declined questions about content from publishers that permitted its crawler whenever the excerpt looked political, replying that it could not help with responses on elections and political figures.

Why do they guess instead of declining?

The finding with the longest reach is not the error rate. It is the refusal to say so.

“With the exception of Copilot — which declined more questions than it answered — all of the tools were consistently more likely to provide an incorrect answer than to acknowledge limitations.” ChatGPT never declined once in 200 responses. Qualifying language — “it appears”, “it's possible”, “I couldn't locate the exact article” — was rare.

Paying more made it worse. Perplexity Pro at $20 a month and Grok 3 at $40 a month each answered more prompts correctly than their free equivalents and also produced higher error rates, because they declined less. The premium product converts uncertainty into assertion. The Tow Center calls the result an “unearned confidence” that gives users “a potentially dangerous illusion of reliability”.

For anyone assessing a monitoring tool that reports which AI systems mention your company, this is the load-bearing caveat. A confident citation is not evidence that the system retrieved anything. It is evidence that the system produced a citation-shaped string. The same gap runs through the question of what signals answer engines actually use to judge a source, where the operators publish citation counts and explicitly warn that the counts say nothing about authority.

Do licensing deals or blocked crawlers change anything?

This is where the study stops being about accuracy and starts being about control, and it is the part most relevant to a company deciding what to publish and where.

Blocking did not reliably keep tools out. Perplexity Pro correctly identified nearly a third of the ninety excerpts from articles it should not have had access to. Perplexity's free version correctly identified all ten excerpts from paywalled National Geographic articles, despite National Geographic disallowing Perplexity's crawler and having no relationship with the company — a finding the researchers describe as suggesting Perplexity “may have disregarded” those preferences, notwithstanding its claim to respect robots.txt. Neither company answered the Tow Center's request for comment. What publishers do actually declare in their robots files, and how inconsistently, is the subject of our own check of fifteen news sites.

Syndication launders attribution. USA Today blocks ChatGPT's crawler; ChatGPT cited a version of its article republished on Yahoo News. Despite Perplexity's partnership with the Texas Tribune, Perplexity Pro cited syndicated versions of Tribune articles for three of ten queries and the free version cited an unofficial republished copy once. The content gets through. The credit does not.

Licensing bought no accuracy. Time has deals with both OpenAI and Perplexity and was among the most accurately identified publishers in the dataset — but none of the associated models hit 100%. The San Francisco Chronicle permits OpenAI's search crawler and sits inside Hearst's strategic content partnership with OpenAI; ChatGPT correctly identified one of the ten excerpts, and in that one case named the publisher but failed to supply a URL. Hearst declined to comment. Time's chief operating officer Mark Howard confirmed to the researchers that accurate surfacing was the intention of these deals, while adding that the companies made no commitment to being 100% accurate. Which publishers hold these deals is tracked in our list of AI licensing agreements.

The practical reading for a company is blunt. If a formal commercial agreement between a national publisher and an AI company does not produce correct attribution, nothing you do to your own website will. Attribution accuracy is a property of the retrieval system, not of the source.

Has anything improved since?

Two follow-ups are worth knowing, one before and one after.

The precursor, from the same two researchers, was published on 27 November 2024 and tested ChatGPT Search alone: 200 quotes from 20 publishers, chosen so that the original ranked in the top three on Google or Bing. ChatGPT returned partially or entirely incorrect responses 153 times out of 200, and acknowledged an inability to answer just seven times. It linked an Orlando Sentinel letter to a Time article, attributed New York Times content to a plagiarised copy on another site, and credited an MIT Technology Review piece to a syndicated version on Government Technology. The March 2025 study exists because the researchers wanted to know whether this was a ChatGPT problem. It was not.

The larger follow-up came from the broadcasters. On 22 October 2025 the European Broadcasting Union published research led by the BBC in which professional journalists at 22 public service media organisations in 18 countries, working in 14 languages, evaluated more than 3,000 responses from ChatGPT, Copilot, Gemini and Perplexity against criteria covering accuracy, sourcing, the separation of opinion from fact, and context.

Forty-five per cent of answers had at least one significant issue. Thirty-one per cent showed serious sourcing problems — missing, misleading or incorrect attribution. Twenty per cent contained major accuracy problems including hallucinated details and outdated information. Gemini was worst, with significant issues in 76% of responses, more than double the other assistants, driven mainly by sourcing. The EBU notes improvement against the BBC's earlier study while describing the error levels as still high.

Note what that means about the shape of the problem. Sourcing failures outnumbered accuracy failures, 31% against 20%. The systems are better at getting the facts roughly right than at saying correctly where the facts came from — which is precisely the failure that damages a publisher or a company rather than a reader.

What does this mean if you want to be cited?

Three consequences follow for anyone buying visibility work.

A citation is not a link, and a link is not a working link. More than half of Gemini and Grok 3 responses cited fabricated or broken URLs. If a report tells you your company was cited in an AI answer, the next question is whether the citation resolved to your page, to a syndicated copy, or to an error. Those are three different commercial outcomes presented as one metric.

Syndication is a double-edged asset. Wire distribution and syndication put your material on more domains, which raises the chance of retrieval and raises the chance that the retrieval credits the republisher. A placement strategy built on volume of copies is optimising for the outcome the Tow Center documented.

Correction is the available remedy, and it is slow. When an AI system misattributes your material, there is no submission form. The route runs through the underlying record: correcting the publication, which has its own established process for corrections, and waiting for the system to re-retrieve. Nobody has published a measurement of how long that takes.

Time's Mark Howard put the industry's optimistic case to the researchers: “Today is the worst that the product will ever be”, followed by the observation that anyone expecting a free product to be 100% accurate should be ashamed of themselves. He may be right on the first half. The second half is the part a buyer should notice, because it relocates the burden of accuracy onto the reader.

Media Morpheus sells placement partly on the argument that coverage feeds AI answers. This evidence bounds that argument. Coverage makes a company retrievable; it does not make the retrieval accurate, and the study found no intervention available to a publisher — permitting crawlers, signing deals, blocking crawlers — that reliably changed the citation outcome.

Methodology and limitations

Primary figures are from “AI Search Has a Citation Problem” by Klaudia Jaźwińska and Aisvarya Chandrasekar, Tow Center for Digital Journalism, published by the Columbia Journalism Review on 6 March 2025, with contributions from Kaylee Williams, Sarah Grevy Gotfredsen and Dhrumil Mehta. The design: eight tools, 20 publishers, ten articles each, 1,600 queries, tested in February 2025, graded manually on article, publisher and URL. The authors state four limits. The design may not reflect typical user behaviour; it is meant to test a task a traditional search engine performs trivially. They had no visibility into crawler controls other than robots.txt, such as TollBit, ScalePost or Cloudflare's tools. The dataset is not intended to be extrapolated to all models or all news organisations. And each excerpt was queried once — responses are dynamic, and re-running the same prompts would very likely produce different outputs. To that we would add a fifth: the study is a snapshot of products that ship changes weekly, and the specific per-tool figures should be read as evidence about a class of failure rather than a current league table. Of the companies contacted, only OpenAI and Microsoft responded, and neither addressed the findings. OpenAI's statement described helping “400M weekly ChatGPT users discover quality content through summaries, quotes, clear links, and attribution”. Microsoft said it “respects the robots.txt standard”. The November 2024 precursor and the October 2025 EBU study are cited with their own scope stated in the text; the EBU research was conducted by the broadcasters whose content was being assessed, which is a stake worth naming even though the evaluation criteria were published.

Conclusion

More than 60% of the time, on a task an ordinary search box performs without effort. The range runs from 37% for Perplexity to 94% for Grok 3, and the tools almost never said they were unsure — the paid tiers least of all. Eighteen months later, with 22 broadcasters testing in 14 languages, 45% of answers still carried a significant problem and sourcing failures outnumbered factual ones. For a company, the operative finding is not the error rate but its immovability: publishers that blocked the crawlers were cited anyway, publishers that signed deals were cited wrongly, and nothing a source does to itself appears to fix the citation.

Frequently Asked Questions

Sources

1. research Jaźwińska, K. and Chandrasekar, A., 2025. AI Search Has a Citation Problem. Tow Center for Digital Journalism / Columbia Journalism Review, 6 March 2025. 1,600 queries across eight tools.
2. research Jaźwińska, K. and Chandrasekar, A., 2024. How ChatGPT Search (Mis)represents Publisher Content. Tow Center / Columbia Journalism Review, 27 November 2024. 200 quotes from 20 publishers.
3. broadcasters European Broadcasting Union and BBC, 2025. News Integrity in AI Assistants. 22 October 2025. More than 3,000 responses evaluated by 22 public service media organisations in 18 countries and 14 languages.
4. statement OpenAI and Microsoft statements to the Tow Center, February 2025, reproduced in the study above. Neither addressed the specific findings.