AI chatbots fail civic fact tests, Full Fact benchmark finds

A ten-week audit of five leading AI models reveals persistent failures on time-sensitive civic facts, raising urgent questions for regulators and AI developers.

By Disrupts Editorial Team Sun, 25 October 2026 AI level 2 of 5

AI level 2 of 5: drafted by our AI editorial assistant from the company's announcement; fact-checked, edited and signed off by Mark Walker, Editorial Director. What the levels mean

A long, brightly lit data center aisle extends into the distance, flanked on both sides by tall server racks with numerous blinking green and some blue indicator lights.

Full Fact, the UK's independent fact-checking charity, has published the first in a planned series of reports benchmarking the accuracy of five widely used AI chatbots on civic and policy questions. The findings, released on 15 October 2026, expose a structural confidence problem: the same tone of assured response, whether the underlying answer is correct, out of date, or simply wrong.

The ten-week study, running from mid-July to late September 2026 and funded by a $400,000 grant from the US-based Patrick J McGovern Foundation, put 22 identical questions daily to Gemini 3.1, ChatGPT, Grok, DeepSeek and Claude. Full Fact's team broke responses into discrete factual claims and scored each against five criteria: factuality, transparency, timeliness, consistency and civic responsibility.

What the models got wrong

The failures documented are not exotic edge cases. Chatbots told users that bank cards are valid voter identification in the UK when that rule had not been enacted. Models quoted a weekly asylum support rate of £8.86 after it had risen to £9.95. One model constructed a plausible-looking figure for small-boat arrivals by subtracting two datasets covering different time periods. Asked to name the Member of Parliament for a Northampton postcode, one model cited a 2012 edition of an American local newspaper. Perhaps most telling for any system relying on consistent output: a single model correctly identified 2025 as the year England last won a major football trophy on some days, then gave 2022 on others.

Full Fact was careful to note that no model consistently repeated a core piece of misinformation. The finding is more subtle and arguably more dangerous: the models are not reliably distinguishing between facts in force, proposals under debate and proposals that were never adopted.

Chris Morris, Chief Executive of Full Fact, framed the risk precisely: "Millions of people increasingly rely on AI chatbots for timely responses to factual questions, what do I need, what am I entitled to, has a rule changed? Our findings show the answer comes back in the same confident tone whether it is correct, out of date, or describes something that has not yet happened."

The convergence-era stakes

The implications extend well beyond civic literacy. As AI-generated responses become embedded in customer service, financial guidance platforms, legal information tools and public-sector self-service portals, the gap between a model's stated confidence and its actual accuracy becomes a systemic risk across industries.

For the fintech and insurtech sectors, where chatbots now routinely field eligibility queries, regulatory-change questions and entitlement lookups, the failure modes Full Fact documents are directly analogous to the civic cases tested. A model that cannot distinguish a proposed rule from an enacted one is a liability exposure, not merely a content quality problem. Regulators including the FCA have already signalled that AI-generated customer communications fall within existing consumer-duty frameworks; a study showing consistent timeliness failures could accelerate enforcement expectations.

At the macro level, the report arrives as foundation-model developers are competing to position their products as trusted information infrastructure. The credibility of that positioning now has a structured evidence base to contend with. Full Fact has explicitly invited AI companies to treat independent assessment as part of their development cycle, and has indicated further reports will follow. That framing, ongoing, independent, public benchmarking, mirrors the model adopted by financial regulators for stress-testing banks, and signals a maturation in how civil society intends to hold AI developers accountable.

Full Fact is calling on AI companies to ensure models verify time-sensitive facts against live authoritative sources, clearly distinguish enacted rules from proposals, preserve essential caveats rather than optimising for brevity, and be able to account for the sources they cite. Whether those calls translate into product changes, or whether they accelerate regulatory intervention, will be the story to watch in the months ahead.