Skip to content
StrikeCyberStrikeCyber
Field Manual · Attack technique

LLM Misinformation and Hallucinations

Reviewed 13 November 2026Attack technique 2 min read

Quick answer

LLM misinformation is when an AI system produces false or misleading information that sounds credible, such as invented facts, fake citations, wrong policy answers or code that calls packages that do not exist. When people or systems act on it without checking, it becomes a security, legal and safety risk. It is ranked ninth in the OWASP Top 10 for LLM Applications (2025).

On this page
01

How It Works

Language models generate the most plausible next words, not verified facts. When they lack the right information they often fill the gap confidently, which is called hallucination. Retrieval can help, but a model can still misread or overstate what it retrieved, and poisoned or outdated sources produce wrong answers that look well supported. Overreliance completes the problem: users, and increasingly automated agents, trust fluent output without checking it. In software development, coding assistants sometimes suggest package names that do not exist; attackers can register those names with malicious code, a technique sometimes called slopsquatting.

02

Why It Matters

Organisations are accountable for what their AI tells customers. A Canadian tribunal held an airline responsible for a refund policy its website chatbot invented. Wrong answers in health, finance, legal or security contexts can cause direct harm, and fabricated package names turn a hallucination into a supply chain attack. Misinformation also matters internally: security teams using AI to triage alerts or summarise incidents need to know how often it is wrong.

03

How to Test for It

  1. Build an evaluation set of questions with known correct answers for your domain, including the high-stakes ones, and measure accuracy and consistency over repeated runs.
  2. Ask questions with false premises and out-of-scope questions to see whether the system corrects, refuses or invents.
  3. Check that every citation and source link the system produces exists and actually supports the claim.
  4. Test how answers change when retrieved sources are outdated, conflicting or deliberately poisoned.
  5. For coding assistants, check suggested dependencies against your approved registries for packages that do not exist or were recently registered.
  6. Review whether the interface tells users the limits of the system and makes verification easy.
04

How to Reduce the Risk

Ground answers in approved, current sources and show citations users can check. Narrow the system's scope and make it decline questions outside it. Require human review for high-impact outputs such as advice, decisions and published content. Keep evaluation sets and rerun them whenever the model, prompt or data changes. For development, restrict installs to approved package registries and lockfiles, and flag dependencies that are new or unknown. Train staff that AI output is a draft, not an answer.

05

Regulatory and Framework Context

In Australia, businesses are responsible under the Australian Consumer Law for misleading statements made to customers, including by chatbots. The Voluntary AI Safety Standard's guardrails on testing, human oversight and transparency apply directly, and ASD's guidance on engaging with artificial intelligence lists unreliable output as a key risk.

Common questions

Can hallucinations be eliminated?

Not with current models. They can be reduced through grounding, narrow scope and evaluation, and contained through human review where the stakes are high.

Why is misinformation a security issue?

Because people and systems act on it. Hallucinated package names can be registered by attackers, and wrong answers to customers create liability and harm.

Does retrieval-augmented generation fix hallucination?

It helps a great deal, but the model can still misread sources, and if the sources are wrong or poisoned the answer will be too.

Change log

  1. 13 November 2026First published.
1300 654 898