Skip to content
StrikeCyberStrikeCyber
Field Manual · Concept

AI Red Teaming vs Penetration Testing

Reviewed 13 November 2026Concept 2 min read

Quick answer

Penetration testing finds exploitable flaws in software and infrastructure, such as broken access control, injection and misconfiguration. AI red teaming attacks the behaviour of AI models and the applications built on them, using techniques such as prompt injection, jailbreaking and data extraction, and measures how often they succeed. An AI feature is both, so most need both forms of testing.

On this page
01

The Short Version

A penetration test treats a system as code and configuration and asks whether an attacker can break it. AI red teaming treats a model as a component whose behaviour can be steered by language, and asks what an attacker can make it say, reveal or do. An AI chatbot is still a web application, with APIs, authentication and data stores underneath, so it inherits every traditional weakness and adds new ones.

02

Side by Side

Penetration testingAI red teaming
TargetApplications, APIs, networks, cloudModels, prompts, retrieval, agents and tools
Typical flawsAccess control, injection, misconfiguration, outdated softwarePrompt injection, jailbreaks, data leakage, excessive agency, harmful or false output
ResultsDeterministic: a flaw exists or it does notProbabilistic: an attack succeeds some of the time
MethodManual exploitation with supporting toolsAdversarial prompting, automated attack generation and repeated trials
Reference listsOWASP Top 10, OWASP API Security Top 10OWASP Top 10 for LLM Applications, MITRE ATLAS
FixPatch or reconfigure, then retestReduce access and contain impact; model behaviour cannot be fully fixed
03

Why One Is Not Enough

A penetration test alone will check authentication and APIs but miss an assistant that can be talked into emailing a customer list through prompt injection. AI red teaming alone may find jailbreaks but miss that the chat API has a broken object level authorisation flaw exposing every conversation. The most serious real findings in AI systems usually chain the two: a language-level attack that reaches a conventional weakness, such as improper output handling or a tool with excessive agency.

04

What a Combined Test Covers

An LLM application test typically covers each item in the OWASP Top 10 for LLM Applications alongside standard web and API testing: direct and indirect prompt injection, system prompt leakage, data disclosure through retrieval, unsafe handling of model output, agent and tool permissions, cost and rate limits, and the authentication, authorisation and infrastructure around the model. Because results vary between attempts, AI findings are reported with how reliably they reproduce, not just whether they worked once.

05

Safety Testing Is Different Again

Some AI red teaming focuses on safety rather than security: whether a model produces harmful, biased or prohibited content. That matters for model developers and public-facing systems, but it is a different exercise from testing whether an attacker can use your AI feature to steal data or take actions. Be clear which one you are buying.

06

Regulatory and Framework Context

In Australia, the Voluntary AI Safety Standard calls for testing AI systems before deployment and monitoring them afterwards, and ASD's guidance on engaging with artificial intelligence and on deploying AI systems securely describes threats such as prompt injection and data poisoning. Combined testing provides evidence for ISO/IEC 42001 and for existing obligations under APRA CPS 234 where AI touches regulated systems.

Latest intel on this topic

Updated hourly from the daily briefing.

Common questions

Is AI red teaming just a penetration test of a chatbot?

No. It adds adversarial testing of model behaviour, which a conventional penetration test does not cover, and it measures how often attacks succeed rather than whether a single flaw exists.

Do we need AI red teaming if we use a vendor's model?

Yes. The vendor tests the model; you still need to test your prompts, data connections, tools and permissions, which is where most real risk sits.

Can AI red teaming be fully automated?

Automated attack generation helps with scale and repeatability, but finding the chains that matter for your use case still needs experienced human testers.

Change log

  1. 13 November 2026First published.
1300 654 898