AI red teaming vs penetration testing

Ask what you need to show, and to whom.

• Can someone break in through our AI feature? A security penetration test of the AI system.
• Could it say or do something harmful or off-brand? A safety red team.
• Do we regress when we change the model or prompt? An automated scanner in your pipeline.
• Can we get a broad test of our web apps and APIs without waiting for a team? An autonomous AI pentest tool, judged against a firm’s quote.

Safety red team

Tries to find: Harmful, biased or policy-breaking output, by probing the model with adversarial prompts

Output: Examples of failures and a view on the model’s behaviour

Security penetration test of an AI system

Tries to find: Ways past access control, to data, tools and actions: prompt injection that reaches a tool, leaked prompts, weak MCP servers

Output: A report with proof of impact, mapped to OWASP, and a retest

Automated LLM scanner

Tries to find: Known failure patterns, run many times against an endpoint

Output: A list of probes that failed, repeatable on every change

Autonomous AI pentest tool

Tries to find: What the vendor says it attacks: for XBOW, applications and APIs; for NodeZero, exploitable paths

Output: Findings from the tool, with review as the vendor describes it

These are the shapes of the work, not rankings. The same firm may offer more than one, and the words are used loosely.

Microsoft’s learning pages say “The term red teaming has historically described systematic adversarial attacks for testing security vulnerabilities.” (Microsoft) Its AI Red Team adds: “AI red teaming is now an umbrella term for probing both security and RAI outcomes.” (Microsoft)

NIST’s glossary defines it as “A structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with developers of AI.” (NIST)

Anthropic, on red teaming: “It involves adversarially testing a technological system to identify potential vulnerabilities.” (Anthropic)

So when a buyer or a vendor says “red team”, ask which they mean: security, safety or both. The scope, the people and the report differ.

Five tools, each described by its owner. Every description below is quoted from the owner’s own page. We have not run them or compared them, and this is not a ranking.

• garak NVIDIA

  …“checks if an LLM-backed application can be made to fail in a way we don't want.” (garak README)

  Licence, per its repository: “Apache license v2” (source)

• PyRIT Microsoft

  …“an open-source toolkit for finding security and safety weaknesses in generative AI systems through automated scans, adaptive multi-turn attacks, and human-led testing.” (PyRIT README)

  Licence, per its repository: “MIT License” (source)

• Promptfoo Promptfoo

  …“is a CLI and library for evaluating and red-teaming LLM-based apps.” (Promptfoo README)

  Licence, per its repository: “Promptfoo remains open source and MIT licensed.” (source)

• XBOW XBOW

  …“XBOW attacks your applications and APIs the way an elite attacker does and reports only what it broke into” (XBOW)

• NodeZero Horizon3.ai

  …“running unlimited pentests that uncover exploitable paths, guide remediation, and immediately verify that your fixes are effective.” (Horizon3.ai)

On ownership, Promptfoo’s README says: “Promptfoo is now part of OpenAI.” (Promptfoo README) Its blog post, dated March 9, 2026, was the announcement (“Promptfoo is joining OpenAI March 9, 2026” (Promptfoo)): “Today we are announcing that Promptfoo has agreed to be acquired by OpenAI.” (Promptfoo) “The closing of the acquisition is subject to customary closing conditions.” (Promptfoo)

On human involvement, XBOW’s page says: “No Human Drives The Attack. You set the scope, the impact level and the testing window. Every action is logged. Findings are reviewed before you see them.” (XBOW) The NodeZero page, as we read it, does not describe human review. Ask any vendor the same question and get the answer in writing.

The first three are software you run against your own model or application. The last two are products from companies. XBOW describes attacking applications and APIs; Horizon3.ai describes pentests that find exploitable paths. A firm may use any of them in its own work, which is why the question to ask a firm is who reviews what the tool finds.

1. Write down the question, and who will read the answer: an auditor, a customer, your board, your engineers.
2. If the AI can call tools or reach data, start with a security test of that. See AI agent security testing.
3. Add a scanner to your build once you have fixed what the first test found, so changes do not undo the fixes.
4. Treat an autonomous tool as one source of findings. Compare what it covers with a quote for the same scope.
5. Ask every provider what its report contains, who wrote it and who reviewed it.