What Are AI Red Teaming and LLM Pentest? Security Testing for AI Applications

Introduction: Are AI Systems Being Tested Against Real Attack Scenarios?

Enterprise use of artificial intelligence is spreading rapidly.

Companies are using large language models in their customer support processes.

Software teams are using code assistants.

Finance departments are developing AI-supported analysis systems.

Human resources teams are carrying out document analysis.

In cyber security operations, AI-supported incident assessment systems are being used.

In addition, autonomous AI Agent systems are being designed to perform actions across different enterprise applications.

However, these developments bring an important security question with them:

Is the security of AI systems being validated with realistic attack scenarios?

An AI application answering normal user requests correctly does not mean that it is secure.

The model may violate security policies when faced with certain inputs.

It may be influenced by malicious instructions in external documents.

It may try to access data it is not authorised to see.

It may include sensitive information in its output.

It may call the tools it is connected to with incorrect parameters.

Or it may initiate actions on behalf of a user that it should not perform.

Some of these risks can be detected with traditional application security testing.

However, the probabilistic behaviour of AI systems, the fact that they are steered through natural language, and their interaction with external data sources call for additional testing methods.

AI Red Teaming and LLM Pentest are security assessment approaches that meet this need.

In this article, we will discuss how AI systems should be tested, which attack surfaces should be assessed, and how the results can be turned into enterprise security improvements.

1. What Is AI Red Teaming?

AI Red Teaming is a controlled testing approach that aims to assess the security, reliability and misuse risks of AI systems through realistic attacker behaviour.

These engagements do not only look for known vulnerabilities.

How the system behaves under different conditions is also examined.

For example, an AI Red Team engagement may assess the following questions:

Can the model maintain its security boundaries when faced with manipulative inputs?

Does it interpret instructions in external data sources as trusted commands?

Can unauthorised information be accessed?

Can the AI Agent perform a critical action without user approval?

Do the model's outputs create security problems in other systems?

Do security controls work consistently against different attack variations?

AI Red Teaming aims to answer these questions through controlled and measurable tests.

AI Red Teaming is not just Prompt Injection testing

Prompt Injection is an important attack category.

However, a comprehensive AI Red Team engagement can also assess data security, model security, authorisation, integrations, tool calls and operational controls.

For this reason, AI Red Teaming should be approached as a risk-based engagement that covers the entire AI application.

2. What Is LLM Pentest?

LLM Pentest is a penetration testing approach that assesses the vulnerabilities and misuse risks of applications that use large language models through controlled test scenarios.

Within the scope of an LLM Pentest, both traditional application security and AI-specific security risks can be examined.

Components that may fall within the test scope

LLM API integration.

User interface.

Prompt processing mechanism.

System Prompt and application instructions.

RAG infrastructure.

Vector Database.

Document upload processes.

AI Agent tools.

MCP integrations.

Authentication.

Authorisation.

Data protection controls.

Systems that process model outputs.

If these components are not assessed together, the real security risk of the application may be only partially understood.

3. The Difference Between AI Red Teaming and LLM Pentest

The two approaches are related but not entirely the same.

Characteristic	LLM Pentest	AI Red Teaming
Main objective	Identify vulnerabilities	Measure resilience against realistic attack objectives
Approach	Defined test scope and controls	Threat-driven scenarios
Scope	The LLM application and its integrations	Model, application, processes and operations
Outcome	Security findings and remediation recommendations	Attack paths, security impacts and defence assessment
Test method	Manual and automated tests	Scenario-based controlled attack simulations

In practice, the scopes of these engagements may overlap.

For example, a comprehensive LLM Pentest may include AI Red Teaming scenarios.

Similarly, traditional penetration testing techniques may be used during AI Red Teaming.

What matters more than the name of the engagement is that the objectives, authorisations, test scope and success criteria are clearly defined.

4. Is Traditional Pentesting Sufficient for AI Security?

Traditional pentesting can detect important vulnerabilities in AI applications.

For example:

SQL Injection.

Broken Access Control.

Authentication problems.

API authorisation errors.

SSRF.

Insecure file upload.

Misconfigured cloud services.

However, some AI risks may not be adequately addressed by traditional testing methodologies.

For example:

Prompt Injection.

Jailbreak.

RAG document poisoning.

AI Agent privilege escalation.

Tool Calling manipulation.

Model Extraction.

Model Inversion.

Adversarial Model Attacks.

For this reason, for enterprise AI systems, traditional application security testing needs to be supported by AI-specific testing.

An AI application is a software system; however, its security assessment should not be limited to traditional software vulnerabilities.

5. How Is the Scope of AI Security Testing Determined?

A successful security test depends first of all on defining the right scope.

This is because AI systems can run on different architectures.

Some applications only use an external LLM API.

Some run a model in-house.

Some access enterprise documents through a RAG architecture.

Others perform actions across different systems through an AI Agent.

Scoping questions

Which AI model is being used?

Is the model running inside the organisation?

Is there an external service provider?

Is there a RAG infrastructure?

Which data sources are accessed?

Is an AI Agent being used?

Are there Tool Calling capabilities?

Are MCP servers being used?

Which user roles exist?

Which sensitive data is processed?

Do model outputs trigger automated actions?

These questions help in building the test plan.

6. What Is AI Threat Modeling?

AI Threat Modeling is the systematic assessment of an AI system's attack surfaces, critical assets and trust boundaries.

This work can be carried out before testing.

Critical assets

User data.

System Prompt.

Model files.

Training data.

API keys.

RAG documents.

Vector Database.

AI Agent privileges.

MCP integrations.

Enterprise application connections.

Trust boundaries

The boundary between the user and the application.

The boundary between the application and the model.

The boundary between the model and the tools.

The boundary between the RAG data source and the model.

The boundary between the Agent and enterprise systems.

Identifying these boundaries makes it possible to build more realistic attack scenarios.

7. Prompt Injection Security Testing

Prompt Injection is an attempt to manipulate model behaviour through instructions contained in the content that the AI system processes.

It is one of the key test areas in LLM Pentest engagements.

Test objectives

Does the model interpret untrusted content as instructions?

Can user inputs influence system policies?

Can external documents change the model's behaviour?

Can tool outputs cross security boundaries?

Does Prompt Injection result in an unauthorised action?

Testing approach

First, the model's normal behaviour is established.

Then controlled manipulative content is prepared.

How the system handles this content is examined.

Successful and unsuccessful scenarios are recorded.

Findings are assessed according to their real impact on the system.

An important distinction

The model producing an unexpected sentence does not always mean a critical vulnerability.

The security impact should be determined by which data the system discloses or which actions it performs.

8. Direct Prompt Injection Tests

Direct Prompt Injection is when an attacker tries to influence model behaviour directly through user input.

For example, a user may send requests aimed at making the application behave outside its defined function.

Test areas

Instruction priority.

Application policies.

Role boundaries.

Sensitive information disclosure.

Tool usage privileges.

Output control.

Test evaluation

What should be examined is not only whether the model answers the request, but whether the application's security boundaries are maintained.

For example, unauthorised data access should be prevented by backend authorisation controls, not by the model's response.

9. Indirect Prompt Injection Tests

Indirect Prompt Injection is an attempt to influence the model's behaviour through the external content it processes.

This type of attack is particularly important in RAG and AI Agent systems.

Test sources

Web pages.

PDF documents.

Emails.

Database records.

Search results.

Tool outputs.

RAG content.

Example scenario

An organisation's document analysis assistant summarises PDF files.

Within the test scope, an instruction unrelated to summarising the document is placed inside a controlled PDF.

Then it is examined whether the system treats this instruction as data or as a command to be executed.

Security objective

Untrusted document content is expected not to alter the application's instruction hierarchy or its action privileges.

10. Jailbreak Security Tests

Jailbreak refers to manipulation attempts aimed at bypassing an AI model's behavioural limits or safety restrictions.

Jailbreak tests can assess the model's resilience against certain policy boundaries.

Test areas

Role-switching attempts.

Multi-stage steering.

Context manipulation.

Indirect instructions.

Encoded or transformed content.

Long conversation context.

Evaluation

The outcome of the tests should not only measure whether the model refused.

The application's function, data access and tool privileges must also be taken into account.

In critical enterprise systems in particular, security should not be left solely to the model's behavioural constraints.

11. System Prompt Disclosure Tests

The System Prompt may contain the high-level instructions the application uses to steer model behaviour.

In some applications, disclosure of the system instructions may be undesirable.

However, the System Prompt is not, on its own, a secure mechanism for keeping secrets.

Test objectives

Are the system instructions disclosed in unexpected ways?

Does the prompt contain sensitive information?

Have API keys or credentials been embedded in the prompt?

Does disclosure of the system instructions lead to other security impacts?

Security approach

Access keys or passwords should not be placed in the System Prompt.

Authorisation controls should be enforced on the backend.

Critical security decisions should not be left to the model's instructions.

For this reason, System Prompt Disclosure findings should be assessed together with their real security impact.

12. Sensitive Data Disclosure Tests

The risk of sensitive data disclosure is one of the key test areas in AI applications.

Possible data types

Personal data.

Customer information.

Financial data.

Source code.

Trade secrets.

API keys.

Enterprise documents.

Testing approach

Controlled and synthetic data can be used in the test environment.

Different user roles are created.

Data access boundaries are defined.

It is assessed whether the model discloses unauthorised data.

Key point

Unnecessarily transferring real personal data into the test environment should be avoided.

The tests themselves must also be conducted in line with data security obligations.

13. What Is RAG Security Testing?

RAG is the abbreviation of the Retrieval-Augmented Generation approach.

It enables the model to generate answers using content retrieved from external knowledge sources.

The RAG architecture is useful for access to enterprise knowledge.

However, it creates new attack surfaces.

RAG test areas

Document access control.

Retrieval authorisation.

Vector Database security.

Document poisoning.

Metadata manipulation.

Source verification.

Sensitive data leakage.

Indirect Prompt Injection.

Critical security principle

The fact that a document can be found in a retrieval result does not mean the user is authorised to access that document.

Authorisation should be enforced in trusted application layers that are independent of the model.

14. RAG Document Poisoning Tests

RAG systems can ingest documents from different data sources.

The reliability of these documents may vary.

If content controlled by an attacker is included in the retrieval process, attempts may be made to influence the model's behaviour.

Test scenario

A controlled document is added to a data source.

The document contains instructions unrelated to the application's normal task.

How the system retrieves and interprets the document is examined.

Measurable outcomes

Was the document included in the retrieval result?

Did the model follow the instruction in the document?

Did sensitive data disclosure occur?

Did an unauthorised tool call take place?

Did a security control detect the event?

This approach ensures that RAG security is assessed not only through the model's answer, but together with the data and action flow.

15. Vector Database Security Testing

A Vector Database can be used to store embedding data and perform similarity search.

However, vector database security is also important in enterprise systems.

Test areas

Authentication.

Authorization.

Tenant isolation.

Collection access.

Metadata filters.

API security.

Data deletion processes.

Network access.

Example risk

While a user should only have access to documents belonging to their own department, content from another department may be included in the retrieval result.

In this case, even if the model tries to answer in line with the security policy, the data access boundary may already have been violated.

For this reason, authorisation controls at the retrieval stage must be tested separately.

16. What Is AI Agent Security Testing?

AI Agent systems may not be limited to generating answers.

They can call tools.

They can read files.

They can use APIs.

They can start workflows.

They can perform actions in enterprise systems.

For this reason, AI Agent security tests go beyond model behaviour.

Test objectives

Which tools can the Agent access?

On behalf of which user does it act?

Are its privileges properly restricted?

Do critical actions require approval?

Can external content influence action decisions?

Does the Agent perform unnecessary data transfers?

Are action logs kept?

These tests are particularly important for systems that perform actions autonomously.

17. Tool Calling Security Testing

Tool Calling allows the model to perform certain actions by calling defined tools.

For example, an AI Agent can retrieve customer information from a CRM system.

It can create a calendar entry.

It can open a support ticket.

Or it can prepare specific reports.

Security risks

Unauthorised tool calls.

Excessive privileges.

Parameter manipulation.

Insecure API integration.

Sensitive data transfer.

Bypassing of critical action approval.

Testing approach

A privilege matrix is created for the tools.

Different user roles are defined.

Critical action scenarios are identified.

The model's tool calls are monitored.

Backend authorisation controls are verified.

Core principle

The model wanting to call a tool should not mean it has sufficient authority for that action to be carried out.

18. MCP Security Testing

MCP is the abbreviation of Model Context Protocol.

It is a protocol that enables AI applications to interact with tools and data sources in a standardised way.

MCP integrations can create different security risks.

Test areas

MCP server security.

Tool authorisation.

Server identity.

Tool metadata trustworthiness.

Untrusted tool outputs.

Sensitive data access.

Third-party integrations.

Key point

Using MCP does not automatically mean that secure authentication or authorisation is in place.

The real security level depends on the architecture used, the protocol features and the application controls.

For this reason, MCP servers and clients should be assessed separately.

19. Excessive Agency Tests

Excessive Agency is a security problem related to an AI agent having broader action privileges than necessary.

For example, if a reporting assistant only needs to read data but has the privilege to delete records, this creates unnecessary risk.

Testing approach

All of the Agent's tools are inventoried.

The required privilege level is determined for each tool.

It is compared against user roles.

Approval mechanisms for critical actions are assessed.

Unauthorised action attempts are tested in a controlled manner.

Protection approach

The principle of least privilege should be applied.

Read and write privileges should be separated.

Critical actions should require independent approval.

Backend policies should be enforced.

20. AI Agent Privilege Escalation Scenario

Let us consider an organisation that uses an AI sales assistant connected to its CRM system.

The assistant's task is to summarise customer records and offer recommendations to the sales representative.

However, the system also technically has the privilege to modify customer records.

Test questions

Can the Agent use this privilege?

Is the user's authority checked?

Can external content influence the Agent's action decisions?

Do critical actions require approval?

Are unauthorised changes prevented?

Expected security approach

The Agent should have only the privileges required for its function.

Critical changes should be checked in a trusted application layer.

Actions that require user approval should be verified independently.

This scenario shows why relying on the model to behave correctly is not sufficient on its own.

21. Model Security Testing

In some projects, AI Red Teaming engagements can also cover attacks at the model level.

Test areas

Data Poisoning.

Adversarial Attacks.

Model Extraction.

Model Inversion.

Membership Inference.

Backdoor behaviours.

Model integrity.

Scope distinction

Not every LLM Pentest engagement needs to cover all model security tests.

For example, an organisation that only uses a third-party LLM API may not have access to the model weights or training data.

In this case, the test scope may be limited to the application and integration level.

In organisations that develop models in-house, more comprehensive model security assessments can be carried out.

22. Model Extraction Security Tests

Model Extraction is an attack approach that aims to extract the model's behaviour or certain characteristics through queries.

Test objectives

Can the model API be queried without control?

Are usage quotas in place?

Is rate limiting applied?

Is anomalous query behaviour detected?

Is unnecessary model information disclosed?

Evaluation

When assessing Model Extraction risk, the number of test queries and resource consumption should be limited.

The service continuity of the production environment must be protected.

Tests should be carried out within pre-defined authorisation and scope.

23. Adversarial Model Testing

Adversarial Model Testing is the assessment of a model's resilience against manipulative inputs.

These tests may differ depending on the model type.

Example areas

Image classification.

Anomaly detection.

Fraud detection.

Content classification.

Large language models.

Evaluation criteria

The model's normal performance.

Its behaviour against manipulative inputs.

False positive rate.

False negative rate.

The effect of security controls.

The potential impact on the real business process.

These tests should be designed according to the model's intended use.

24. AI Red Teaming Methodology

A structured methodology should be used in enterprise AI Red Teaming engagements.

Phase 1: Scope and authorisation

The systems to be tested are identified.

Written authorisation is obtained.

Test boundaries are defined.

Production environment constraints are determined.

Phase 2: Architecture analysis

Model integrations are examined.

Data flows are identified.

Tool and service connections are assessed.

Phase 3: Threat modelling

Critical assets are identified.

Trust boundaries are defined.

Potential attacker capabilities are assessed.

Phase 4: Test scenarios

Prompt Injection.

Jailbreak.

RAG security.

Agent authorisation.

Data leakage.

API security.

Model security.

Scenarios are prepared in areas such as these.

Phase 5: Controlled testing

Tests are carried out within pre-determined boundaries.

System behaviour is observed.

Evidence is recorded.

Phase 6: Risk assessment

The technical impact and business impact of the findings are assessed.

Phase 7: Reporting

Security findings, evidence and remediation recommendations are presented.

Phase 8: Retest

After fixes are applied, the security controls are verified again.

25. Test Matrix for AI Red Teaming

Test area	Example security objective
Prompt Injection	Maintaining instruction boundaries
Jailbreak	Resilience of behavioural policies
Sensitive Data Disclosure	Preventing unauthorised data disclosure
RAG Security	Document access control
Vector Database	Tenant and collection isolation
AI Agent	Restricting action privileges
Tool Calling	Authorisation in tool calls
MCP Security	Server and tool security
Model Extraction	Reducing uncontrolled model querying
API Security	Authentication and authorisation
Output Handling	Secure processing of model outputs
Monitoring	Monitoring of suspicious behaviour

This matrix is illustrative.

The actual test scope should be built according to the application's architecture and risk level.

26. Automated LLM Security Testing

Automation can be used in LLM security testing.

For example, specific prompt variations can be tested systematically.

Model responses can be compared.

The behaviour of security controls against different inputs can be measured.

Advantages of automation

Repeatable tests.

Broad scenario coverage.

Regression testing.

Comparison between model versions.

Validation of security policy changes.

Limitations

Automated tests may not always assess the real security impact correctly.

Model responses can be probabilistic.

False positive results may occur.

Some security problems require business process context.

For this reason, automation should be supported by manual analysis.

27. Why Is Manual AI Red Teaming Important?

Manual tests can help assess the system's business logic and enterprise usage scenarios in more detail.

For example, an automated test can check whether the model produces a specific phrase.

However, an experienced security specialist can also assess the following questions:

Does the model's response constitute a real authorisation violation?

Did the tool call actually take place?

On behalf of which user was the action performed?

Did the backend control come into play?

Was the data really transferred to an unauthorised person?

Was the security event logged?

For this reason, manual tests are particularly important in complex RAG and AI Agent systems.

28. AI Red Teaming and OWASP

OWASP offers open-source guides and projects that are widely used in the field of application security.

The OWASP LLM Top 10 work for large language model applications can be used in assessing AI security risks.

Within this scope:

Prompt Injection.

Sensitive data disclosure.

Supply chain risks.

Insecure output handling.

Excessive agency.

topics such as these can be assessed.

However, OWASP categories alone are not a complete testing methodology.

They should be extended by taking the organisation's architecture and business processes into account.

29. AI Red Teaming and MITRE ATLAS

MITRE ATLAS is a knowledge base that can be used to assess adversarial tactics and techniques targeting AI systems.

This approach can help structure AI Red Teaming scenarios.

Use cases

Threat modelling.

Attack scenario creation.

Security control mapping.

Test scope analysis.

Defence assessment.

MITRE ATLAS can provide a threat-driven perspective in AI security work.

However, actual test scenarios should be adapted to the organisation's application architecture.

30. AI Red Teaming and NIST AI RMF

The NIST AI Risk Management Framework provides a framework for managing AI risks.

AI Red Teaming can support these risk management processes in terms of technical validation.

For example, the organisation may have defined certain AI risks.

An AI Red Teaming engagement can assess the impact of these risks on the actual system.

An important distinction

Establishing a risk management framework and carrying out technical security testing are different activities.

When the two are used together, a more comprehensive security approach can be created.

31. How Are AI Red Teaming Findings Rated?

Rating AI security findings should be done carefully.

This is because not every unexpected model response has the same security impact.

Assessment criteria

Prerequisites of the attack.

The attacker's access level.

Repeatability.

Affected data types.

Whether an unauthorised action occurred.

Business impact.

Existing security controls.

Example risk levels

Critical: Unauthorised users being able to perform high-impact actions in critical systems.

High: Unauthorised access to sensitive data or a significant escalation of action privileges.

Medium: Weakening of security boundaries under certain conditions, or limited data disclosure.

Low: Configuration or behaviour problems with limited direct security impact.

This classification is illustrative.

The actual risk level should be determined according to the organisation's system and business impact.

32. How Should an AI Red Teaming Report Be Prepared?

A professional AI Red Teaming report should not consist only of successful prompt examples.

Technical evidence and business impact should be presented together.

Recommended report structure

Executive summary

The purpose, scope and main results of the test.

Architecture assessment

The model and integration components that were tested.

Methodology

The testing approach and reference frameworks used.

Findings

Technical descriptions of the security problems.

Evidence

Test inputs, system responses and related action logs.

Risk assessment

Likelihood, impact and relationship to the business process.

Remediation recommendations

Technical and administrative control recommendations.

Retest results

Verification of the fixes.

This approach ensures the report is usable not only for technical teams, but also for management and risk teams.

33. What Should Be Done After AI Red Teaming?

Once the security test is complete, the findings need to be closed.

However, the remediation approach for AI systems can differ from that for traditional software systems.

For example, a Prompt Injection problem may not be permanently resolved just by changing the System Prompt.

Similarly, the model refusing a particular response does not eliminate a backend authorisation problem.

Improvement areas

Backend authorization.

Tool permission management.

RAG access control.

Data masking.

Input validation.

Output validation.

Human approval.

AI Gateway policies.

Logging and monitoring.

Retest

After fixes are applied, the same scenarios should be tested again.

It should also be assessed whether the changes have created new security problems.

34. How Is an AI Gateway Assessed in Security Testing?

An AI Gateway can be used as a central control layer in the communication between AI applications and model services.

However, the presence of a security gateway does not mean that all risks have been eliminated.

Test areas

Sensitive data detection.

Data masking.

Prompt Injection controls.

Jailbreak controls.

Model access policies.

Tool Calling controls.

Approval mechanisms.

Audit logging.

Example assessment

Test inputs containing specific sensitive data can be prepared.

The system's masking or blocking policies can be verified.

The consistency of the security control can be measured with different prompt variations.

It can be examined whether the approval mechanism comes into play for critical tool calls.

These tests make it possible to assess the effectiveness of AI Gateway controls in the actual application flow.

35. Example Scenario: Enterprise Document Assistant

Let us consider an organisation that uses a RAG-based document assistant.

Employees query company policies and procedures through this system.

Test scenario 1: Unauthorised document access

Which documents different user roles can access is assessed.

Test scenario 2: Indirect Prompt Injection

It is examined whether the model's behaviour can be influenced through a controlled document.

Test scenario 3: Sensitive data disclosure

It is assessed whether the model presents unauthorised data to the user.

Test scenario 4: Retrieval security

It is tested whether metadata filters and access controls are applied correctly.

Test scenario 5: Audit logging

It is assessed whether suspicious actions are recorded appropriately.

This example shows that RAG security tests should cover the data access architecture as well, not just the model's response.

36. Example Scenario: SOC AI Agent

Let us consider an organisation that uses an AI agent in its SOC operations.

The Agent analyses security incidents.

It queries SIEM records.

It evaluates threat intelligence data.

Under certain conditions, it generates recommendations for security operations.

Test scenario 1: Privilege boundaries

Which systems the Agent can access is assessed.

Test scenario 2: Tool Calling

It is examined whether tool calls pass through the correct authorisation controls.

Test scenario 3: External data manipulation

It is tested whether untrusted content in threat intelligence or incident descriptions influences the Agent's behaviour.

Test scenario 4: Critical action approval

It is assessed whether the Agent can perform high-impact actions such as isolation or blocking without independent approval.

Test scenario 5: Audit records

The traceability of the Agent's decision and action flows is examined.

This scenario shows that AI Agent security is particularly important in systems that can have an operational impact.

37. SECURESYS's Approach to AI Red Teaming and LLM Pentest

At SECURESYS, we believe that AI security cannot be assessed merely by checking a model's responses.

This is because modern AI systems work together with data sources, API services, enterprise applications and autonomous agents.

For this reason, the security assessment must cover the system's entire architecture.

Depending on the project scope, the following areas can be assessed:

LLM Penetration Testing

Security testing of large language model applications.

Prompt Injection Assessment

Assessment of direct and indirect prompt manipulation risks.

Jailbreak Resistance Testing

Examination of the model's behavioural boundaries through controlled tests.

RAG Security Testing

Assessment of retrieval processes, document security and access controls.

AI Agent Security Testing

Examination of autonomous agents' action privileges and security boundaries.

MCP Security Assessment

Security assessment of MCP servers, tools and integrations.

AI Model Security Testing

Assessment of model confidentiality, integrity and adversarial resilience.

AI Gateway Security Validation

Testing the effectiveness of centralised AI security controls.

AI Red Teaming

Assessment of the security resilience of AI systems through realistic attack scenarios.

The main purpose of this approach is not merely to identify vulnerabilities.

It is to help organisations understand under which conditions their AI systems may create risk, and to improve their security controls in a measurable way.

What matters in AI security is testing not only what a system is able to do, but also under which conditions it can perform actions it should not.

38. AI Red Teaming Checklist

Architecture and scope

• Has the AI architecture been identified?
• Have the model integrations been examined?
• Have the data sources been inventoried?
• Have the AI Agent and Tool Calling capabilities been identified?

Prompt security

• Has Direct Prompt Injection been tested?
• Has Indirect Prompt Injection been assessed?
• Have Jailbreak scenarios been applied?
• Does the System Prompt contain sensitive information?

Data security

• Have sensitive data disclosure tests been performed?
• Have user roles been assessed?
• Have RAG access controls been verified?
• Have data masking policies been tested?

Agent security

• Have tool privileges been restricted?
• Have MCP integrations been assessed?
• Do critical actions require approval?
• Are Agent action logs kept?

Model security

• Has Model Extraction risk been assessed?
• Has the need for adversarial testing been determined?
• Has model artifact security been examined?
• Has model API security been tested?

Operational security

• Are security events being monitored?
• Have the test results been reported?
• Has an action plan been created for the findings?
• Has a retest been carried out?

This checklist can be extended according to the organisation's AI use cases.

Frequently Asked Questions

What is AI Red Teaming?

AI Red Teaming is the assessment of AI systems against realistic attack scenarios in terms of security and resilience.

What is LLM Pentest?

LLM Pentest is the assessment of the vulnerabilities and misuse risks of applications that use large language models through controlled tests.

What is the difference between AI Red Teaming and LLM Pentest?

LLM Pentest generally focuses on defined application security tests, whereas AI Red Teaming can cover broader threat and attack scenarios.

Is traditional pentesting sufficient for AI?

On its own, it is not always sufficient. AI-specific Prompt Injection, RAG, Agent and model security tests may also be required.

What is a Prompt Injection test?

It is the assessment of whether the model is influenced by manipulative instructions in user input or external content.

What is a Jailbreak test?

It is the assessment of the model's resilience against manipulation attempts aimed at bypassing its behavioural limits.

What is RAG Security Testing?

It is the testing of document access, retrieval, data security and Prompt Injection risks in a RAG architecture.

What is AI Agent Security Testing?

It is the assessment of AI agents' tool privileges, data access and critical action controls.

What is MCP Security Testing?

It is the assessment of MCP servers, tools and integrations from a security perspective.

What is a Model Extraction test?

It is the assessment of the existing security controls against extraction of the model's behaviour through uncontrolled queries.

Can AI Red Teaming be carried out in a production environment?

With appropriate authorisation, scope, security boundaries and operational safeguards, some tests can be carried out in a production environment. For high-risk scenarios, an isolated test environment should be preferred.

Can AI Red Teaming be automated?

Some tests can be automated. However, manual analysis is important when assessing complex security impacts.

How often should AI Red Teaming be carried out?

The frequency should be determined according to the organisation's risk level and system changes. Reassessment after significant model, integration or privilege changes is beneficial.

What should an AI Red Teaming report contain?

It should contain the scope, methodology, technical findings, evidence, risk ratings, remediation recommendations and, where available, retest results.

Does using an AI Gateway remove the need for LLM Pentest?

No. An AI Gateway can provide certain security controls, but application, data access and Agent authorisation risks still need to be tested separately.

Conclusion: AI Security Must Be Validated by Testing, Not by Assumptions

AI systems are being integrated into more and more enterprise business processes.

Models do not just generate answers.

They access documents.

They query databases.

They communicate with enterprise applications.

They call tools.

And in some cases they perform actions on behalf of users.

These developments increase the security impact of AI systems.

A model working correctly under normal conditions does not mean it will behave securely under attack conditions.

Prompt Injection attacks can influence model behaviour.

RAG systems can create unauthorised data access.

AI Agents can have broader privileges than necessary.

MCP integrations can create new trust boundaries.

Model APIs can be exposed to uncontrolled use.

For this reason, AI security should not be assessed solely on the basis of policy documents or the model provider's security statements.

Controlled security tests should be carried out on the real system.

AI Red Teaming and LLM Pentest engagements help organisations understand these risks.

However, for the tests to be effective, the scope must be defined correctly.

Architecture analysis must be performed.

A threat model must be built.

Realistic scenarios must be prepared.

Technical evidence must be collected.

Risks must be assessed together with their business impact.

And the remediation of findings must be verified with a retest.

Secure AI is not merely a system that is believed to be secure; it is a system whose security boundaries have been validated with realistic attack scenarios.