# What Is LLM Security? Cyber Attacks on Large Language Models and How to Defend Against Them

**URL:** https://securesys.com.tr/en/learning/software-and-ai-security/what-is-llm-security-cyber-security-risks-in-large-language-models

![What Is LLM Security? Cyber Attacks on Large Language Models and How to Defend Against Them](/images/bilgi-merkezi/covers/cover-yzguv-03.webp)

### Introduction: As AI Models Grow More Powerful, Security Responsibility Grows Too

Imagine an organisation that uses a large language model for customer service.

At the start, the system's job is quite simple.

Answering customers' frequently asked questions.

Explaining product information.

Classifying support requests.

Over time, however, new features are added.

First it is connected to the customer database.

Then it begins to query order information.

Later it becomes able to update support records.

In the final stage, it can carry out certain customer transactions automatically.

At this point, the AI application is no longer just a system that generates text.

It is a software component that reaches corporate information sources and affects real business processes.

Consequently, the security assessment must change as well.

In a traditional application, the developer largely defines in advance which function receives the user input and which operation runs.

In systems that use large language models, however, requests that users give in natural language are interpreted, and the outputs the model produces can influence application behaviour.

This creates new security risks.

For example, an attacker may try to bypass the model's security instructions.

They may plant malicious instructions inside an untrusted document.

They may ask the model to disclose sensitive information.

They may aim for the model's output to be processed insecurely by another system.

Or they may try to perform unauthorised operations by using the tools the model is connected to.

All of these risks fall within the scope of LLM Security, that is, Large Language Model Security.

There is an important point here, however:

**LLM security is not only about preventing the model from producing harmful answers. It is about protecting the security of the application, data and transaction systems that the model is connected to.**

This article examines the core concepts of LLM security, the attack surfaces of large language models, the critical risks and the protection methods organisations should apply.

### \1. What Is an LLM?

LLM is the abbreviation of Large Language Model.

In Turkish it is called Büyük Dil Modeli.

Large language models are AI models trained on large datasets that can perform a variety of natural language tasks.

These models can:

- Generate text.
- Answer questions.
- Summarise documents.
- Write source code.
- Classify text.
- Extract information.
- Perform translation.
- Produce structured output.
- Work integrated with applications.

LLM technologies are not used only in chat applications.

Today they are also used in software development, customer service, legal, finance, human resources, knowledge management and operations.

However, a model's ability to perform a task does not guarantee that it performs it securely.

For example, a model may summarise a customer's order information correctly.

But without the necessary authorisation checks, it may also disclose another customer's order information.

In this case the problem stems not from the model's language ability but from the application's security architecture.

### \2. What Is LLM Security?

LLM security is the set of technical and organisational controls applied to protect large language models, and the applications that use them, against security threats.

In this scope, not only the model itself but the entire system in which the model operates is assessed.

For example:

**User → Application → LLM → Data Source → Tools → Output**

In an architecture like this, security risks can arise at different points.

The user may send a malicious instruction.

The application may implement authorisation checks incompletely.

The model may interpret untrusted content as an instruction.

The data source may expose sensitive information to an unauthorised user.

The tool integration may run with more privileges than necessary.

The model output may trigger an insecure operation in another system.

Therefore, LLM security cannot be achieved with a single security filter.

**A secure LLM application is the result of the security architecture built around the model.**

### \3. The Difference Between LLM Security and Traditional Application Security

In traditional application security, user inputs are processed within the framework of specific functions and programming rules.

For example, in a web application, a customer_id parameter received from the user may be used in a database query.

Security controls include validating the parameter, building the query securely and checking the user's authorisation to access the record.

In LLM applications, however, natural language inputs can also influence the application's behaviour.

For example, the user may give the following instruction:

“Prepare the sales report for the last three months.”

The model may interpret this request and try to determine which data is needed.

If tool use is enabled, it may suggest the relevant API calls.

This raises new questions from a security standpoint.

Which data may the model request?

Which reports is the user authorised to access?

Is the tool call suggested by the model trustworthy?

Can the documents the model reads manipulate its behaviour?

For this reason, traditional application security and LLM security must be applied together.

### \4. What Is a Trust Boundary in LLM Applications?

A trust boundary is the point at which components with different trust levels are separated from one another.

For example, user input coming from the internet is treated as untrusted data.

By contrast, the application's server-side authorisation policy has a higher trust level.

In LLM systems, trust boundaries can become more complex.

This is because the model can process both trusted instructions and untrusted content within the same context.

For example, the system instruction of a corporate assistant might be:

“Summarise only the corporate documents the user is authorised to access.”

However, a document the model reads may contain a different instruction.

That instruction is part of the document's content.

It is not the actual system policy.

It should not be assumed that the model will make this distinction correctly in every case.

#### Protecting trust boundaries

In a secure architecture:

- System policies should be backed up at the application layer.
- User permissions should be verified by the backend.
- External documents should be treated as untrusted content.
- Model outputs should not directly trigger insecure operations.
- Critical tool calls should pass through independent controls.

These principles form the foundation of LLM application security.

### \5. What Are the System Prompt, Developer Prompt and User Prompt?

In LLM applications, instructions can be provided to the model from different sources.

Depending on the model and API used, these instructions can be represented by different roles.

#### System Prompt

The System Prompt refers to the high-level instructions used to steer the model's general behaviour.

For example, it can define which task the model will be used for or which behavioural limits are expected.

#### Developer Prompt

The Developer Prompt can refer to the task and behaviour instructions that the party developing the application provides to the model.

The availability and priority of this role depend on the model provider.

#### User Prompt

The User Prompt is the instruction or question that the end user sends to the model.

For example:

“Summarise this document.”

“Explain this code.”

“Create a sales report.”

Requests like these constitute user input.

#### An important distinction from a security perspective

The existence of these roles does not mean the application is automatically secure.

How the model applies instruction priorities depends on the model used.

In addition, untrusted documents or tool outputs should not be treated as high-priority instructions.

**Prompt roles help steer behaviour; however, they do not replace access control and authorisation mechanisms.**

### \6. Prompt Injection: One of the Core Threats to LLM Security

Prompt Injection is an attack approach that aims to have untrusted content used as an instruction in order to change the model's behaviour.

These attacks are among the most important topics in LLM security.

For example, an AI assistant summarises product documents for a user.

An attacker can plant a different instruction in a document the assistant will read.

If the model interprets this content as an instruction instead of treating it as data to be summarised, a security problem can arise.

#### Why does Prompt Injection matter?

Because large language models are designed to interpret natural language instructions.

However, hard technical boundaries do not always separate trusted instructions from data to be processed within natural language content.

For this reason, writing a security rule into the prompt alone is not sufficient.

The application architecture must also limit what the model can do.

Prompt Injection attacks, direct and indirect methods, example scenarios and protection techniques are covered in chapter four of our series.

### \7. Jailbreak Attacks

Jailbreak refers to attempts to bypass the behavioural or safety restrictions of a large language model.

These attacks can be carried out with different methods.

For example, an attacker may try to make the model ignore certain restrictions.

However, an important distinction must be made in the security assessment.

An unexpected model answer is not the same as an unauthorised operation in the real system.

For example, the model may claim that it can generate a report the user has no access permission for.

But if backend authorisation is correct, access to the report data is still blocked.

Therefore, Jailbreak tests must assess the real security impact on the application, not only the model's answers.

### \8. What Is LLM Data Leakage?

LLM data leakage is the AI application disclosing or transferring sensitive or unauthorised information to inappropriate persons or systems.

This risk can arise from different sources.

For example:

- Personal data belonging to the user.
- Corporate financial information.
- Customer records.
- Source code.
- API keys.
- System configurations.
- Trade secrets.
- Confidential documents.

In an LLM application, data leakage does not appear only in the model's response text.

A security problem can also arise when data is sent to an external model service.

For example, an in-house application may send more customer data than necessary to an external LLM API to fulfil a request.

In this case, it is not enough that the model avoids disclosing sensitive information to the user.

Where the data is sent and how it is processed must also be assessed.

### \9. Sensitive Information Disclosure in Model Output

An LLM application can access corporate information sources.

For example, a customer support system can query customer records.

However, if the application does not apply user permissions correctly, the model may disclose information belonging to other customers.

#### Example scenario

A customer support assistant is connected to an order lookup API.

The user asks the following question:

“Show the delivery address of order number 12345.”

If the application fetches data solely by order number without checking that the user owns the order, another customer's address may be disclosed.

This is not purely an LLM behaviour problem.

At its core, it is a lack of object-level authorisation.

#### Secure approach

The user's identity must be verified.

It must be checked that the order belongs to the relevant user.

The scope of data the model can access must be limited.

Unnecessary sensitive fields must not be sent to the model.

This approach shows that data security must be ensured at the application level, not at the prompt level.

### \10. What Is System Prompt Leakage?

System Prompt Leakage refers to the risk of the model disclosing some parts of its system or developer instructions without authorisation.

Some organisations put detailed instructions defining application behaviour inside the system prompt.

For example:

- The role of the application.
- Workflows.
- Response format.
- Usage policies.
- Tool-calling rules.

However, passwords, API keys or other sensitive access credentials must not be kept inside the system prompt.

This is because there is no guarantee that the model will never disclose the prompt content.

#### Secure approach

The system prompt must not be used in place of a secure secrets management system.

Sensitive access credentials must be protected at the application layer.

Authorisation decisions must not be made through the prompt.

Security-critical information in the system prompt should be minimised.

### \11. Why Should LLM Outputs Be Treated as Untrusted Data?

A key security principle in an LLM application is that model outputs are not automatically trusted.

The model can generate text.

It can create JSON.

It can suggest a SQL query.

It can prepare HTML content.

It can produce a terminal command.

However, these outputs must not be executed without security validation.

For example, inserting model-produced HTML directly into a web page can create an XSS risk when output encoding and content security controls are absent.

Similarly, directly executing a model-generated SQL query can create a risk of unnecessary data access or modification.

#### Secure output handling

Model outputs must be validated for the context in which they will be used.

HTML outputs must be processed securely.

SQL operations must be restricted by permitted query types and data access policies.

Shell commands must not be executed directly.

Structured outputs must be validated against the expected schema.

Independent authorisation and approval mechanisms must be applied for critical operations.

### \12. Insecure Output Handling

Insecure Output Handling refers to the risks of the application using model output in an insecure manner.

This risk is closely related to traditional software security problems.

For example, an application can use LLM output to prepare an automatic email.

Another application can run a command generated by the model in a terminal.

Yet another system can perform a transaction with API parameters suggested by the model.

Each of these operations carries a different security level.

#### Example: SQL reporting assistant

Suppose an organisation has developed an AI system that prepares reports in natural language.

The user:

“Show the sales for the last three months.”

sends this request.

The model generates a SQL query.

If the application runs the query produced by the model on the production database without any checks, a serious risk can arise.

A more secure approach is to restrict permitted query types, grant the database account only necessary privileges and pass queries through independent security controls.

### \13. LLM Tool Calling Security

Tool Calling is the mechanism by which the model proposes calls to functions or tools defined by the application.

For example, an LLM can:

Query customer information.

Create a support ticket.

Prepare a calendar entry.

Call a reporting service.

However, the model proposing a tool call does not mean that the operation is authorised.

#### Secure Tool Calling principles

A separate permission must be defined for each tool.

The user's authority to perform the operation must be verified.

Tool parameters must be checked.

An approval mechanism must be applied for critical operations.

Tool calls must be logged.

Unnecessary tools must not be exposed to the model.

In particular, operations such as money transfers, data deletion, changing user permissions and infrastructure management must be considered high-risk.

### \14. What Is Excessive Agency?

Excessive Agency is the security risk of giving an AI system operational privileges beyond business need.

For example, a corporate assistant's job may be solely to prepare reports.

However, if the same assistant is granted the authority to delete or modify customer records, unnecessary risk is created.

This is especially important in AI Agent systems.

#### Example scenario

An employee gives an AI assistant the following instruction:

“Summarise this customer's most recent orders.”

is the instruction given.

The assistant accesses the order system.

However, the API account it uses has not only read permission but also the authority to cancel orders.

In this case, Prompt Injection or a model error, combined with unnecessary operational privileges, can lead to more serious consequences.

#### Protection methods

Least privilege must be applied.

Read and write operations must be separated.

Human approval should be considered for critical operations.

The access scope of tools must be limited.

Authorisation appropriate to the user context must be applied.

### \15. RAG Security in LLM Applications

RAG stands for the Retrieval-Augmented Generation approach.

In this architecture, the model produces answers using content retrieved from external information sources.

RAG systems can provide significant advantages in corporate knowledge management.

However, they involve different risks from a security standpoint.

For example:

Unauthorised document access.

Document poisoning.

Instructions coming from untrusted content.

Sensitive information leakage.

Tenant isolation problems.

#### The core principle of RAG security

The fact that the search system can find a document does not mean the user is authorised to access that document.

For this reason, document access policies must be enforced at the search and data retrieval stage.

It is not sufficient to expect the model to hide an unauthorised document through the prompt alone.

### \16. Vector Database and Embedding Risks

Vector Database systems can be used to store embedding data and perform similarity searches.

These systems are common in RAG architectures.

However, embedding data must also be assessed from a data security perspective.

In particular:

- Data access control.
- Tenant isolation.
- Data retention.
- Encryption.
- Audit logging.
- Data deletion.
- Search result authorisation.

and similar topics are important.

An organisation may keep documents belonging to different customers on the same infrastructure.

However, data isolation must be ensured between customers.

It is not right to leave this control solely to the model instruction.

### \17. LLM Denial of Service and Resource Consumption

LLM systems can consume resources differently from traditional web applications.

For example, large context windows, long outputs or heavy model calls can be costly to process.

An attacker may aim to exhaust the application's resources.

This can degrade service quality or increase costs.

#### Risk examples

A large number of concurrent model requests.

Excessively long inputs.

Unnecessarily repeated tool calls.

Uncontrolled token consumption.

Long-running agent workflows.

#### Protection methods

Rate limiting must be applied.

Per-user quotas must be set.

Token and cost limits must be defined.

The number of tool calls must be limited.

Request timeout policies must be established.

Abnormal usage behaviour must be monitored.

These controls are important for both security and operational sustainability.

### \18. The Difference Between Model Denial of Service and Traditional DDoS

In traditional DDoS attacks, the goal is mostly to exhaust network or application resources with heavy traffic.

In LLM systems, an attacker can exploit requests that increase the model's computational cost.

For example, very long contexts or repeated model operations can create high resource consumption.

This does not always look like a classic network-level DDoS.

Therefore, firewall or WAF controls alone may not be sufficient.

Model usage policies at the application level are also necessary.

For example, a per-user token quota and a concurrent request limit can be applied.

### \19. Model Extraction and Model Theft

Model Extraction refers to attempts to extract a model's behaviour or functionality without authorisation.

Model Theft is a broader concept associated with obtaining the model itself or its valuable properties without authorisation.

These risks are particularly important for models that are developed commercially or customised with proprietary data.

For example, an attacker may send a large number of queries to a model and try to build another system that imitates its behaviour.

However, the success of such attacks depends on how the model is accessed, its protections and the attacker's resources.

#### Protection methods

API access must be protected with authentication.

Rate limiting must be applied.

Abnormal query behaviour must be monitored.

Access to model files must be restricted.

Model and infrastructure access must be logged.

These controls can also contribute to protecting intellectual property.

### \20. Training Data Extraction

Training Data Extraction refers to the risks of attempts to obtain specific information from a model's training data through its outputs.

This risk is not at the same level in every model.

The model's training method, data content and manner of use matter.

Data protection controls are necessary, particularly when sensitive data is used in training or fine-tuning.

#### Secure approach

Training data must be classified.

Unnecessary personal data must not be used.

Data usage permissions must be verified.

Appropriate privacy and data minimisation methods must be considered.

Sensitive information disclosure in model outputs must be tested.

This approach combines model security with data protection management.

### \21. LLM Supply Chain Security

LLM applications can use many third-party components.

For example:

Model files.

Tokenizer components.

Inference frameworks.

Python packages.

Container images.

Vector database systems.

Model serving services.

External API providers.

Each of these components must be assessed from a supply chain security perspective.

#### Trusted model source

The source from which the model was obtained must be verified.

#### Model integrity

The integrity of model files must be checked with appropriate mechanisms.

#### Dependency security

Third-party software components must be included in SCA controls.

#### Secure model loading

The mechanisms used while loading model files must be assessed.

#### Update management

Security updates for the model serving infrastructure and related components must be tracked.

### \22. Logging and Monitoring in LLM Security

Appropriate monitoring and logging are necessary to operate LLM applications securely.

However, care must be taken not to record sensitive data unnecessarily while logging.

For example, storing all user prompts in plain text creates new data protection risks.

For this reason, the logging policy must be designed on a risk basis.

#### Events that can be monitored

Failed authentication attempts.

Unauthorised tool calls.

Abnormal model usage volume.

High token consumption.

Critical operation approvals.

Security policy violations.

Suspicious data access.

#### Secure logging

Sensitive data must be masked appropriately.

Log access must be restricted.

Retention periods must be defined.

The integrity of audit records must be protected.

Where necessary, integration with SIEM or SOC processes can be provided.

### \23. What Is LLM Security Testing?

LLM Security Testing is the systematic assessment of the security risks of applications that use large language models.

These tests are not limited to examining the model's answers.

The application's architecture, data access, tool integrations and security controls must also be assessed.

#### Test scope

Prompt Injection resilience.

Sensitive data disclosure.

System Prompt Leakage.

Unauthorised tool calls.

RAG access control.

Insecure output handling.

Resource consumption.

Model and dependency security.

#### Testing approach

First, the application's architecture must be understood.

Then trust boundaries must be identified.

The data and tools the model can access must be inventoried.

Risk-based test scenarios must be prepared.

The real system impact of findings must be verified.

Finally, remediation recommendations and a retest must be carried out.

### \24. The Difference Between LLM Pentest and Traditional Pentest

A traditional pentest assesses vulnerabilities in web, mobile, API, network and system components.

An LLM Pentest, in addition to these, addresses model behaviour and AI-specific attack surfaces.

For example, an API authentication problem may be identified during a traditional pentest.

During an LLM Pentest, by contrast, the model's attempts to use the tools it is connected to without authorisation can be tested.

However, the two approaches are not alternatives to one another.

Using them together in critical AI applications gives more comprehensive security visibility.

### \25. Defense in Depth in LLM Security

A single security control must not be relied upon in LLM applications.

A layered security approach must be applied.

For example:

#### Authentication

The user's identity is verified.

#### Authorisation

Data and operation access is restricted.

#### Data Security

The use of sensitive data is controlled.

#### Prompt and Model Security

Efforts are made to reduce the influence of untrusted content.

#### Tool Security

Tool Calling permissions are restricted.

#### Output Security

Model outputs are validated according to the context in which they will be used.

#### Monitoring

Critical operations and abnormal behaviour are tracked.

#### Continuous Testing

Security assessments are repeated regularly.

When these layers work together, the failure of a single control can have a more limited impact.

### \26. AI Gateway and LLM Security

An AI Gateway can place communication between AI applications and model services under central security and management policies.

For example, an organisation may be using different LLM services.

Some applications benefit from cloud-based models.

Some use in-house models.

Others switch between different providers.

In this situation, central visibility and policy management become important.

Depending on product capabilities, AI Gateway solutions can offer:

- Identity and access control.
- Model usage policies.
- Sensitive data detection.
- Data masking.
- Prompt security controls.
- Model routing.
- Usage monitoring.
- Audit logging.
- Cost and quota management.

and similar features.

However, an AI Gateway does not eliminate all security risks on its own.

For example, a backend authorisation problem or an insecure API integration must be fixed separately.

### \27. How to Build a Corporate LLM Security Policy

A corporate LLM Security Policy defines the security rules governing AI applications.

This policy should include both technical and organisational controls.

#### Model selection

The models and providers to be used must be assessed.

#### Data sharing

The types of data that may be sent to the model must be determined.

#### Access management

User, service and tool permissions must be defined.

#### Security testing

LLM applications must be tested according to their risk levels.

#### Monitoring

Critical operations and security events must be logged.

#### Change management

Changes to models, prompts, data sources and tools must be assessed for their security impact.

#### Incident response

Response processes must be established for security incidents specific to LLM applications.

This policy can be part of the organisation's AI Governance programme.

### \28. Example Scenario: A Financial LLM Assistant

Suppose a financial institution has developed an LLM-based reporting assistant for its employees.

The assistant summarises financial reports.

It can also perform certain database queries.

In the first stage, the system only provides information.

Over time, however, new tools are added.

For example, report export and email sending features are added to the system.

In this case, security risks may increase.

#### Risk 1: Unauthorised data access

An employee may query financial reports they have no access permission for.

#### Risk 2: Prompt Injection

A document the system reads may try to manipulate the model's behaviour.

#### Risk 3: Insecure tool use

The model may suggest sending a report to unexpected recipients.

#### Risk 4: Data leakage

Sensitive financial information may be unnecessarily transferred to an external model service.

#### Risk 5: Insufficient monitoring

Critical report accesses may not be logged.

#### Recommended security approach

User-based access control must be applied.

Data sources must be authorised.

Tool calls must be restricted.

Sensitive data transfer must be controlled.

An approval mechanism for critical operations should be considered.

Security testing and monitoring processes must be established.

This example shows that LLM security is more than just checking the model's answer.

### \29. The SECURESYS Approach to LLM Security

At SECURESYS, we believe that LLM security should not be assessed independently of traditional application security.

As large language models connect to real business processes, they interact more with classic application security problems.

For example, an LLM application may carry a Prompt Injection risk.

But the same application's API authorisation may also be flawed.

Tenant isolation may be missing in the RAG data source the model uses.

Or the Tool Calling integration may run with more privileges than necessary.

For this reason, the security assessment must cover the entire application architecture.

Depending on project needs, the following security areas can be addressed together:

#### LLM Security Assessment

Assessment of model behaviours and application security controls.

#### Prompt Injection Testing

Examination of the effect of untrusted content on model behaviour.

#### Data Leakage Assessment

Assessment of sensitive data access and transfer risks.

#### API Security Testing

Examination of the security controls of the services the LLM application is connected to.

#### AI Agent Security

Assessment of Tool Calling, MCP and agent authorisation mechanisms.

#### RAG Security

Examination of corporate information sources and document access controls.

#### Source Code Security

Assessment of application source code with SAST and manual analysis methods.

#### AI Gateway Architecture

Design of central policy, data protection and model usage controls.

The main goal of this approach is not merely to find vulnerabilities.

It is to help the organisation develop its LLM applications in a more controlled and secure way.

Because a powerful model alone does not make a secure application.

**A secure LLM application is built with the right architecture, the right authorisation and continuous security controls.**

### \30. LLM Security Checklist

The following checklist can be used as a starting point for corporate LLM applications.

#### Architecture security

- Have the model and application components been inventoried?
- Have trust boundaries been identified?
- Have external data sources been assessed?
- Have tool integrations been documented?

#### Identity and access

- Is the user's identity verified?
- Is data access limited according to user permissions?
- Do tool calls pass through independent authorisation?
- Is tenant isolation ensured?

#### Data security

- Has sensitive data been classified?
- Have the model provider's data processing terms been assessed?
- Is unnecessary data transfer prevented?
- Is sensitive data protected in logs?

#### Model and prompt security

- Have Prompt Injection scenarios been tested?
- Has System Prompt Leakage been assessed?
- Are untrusted documents handled appropriately?
- Are model outputs processed securely?

#### Operational security

- Is rate limiting applied?
- Are there token and cost limits?
- Is abnormal usage monitored?
- Are critical operations logged?
- Is there an incident response plan for security events?

#### Security testing

- Has LLM Security Testing been performed?
- Has API security been assessed?
- Have RAG access controls been tested?
- Have Tool Calling permissions been reviewed?
- Has a retest been performed for critical findings?

This checklist can be expanded according to the organisation's use case and risk level.

### Frequently Asked Questions

#### What is LLM security?

LLM security is the protection of large language models, and the applications that use them, against manipulation, data leakage, unauthorised access and other security risks.

#### What is the difference between LLM Security and AI Security?

AI Security is a broader field. LLM Security focuses on the security risks specific to large language models and the applications that use them.

#### What is Prompt Injection?

Prompt Injection is an attack approach that aims to have untrusted content used as an instruction in order to change the model's behaviour.

#### What is Jailbreak?

Jailbreak refers to attempts to bypass the model's behavioural or safety restrictions.

#### What is System Prompt Leakage?

System Prompt Leakage is the risk of the model disclosing some parts of its system or developer instructions without authorisation.

#### How does LLM data leakage occur?

It can occur through causes such as unauthorised data access, excessive data sharing, insecure tool integrations or sensitive information disclosure in model outputs.

#### Are LLM outputs trustworthy?

Model outputs must be validated according to the context in which they will be used. In particular, they must not be treated as directly trustworthy for SQL, HTML, command or API operations.

#### What is an LLM Pentest?

An LLM Pentest is the assessment of the security risks of applications that use large language models through realistic attack scenarios.

#### Is a WAF sufficient for LLM security?

No. A WAF can provide protection against some traditional web attacks but does not, on its own, cover all LLM-specific risks.

#### Does an AI Gateway ensure LLM security?

An AI Gateway can centralise certain security and management controls. However, it does not replace other controls such as application authorisation, data access and secure coding.

#### Why is RAG security important in LLM applications?

RAG systems use external information sources. For this reason, document authorisation, data isolation and the management of untrusted content are critically important.

#### Is ISO/IEC 42001 sufficient for LLM security?

ISO/IEC 42001 is a standard for AI management systems. It does not replace technical LLM security testing.

#### How often should LLM security tests be performed?

Test frequency should be determined according to the application's criticality level, changes and risks. Changes to the model, prompt, data source or tools may require additional assessment.

### Conclusion: LLM Security Starts with the System Around the Model

Large language models are transforming the modern software world.

Organisations can access information through natural language interfaces.

They can analyse their documents.

They can improve customer service.

They can speed up software development processes.

And they can carry out some operational tasks with AI-supported systems.

However, these capabilities create new security responsibilities.

Because in LLM applications, the risk is not only a wrong model answer.

The model may interpret untrusted content as an instruction.

It may access sensitive data more than necessary.

It may suggest insecure tool calls.

Model outputs may be processed incorrectly by the application.

Resource consumption may become uncontrolled.

For this reason, a secure LLM architecture cannot be built on a single control.

Authentication is required.

Authorisation is required.

Data security is required.

Prompt and model security is required.

Tool access controls are required.

Secure output handling is required.

Continuous security testing and monitoring are required.

Most importantly, the decisions made by the AI model must not be allowed to override the application's security policies.

**The model may propose an operation; but whether that operation is carried out must be decided by trusted application controls.**

This distinction lies at the foundation of corporate LLM security.

Because in AI applications, security is not only about what the model says.

**It is about which information the model can access, which tools it can use and which operations it can perform.**