What Is Software and AI Security? New-Generation Cyber Threats in Modern Applications

Introduction: Software Is No Longer Made of Code Alone

There was a time when software security mostly meant source code analysis, SQL Injection, Cross-Site Scripting, authorisation flaws and secure software development processes.

To assess whether an application was secure, its source code was reviewed, penetration tests were run against the live application, and the third-party components it used were checked for known vulnerabilities.

These methods remain important today.

However, the software world is going through a new transformation.

Applications no longer simply execute rules defined in advance by developers. They communicate with large language models, interpret instructions that users give in natural language, collect information from corporate documents, connect to external systems and, in some cases, perform actions on behalf of users.

A customer service application can answer questions through artificial intelligence.

A finance application can generate reports from instructions given in natural language.

A software development assistant can write source code.

An AI Agent can access a calendar system to create meetings, pull information from a corporate database or start operations through the tools it has been authorised to use.

These developments offer businesses significant productivity opportunities.

But they also raise a new security question:

If software no longer merely executes commands but can interpret instructions and make decisions, how do we make it secure?

This question is at the heart of the concept of Software and AI Security.

New components are now being added to what traditional application security tries to protect (source code, data, user accounts and system resources): AI models, prompt structures, embedding data, vector database systems, model contexts, tool calls and autonomous agent behaviours.

The security approach therefore needs to expand as well.

In this article, we examine in detail what software and AI security means, where it differs from traditional application security, what the new attack surfaces are, and what kind of security architecture organisations should build.

1. What Is Software Security?

Software security is the management of security risks throughout the entire life cycle of an application, from design and development through testing to operation in production.

The core objective is to make the software resilient against unauthorised access, data leaks, manipulation, service outages and other cyber threats.

Three fundamental properties stand out in traditional software security:

Confidentiality: Only authorised users and systems can access the data.

Integrity: Data and transactions cannot be altered without authorisation.

Availability: The application can deliver service reliably whenever it is needed.

These three properties form the foundation of information security.

In modern applications, however, requirements such as authentication, authorisation, accountability, traceability and secure transaction execution are also important.

For example, in a banking application a user must be able to view their own account but must not be able to access another customer's account details.

In an e-commerce application, a product price must not be changeable without authorisation through a request sent by a user.

In a corporate portal, a standard user must not be able to perform operations that require administrator privileges.

These security requirements are met through various controls applied to the source code, the application architecture and the runtime environment.

Methods used in traditional software security

Software security draws on different testing and control methods.

SAST (Static Application Security Testing) helps identify potential security weaknesses by analysing source code without executing it.

DAST (Dynamic Application Security Testing) evaluates the externally observable security behaviour of a running application.

SCA (Software Composition Analysis) examines the third-party libraries, dependencies and known vulnerabilities in use.

Manual Source Code Analysis investigates business logic, complex authorisation and architectural problems through expert assessment.

Penetration Testing (Pentest) evaluates an application's security posture against realistic attack scenarios.

Secure Coding aims to ensure that developers use secure programming practices.

DevSecOps integrates security controls into software development and CI/CD processes.

These methods remain valid for applications that use AI. However, additional security controls are needed for the risks specific to AI components.

2. What Is AI Security?

AI security is the protection of AI models, the applications that use them, data processing workflows and AI-dependent systems against misuse, unauthorised access, manipulation and security breaches.

In English this field is most often referred to as AI Security.

However, AI Security is not a single security technology.

It is a combination of several disciplines.

For example, the following topics can be assessed within the scope of AI security:

• Use of the model by unauthorised persons.
• Manipulation of the model through malicious inputs.
• Leakage of corporate data through the model.
• AI agents performing actions beyond their authority.
• Manipulation of the model's training or knowledge sources.
• Insecure handling of model outputs by the application.
• Software and supply chain vulnerabilities in the AI infrastructure.

There is an important distinction here.

AI security is not the same thing as using AI for cyber security purposes.

A SOC system performing threat analysis with AI is the use of AI in security operations.

Protecting an LLM application against Prompt Injection attacks, on the other hand, is the security of the AI system.

Although the two fields are related, they address different security problems.

In this content series we will focus mainly on the second topic, that is, the security of software and systems that use AI.

3. The Difference Between Traditional Software Security and AI Security

Traditional applications mostly follow logical flows defined by developers.

For example, when a user logs in, the application checks the credentials and grants access if the defined conditions are met.

In AI-enabled systems, however, some behaviours stem not from strict rules written in advance but from outputs the model produces by interpreting its inputs.

This makes security assessment more complex.

Security Area	Traditional Software	AI-Enabled Software
Core processing	Predefined code flow	Code flow and model inference
User input	Form, API, parameter	Natural language, file, image, API
Main risks	Injection, XSS, authorisation	In addition to these, Prompt Injection and model manipulation
Data sources	Database and services	Database, model context, RAG, tools
Privilege management	User and service privileges	User, service and AI Agent privileges
Testing approach	SAST, DAST, Pentest	Traditional tests and AI Security Testing
Output behaviour	Largely deterministic	Can be probabilistic and context-sensitive

This comparison does not mean that traditional security controls have become invalid.

On the contrary, AI systems add new attack surfaces on top of the traditional security infrastructure.

For example, if an AI Agent uses an API that accesses a database, that API's authorisation controls are still mandatory.

The fact that a model tells a user which action to take is not enough for the application to permit that action automatically.

Decisions produced by AI must not replace security authorisation.

4. Why Does AI Create a New Attack Surface?

An attack surface refers to all the access points and components of a system that attackers can target.

In a traditional web application, the attack surface mostly consists of HTTP endpoints, user input fields, API services, file upload features and system integrations.

In an AI-enabled application, new areas are added to these.

Consider, for example, a corporate AI assistant.

The user asks a question in natural language.

The assistant connects to the corporate document search system.

It finds the relevant documents.

It sends the text taken from those documents to the model.

The model produces an answer.

If necessary, it calls another API.

The result is presented to the user.

During this process, different trust boundaries emerge from a security perspective.

The prompt written by the user may not be trustworthy.

The document content may have been altered by an attacker.

The answer generated by the model may be wrong or manipulated.

The connected API may have broader privileges than expected.

Therefore, checking only the user's input is not enough.

All data that the model reads, interprets, produces and uses to perform actions must be assessed from a security perspective.

Shifting trust boundaries

In traditional software, the separation between untrusted user data and application commands is mostly determined by the programming language and the application architecture.

In LLM systems, however, instructions and the content to be processed can exist in the same natural-language context.

For example, a document that the model is supposed to summarise may contain expressions that look like instructions.

These expressions are not genuine user instructions.

But the model may interpret them as commands that change its behaviour.

This problem forms the basis of the Prompt Injection risk.

5. What Is LLM Security?

LLM stands for Large Language Model.

These models can be used for natural language processing, text generation, summarisation, classification, code generation and similar tasks.

LLM Security addresses the security risks of these models and of the applications that use them.

In an LLM application, it is not only the model itself but all the components around it that must be assessed.

For example, consider the flow:

User Interface → Application Backend → Prompt Construction → LLM → Output Processing → User

as a whole.

In this architecture, each stage needs different security controls.

The user interface may need authentication and input controls.

The backend must apply authorisation and data access controls.

The prompt construction layer must keep trusted instructions separate from data coming from external sources.

LLM output must be evaluated before the application executes it or passes it on to other systems.

In particular, using model outputs in security-critical areas such as SQL, shell commands, HTML or automated API operations can introduce additional risk.

For this reason, LLM security is not limited to strengthening the system instruction given to the model.

A secure LLM application requires strong security boundaries to be built around the model.

6. What Is Prompt Injection?

Prompt Injection is an attack approach that aims to have untrusted content treated as instructions in order to change the behaviour of an AI system.

These attacks can be direct or indirect.

Direct Prompt Injection

The attacker sends specific instructions directly to the AI application in an attempt to bypass the system's rules.

For example, a user may ask the corporate assistant to disclose information it would normally not share.

Such attempts aim to push the boundaries of the model's defined usage limits.

Indirect Prompt Injection

In indirect attacks, the malicious instruction is not in the user's message but in other content that the model reads.

This content can be:

• a web page,
• a PDF document,
• an e-mail,
• a database record,
• a search result,
• a repository file

.

For example, while summarising a document it has obtained from an external source, an AI assistant may also interpret instructions that have been planted inside the document.

This is particularly important in RAG systems and AI Agent architectures.

Why does Prompt Injection differ from traditional Injection attacks?

In a SQL Injection attack, the core problem is the application interpreting data as a SQL command.

In a Prompt Injection attack, the aim is for the model to interpret untrusted content as an instruction that steers its behaviour.

The technical mechanisms of the two attacks are different.

However, the underlying security principle is the same:

Untrusted data must never become an authorised command.

We will cover Prompt Injection in detail in the fourth chapter of our series.

7. What Is a Jailbreak?

A jailbreak refers to attempts to bypass the security or behavioural restrictions of an AI model.

Although it is related to Prompt Injection, it is not the same concept.

Prompt Injection specifically aims for untrusted content to influence the instruction hierarchy.

A jailbreak, on the other hand, is a broader category of behavioural manipulation aimed at making the model break the restrictions it is normally supposed to follow.

This distinction matters in enterprise applications.

That is because a model producing a response that breaks its safety rules is not the same outcome as genuinely gaining unauthorised access to corporate data.

A real security assessment must examine not only what the model says but also which data and operations the application gives it access to.

For example, a model may claim that a user can retrieve information belonging to another customer.

However, if backend authorisation is implemented correctly, this claim may not translate into actual data access.

For this reason, in LLM security tests, model behaviour must be separated from the real security impact on the system.

8. What Is AI Agent Security?

An AI Agent refers to an AI-based system that can gather information, make decisions and carry out actions using the tools defined for it in order to achieve specific goals.

A classic chatbot mostly produces answers.

An AI Agent, if authorised, can perform actions.

For example, an AI Agent can:

• create calendar entries,
• prepare e-mail drafts,
• query customer records,
• open support tickets,
• generate reports,
• start workflows.

These capabilities provide significant operational advantages.

However, a wrong or manipulated decision by the model can turn into real system operations.

For example, it is an unnecessary risk for an assistant that is only supposed to present reports to a user to have the authority to modify customer records.

That is why Least Privilege, the principle of least privilege, is critically important in AI Agent systems.

Why does an AI Agent require specific security controls?

When assessing the security of an AI Agent, the model's responses alone should not be examined.

The following questions must also be answered:

Which tools can the agent use?

What is the privilege of each tool?

Does the agent act on behalf of the user or with a system account?

Is the user's authorisation re-verified before the action?

Is human approval required for critical operations?

Are the operations performed by the agent logged?

Without answering these questions, simply writing a strong system prompt does not provide sufficient security.

9. Tool Calling and MCP Security

AI applications can use tool-calling mechanisms to interact with external systems.

With Tool Calling, the model can request that functions exposed by the application be executed with specific parameters.

For example, a model may interpret the request:

"Retrieve this customer's last three orders."

and propose calling the relevant API function.

In a secure architecture, however, the model alone should not decide whether the API call is actually made.

The application layer must check the user's identity, access authorisation, the parameters and the transaction policy.

What is MCP?

The Model Context Protocol (MCP) is a protocol that makes it easier for AI applications to interact with external data sources and tools in a standardised way.

Through MCP integrations, AI systems can connect to different tools and resources.

However, as the number of integrations grows, the security surface can expand too.

For example, a malicious or compromised tool server can manipulate the information the model uses.

A tool may be configured with more privileges than necessary.

An integration may provide broader data access than expected.

For this reason, in MCP security the origin of tools, their authorisation, the scope of their data access and the auditing of their operations are important.

In the fifth chapter of our series, we will examine AI Agent, Tool Calling and MCP security in detail.

10. What Is RAG Security?

RAG stands for Retrieval-Augmented Generation.

In this architecture, the large language model does not build its answers solely from what it learned during its own training.

It can also use content retrieved from external knowledge sources.

For example, an organisation can connect its internal procedures, technical documents and product manuals to an AI assistant.

When a user asks a question, the system finds the relevant documents and helps the model build its answer.

This approach is highly valuable, particularly in corporate knowledge management.

However, RAG systems carry significant security risks.

Unauthorised document access

All of an organisation's documents may sit in the same knowledge pool.

But not all employees may be authorised to access all of those documents.

For example, human resources documents, financial reports and board documents may have different access levels.

If the RAG system does not filter search results according to the user's actual privileges, unauthorised information disclosure can occur.

Document poisoning

An attacker can add manipulated content to the knowledge source.

This content can cause the model to produce incorrect information or follow unwanted instructions.

Trustworthiness of the data source

In a RAG system, the freshness, integrity and source of the retrieved content are important.

For the model to produce a correct answer, it is not enough merely to find the relevant document.

The document must also be trustworthy and appropriate in terms of access.

The core principle in RAG security is that a search result being discoverable does not mean that access is permitted.

11. Vector Database Security

One of the components frequently used in RAG architectures is the Vector Database.

These systems can store embedding data, which are numerical representations of content, and perform similarity searches.

Embedding data is not always a direct copy of the original text.

In some circumstances, however, it can allow inferences about sensitive information or contribute to information leakage risks.

For this reason, vector database security must also be part of the data security programme.

In particular, the following controls should be considered:

• User- and service-based access control.
• Tenant isolation.
• Data source authorisation.
• Encryption.
• Audit logging.
• Data retention and deletion policies.
• Document-level authorisation checks in search results.

For example, in a multi-tenant SaaS application, the embedding data of different customers may be held on the same infrastructure.

In this case, the application must prevent one customer from seeing search results that belong to another customer.

Simply writing "do not show other customers' data" inside the prompt is not enough.

Authorisation must be enforced in the application and data access layers.

12. What Is Shadow AI?

Shadow AI is the use of AI services that have not been approved by the organisation, or are not adequately governed, by employees or teams in business processes.

This use is not always malicious.

An employee may want to prepare a report faster.

A developer may have an AI tool review their code.

A sales employee may use an external LLM service to summarise a customer proposal.

In the process, however, corporate data can be transferred to third-party systems without control.

For example, a developer may send the application's source code to an external AI tool for debugging.

The code may contain:

• customer information,
• access tokens,
• connection details,
• internal business logic

.

This can pose a risk to the organisation's data security and intellectual property protection policies.

How should Shadow AI be managed?

Simply banning all AI tools may not be sustainable in most organisations.

A more effective approach is to create secure and approved usage options.

For example, the organisation can:

Define approved AI services.

Classify which data may be shared.

Apply corporate account and contract controls.

Monitor sensitive data transfers.

Use AI Gateway and DLP controls where necessary.

The goal here is not to block AI use entirely but to bring it into line with corporate security policies.

13. AI and Sensitive Data Security

AI applications can work with large volumes of data.

These data can include personal data, trade secrets, financial records and customer information.

AI security is therefore not just about preventing attackers from taking over the system.

It also matters for what purpose, in which system, for how long and under whose access the data is processed.

For example, an organisation may analyse customer support conversations with AI.

These conversations may contain names, phone numbers, e-mail addresses, address details or other personal data.

The data processing workflow must be assessed in terms of KVKK (Turkish Personal Data Protection Law) and other applicable data protection obligations.

Data minimisation

Data that the model does not need for processing should not be sent to the model.

For example, to categorise a customer support request, it may not be necessary to pass the customer's entire financial history to the model.

Masking and anonymisation

Depending on the purpose of use, sensitive data can be masked or anonymised using suitable methods.

However, masking does not always mean anonymisation.

Whether the data can be re-identified, and the purpose of the processing, must be assessed separately.

Data retention

It should be examined how long the AI service provider retains the data sent to it, for what purposes it uses it and which sub-processors it shares it with.

These matters are particularly important for organisations that use external LLM APIs.

14. Security of AI Models

So far we have mainly addressed the security of AI applications.

However, the model itself also needs a security assessment.

Different threats can arise while an AI model is being developed, trained, stored and deployed.

Some of these are classified as Data Poisoning, Model Poisoning, Model Theft and Adversarial Attack.

Data Poisoning

Data Poisoning targets the malicious manipulation of the data used in model training or in related data processing workflows.

For example, an attacker may add misleading examples to the training data in order to make the model behave incorrectly in certain situations.

Model Poisoning

Model Poisoning can cover attacks related to manipulating a model's parameters, weights or training process.

This risk is particularly important for models obtained from external sources and for uncontrolled model supply chains.

Model Theft

Model Theft refers to risks such as a model being copied or extracted without authorisation, or its behaviour being imitated.

For commercially developed models, it can lead to intellectual property and economic losses.

Adversarial Attack

Adversarial attacks can make use of specially crafted inputs that aim to make the model misclassify or produce wrong decisions.

The form of these attacks varies according to the type of model used.

For example, attacks on an image classification system and on a large language model do not share the same technical characteristics.

Model security therefore requires a testing methodology suited to the model type and the usage scenario.

15. AI Model Supply Chain Security

In traditional software, third-party libraries can pose significant security risks.

In AI systems, the model and data supply chain is added to these risks.

For example, an organisation may use a ready-made open-weight model obtained from outside.

The model's files, loading mechanism, dependencies or distribution package must be reviewed from a security perspective.

Some model file formats and loading processes can lead to risks such as code execution when used insecurely.

For this reason, choosing models solely on the basis of performance is not enough.

The model's source and integrity must also be assessed.

Trusted model source

It should be verified which organisation published the model and from which source it was downloaded.

Integrity check

Whether model files match the expected content can be verified with suitable integrity mechanisms.

Secure loading

The software libraries and file formats used while loading the model should be assessed from a security perspective.

Dependency security

Model serving infrastructure, Python packages, container images and other related components should be included in SCA and infrastructure security controls.

This approach shows that the AI supply chain is not independent of the traditional software supply chain.

16. Authorisation in AI Applications

One of the most critical security principles in AI applications is that authorisation controls are applied outside the model.

For example, an employee might ask the corporate AI assistant the following question:

"Show me the salary list of all the company's employees."

The fact that the model can understand this request does not mean that the request should be allowed to be carried out.

The application layer must check the user's authorisation to access the relevant data.

This approach is particularly important in RAG and AI Agent systems.

The model is not an authentication system

An LLM deciding whether a user appears trustworthy is no substitute for secure authentication.

A prompt is not an authorisation policy

Writing "only give information to authorised users" in the system prompt is not sufficient on its own.

Tool calls must be re-authorised

When the model proposes an API call, the application must check the real user identity and the authority to perform the operation.

Data-level access must be enforced

Particularly in multi-user and multi-tenant systems, data access must be assessed at the level of user, role, tenant and object ownership.

In a secure AI architecture the model may suggest a decision; however, authority must be determined by trusted application controls.

17. What Is Human-in-the-Loop?

Human-in-the-Loop is the inclusion of human review or approval in certain AI operations.

It is an important security control, especially for high-impact operations.

For example, an AI Agent preparing an e-mail draft may be low risk.

But the same agent automatically sending e-mails to thousands of customers poses a higher risk.

Similarly, producing a suggestion for a financial transaction and carrying out an actual money transfer should not be assessed at the same privilege level.

Operations can therefore be classified according to their risk.

Low-risk operations can be executed automatically.

Medium-risk operations may require additional verification.

For high-risk operations, human approval can be made mandatory.

However, human approval is not sufficient on its own either.

The approval screen must clearly show what the operation will do, which data it will use and its possible impacts.

Otherwise, the user may not understand what they are actually approving.

18. What Is an AI Gateway?

An AI Gateway is an architectural component that ensures communication between AI applications and model services is carried out under defined security and governance policies.

Depending on the use case and product capabilities, an AI Gateway can support the following controls:

• Identity and access management.
• Model usage policies.
• Sensitive data detection and masking.
• Prompt Injection risk analysis.
• Model routing.
• Usage and cost visibility.
• Audit logging.
• Data transfer controls.
• Agent and Tool Calling policies.

There is, however, an important point.

An AI Gateway alone does not solve all AI security problems.

For example, an AI Gateway can inspect model requests.

But it may not automatically fix faulty authorisation controls on the application's backend.

Likewise, if tenant isolation is missing in the RAG database the model uses, prompt-level controls alone may not be enough.

For this reason, an AI Gateway should be considered as part of a layered security architecture.

19. Defense in Depth in AI Security

Defence in Depth is an approach that aims to ensure security does not depend on a single control.

This principle is especially important in AI applications.

For example, a corporate LLM assistant may have the following layers:

First layer: Authentication

The user's identity is verified using trusted mechanisms.

Second layer: Authorisation

The data and operations the user may access are determined.

Third layer: Data security

The use, transfer and storage of sensitive data are controlled.

Fourth layer: AI security policies

Model inputs, outputs and tool calls pass through appropriate risk controls.

Fifth layer: Application security

Backend APIs, source code, dependencies and the runtime environment are protected.

Sixth layer: Monitoring and auditing

Security events, critical operations and anomalous behaviours are logged.

Seventh layer: Continuous testing

Penetration tests, AI Red Teaming and other security assessments are carried out.

These controls complement one another.

The failure of one control should not automatically lead to the compromise of the entire system.

20. What Is AI Red Teaming?

AI Red Teaming is the systematic assessment of AI systems against misuse and attack scenarios.

In traditional Red Teaming exercises, an attacker's attempts to bypass network, user, system and application controls can be examined.

In AI Red Teaming exercises, attack surfaces specific to AI are assessed as well.

For example:

Prompt Injection resilience.

Jailbreak attempts.

Unauthorised data access.

Tool Calling manipulation.

Agent privilege boundaries.

Security problems originating from RAG.

Sensitive information disclosure.

However, AI Red Teaming is not just about asking the model difficult questions.

The real application architecture and the systems the model can reach must be assessed.

The difference between model testing and system testing

A model producing text that violates a safety rule in response to a particular prompt is a finding about model behaviour.

The same model performing an unauthorised operation through an API, on the other hand, is a broader finding that concerns application and system security.

For this reason, in AI security tests:

Model Behavior

and

System Impact

must be assessed separately.

21. Is Traditional Pentesting Sufficient for AI Systems?

Traditional pentesting can identify important security risks in AI-enabled applications.

For example:

• SQL Injection.
• Broken Access Control.
• SSRF.
• Insecure APIs.
• Authentication problems.
• Misconfiguration.
• Sensitive data exposure.

However, additional test scenarios may be needed to assess Prompt Injection, Agent Manipulation, RAG Poisoning and risks specific to model behaviour.

For this reason, in critical AI applications, traditional pentesting and AI Security Testing should complement each other.

For example, in a corporate AI Agent project, the test scope should cover both the security of the connected APIs and how the agent uses those APIs.

Testing only the model's answers is not enough.

Testing only the API endpoints may not cover all AI risks either.

22. DevSecOps and MLOps in AI Security

DevSecOps aims to integrate security controls into software development processes.

MLOps, on the other hand, manages the development, training, deployment and operation of machine learning models.

In AI systems, these two approaches should be considered together.

That is because a modern AI application consists of both traditional software components and model and data components.

For example, the application's backend code may be secure.

But the model file in use may have been downloaded from an untrusted source.

Or the model may be trustworthy.

But the API the model connects to may be over-privileged.

The security life cycle must therefore cover not only source code commits but also model and data changes.

Example of a secure AI pipeline

The following controls can be used in the development process of an AI application:

Source Code Control

SAST and Secret Scanning.

Dependency Control

SCA and known vulnerability analysis.

Model Control

Assessment of model source, integrity and secure loading.

Data Control

Data source, sensitivity and access policies.

AI Security Testing

Prompt Injection, model behaviour and Agent privilege scenarios.

Release Control

Risk-based security approval.

Production Monitoring

Tracking security events and anomalous transaction behaviour.

This approach forms the basis of the Secure AI SDLC concept.

23. What Is Secure AI SDLC?

Secure AI SDLC is the practice of addressing the security requirements of AI-enabled software throughout all the stages of planning, design, development, testing, deployment and operation.

It extends the traditional Secure SDLC approach with AI-specific components.

Planning stage

The business problem the application will solve, the data it will use and the operations it may perform are determined.

Design stage

The trust boundaries between the model, users, data sources and tools are defined.

Development stage

Secure coding, dependency security, secrets management and authorisation controls are applied.

Testing stage

In addition to traditional application security tests, AI-specific test scenarios are assessed.

Deployment stage

The trustworthiness of the model, data and application components is verified.

Operation stage

Model behaviours, critical operations, accesses and security events are monitored.

Continuous improvement

Security findings and production incidents are fed back into the development processes.

This approach ensures that security is not merely a test carried out before the model is released.

24. ISO/IEC 42001 in AI Security

ISO/IEC 42001 is an international standard for AI management systems.

It helps organisations address their use of AI with a management system approach.

Within this scope, policy, responsibility, risk management, processes and continual improvement related to AI are assessed.

However, holding an ISO/IEC 42001 certificate does not mean that an AI application is free of all technical security vulnerabilities.

Management system conformity and technical security testing are different matters.

For example, an organisation may establish an AI risk management process under ISO/IEC 42001.

Yet the LLM application it develops may still contain Prompt Injection or Broken Access Control problems.

For this reason, the management system and technical security controls must complement each other.

AI Governance is not an alternative to AI Security Testing.

25. NIST AI RMF and AI Risk Management

The NIST AI Risk Management Framework is a framework that helps manage the risks associated with AI systems.

AI risks do not consist of cyber attacks alone.

The system's reliability, usage context, performance, data governance and other potential impacts must also be assessed.

For this reason, the AI RMF provides a broader risk management perspective that goes beyond technical security testing.

In an enterprise approach, for example:

NIST AI RMF for general AI risk management,

ISO/IEC 42001 for the AI management system,

OWASP LLM security resources for application-specific technical risks,

Secure SDLC and DevSecOps for software development controls

can be used together.

Each of these sources meets a different need.

26. Who Is Responsible for AI Security?

AI security is not the responsibility of the cyber security team alone.

A successful programme requires different teams to work together.

Software development teams

They develop secure APIs, authorisation controls and secure data processing mechanisms.

AI and data teams

They manage model selection, data quality, the model life cycle and the AI application architecture.

Cyber security teams

They contribute to threat modelling, security testing, access policies and security monitoring processes.

Legal and compliance teams

They carry out assessments in terms of data protection, contractual obligations and applicable regulations.

Business units

They determine the purpose of AI use, its business impact and the acceptable level of risk.

Senior management

It makes decisions on risk appetite, resource allocation, responsibilities and corporate policy.

AI security should be treated as the shared responsibility of all these teams.

27. Where Should Organisations Start with AI Security?

For an organisation that wants to build an AI security programme, the first step is not necessarily to buy a new security product.

First, existing AI use needs to be made visible.

Stage one: AI inventory

Which AI services does the organisation use?

Which applications contain an LLM?

Which employees use external AI services?

In which systems are AI Agents used?

These questions need to be answered.

Stage two: Data classification

Which data do the AI systems process?

Is this data personal data?

Does it contain trade secrets?

Is customer data involved?

Is it transferred outside the organisation?

Stage three: Risk assessment

Not every AI use case carries the same level of risk.

For example, a general text-editing application and a financial AI Agent with access to customer accounts should not be treated at the same security level.

Stage four: Security policies

Approved model services, data sharing rules, tool privileges and critical operation approvals should be defined.

Stage five: Technical controls

In suitable use cases, authentication, authorisation, DLP, AI Gateway, logging and security testing can be applied.

Stage six: Continuous monitoring

Model and application changes, new security risks and incidents should be tracked.

This structure can be matured over time.

28. Example Scenario: A Corporate AI Assistant

Imagine an organisation developing an AI-powered knowledge assistant for its employees.

The assistant's purpose is to help employees quickly find corporate procedures and technical documents.

At first glance, this application may look low risk.

Over time, however, the system may gain new capabilities.

First it starts reading PDF documents.

Later it connects to SharePoint-like document systems.

Then it gains access to customer records.

Finally, the ability to create support tickets is added.

At this point, the application's security profile has changed significantly.

First risk: Unauthorised document access

An employee may obtain summaries of management documents they are not authorised to access.

The solution is document-level authorisation and search controls appropriate to the user context.

Second risk: Prompt Injection

A document that the assistant reads may contain malicious instructions.

The solution is to apply architectural controls and behavioural tests that limit untrusted content from gaining instruction authority.

Third risk: Over-privileged tools

If the assistant only needs to open support tickets but has the authority to delete existing records, this creates an unnecessary risk.

The solution is to restrict tool privileges according to business need.

Fourth risk: Sensitive data transfer

The assistant may send customer data to an external model service.

The solution is data classification, appropriate transfer policies and the necessary technical safeguards.

Fifth risk: Lack of auditing

If it is not recorded which operation the assistant performed on behalf of which user, incident investigation becomes difficult.

The solution is secure audit logging and transaction traceability.

This example shows us an important reality:

The risk of an AI application stems not only from the model it uses but also from the data and operational privileges connected to the model.

29. SECURESYS Software and AI Security Approach

At SECURESYS, we believe that software security and AI security should not be treated as two entirely independent fields.

That is because AI applications also run on software infrastructure.

They use API services.

They connect to databases.

They need authentication and authorisation mechanisms.

They make use of third-party components.

Traditional application security controls therefore remain important in AI systems too.

In AI-enabled applications, however, new risk areas must be assessed in addition to these.

Depending on the project scope and application architecture, an integrated security approach may consist of the following components:

Source Code Security

SAST, manual source code analysis and Secure Coding controls.

Application and API Security

Web, mobile and API security tests.

LLM Security

Assessment of Prompt Injection, model behaviour and sensitive information disclosure risks.

AI Agent Security

Review of tool privileges, operation limits, MCP integrations and approval mechanisms.

Data Security

Sensitive data processing, masking, data access and transfer controls.

AI Gateway

A security architecture for managing AI use under centralised policies.

AI Red Teaming

Assessment of the security resilience of AI systems through realistic misuse scenarios.

Secure AI SDLC

Including security in the development and operation life cycle of AI applications.

The core goal of this approach is not merely to report vulnerabilities.

It is to make the organisation's use of AI more controlled, traceable and sustainable.

Particularly in corporate AI applications, which data the model can access, which tools it can use and which operations it can perform must be clearly defined.

Because secure AI is not simply AI that produces the right answer.

It is AI that operates within the right privilege boundaries, protects sensitive data and carries out critical operations in an auditable way.

30. The Future of Software and AI Security

As AI technologies develop, the way applications behave is changing too.

In the coming period, organisations can be expected not only to develop applications that use AI but also to give these applications ever more operational responsibility.

For example, AI Agent systems may be integrated into more business processes.

Broader connections may be established with corporate document and data systems.

The areas of use for AI-assisted software development tools may expand.

These developments bring two important consequences from a security standpoint.

First, the importance of traditional software security will not diminish.

On the contrary, the security of the APIs, databases, identity management and infrastructure components that AI systems connect to may become even more critical.

Second, AI-specific security controls will need to mature.

For example, it will no longer be enough for an organisation to ask only "Do we have Prompt Injection protection?"

More comprehensive questions will be needed:

Which data can the model access?

Which operations can the agent perform?

How are critical operations approved?

How are untrusted documents handled?

How are security tests repeated when the model or tools change?

Which logs can be examined when an incident occurs?

The answers to these questions will determine an organisation's AI security maturity.

Frequently Asked Questions

What is software and AI security?

Software and AI security is the integrated management of the security risks of traditional application components, AI models, data sources and AI Agent systems.

Are AI Security and Cybersecurity the same thing?

AI Security is a specialised field related to cyber security. It concentrates on the security of AI systems and the risks specific to them. Cyber security is broader in scope.

Why are AI applications vulnerable to cyber attacks?

AI applications use traditional software components and add new attack surfaces to them, such as models, prompts, data sources and tool integrations.

What is Prompt Injection?

Prompt Injection is a manipulation approach that aims to make an AI system interpret untrusted content as instructions.

What is the difference between a Jailbreak and Prompt Injection?

A jailbreak refers to attempts to bypass a model's behavioural restrictions. Prompt Injection aims for untrusted inputs to influence how the model interprets instructions.

What is LLM Security?

LLM Security is the assessment and protection of large language models and the applications that use them against manipulation, data disclosure, unauthorised operations and other security risks.

Why is AI Agent security important?

AI Agent systems do not merely produce information; they can perform operations through tools. Authorisation, operation approval and auditing controls are therefore critically important.

What is MCP security?

MCP security is the management of authentication, authorisation, tool trustworthiness and data access controls in the integrations that AI applications establish with external tools and data sources.

What is RAG security?

RAG security is the protection of AI applications against risks such as unauthorised access, document manipulation and data leakage when they retrieve data from external knowledge sources.

What is Shadow AI?

Shadow AI refers to AI services and applications used outside the organisation's approval or adequate oversight processes.

What is an AI Gateway used for?

An AI Gateway can help manage communication between AI applications and model services through centralised policy, access, data security and monitoring controls.

What is AI Red Teaming?

AI Red Teaming is the systematic assessment of the security resilience of AI systems against realistic attack and misuse scenarios.

Is traditional pentesting sufficient for AI applications?

Traditional pentesting can identify important vulnerabilities. However, additional tests may be needed for AI-specific Prompt Injection, Agent Manipulation and model behaviour risks.

Does ISO/IEC 42001 guarantee AI security?

No. ISO/IEC 42001 is a standard for AI management systems. It does not guarantee that there are no technical vulnerabilities. It must be supported by technical testing.

What is Secure AI SDLC?

Secure AI SDLC is the integration of the security controls of AI-enabled software into all the stages of planning, design, development, testing, deployment and operation.

Conclusion: AI Security Is the New Frontier of Software Security

A major shift is under way in the software world.

Applications no longer simply run commands written by developers.

They interpret natural language.

They gather information from corporate documents.

They try to understand users' intentions.

They communicate with tools.

And in some cases they can initiate real system operations.

These developments expand the capabilities of software while also increasing its security responsibilities.

Traditional application security is still a fundamental requirement.

SQL Injection, Broken Access Control, SSRF, insecure APIs and sensitive data leaks have not gone away.

But new risks have been added to them.

Prompt Injection.

Jailbreak.

RAG Poisoning.

AI Agent privilege problems.

Model and data supply chain risks.

Shadow AI.

Uncontrolled tool calls.

Modern organisations' security approach must therefore broaden as well.

It is not enough for the source code alone to be secure.

Nor is it enough for the model alone to produce safe answers.

The data the model works with, the tools it uses, the operations it performs and the trust boundaries within which it operates must be assessed together.

Three principles lie at the foundation of a secure AI architecture:

First: Untrusted content must not turn into an authorised instruction.

Second: The decision of an AI model must not replace application authorisation.

Third: The data and operations that AI can access must be limited to genuine business need.

When these principles are applied, AI systems can be used in a more controlled, auditable and secure way.

The future of software and AI security does not depend on the success of a single product or a single testing method.

Success depends on secure software development, data protection, model security, agent authorisation, continuous testing and corporate governance being applied together.

Because in AI systems, security is determined not only by what the model says but by what the model can access and what it can do.