On Monday the sales team shows the board an assistant that answers questions about contract terms in seconds. On Tuesday someone notices that the same assistant quotes a passage from another client’s contract, because its document index was built from an entire network share. On Wednesday it turns out that the API key for the model service has been sitting in a public code repository for two months. None of these incidents required an attack on the language model itself.
This is what AI security usually looks like in practice: the risk rarely sits in the model and almost always sits in the environment around it — in identities, keys, data, integrations, vendors and monitoring gaps. This guide shows how to secure AI in a company domain by domain, with the cybersecurity controls you already have, and ends with a risk and control map and an action plan for the first three months.
How does securing AI environments differ from protecting the models themselves?
Securing an AI environment covers everything around the model: who authenticates to it and how, what data reaches it, which systems it is connected to, who supplies it and whether the security team can see what happens to it. Protecting the model itself is a narrower slice: the integrity of training data, resistance to adversarial attacks, model theft. We cover that slice separately in the article on how to protect AI models and training data from attacks.
The difference matters in practice. Most companies do not train their own models. They buy access to a model through an API, switch on AI features in SaaS applications, or build an application that connects a ready-made model to their own documents. In each of these cases the company does not control the model weights, but it does control access, data and integrations. That is where most of the decisions the CISO is accountable for are made.
The term AI security therefore has two levels. The first is the security of AI systems as company assets, which is the subject of this guide. The second is the use of AI in defence, for example in event analysis in the SOC, which is a separate topic. Mixing the two levels leads to a situation where the company buys a tool with “AI” in its name while its own assistant still has access to every file.
The good news is that you do not have to build a programme from scratch. NIST’s AI Risk Management Framework (AI RMF 1.0, released on 26 January 2023) is voluntary and rests on four functions: GOVERN, MAP, MEASURE and MANAGE. The first layer of ENISA’s three-layer Multilayer Framework for Good Cybersecurity Practices for AI, published on 7 June 2023, is simply cybersecurity foundations. Both frameworks say the same thing: AI inherits existing controls and adds a few of its own.
How do you build an inventory of AI systems and AI use in a company?
An AI inventory is a register of three categories: AI systems the company builds, AI services it buys, and AI features and tools employees use — with or without the IT department’s knowledge. Without such a list every subsequent control is partial, because nobody knows what it applies to.
The first category is the easiest to record: applications with a language model, customer chatbots, internal assistants, automations with agents. Each of these systems has a code repository, a runtime environment and a team, so it is enough to add it to the application register and flag that it uses a model.
The second category is purchased services: access to models through an API, platforms for building AI applications, and AI features that vendors have added to applications the company has used for years. That last group is the one most often missed. The vendor of a CRM, a helpdesk system or an office suite switches on an assistant as part of an update, and the company learns about it from a newsletter. That is why the inventory needs refreshing at every review of SaaS vendor contracts.
The third category is tools employees use on their own: chatbots on private accounts, browser extensions, meeting transcription apps. This layer is called shadow AI, and detecting it from DNS, proxy and CASB logs is covered in detail in the guide on how to detect and reduce shadow AI.
For every inventory entry it is worth recording at least: the business owner, the technical owner, the model vendor, the categories of data the system processes, the authentication method, integrations with other systems and the company’s role under the AI Act. The last field comes back in the section on legal requirements. The register can live in an existing CMDB — what matters more than the tool is that there is one register and that someone is responsible for keeping it current.
The inventory immediately suggests the order of further work. Start with systems that meet at least two of three conditions: they process confidential or personal data, they are available to customers or partners outside the company, and they can perform operations in other systems rather than just answer. An internal assistant with no document access is a different risk class from a customer service chatbot connected to the CRM. Simple criteria like these prevent a situation in which the security team spends a quarter analysing the most visible project while the riskiest one runs without oversight.
How do you control identity, API keys and permissions in AI systems?
AI systems must be brought under the same identity management as other applications, with extra attention to three elements: API keys for model services, service accounts used by integrations, and the permissions with which the model reaches data. This domain produces the most incidents that have nothing in common with AI except the name.
An API key for a model service works like a password to an account with a budget. Whoever holds it can send requests at the company’s expense, and sometimes also read conversation history or files uploaded to the service. Keys must be kept in a secrets vault, not in code or configuration files, rotated on a schedule and immediately after the departure of anyone who knew them, and repositories should be scanned for leaked secrets. Separate keys for test and production environments also make analysis easier when something goes wrong.
Service accounts are the second trap. An integration that pulls documents into a search index or performs operations in a ticketing system usually runs on a technical account. Such an account often receives permissions “just in case”, because that was faster at first launch. The principle of least privilege applies here more than anywhere else, because a service account is used automatically, around the clock and without a human who would notice an unusual request.
The third element is the permissions with which the model sees data. An assistant that answers from documents should see only what the person asking has access to — not everything the technical account can see. If the application architecture does not ensure this, every user of the assistant indirectly gains access to the whole index. The same applies to off-the-shelf office assistants: they inherit the user’s permissions, so excessive sharing in file repositories becomes visible in their answers.
Permissions of AI service accounts and integrations should be reviewed in the same cycle as privileged access, not once at launch. AI projects grow quickly: an integration that was meant to read one document library is serving three more a few weeks later, and the technical account keeps every permission granted along the way.
Finally, user identity. Access to the admin consoles of AI services, application-building platforms and model vendor portals requires multi-factor authentication and sign-in through the company identity provider. An account created with a private email address on a model vendor’s portal is an account the company cannot disable when the employee leaves.
How do you protect the data that reaches AI models and assistants?
Data is protected at three points: at input, meaning prompts and uploaded files; in the indexes and knowledge bases the model uses; and wherever AI services store conversation history and logs. At each of these points the same foundation applies: knowing which data is sensitive, and rules on where it may be sent.
The starting point is data classification. A company that has not labelled contracts, personal data and source code will not configure meaningful rules for AI, because the rules have nothing to refer to. Sensitivity labels and data flow controls that already work for email and cloud storage should be extended to AI services: block sending data labelled as confidential to unapproved tools, warn when large blocks of code are pasted, and record exceptions. A detailed review of such controls is in the article on how to stop data leaks to ChatGPT and AI assistants.
The second point is indexes and knowledge bases. A RAG application, meaning one that combines a model with search across company documents, is only as secure as the documents let into it. If HR files or contracts with a single client have made it into the index, the assistant may quote them to someone who should never see them. The scope of the index must therefore be set deliberately, and document permissions must be carried over into the search layer.
A separate issue is the integrity of the data models use. A maliciously altered document in a knowledge base or a manipulated dataset for fine-tuning a model is data poisoning. CISA, the NSA, the FBI and international partners described the risks arising from data security and integrity issues in AI Data Security: Best Practices for Securing Data Used to Train & Operate AI Systems, published on 22 May 2025, and stress that they span all phases of the AI lifecycle. In practice this means controlling who can add and change documents in RAG sources, and logging those changes.
The third point is retention. AI services store conversation history, uploaded files and query logs according to their own rules. The company should know where this data sits, for how long and who can access it — both on the vendor’s side and in its own systems that receive application logs.
Key takeaways:
- An AI inventory covers systems you build, services you buy and tools employees use without IT’s knowledge.
- API keys for model services need a secrets vault, rotation and repository scanning.
- An assistant must see data with the permissions of the user asking, not of the technical account.
- Without data classification, AI rules have nothing to refer to.
What risks do LLM and RAG applications carry, and how do you test them?
The best-documented list of risks for applications built on large language models is the OWASP Top 10 for LLM Applications 2025 — and it is worth adopting as the common language between the security team, developers and the testing provider. The list covers ten categories, from prompt injection to unbounded consumption of resources.
In first place is prompt injection: injecting instructions that change the model’s behaviour against the intent of the application’s creator. In the direct form, the user does it in the chat window. In the indirect form, the instruction is hidden in a document, web page or email the model processes. The indirect form is more dangerous for companies, because the attacker does not need access to the application — it is enough for their content to reach the sources the application reads.
The other categories can be grouped by what they concern. Sensitive information disclosure (LLM02) and system prompt leakage (LLM07) are confidentiality risks. Data and model poisoning (LLM04) and vector and embedding weaknesses (LLM08) concern the integrity of the sources RAG relies on. Improper output handling (LLM05) and excessive agency (LLM06) are risks that surface when the model’s answer travels further — into a browser, a database or a tool that performs operations.
| OWASP 2025 category | Test question before release | Typical control |
|---|---|---|
| LLM01 Prompt Injection | Does the content of a document or page change the assistant’s behaviour? | separating instructions from data, filters, restricting tools |
| LLM02 Sensitive Information Disclosure | Does the assistant return data beyond the user’s permissions? | permissions in the search layer, control of index scope |
| LLM04 Data and Model Poisoning | Who can add or change a document in RAG sources? | change control and a log of source modifications |
| LLM05 Improper Output Handling | Does the model’s answer reach HTML, SQL or a shell without validation? | output validation and encoding, as for user input |
| LLM06 Excessive Agency | Which operations can the model perform without human confirmation? | an allow-list of tools, confirmations for state-changing operations |
| LLM07 System Prompt Leakage | Does the system prompt contain secrets or access rules? | secrets kept out of the prompt, authorisation outside the model |
| LLM10 Unbounded Consumption | Can a single user generate an arbitrarily large cost? | request and budget limits per key |
A table like this is not filled in once. The behaviour of an LLM application depends on the model, the system prompt, the data scope and the available tools, so any change to one of these elements also changes the attack surface. Tests must therefore be repeated after a model change, a new data source or a new tool, not only before the first launch.
Functional tests alone are not enough, because they check whether the application does what it should, not whether it can be forced to do something else. That is the purpose of AI/LLM application penetration testing, in which the tester attempts prompt injection, system prompt extraction, bypassing document permissions and tool abuse. Frameworks for assessing LLM risk across the whole enterprise are covered in a separate article: LLM security — a risk assessment framework.
How do you secure AI agents and their integrations with tools?
An AI agent is secured first and foremost by limiting what it can do: the list of tools it uses, the permissions of the accounts it runs on and the operations that require human confirmation. An agent differs from an assistant in that it does not just answer but takes actions — it sends messages, changes records, runs scripts.
This difference amplifies the consequences of prompt injection. An assistant that has been fed an injected instruction can at worst say something inappropriate. An agent with access to email and the file system can carry the instruction out: send a document outside the company or delete data. OWASP calls this risk excessive agency (LLM06) and points out that its root cause is typically excessive functionality, excessive permissions or excessive autonomy. The breakdown is practical, because each cause has a different control: unnecessary functions are removed from the tool list, permissions are restricted on the account the agent runs on, and autonomy is limited through human confirmation of selected operations.
Agents connect to tools through integrations, increasingly through Model Context Protocol (MCP) servers. Each such server is new code with access to data, often pulled from a public repository. The tool description the agent reads may contain hidden instructions, and a server may change its behaviour after an update. These issues, together with authorisation and server isolation, are developed in the guide on AI agent security and MCP servers.
From the CISO’s perspective, a few rules are enough, and they can be checked in any agent project. The agent runs on a dedicated account with minimal permissions, not on a user account with full access. The list of tools and servers is approved like any other software. Irreversible operations, or operations that reach outside the company, require human confirmation. Every tool call is logged with information on which user and which conversation triggered it.
How do you assess vendors of AI models and services?
An AI vendor is assessed like any vendor processing company data, with additional questions about training models on customer data, retention of prompts and responses, processing location and subprocessors. The answers to these questions change faster than for classic cloud services, because vendors frequently change models and terms.
The most important question concerns training. Can data from prompts, files and conversations be used to train or improve the vendor’s models? The answer often differs between the consumer and business versions of the same service, and between channels: the chat interface and the API. That is why a statement that customer data is not used for training should be written into the contract or into the documentation of the service the company actually bought, not taken from marketing material.
The second question concerns retention and access. How long does the vendor store prompts and responses, can its staff review them, for example as part of abuse monitoring, and can the company shorten or switch off that retention? The third concerns location: in which regions the data is processed and whether the vendor uses subprocessors, including third-party model providers.
These questions have a legal basis. Article 28 of the GDPR requires the controller to use only processors providing sufficient guarantees of data protection. Article 21(2)(d) of the NIS2 Directive lists supply chain security among the risk-management measures, including relationships with direct suppliers and service providers. For the financial sector, Article 28 of DORA treats ICT third-party risk as an integral component of ICT risk, for which the financial entity remains fully responsible.
In practice, an AI vendor assessment ends with three documents: a security questionnaire with AI-specific questions, a data processing agreement with clauses on no training on data and on retention, and an entry in the vendor register with the date of the next review. Without that last item, a change in the vendor’s terms will go unnoticed.
It is also worth recording what happens when the vendor changes the model behind the service. A new model version may react differently to the same prompts, treat instructions hidden in documents differently and follow different data processing rules. The contract or documentation should provide for notifying the customer of such changes, and the company should treat them like any significant change to a production system: with a risk assessment and a new test of the applications that rely on the model.
How do you bring AI systems into SOC monitoring and incident response?
AI systems must be connected to the SIEM like any business application: with logs of authentication, admin activity, tool calls and — where the law and the privacy policy allow — the content of interactions. Without these logs the SOC cannot tell prompt injection from an ordinary question, and cannot reconstruct what an agent did on someone’s behalf.
The Guidelines for secure AI system development, published by the UK NCSC on 27 November 2023 and announced jointly with the US CISA, divide the AI system lifecycle into four areas: secure design, secure development, secure deployment, and secure operation and maintenance. In the last area the guidelines explicitly name logging and monitoring. That is a good argument in a conversation with a project team that wants to launch an assistant before the logs are connected.
What is worth monitoring? Unusual query volumes on a single API key, which may indicate a leaked key. Queries containing known prompt injection patterns or attempts to extract the system prompt. Tool calls by an agent outside the working hours of the user it acts for. Bulk document downloads by an integration’s technical account. Access to model vendor consoles from new locations. Some of these scenarios are classic correlation rules that only need to be extended with new log sources.
The other half of this domain is incident response. The response plan should be extended with AI-specific scenarios: a leaked API key for a model service, disclosure of data by an assistant to an unauthorised person, an agent performing an operation nobody requested, and poisoning of RAG sources. For each scenario you need to decide who can switch off the assistant or agent, how to revoke keys, how to preserve conversation logs as evidence, and when the incident must be reported under the GDPR or NIS2.
For analysing attacker techniques, MITRE ATLAS (Adversarial Threat Landscape for AI Systems) is useful: a public knowledge base of adversary tactics and techniques targeting AI systems. The SOC team can draw on it for exercise scenarios and detection rule ideas instead of inventing attacks on assistants from scratch.
Key takeaways:
- LLM application tests are repeated after every change of model, data source or tool.
- An agent runs on a dedicated account, and irreversible operations require human confirmation.
- No training on data and retention terms belong in the contract, not in the vendor’s marketing material.
- Without tool call logs the SOC cannot reconstruct what an agent did on someone’s behalf.
Which legal requirements apply to AI systems?
For AI systems in a company you need to consider the AI Act, the GDPR and — depending on the sector — NIS2 and DORA at the same time; the AI Act adds obligations that depend on the company’s role and the system’s risk category, while the other acts apply regardless of whether the system is “high-risk”. We describe the legal position as of 10 November 2026.
The AI Act, Regulation (EU) 2024/1689 as amended by Regulation (EU) 2026/1744, distinguishes roles. A provider develops an AI system and places it on the market under its own name, while a deployer uses an AI system under its authority. Most companies that buy AI services are deployers, but a company that builds its own application and offers it to customers may become a provider. The role must be determined for every inventory entry.
Dates (as of 10 November 2026, the AI Act on EUR-Lex): the general provisions, including Article 4 on AI literacy, and the prohibited practices of Article 5 have applied since 2 February 2025. The general date of application of the regulation is 2 August 2026. After the 2026 amendment, the requirements for high-risk systems apply from 2 December 2027 for systems listed in Annex III and from 2 August 2028 for systems covered by Annex I. Article 4 in its current wording requires providers and deployers to take measures to support the development of their staff’s AI literacy, without requiring them to guarantee any specific level. The full timeline of obligations is covered in a separate AI Act guide.
In Poland, the Act of 3 July 2026 on artificial intelligence systems (Journal of Laws 2026, item 1003) is in force, establishing the Commission for the Development and Security of Artificial Intelligence (Komisja Rozwoju i Bezpieczeństwa Sztucznej Inteligencji) as the market surveillance authority. Roles, the system register and the approval process are handled within the company by the structure described in the article on how to organise AI governance in a company.
Regardless of the AI Act, an AI system that processes personal data is subject to the GDPR. Article 32 requires technical and organisational measures that ensure a level of security appropriate to the risk, and Article 35 requires a data protection impact assessment before processing that, in particular when using new technologies, is likely to result in a high risk. An assistant that analyses customer or employee data is a typical candidate for such an assessment. NIS2 and DORA add obligations for risk management, the supply chain and incidents, which cover AI systems just like any other ICT system.
These acts lead to a practical conclusion for governance, risk and compliance: legal requirements do not create a separate programme, they add fields to the inventory (role, risk category, personal data), evidence to the control map and scenarios to the response plan. Bringing this together in one process is the scope of AI Act readiness and AI governance.
What does the risk and control map and the 90-day plan look like?
The risk and control map pairs every AI security domain with its main risk, control, owner and the evidence that will show an auditor the control works. The table below is a starting point — in a given company some rows will split and others will merge.
| Domain | Main risk | Control | Owner | Evidence of operation |
|---|---|---|---|---|
| Inventory | AI system outside IT and security’s knowledge | register of AI systems, services and tools with the AI Act role | CISO with business owners | register with review date |
| Identity and keys | leaked API key, over-privileged service account | secrets vault, rotation, least privilege, MFA for consoles | system’s technical owner | rotation report, repository scan result |
| Data | confidential data in prompts and indexes | classification, labels, data flow controls, RAG index scope | data owner and security team | rules and exception log |
| LLM and RAG applications | prompt injection, data disclosure | tests against the OWASP Top 10 for LLM Applications 2025, output validation | product team | test and retest report |
| Agents | operation performed without a request | tool list, dedicated account, human confirmations | agent’s technical owner | configuration and tool call logs |
| Vendors | training on data, retention, subprocessors | AI questionnaire, data processing agreement, periodic review | procurement with the CISO and DPO | contract and vendor register entry |
| Monitoring | no visibility of misuse | logs in the SIEM, correlation rules for AI | SOC | rules and sample alerts |
| Response | chaos during an incident with an assistant or agent | AI scenarios in the response plan, exercises | incident response team | exercise report |
| Compliance | role and obligations not determined | AI Act role, GDPR DPIA, mapping to NIS2 and DORA | DPO and legal | role assessment and DPIA |
The 90-day plan arranges these rows in the order that produces visible results fastest. Days 1–30 are about visibility: an inventory of the three categories, assigning owners, determining roles under the AI Act and a list of vendors with contracts to review. Days 31–60 are about baseline controls: moving API keys into the vault, reviewing service accounts, extending data protection rules to AI services, connecting the logs of the most important systems to the SIEM. Days 61–90 are about verification: penetration tests of LLM applications that process sensitive data, an exercise of an incident scenario with an assistant, completing vendor contracts and the first review of the map with the board.
The order is not accidental. Tests before the inventory produce a report on one application while the risk lies in five others nobody knew about. Monitoring before the keys and service accounts are in order produces alerts nobody can handle. That is why the first 30 days are spent on visibility alone, even if the project team is pushing for faster testing.
How can nFlo help secure AI environments?
nFlo supports this programme at the three points where companies most often lack an independent assessment: visibility of AI use, application testing and regulatory order. We do not replace the teams that build or buy AI systems — we provide evidence on which those teams and the CISO can base their decisions.
The first service is the shadow AI assessment and AI use policy: establishing which AI tools employees use, what data reaches them and which data protection controls work, and on that basis an AI use policy. This feeds directly into the inventory and the data domain of the control map.
The second is AI/LLM application penetration testing, which checks prompt injection, system prompt leakage, the boundaries of document permissions in RAG and tool abuse, with a report mapped to the OWASP Top 10 for LLM Applications and a retest after fixes. The third is AI Act readiness and AI governance: determining roles, risk categories and obligations, and a register of AI systems. All three areas are brought together on the AI security page, and an overview of the articles in this cluster is in the AI security knowledge base. If you want to decide which domain to start with in your company, contact us.
What else is worth knowing about AI security in a company?
Where should a company start with AI security?
With an inventory: a list of the AI systems the company builds or buys, the AI features switched on in the SaaS applications it uses, and the tools employees use on their own. Without an inventory you cannot assign owners, assess risk or check whether your controls cover a given system at all.
Does AI security require new tools?
Usually not at the start. Most controls are existing cybersecurity mechanisms: identity management, secrets vaults, data classification, data flow controls, SIEM logging and incident response procedures. What is new is the scope and the scenarios those mechanisms must cover, such as prompt injection or excessive agent autonomy.
Who in the company is responsible for the security of AI systems?
Every AI system should have a business owner, responsible for its purpose and data, and a technical owner, responsible for configuration and changes. The CISO owns the control framework and risk oversight, and the data protection officer owns GDPR compliance of personal data processing.
How often should LLM-based applications be tested?
Before they are first made available to users and after every significant change: a new model, a new RAG data source, a new tool for an agent or a change to the system prompt. Model behaviour depends on the whole configuration, so a test result from before a change does not confirm security after it.
Does the AI Act require securing every AI system in the company?
The AI Act places its most far-reaching requirements on high-risk systems, and some obligations depend on the company’s role: provider or deployer. Regardless of the AI Act, security obligations follow from the GDPR, NIS2 and DORA, so the controls in this guide apply to every system that processes company data.
