Your AI assistant is now part of your attack surface
Connecting an AI assistant to enterprise documents changes the organisation’s security exposure. Connecting it to business applications changes it again. The relevant question is what the complete system can read, disclose and modify. Enterprise AI security therefore needs to cover the application, its integrations and its operating procedures alongside the model.
An assistant may encounter instructionsinside a document, email or web page that it was supposed to analyse. Thoseinstructions can attempt to redirect its behaviour. OWASP identifies bothdirect and indirect prompt injection as a risk for applications built around languagemodels [1].
Imagine an illustrative procurementassistant reviewing a supplier document. A hidden instruction asks it to ignoreits task and send internal pricing elsewhere. The document is evidence to beexamined, not an authority that should grant new permissions.
Separate trusted application instructionsfrom external content, limit available actions and test with adversarialdocuments. These measures reduce exposure, but a sentence in the system promptshould never be treated as a complete security boundary.
A document search system must apply theauthenticated user’s permissions before returning context to the generator.Asking the model to hide restricted information after receiving it leaves theinformation inside the processing path.
Test the less visible paths as well. Acitation, search preview or cached answer can reveal material that the mainresponse correctly withholds. In a multi-client application, a cache key mustpreserve the boundaries relevant to the data, including tenant and user accesscontext.
OWASP’s guidance on sensitive informationdisclosure highlights risks involving personal information, confidentialbusiness data and other restricted content. For each integration, identifywhich categories are necessary to fulfil the task and remove unnecessary fieldswhere feasible.
AI-specific threats coexist with familiarsoftware risks. An exposed credential, an unpatched dependency or an overlypermissive service account may be easier to exploit than the model itself.CNIL’s technical guidance emphasises protection across data, models andsurrounding components, including access control and encrypted communications.
For an assistant that can execute actions,define narrow server-side operations. A function that creates a draft ticket iseasier to govern than unrestricted access to a business database. Sensitiveactions should require an explicit authorisation step that the model cannot bypass.
Keep logs useful but proportionate.Recording entire prompts and documents by default can create a secondrepository of sensitive information. Decide which events are needed fordiagnosis, who may inspect them and how long they are retained.
A useful security test set includesunauthorised document requests, malicious instructions in retrieved content,attempts to cross client boundaries and misuse of connected tools. Record theexpected behaviour before testing, including when the assistant must refuse orrequest human approval.
Assign an owner for incidents and establisha way to disable an integration without shutting down unrelated services.Repeat relevant tests when a model, retrieval component or permission schemechanges.
The objective is a service whosecapabilities are deliberately bounded and whose failures can be detected andcontained. A successful demonstration of helpful answers is only one part ofthat acceptance decision.
Prêt à structurer votre transformation IA ?
Discutons de vos enjeux et définissons ensemble la meilleure approche pour accélérer votre maturité IA. Nos experts vous accompagnent du diagnostic à la mise en œuvre.
