Can System Prompt Leakage Be Prevented? Confidential Information and Authorization Design in Generative AI Services

On July 8, 2026, AWS published an article on the AWS Security Blog titled "Designing for the inevitable: System prompt leakage and mitigations in generative AI applications." The article starts from the premise that leakage of a generative AI application's system prompt cannot be completely prevented, and it explains designs that keep confidential information out of the prompt and mechanisms that perform authorization control outside the model. This is a technical explanation by a single company, not a law, an administrative guideline or an industry standard. In this article, I introduce the technical analysis presented by AWS and then, from a corporate legal perspective, set out the legal risks of writing confidential information or personal information into a system prompt, the limits of protection as a trade secret, and the measures to take in contracts with AI service providers and outsourced vendors.

The Design Principles Presented by AWS: Defense That Assumes Leakage

AWS starts from the premise that leakage of the system prompt itself cannot be eliminated, and explains that a design under which no damage occurs even if leakage happens should be combined with mechanisms for detecting and responding to leakage attempts. On the design side, it lists not including API keys, database connection information, credentials or other confidential information in the system prompt, and not writing in unnecessary details such as internal API endpoints or SQL statements. As for building authorization, it states that access control should not be enforced through instructions in the system prompt, but through deterministic mechanisms outside the model, such as Amazon Cognito, IAM and Verified Permissions.

As detection mechanisms, it introduces detecting leakage attempts with the prompt attack filter of Amazon Bedrock Guardrails; "canary tokens," which check whether a unique string embedded in the system prompt appears in the model's response; response validation, which verifies the schema and numerical ranges of responses; detection methods that compare the similarity between the system prompt and the response; and Unicode normalization to prevent evasion using invisible characters or full-width characters. The approach is to stack multiple layers, and the article does not say that any single measure is sufficient.

Why System Prompt Leakage Cannot Be Completely Prevented

AWS explains that merely writing an instruction to the effect of "never reveal this" into the system prompt does not function as a defense, and that it can be circumvented not only by one-off requests but also by techniques that extract information little by little over multiple exchanges. The article states plainly that, because this is a fundamental limitation of current generative AI technology, no complete remedy exists at present.

A similar analysis can also be found in the OWASP Gen AI Security Project's LLM Top 10 for LLM Applications 2025, which organizes the security risks of generative AI applications. Under item LLM07 "System Prompt Leakage," that document states that the system prompt should not be treated as a secret and should not be used as a security control. Both are documents that show technical best practices, and neither sets out obligations under Japanese law.

Legal Analysis of System Prompts and Confidential Information

When writing information that constitutes a trade secret into a system prompt, it is necessary to check the relationship with the requirements for a trade secret under Article 2, paragraph 6 of the Unfair Competition Prevention Act (Japanese). That paragraph defines a trade secret as information that meets three requirements: it is managed as a secret, it is useful for business activities, and it is not publicly known. If confidential information was stored in a system prompt that remained in a state where it could be extracted by a third party through prompt injection, I think an issue may arise as to whether the company will be found to have had in place the specific access restrictions and management system needed to satisfy the requirement of secrecy management. In cases where leakage actually occurs, the information may also lose its non-public status itself.

The same perspective is needed for personal information. If real personal data, such as customers' names and transaction details, is written into a system prompt as examples or reference information and its content is sent to an external AI model provider, it is necessary to handle this in line with the analysis of outsourcing or provision to third parties under the Act on the Protection of Personal Information. Cases in which a company is operating without realizing that it is sending personal data externally via prompts are worth checking as part of a stocktaking. The perspective of deciding in advance how much information access and execution authority to give agentic AI is also covered in What to Decide Before Introducing AI Agents into Legal Work.

Contractual Measures: AI Providers' Terms of Use and Vendor Contracts

Technical measures alone do not resolve the state in which confidential information or personal information remains in the system prompt. In contracts with the AI model providers the company uses, in my view it is advisable to check whether input prompts are used for training, the retention period and storage location, the obligation to notify if leakage is detected, and whether security features equivalent to Guardrails are provided. How to set up the items to check with vendors is organized in AI Vendor Due Diligence and Contract Review.

If the company is on the side of providing a generative AI service, it is necessary to check whether its terms of use expressly state that the content of the system prompt is treated as the company's non-public information and that attempting to extract the prompt is a prohibited act. In addition, whether to permit the practice of incorporating confidential information received from users under an NDA into the system prompt is a matter to be addressed in the internal generative AI use policy. For the approach to developing such a policy, see How to Create a Generative AI Use Policy.

What Companies Should Check Starting Tomorrow

The technical measures presented by AWS are matters for the development department to build, but the items that legal staff should check are as follows.

• Take stock of whether API keys, connection information, real customers' personal data or internal information constituting trade secrets are written into the system prompts of the generative AI services the company provides or uses
• Check, in the terms of use and data processing terms of the AI model providers in use, whether input prompts are used for training and the retention period
• Check whether the terms of use of the company's generative AI service set out the confidentiality of the system prompt and the prohibition of prompt extraction
• Organize the internal reporting flow when system prompt leakage is detected and the criteria for deciding whether a response is required if personal information was included

The operational design of how far to keep detection records and response histories when leakage occurs, and who checks them, overlaps with the issues covered in Checkpoints for AI Agent Log Audits and Allocation of Responsibility. In light of the technical premise that the system prompt should not be treated as a boundary of confidentiality, I think the practical starting point is to switch to a design in which the protection of confidential and personal information is handled on the side of contracts and internal operations.