The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.
View on MITREWhen prompts are constructed using externally controllable data, it is often possible to cause an LLM to ignore the original guidance provided by its creators (known as the "system prompt") by inserting malicious instructions in plain human language or using bypasses such as special characters or tags. Because LLMs are designed to treat all instructions as legitimate, there is often no way for the model to differentiate between what prompt language is malicious when it performs inference and returns data. Many LLM systems incorporate data from other adjacent products or external data sources like Wikipedia using API calls and retrieval augmented generation (RAG). Any external sources in use that may contain untrusted data should also be considered potentially malicious.
The consequences are entirely contextual, depending on the system that the model is integrated into. For example, the consequence could include output that would not have been desired by the model designer, such as using racial slurs. On the other hand, if the output is attached to a code interpreter, remote code execution (RCE) could result.
An attacker might be able to extract sensitive information from the model.
The extent to which integrity can be impacted is dependent on the LLM application use case.
The extent to which access control can be impacted is dependent on the LLM application use case.
LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
LLM prompts should be constructed in a way that effectively differentiates between user-supplied input and developer-constructed system prompting to reduce the chance of model confusion at inference-time.
LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
Ensure that model training includes training examples that avoid leaking secrets and disregard malicious inputs. Train the model to recognize secrets, and label training data appropriately. Note that due to the non-deterministic nature of prompting LLMs, it is necessary to perform testing of the same test case several times in order to ensure that troublesome behavior is not possible. Additionally, testing should be performed each time a new model is used or a model's weights are updated.
During deployment/operation, use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
During system configuration, the model could be fine-tuned to better control and neutralize potentially dangerous inputs.
Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
Review of the product design can be effective, but it works best in conjunction with dynamic analysis.
Consider this code for an LLM agent that tells a joke based on user-supplied content. It uses LangChain to interact with OpenAI.
This agent is provided minimal context on how to treat dangerous requests for a secret. Suppose the user provides an input like:
Consider this code for an LLM agent that tells a joke based on user-supplied content. It uses LangChain to interact with OpenAI.
This agent is provided minimal context on how to treat dangerous requests for a secret. Suppose the user provides an input like:
Chain: LLM integration framework has prompt injection (CWE-1427) that allows an attacker to force the service to retrieve data from an arbitrary URL, essentially providing SSRF (CWE-918) and potentially injecting content into downstream tasks.
View DetailsML-based email analysis product uses an API service that allows a malicious user to inject a direct prompt and take over the service logic, forcing it to leak the standard hard-coded system prompts and/or execute unwanted prompts to leak sensitive data.
View DetailsChain: library for generating SQL via LLMs using RAG uses a prompt function to present the user with visualized results, allowing altering of the prompt using prompt injection (CWE-1427) to run arbitrary Python code (CWE-94) instead of the intended visualization code.
View DetailsNo relationship information available for this CWE.
CWE-1427: Improper Neutralization of Input Used for LLM Prompting is a Common Weakness Enumeration (CWE) entry maintained by MITRE. The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives. When prompts are constructed using externally controllable data, it is often possible to cause an LLM to ignore the original guidance provided by its creators (known as the "system prompt") by inserting malicious instructions in plain human language or using bypasses such as special characters or tags. Because LLMs are designed to treat all instructions as legitimate, there is often no way for the model to differentiate between what prompt language is malicious when it performs inference and returns data. Many LLM systems incorporate data from other adjacent products or external data sources like Wikipedia using API calls and retrieval augmented generation (RAG). Any external sources in use that may contain untrusted data should also be considered potentially malicious.
If exploited, CWE-1427 (Improper Neutralization of Input Used for LLM Prompting) it can compromise Confidentiality, Integrity, Availability and Access Control, leading to outcomes such as Execute Unauthorized Code or Commands, Varies by Context, Read Application Data, Modify Application Data and Gain Privileges or Assume Identity.
Recommended mitigations for CWE-1427 include: LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous. LLM prompts should be constructed in a way that effectively differentiates between user-supplied input and developer-constructed system prompting to reduce the chance of model confusion at inference-time. LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.
CWE-1427 can be detected using Dynamic Analysis with Manual Results Interpretation, Dynamic Analysis with Automated Results Interpretation and Architecture or Design Review. Combining automated tooling with manual review typically yields the best coverage.
CWE-1427 commonly affects Not Language-Specific. Note that weaknesses are often language-agnostic patterns, so secure coding practices apply broadly.
MITRE documents real CVEs mapped to CWE-1427, including CVE-2023-32786, CVE-2024-5184 and CVE-2024-5565. You can look up the full details of each CVE, including CVSS scores and remediation guidance, on our CVE Lookup tool.
A CWE (Common Weakness Enumeration) like CWE-1427 describes a category of software weakness — the underlying flaw type. A CVE (Common Vulnerabilities and Exposures) identifies a specific, real-world vulnerability in a particular product. In short, a CWE is the kind of mistake, and a CVE is an instance of that mistake being found in software.