The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.
View on MITREIn an agent-oriented setting, output could be used to cause unpredictable agent invocation, i.e., to control or influence agents that might be invoked from the output. The impact varies depending on the access that is granted to the tools, such as creating a database or writing files.
Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space.
Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar.
Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
During model training, use an appropriate variety of good and bad examples to guide preferred outputs.
Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.
Review of the product design can be effective, but it works best in conjunction with dynamic analysis.
chain: GUI for ChatGPT API performs input validation but does not properly "sanitize" or validate model output data (CWE-1426), leading to XSS (CWE-79).
View DetailsNo relationship information available for this CWE.
CWE-1426: Improper Validation of Generative AI Output is a Common Weakness Enumeration (CWE) entry maintained by MITRE. The product invokes a generative AI/ML component whose behaviors and outputs cannot be directly controlled, but the product does not validate or insufficiently validates the outputs to ensure that they align with the intended security, content, or privacy policy.
If exploited, CWE-1426 (Improper Validation of Generative AI Output) it can compromise Integrity, leading to outcomes such as Execute Unauthorized Code or Commands and Varies by Context.
Recommended mitigations for CWE-1426 include: Since the output from a generative AI component (such as an LLM) cannot be trusted, ensure that it operates in an untrusted or non-privileged space. Use "semantic comparators," which are mechanisms that provide semantic comparison to identify objects that might appear different but are semantically similar. Use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.
CWE-1426 can be detected using Dynamic Analysis with Manual Results Interpretation, Dynamic Analysis with Automated Results Interpretation and Architecture or Design Review. Combining automated tooling with manual review typically yields the best coverage.
CWE-1426 commonly affects Not Language-Specific. Note that weaknesses are often language-agnostic patterns, so secure coding practices apply broadly.
MITRE documents real CVEs mapped to CWE-1426, including CVE-2024-3402. You can look up the full details of each CVE, including CVSS scores and remediation guidance, on our CVE Lookup tool.
A CWE (Common Weakness Enumeration) like CWE-1426 describes a category of software weakness — the underlying flaw type. A CVE (Common Vulnerabilities and Exposures) identifies a specific, real-world vulnerability in a particular product. In short, a CWE is the kind of mistake, and a CVE is an instance of that mistake being found in software.