Insecure Setting of Generative AI/ML Model Inference Parameters

Opis

The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.

Generative AI/ML models, such as those used for text generation, image synthesis, and other creative tasks, rely on inference parameters that control model behavior, such as temperature, Top P, and Top K. These parameters affect the model's internal decision-making processes, learning rate, and probability distributions. Incorrect settings can lead to unusual behavior such as text "hallucinations," unrealistic images, or failure to converge during training. The impact of such misconfigurations can compromise the integrity of the application. If the results are used in security-critical operations or decisions, then this could violate the intended security policy, i.e., introduce a vulnerability.

Skutki

• Integrity, Other

  Varies by Context; Unexpected State

  The product can generate inaccurate, misleading, or nonsensical information.

• Other

  Alter Execution Logic; Unexpected State; Varies by Context

  If outputs are used in critical decision-making processes, errors could be propagated to other systems or components.

Zapobieganie

• Implementation, System Configuration, Operation

  Develop and adhere to robust parameter tuning processes that include extensive testing and validation.

• Implementation, System Configuration, Operation

  Implement feedback mechanisms to continuously assess and adjust model performance.

• Documentation

  Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.

Słabość nadrzędna

• CWE-440 Expected Behavior Violation
• CWE-665 Improper Initialization

Może prowadzić do

• CWE-684 Incorrect Provision of Specified Functionality

Pokrewna

• CWE-691 Insufficient Control Flow Management

Kiedy powstaje

• Build and Compilation — During model training, hyperparameters may be set without adequate validation or understanding of their impact.
• Installation — During deployment, model parameters may be adjusted to optimize performance without comprehensive testing.
• Patching and Maintenance — Updates or modifications may be made to the model that alter its behavior without thorough re-evaluation.

Dotyczy

Not Language-Specific, Not Architecture-Specific, AI/ML, Not Technology-Specific

Uwagi

This weakness might be under-reported as of CWE 4.18, since there are no clear observed examples in CVE. However, inference parameters may be the root cause for various vulnerabilities - or important factors - but the vulnerability reports may concentrate more on the negative impact (e.g. code execution) or the weaknesses that the insecure settings contribute to. Alternately, dynamic techniques might not reveal the root cause if the researcher does not have access to the underlying source code and environment.

Źródła

1. We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs — Joseph Spracklen, Raveen Wijewickrama, A H M Nazmus Sakib, Anindya Maiti, Bimal Viswanath, Murtuza Jadliwala, 2025