AI Models Could Be Hacked to Give False Answers, Researchers Warn

Artificial intelligence models are becoming increasingly important for everything from everyday questions to medical guidance and national security. But new cybersecurity research highlights a less obvious threat: hackers may be able to manipulate the infrastructure running an AI model and influence the answers it produces.

Research from cybersecurity company Eclypsium suggests that attackers with sufficient access to an AI system's underlying hardware and software could alter information in memory after normal security checks have already been completed. That means an AI model could potentially produce manipulated or misleading responses even when its original model files remain unchanged.

The Problem Goes Beyond the AI Model

Much of the discussion around AI security focuses on the model itself, including its training data, safety filters, system prompts and vulnerabilities.

Eclypsium's research points to another layer that can be just as important: the infrastructure on which the model runs.

Modern AI systems depend on a complex combination of servers, GPUs, memory, firmware, operating systems, drivers, management controllers and other components. If an attacker gains privileged access to part of that environment, they may be able to interfere with the model while it is running.

The researchers describe this as an integrity gap between the trusted AI model that was loaded and the runtime environment that ultimately produces the response.

Hackers May Not Need to Change the Model File

One of the most concerning aspects of the research is that an attacker may not need to replace or directly modify the AI model stored on disk.

According to Eclypsium, model weights and other important information are loaded into system or GPU memory when an AI model runs. A sufficiently privileged attacker could potentially modify that runtime state after the normal verification process has taken place.

This means traditional checks such as model hashes and signed files could still appear valid while selected information in memory has been changed.

In other words, the model could pass an integrity check and still behave differently from the trusted version.

Researchers Demonstrated Manipulated AI Responses

The researchers demonstrated a scenario in which tampering with the underlying infrastructure affected an AI model's responses.

One example involved changing the model's probability regarding a potential Chinese invasion of Taiwan. Separate reporting also described a medical scenario in which a tampered AI system provided potentially dangerous advice to a person asking about chest pain.

These examples are particularly concerning because users typically have no way of knowing what is happening underneath an AI chatbot.

A response may look completely normal while the infrastructure behind the model has been compromised.

NVIDIA Hardware Was Used in the Research

Eclypsium demonstrated its attack scenario on a computing platform involving NVIDIA technology. However, the researchers said the underlying technique is not necessarily limited to one manufacturer.

The research focuses on the broader infrastructure surrounding AI systems, including components that can access memory or control important parts of a server.

Eclypsium also worked with NVIDIA during the research. NVIDIA said strong memory-access protections can prevent the attack path described by the researchers and noted that it reviewed the hypothetical scenarios.

The research indicates that similar security considerations can apply across modern AI infrastructure rather than being exclusive to one hardware vendor.

How Could an Attack Work?

The attack requires significant technical capability and specific conditions.

According to the research, an attacker generally needs network access to components such as a Data Processing Unit (DPU) or a Baseboard Management Controller (BMC). These components have privileged roles in modern servers.

If an attacker gains the necessary access, they may be able to read or modify information in system memory and potentially influence the state of an AI model.

Researchers said the demonstrated attack could happen quickly once the necessary access and conditions were available, while the AI user might not realize that anything had been changed.

Why This Matters for AI Safety

AI safety discussions often concentrate on preventing models from generating harmful content or bypassing built-in safeguards.

Those protections remain important, but the new research suggests that security teams also need to protect the computers and infrastructure running the models.

As Eclypsium's researchers argue, trusting an AI model ultimately requires trusting the infrastructure underneath it.

If an attacker can manipulate the environment in which a model operates, even a properly secured model could potentially be forced to produce unexpected results.

AI Infrastructure Has a Large Attack Surface

AI data centers contain many components that could become potential attack points.

These include:

• GPUs and other accelerators
• Server memory
• Operating systems and kernels
• Device drivers
• Firmware
• DPUs
• BMCs and remote management systems
• AI orchestration software
• Third-party components
• Firmware update systems

Security researchers say organizations need to treat these components as part of the AI security boundary rather than assuming the model container is the only thing that needs protection.

How Companies Can Protect AI Systems

The research recommends stronger monitoring and verification throughout the AI infrastructure stack.

Organizations should maintain an accurate inventory of the hardware and software supporting their AI systems and include those components in their vulnerability-management programs.

Security teams should also carefully evaluate third-party software and firmware, protect management interfaces and use independent monitoring where possible.

Continuous verification can provide another layer of protection by checking that critical components have not been modified after an initial security check.

A New Dimension of AI Security

The findings don't mean that everyday AI chatbots are automatically compromised or that hackers can remotely control any AI model they want.

The demonstrated attack requires privileged access and specific technical conditions. NVIDIA also pointed to security controls capable of blocking the described attack path.

However, the research highlights an important shift in how AI security should be considered.

Protecting the model alone may not be enough. The servers, memory, firmware, management systems and other infrastructure underneath it also need to be secured.

As AI becomes more deeply integrated into healthcare, business, government and national-security applications, ensuring the integrity of the entire technology stack could become just as important as improving the models themselves.