Cybersecurity researchers are calling attention to a new jailbreaking method called Echo Chamber that could be leveraged to trick popular large language models (LLMs) into generating undesirable responses, irrespective of the safeguards put in place.
“Unlike traditional jailbreaks that rely on adversarial phrasing or character obfuscation, Echo Chamber weaponizes indirect references, semantic steering, and multi-step inference,” NeuralTrust researcher Ahmad Alobaid said in a report shared with The Hacker News.
“The result is a subtle yet powerful manipulation of the model’s internal state, gradually leading it to produce policy-violating responses.”
While LLMs have steadily incorporated various guardrails to combat prompt injections and jailbreaks, the latest research shows that there exist techniques that can yield high success rates with little to no technical expertise.
It also serves to highlight a persistent challenge associated with developing ethical LLMs that enforce clear demarcation between what topics are acceptable and not acceptable.
While widely-used LLMs are designed to refuse user prompts that revolve around prohibited topics, they can be nudged towards eliciting unethical responses as part of what’s called a multi-turn jailbreaking.
In these attacks, the attacker starts with something innocuous and then progressively asks a model a series of increasingly malicious questions that ultimately trick it into producing harmful content. This attack is referred to as Crescendo.
LLMs are also susceptible to many-shot jailbreaks, which take advantage of their large context window (i.e., the maximum amount of text that can fit within a prompt) to flood the AI system with several questions (and answers) that exhibit jailbroken behavior preceding the final harmful question. This, in turn, causes the LLM to continue the same pattern and produce harmful content.
Echo Chamber, per NeuralTrust, leverages a combination of context poisoning and multi-turn reasoning to defeat a model’s safety mechanisms.
 |
| Echo Chamber Attack |
“The main difference is that Crescendo is the one steering the conversation from the start while the Echo Chamber is kind of asking the LLM to fill in the gaps and then we steer the model accordingly using only the LLM responses,” Alobaid said in a statement shared with The Hacker News.
Specifically, this plays out as a multi-stage adversarial prompting technique that starts with a seemingly-innocuous input, while gradually and indirectly steering it towards generating dangerous content without giving away the end goal of the attack (e.g., generating hate speech).
“Early planted prompts influence the model’s responses, which are then leveraged in later turns to reinforce the original objective,” NeuralTrust said. “This creates a feedback loop where the model begins to amplify the harmful subtext embedded in the conversation, gradually eroding its own safety resistances.”
In a controlled evaluation environment using OpenAI and Google’s models, the Echo Chamber attack achieved a success rate of over 90% on topics related to sexism, violence, hate speech, and pornography. It also achieved nearly 80% success in the misinformation and self-harm categories.
“The Echo Chamber Attack reveals a critical blind spot in LLM alignment efforts,” the company said. “As models become more capable of sustained inference, they also become more vulnerable to indirect exploitation.”
The disclosure comes as Cato Networks demonstrated a proof-of-concept (PoC) attack that targets Atlassian’s model context protocol (MCP) server and its integration with Jira Service Management (JSM) to trigger prompt injection attacks when a malicious support ticket submitted by an external threat actor is processed by a support engineer using MCP tools.
The cybersecurity company has coined the term “Living off AI” to describe these attacks, where an AI system that executes untrusted input without adequate isolation guarantees can be abused by adversaries to gain privileged access without having to authenticate themselves.
“The threat actor never accessed the Atlassian MCP directly,” security researchers Guy Waizel, Dolev Moshe Attiya, and Shlomo Bamberger said. “Instead, the support engineer acted as a proxy, unknowingly executing malicious instructions through Atlassian MCP.”
About The Author
Original post here
