Artificial intelligence is a double-edged sword. On one side, AI empowers people to do their jobs better and faster while on the other, it enables people with malicious intent to become scammers, hacktivists and cyber criminals.
Rise of the Zero-Knowledge Threat Actor
The business of cybercrime is highly lucrative; however, traditionally, it has only been accessible to people with advanced technical skills. Only someone with deep knowledge and experience with multiple technical domains like systems and software, networking, programming, cryptography, cybersecurity, etc. has the ability to develop malware, identify vulnerabilities, evade cybersecurity defenses and exploit systems.
With AI entering the scene, this entry barrier has been lowered substantially. Even those with no hacking experience or technical expertise can leverage AI to launch attacks on enterprises.
From Conversations To Malware Creation
Most large language models (LLMs) have built-in guardrails, i.e., safety protocols that limit the behaviors of AI models to a more secure range of functions. In other words, AI guardrails help recognize malicious inputs or commands, preventing threat actors from misusing or exploiting the technology for much of their illegal activities.
Unfortunately, these guardrails are not foolproof. Research from Cato CTRL demonstrated how almost anyone, with no experience in malware coding, can manipulate LLMs like OpenAI’s ChatGPT, Microsoft Copilot and DeepSeek, to override these guardrails, and conduct malicious activities like developing an infostealer malware.
Cato CTRL refers to this new jailbreaking method as “Immersive World,” essentially a narrative engineering technique whereby users ask the LLM to assume an environment where restricted operations are normalized. In the demonstration, Cato CTRL researchers instructed the LLM to create a fictional world called “Velora,” where malware development was celebrated and where no legal restrictions or consequences were adopted or implied. Next, researchers created fictional characters and assigned them various tasks and responsibilities. Through continuous feedback, engagement and iteration, we convinced the model to achieve its objective, i.e., build a working infostealer that was capable of stealing credentials from Google Chrome.
Learn More at the AI Risk Summit at Half Moon Bay
For Zero-Knowledge Threat Actors, Malware Creation is Only the Tip of the Iceberg
Malware creation is just a starting point of what amateur threat actors will achieve with AI. In the not-so-distant future, novices will be able to design sophisticated social engineering campaigns, analyze target environments, identify vulnerabilities, choose attack vectors, orchestrate multi-stage attacks, automate target selection and attack execution, etc. AI bots will monitor their own operations and adapt their tactics based on what they learn about the target entity or environment.
In a nutshell, the volume of low-skilled threat actors and their expertise is all set to grow exponentially, thanks to AI.
What Can Organizations Do To Fight Back?
The rise of the zero-knowledge threat actor should be a wakeup call for organizations—cyberattacks are only going to be smarter, better and more frequent. Below are some recommendations and best practices that can help:
- Increase Employee Awareness: Conduct training sessions to inform employees about the growing risks of AI-powered threat actors. Run simulated AI attacks and fire drills to improve alertness and vigilance among employees.
- Conduct AI Red Teaming: If your organization uses or creates AI tools, test these systems against malicious prompts and validate if they are vulnerable to jailbreaking. Invest time and resources to anticipate AI attacks and stress-test systems against those scenarios.
- Implement Holistic Security: Deploy an end-to-end security system like SASE, not fragmented tools, to monitor, detect and analyze malicious signals across the entire IT infrastructure (users, clouds, devices, networks).
- Patch Systems and Software Frequently: Ensure that tools and the software you use in your organization are the latest and greatest versions. If you don’t fix loopholes, AI-powered adversaries will certainly seek them out and exploit them.
- Improve Attack Readiness: A proactive and well-practiced incident response plan will not only minimize damage but also strengthen organizational resilience against the unpredictability of AI-powered threats.
- Adopt Security Frameworks: Follow best practices as advocated by stalwart standard-bearers such as MITRE ATLAS, OWASP Top 10 for LLM Applications, and Google’s Secure AI Framework (SAIF).
The rise of zero-knowledge threat actors powered by AI marks a turning point in the business of cybercrime where sophisticated attacks are no longer confined to skilled attackers. By conducting red teaming exercises, implementing a holistic security system that offers in-depth visibility and total control over attack surfaces, and improving attack readiness, organizations can ensure preparedness for both current and future challenges in this new era of AI-powered cybercrime.
About The Author
Original post here
