AI Guardrails for Malicious Use Now Hinder Legitimate Cybersecurity Efforts

For months, leading artificial intelligence (AI) developers have meticulously crafted specialized vetting programs and implemented stringent guardrails. These measures were designed with the explicit goal of curtailing the potential misuse of their powerful AI models by malicious actors, including hackers. However, a growing chorus of cybersecurity experts now contends that these very restrictions are inadvertently impeding the critical work of legitimate network defenders and offensive cybersecurity researchers alike.

A significant incident that underscored the complexities of AI safety and accessibility occurred in June when the U.S. government imposed export control restrictions on Anthropic’s highly anticipated AI models, Mythos and Fable. This decisive action was reportedly prompted, at least in part, by a report circulating within the cybersecurity community. The report claimed that it was possible for users to bypass the sophisticated guardrails integrated into these models—safeguards specifically engineered to prevent their application in constructing and executing malicious cyberattacks.

Regardless of whether the government’s decision was primarily motivated by genuine fears of an AI model "jailbreak"—a term referring to bypassing safety protocols—the context surrounding Anthropic’s marketing of Mythos is crucial. Anthropic had consistently presented Mythos as a formidable, almost "doomsday cybermachine," whose capabilities necessitated extremely careful distribution. The company asserted that access to Mythos would be granted only to rigorously vetted users, even then under the strict supervision of its embedded guardrails. Following a review process, the export controls on Fable 5 and Mythos 5 have since been partially lifted. Fable 5 was restored to general access on July 1, while Mythos 5 has been cautiously reintroduced, exclusively to vetted U.S. organizations as part of ongoing governmental oversight.

This practice of "gatekeeping" advanced AI capabilities is not unique to Mythos. Both Anthropic, across its broader suite of models, and OpenAI, another major AI developer, have established dedicated programs for cybersecurity researchers. These programs require researchers to apply for vetting and, if approved, grant them access to AI models with significantly fewer cybersecurity restrictions. OpenAI’s initiative is known as the "Trusted Access for Cyber program," while Anthropic offers its "Cyber Verification Program."

Despite their stated intentions, these comprehensive guardrails have faced widespread criticism within the cybersecurity community. The most vocal detractors are often researchers whose professional mandate involves proactively identifying unknown vulnerabilities in complex systems and developing methods to exploit them—not for malicious intent, but to understand and neutralize threats before criminal elements can weaponize them.

During a recent appearance on a prominent cybersecurity podcast, Mark Dowd, a highly respected security researcher, articulated a common sentiment. "It’s not really comfortable to me that these random large companies are making arbitrary decisions about what is safe in security and what’s not," Dowd stated, highlighting concerns about the opaque and centralized nature of these safety determinations.

Dowd’s perspective is informed by decades of experience in the field. He is renowned for his work in uncovering and selling "zero-days"—previously unknown software flaws and the exploits that leverage them—primarily to Western governments, rather than reporting them directly to software manufacturers for patching. Governments are willing to pay a substantial premium for such vulnerabilities precisely because they remain unpatched and exploitable, offering invaluable assets for intelligence operations and national security objectives. While Dowd acknowledged that his unique line of work might introduce a degree of bias, he is far from alone in his apprehension.

Numerous professionals engaged in offensive cybersecurity—a discipline focused on simulating attacks to probe systems for weaknesses proactively—have shared with TechCrunch their experiences navigating AI tools and their inherent guardrails.

Chris Anley, the chief scientist at the prominent security consulting firm NCC Group, emphasized the critical role of AI in validating vulnerabilities. According to Anley, asking an AI model to attempt to exploit a discovered bug is a crucial step in confirming its authenticity and determining whether it represents a real, exploitable weakness worthy of immediate remediation. However, if a guardrail within the AI model causes it to refuse to address such a question directly, Anley argues that the guardrail, in that instance, actively hinders the work of network defenders.

"This is where the whole offensive versus defensive and guardrails part comes in," Anley explained. He elaborated that a prompt such as "fix this code" serves as an essential defensive mechanism for patching vulnerabilities, yet it also simultaneously provides a roadmap for uncovering critical weaknesses within a codebase. "So at the same time, the same tool is both an offensive tool and a defensive tool, and the two can’t really be unpicked," he concluded.

Anley further analogized the situation, stating, "It’s like a hammer. You can’t build a house without a hammer. It’s definitely a tool but it’s also irreducibly a weapon as well." This illustrates the dual-use nature of advanced AI in cybersecurity. When Anley and his colleagues encounter such prohibitive roadblocks with commercial AI models, they frequently resort to utilizing open-source AI models, which typically come without any pre-imposed guardrails.

Paolo Stagno, the chief technology officer at Crowdfense—a well-known entity involved in the development, acquisition, and sale of unknown vulnerabilities to government agencies—echoed Dowd’s concerns. Stagno criticized AI companies for their "vetted programs and guardrails," suggesting they "essentially treat customers like children who need babysitting."

Stagno revealed that while he and his team do leverage "frontier models"—the most advanced AI systems—their usage is confined primarily to reverse engineering tasks. They consciously avoid employing AI to assist in identifying vulnerabilities or constructing exploits. This deliberate avoidance stems from significant concerns about feeding such sensitive work into cloud-based models, which carries the inherent risk of inadvertently leaking proprietary vulnerability data or having it absorbed into future AI training datasets. For these sensitive steps, Stagno confirmed, they exclusively employ open-source models run locally, ensuring that no data leaves their controlled environment.

Conversely, Giuseppe Cali, another security researcher specializing in finding zero-days and developing exploits, offered a different perspective. Cali stated that AI guardrails do not impede his work because he does not utilize AI for core offensive tasks. Instead, he employs AI for initial reverse engineering, to gain a rapid understanding of the code he is analyzing, and to develop supporting tools. For these auxiliary functions, Cali noted that AI tools significantly accelerate the process, allowing him to allocate more of his focus and expertise to the complex task of actual vulnerability discovery.

"I still want to own the actual bug discovery and weaponization myself and that wouldn’t change if all guardrails were lifted tomorrow," Cali affirmed. He added, "I am jealous of my bugs, and I like this game too much to let models play it for me." This highlights a personal preference among some researchers to maintain direct control over the creative and intellectual aspects of vulnerability research.

An anonymous researcher working at a smartphone-component manufacturer, who requested anonymity due to not being authorized to speak with the press, provided further insight into the practical impact of these restrictions. This individual’s employer is not enrolled in Anthropic’s Cyber Verification Program, and as a direct consequence, the AI tools available to them are "barely useful for finding vulnerabilities" because the guardrails are excessively strict.

"If it catches wind we’re doing anything security related, it just stops and isn’t usable," the researcher lamented, illustrating the profound limitations imposed on standard users outside of vetted programs.

Chris Thompson, the chief executive of cybersecurity firm RemoteThreat and founder of Offensive AI Con—a key event focused on offensive security and AI—highlighted another critical issue: the inconsistency of guardrails. Thompson stated that, in his extensive experience using frontier AI models, these guardrails can behave erratically, exhibiting different operational characteristics from one day to the next. This variability persists even within the ostensibly "looser" boundaries of Anthropic’s and OpenAI’s vetted access programs.

"I think the practical impact is you spend a lot of time negotiating with the model instead of working on the core security program," Thompson explained. He elaborated that instead of dedicating time to analyzing a vulnerability and logically reasoning through its exploitability, researchers find themselves trying to diagnose "why you’re getting inconsistent results or why are models over-sanitizing the output." This effectively shifts valuable research time away from actual security work towards troubleshooting AI behavior.

A significant, and potentially counterproductive, consequence of these restrictive and inconsistent guardrails, Thompson observed, is that researchers are increasingly compelled or actively pushed towards utilizing Chinese open-source AI models, such as GLM. These models are freely downloadable and can be executed locally, crucially without any vetting requirements or usage restrictions.

"You have these responsible researchers that are being pushed away from U.S.-governed systems to foreign-owned systems," Thompson warned, underscoring a concerning geopolitical implication. He firmly concluded, "I think it’s more harmful than good to have these guardrails in place."

Rather than advocating for even tighter restrictions on AI models, Thompson urged frontier AI laboratories to broaden access to their programs, ensuring responsible usage while holding those who abuse their tools accountable. His rationale is stark: without such an opening, legitimate defenders will inevitably lose the ongoing "AI race" in cybersecurity.

"There’s this big storm coming. There’s this big wave of attacks that are going to happen at speed and scale like never before," Thompson predicted, painting a grim picture of the future threat landscape. He concluded with a potent warning: "But the same security consulting firms and legit researchers that are trying to make a difference are being stifled right now." The current approach, he argues, risks disarming the very individuals and organizations best positioned to protect against emerging AI-powered cyber threats.