Adversarial machine learning (AML) represents one of the most sophisticated threats to AI systems today.In simple terms, AML is the art and science of manipulating AI systems to behave in unintended ways. The methods through which AML can lead to harmful outcomes are limited only by the imagination and technical skill of criminal and hostile nation-state actors.
These attacks are not theoretical, and the stakes are only getting higher, as AI systems become more pervasive across critical infrastructure, military applications, intelligence operations, and even everyday technologies used by billions of people. In short: a compromised AI could result in anything from a minor inconvenience to a catastrophic security breach.
The intersection of technology, defense, space and intelligence is critical to future U.S. national security.Join The Cipher Brief on June 5th and 6th in Austin, Texas for the NatSecEDGE conference. Find out how to get an invitation to this invite-only event at natsecedge.com
Unlike traditional cybersecurity concerns, adversarial AI attacks operate in a realm most people cannot visualize, an abstract mathematical space where machine learning systems interpret our world. These attacks not only breach digital defenses, but they also manipulate how AI perceives reality itself.
Imagine a financial institution deploying an AI-powered loan approval system, trained on decades of lending data. Unknown to the bank, an insider has subtly manipulated that training data – not enough to raise alarms, but enough to create a hidden bias. Months later, when the system is operational, it systematically rejects qualified applicants from certain neighborhoods while approving less qualified candidates from others. This is data poisoning, a form of AML attack that changes how the AI evaluates risk.
Or consider an autonomous military drone on a reconnaissance mission. The drone’s vision system has been carefully trained to distinguish friend from foe. Yet an adversary has discovered that placing a specific pattern on their vehicles, even one invisible to human observation, causes the drone to consistently misclassify them as civilian infrastructure. This “evasion attack” requires no hacking whatsoever. It simply exploits the way in which the AI interprets visual information.
The vulnerabilities run deeper still. In a landmark 2020 paper, experts demonstrated how attackers could effectively “steal” commercial facial recognition models. Through a technique called “model inversion,” they were able to extract the actual faces used to train the system simply by querying it strategically. In essence, they recovered recognizable images of specific individuals, revealing how AI systems can inadvertently memorize and expose sensitive training data.
The emergence of large language models has introduced entirely new attack surfaces. While most commercial models make a concerted effort to place guardrails on their use, that is not always the case with open-source models, opening up the opportunity for manipulation and harmful (even illegal) outputs. Indeed, seemingly innocuous prompts can trigger systems to generate dangerous content, from malware code to instructions for illegal activities. Prompt injection has become widely recognized as the top risk for LLM applications.
These are no longer hypothetical scenarios at the edge of technological knowledge. They are documented vulnerabilities being actively researched and, in some cases, exploited. What makes these threats particularly insidious is their capacity to compromise systems without changing a single line of code. The AI continues to function normally in most circumstances, making these changes all but invisible to traditional cybersecurity monitoring.
While these threats affect all AI applications, the national security implications stand out as particularly alarming. Across the U.S. national security landscape, agencies and departments have increasingly flagged adversarial machine learning as a critical vulnerability in military and intelligence operations. Gone are the days when US national security organizations only worried about a capable and sophisticated adversary stealing their sensitive data. Today, they must also worry about an adversary manipulating how machines interpret that data.
Imagine a scenario where an adversary subtly manipulates AI systems supporting intelligence analysis. Such an attack might cause these systems to overlook critical patterns or generate misleading conclusions, something quite difficult to detect yet potentially devastating for decision-making at the highest levels of government. This is no longer science fiction; it’s a growing concern among security professionals who understand how AI vulnerabilities translate to national security risks.
These concerns become even more urgent as the global race for Artificial General Intelligence (AGI) accelerates. The first nation to achieve AGI will undoubtedly gain an unprecedented, once-in-a-lifetime strategic advantage, but only if that AGI can withstand sophisticated adversarial attacks. A vulnerable AGI might prove worse than no AGI at all.
Despite these mounting threats, our defensive capabilities remain woefully inadequate. Researchers from the National Institute of Standards and Technology (NIST) captured this reality bluntly in 2024, noting that “available defenses currently lack robust assurances that they fully mitigate the risks.” This security gap stems from several interconnected challenges that have allowed adversarial threats to outpace our defenses.
From AI to unmanned systems, experts are gathering at The Cipher Brief’s NatSecEDGE conference June 5-6 in Austin, TX to talk about the future of war and national security. Be a part of the conversation.
The problem is fundamentally an asymmetrical one. Attackers need find only a single vulnerability, while defenders must protect against all possible attacks. Adding to this challenge, effective defense requires specialized expertise bridging cybersecurity and machine learning, a rare combination in today’s workforce. Meanwhile, organizational structures often separate AI development from security teams, creating unintentional barriers that hinder effective collaboration.
Many senior leaders and stakeholders remain unaware of AI’s unique security challenges, approaching AI security with the same mindset they bring to traditional systems. This results in a predominantly reactive approach, addressing known attack vectors rather than proactively securing systems against emerging threats.
Moving beyond this reactive posture demands a comprehensive counter-AI strategy that encompasses defensive, offensive, and strategic dimensions. First and foremost, security must be woven into AI systems from the ground up, rather than as an afterthought. This requires cross-training personnel to bridge the divide between AI and cybersecurity expertise, something that is no longer a luxury but an operational necessity.
Effective defense might mean deliberately exposing models to adversarial examples during training, developing architectures inherently resistant to manipulation, and implementing systems that continuously monitor for anomalous behavior. Yet defense alone is not enough. Organizations must also develop offensive capabilities, employing red teams to pressure-test AI systems using the same sophisticated techniques potential attackers would deploy.
At the strategic level, counter-AI demands unprecedented coordination across government, industry, and academia. We need mechanisms to share threat intelligence about emerging adversarial capabilities, international standards establishing common security frameworks, and focused workforce development initiatives that build a pipeline of talent with expertise spanning both AI and cybersecurity domains. Some experts have also suggested a rigorous safety testing regime for frontier models both before deployment and throughout their lifespans. It’s a proposal heavy with political and legal dimensions, since frontier models remain the intellectual property of private companies, but some form of safety assurance is needed.
The challenges are formidable, and the stakes are high. As AI systems increasingly underpin critical national security functions, their safety becomes inseparable from our nation’s security posture. The question is not whether adversaries will target these systems. They will. But will we be ready?
Today, we stand at a crossroads. While the public’s attention remains fixed on AI’s dazzling capabilities, those of us who’ve worked behind the classified walls of national security understand that the invisible battle for AI security may prove decisive.
So where do we go from here?
The future demands more than technical solutions. It requires a fundamental shift in how we approach AI development and security. Counter-AI research needs substantial support and funding, particularly for developing adaptive defense mechanisms that can evolve alongside attack methodologies. But money is not the solution. We need to break down the organizational barriers that have traditionally separated developers from security professionals, creating collaborative environments where security becomes a shared responsibility rather than an afterthought.
As with all challenges across the digital landscape, this one is not just about technology; it’s about talent and culture. Having led a large technical workforce at the CIA, I’ve witnessed firsthand how breaking down these barriers creates not just better products, but more secure ones.
And let’s be clear about what’s at stake. The nation that masters counter-AI will likely determine whether artificial intelligence becomes a guardian of or a threat to freedom itself. This may sound like hyperbole, but it’s the logical conclusion of where this technology is headed.
When I speak of freedom in this context, as I often do in public addresses, I’m referring to something more fundamental than just democratic governance. I mean the essential liberty of citizens to make meaningful choices about their lives, access accurate information, and participate in civic processes without manipulation. An AI ecosystem vulnerable to adversarial manipulation threatens these foundational freedoms in profound ways.
Consider a world where information ecosystems are increasingly AI-mediated, yet these systems remain susceptible to sophisticated adversarial influence. In such a world, who controls the manipulation of these systems effectively controls the information landscape. The potential for mass influence operations, targeted manipulation of decision-makers, and the hidden subversion of critical infrastructure represents a serious threat vector against free societies.
A nation that masters counter-AI develops not just a technical advantage, but resistance to these forms of digital manipulation. It preserves the integrity of its information ecosystem, the reliability of its critical infrastructure, and ultimately, the sovereignty of its decision-making processes. In this sense, counter-AI becomes the shield that protects freedom in the age of artificial intelligence.
The AI race we read about so often is more than a race to build the most powerful AI. It is also a race to build resilient AI that remains faithful to human intent even under adversarial attack. This competition unfolds largely beyond public view, conducted in research labs, classified facilities, and corporate campuses around the world. Yet its outcome may prove the most consequential aspect of the broader AI revolution.
For those of us in national security, building the world’s premier counter-AI capability is a strategic imperative that will shape the balance of power for decades to come. The future belongs not to those who merely create the most capable AI, but to those who can defend it from sabotage.
It is time we recognized this silent battlefront for what it is: one of the most important technological competitions of our time. The security of artificial intelligence can no longer remain an afterthought. It must become central to our national conversation about how we build, deploy, and govern these increasingly powerful systems.
The Cipher Brief is committed to publishing a range of expert perspectives on national security issues submitted by deeply experienced national security professionals.
Opinions expressed are those of the author and do not represent the views or opinions of The Cipher Brief.
Have a perspective to share based on your experience in the national security field? Send it to Editor@thecipherbrief.com for publication consideration.
Read more expert-driven national security insights, perspective and analysis in The Cipher Brief
