The new war room

A decade ago, AI tools started to automate the work a junior intelligence analyst might do—picking out an important signal on a social media or satellite feed from lots of noise. Systems like the US military’s Maven, built mainly on technology created by the surveillance giant Palantir, fed that sort of analysis into tools that allow commanders to select a target to bomb halfway around the world, all through clean interfaces that look more like business software than the machinery of war.

Now large language models are making these systems more interactive and capable of doling out advice. One US defense official told MIT Technology Review that today’s military personnel might give chatbots a list of potential targets to help decide which to strike first. Even though the Pentagon recently labeled Anthropic a supply chain risk, the company’s tool Claude has become so intertwined in military operations that the government says it needs six months to get rid of it. 

And it’s not just the US putting chatbots at commanders’ fingertips; China is commissioning similar tools, according to recent analysis by Georgetown University’s Center for Security and Emerging Technology. 

While older AI systems will still be used to gather and analyze information, it’s clear that AI-generated advice is something militaries around the world want more of, and take seriously.

One problem with this paradigm shift will be obvious to anyone who has used generative AI: It produces different outputs even when you use the same prompt, and its unscripted recommendations are not always useful, precise, or correct. People who use it at work are expected to vet its outputs. But if they’re under enough pressure—like, say, the need to clarify which target to strike in the next five minutes—they may cut corners. 

AI-generated errors are only part of the problem. Military experts have also warned that officers who chat with AI systems that compress the world into a neat battlefield dashboard might trust the system too much, or that such systems might give tech companies undue influence over what information gets seen. And while the military rushes to adopt an effectively brand-new technology, the public has no meaningful way to exercise oversight of its use, or to scrutinize its possible role in mistakes (indeed, a January memo from the Pentagon described “responsible AI,” which is built upon these concepts, as “utopian Idealism”). 

The OpenAIs of the world, though, see an opportunity to win lucrative defense contracts. And the Pentagon, which wants AI companies to offer the military the best models they can, even plans to allow companies to train new models on classified military data. That would mean sensitive intelligence like surveillance or battlefield assessments could go into the models themselves, presenting new security risks. It would also bring Silicon Valley closer to the Pentagon than ever before.