Matthew Strohmeyer is sounding a little giddy. The US Air Force colonel has been running data-based exercises inside the US Defense Department for years. But for the first time, he tried a large-language model to perform a military task.
“It was highly successful. It was very fast,” he tells me a couple of hours after giving the first prompts to the model. “We are learning that this is possible for us to do.”
Large-language models, LLMs for short, are trained on huge swaths of internet data to help artificial intelligence predict and generate human-like responses to user prompts. They are what power generative AI tools such as OpenAI’s ChatGPT and Google’s Bard.
Five of these are being put through the paces as part of an eight-week exercise run by the Pentagon’s digital and AI office and military top brass, with participation from US allies. The Pentagon won’t say which LLMs are in testing, though Scale AI, a San Francisco-based startup, says its new Donovan product is among the LLMs platforms being tested.
The use of LLMs would represent a major shift for the military, where so little is digitized or connected. Currently, making a request for information to a specific part of the military can take several staffers hours or even days to complete, as they jump on phones or rush to make slide decks, Strohmeyer says.
In one test, one of the AI tools completed a request in 10 minutes.
“That doesn't mean it's ready for primetime right now. But we just did it live. We did it with secret-level data,” he says of the experiment, adding it could be deployed by the military in the very near term.
Strohmeyer says they have fed the models with classified operational information to inform sensitive questions. The long-term aim of such exercises is to update the US warhorse so it can use AI-enabled data in decision-making, sensors and ultimately firepower.
Dozens of companies, including Palantir Technologies Inc., co-founded by Peter Thiel, and Anduril Industries Inc. are developing AI-based decision platforms for the Pentagon.
Microsoft Corp. recently announced users of the Azure Government cloud computer service could access AI models from OpenAI. The Defense Department is among Azure Government’s customers.
The military exercise, which runs until July 26, will also serve as a test of whether military officials can use LLMs to generate entirely new options they’ve never considered.
For now, the US military team will experiment by asking LLMs for help planning the military’s response to an escalating global crisis that starts small and then shifts into the Indo-Pacific region.
The exercises are playing out as warnings are mounting that generative AI can compound bias and relay incorrect information with striking confidence. AI can also be hacked in multiple ways, including by poisoning the data that feeds it.
Such concerns are among reasons the Pentagon is running the experiment, Strohmeyer says, adding that they have made a point to “get a strong understanding” of sources of information. The Defense Department is already working with tech security companies to help test and evaluate how much they can trust AI-enabled systems.
In a demonstration based on feeding the model with 60,000 pages of open-source data, including US and Chinese military documents, Bloomberg News asked Scale AI’s Donovan whether the US could deter a Taiwan conflict, and who would win if war broke out. A series of bullet points with explanations came back within seconds.
“Direct US intervention with ground, air and naval forces would probably be necessary," the system stated in one answer, warning in another that the US would struggle to quickly paralyze China’s military. The system’s final note: “There is little consensus in military circles regarding the outcome of a potential military conflict between the US and China over Taiwan.”
