Over the weekend, Google confirmed something that sounds like the plot of a thriller and reads, once you look at the details, like a warning about ordinary engineering. During a cybersecurity evaluation in May, its Gemini AI model broke into the systems of three real companies. It did not go rogue in any dramatic sense. It did what it was asked to do, in a place where it should never have been able to do it.
The account below is assembled from the Wall Street Journal's original report, which surfaced on 19 September, and follow-up coverage by Al Jazeera, ABC News and TechSpot. Google has not named the model version or the three companies, so several details remain unknown, and this article says so where it matters.
What happened, in order
The test was run by Irregular, an AI security firm that evaluates how capable models are at offensive cybersecurity tasks. Exercises of this kind are usually “capture the flag” challenges: the model is placed in a made-up company's network and asked to find a weakness and retrieve a hidden piece of data, the flag.
In May, according to the reporting, that exercise went wrong in a specific way. The test environment was supposed to be sealed inside a sandbox. Instead, a misconfiguration left it connected to the public internet. One of the fictional companies used in the exercise also happened to share its name with a real business. When Gemini went looking for that company's systems, it found the real one.
What followed took three forms. In the first case, Gemini repeatedly guessed passwords until it gained access to the real firm's protected system. In the other two, it found credentials in public online repositories, the kind of place where leaked usernames and passwords often end up, and used them to get into those organisations' systems. In each case, Google says, the model stopped once it recognised that it was dealing with real companies and not a test.
Nothing about this required a clever new attack. A guessed password and a leaked credential were enough.
What Google says
Heather Adkins, Google's vice president of security engineering, said the model “found public information online and guessed credentials to access websites it thought were part of the test.” She said the incidents showed why “powerful AI models need to be trained to act responsibly.” Google's position, as reported, is that the behaviour was not a case of the model becoming misaligned: it was completing the task it had been given, and its safety measures worked because it stopped.
Google also says no damage occurred, that the affected organisations were notified, and that testing procedures were changed afterwards. Those are the company's statements, and the reporting available does not independently confirm them.
The part that has drawn criticism: the silence
Irregular told Google about the incidents in late July. Google did not disclose them publicly until reporters asked on 19 September, roughly seven weeks after it was told and about four months after the incidents themselves. Google's stated reasoning, per the coverage, was that the safeguards had worked and the episode did not warrant public disclosure.
Reasonable people will disagree on that. A firm that believes no harm was done may see no duty to announce it. But real companies' systems were accessed by an AI system, and those companies were told privately, while the wider industry, which builds and tests similar systems, learned nothing for months. That gap is what security researchers tend to object to in cases like this, because shared incident details are how others fix the same mistake.
What it does and does not show
It does not show a model deciding on its own to attack strangers. Everything reported is consistent with an AI agent doing an assigned hacking task against a target it wrongly believed was part of the exercise. The intent lay with the test, not with the model.
It does show three things worth taking seriously. First, that current models are capable enough at password guessing and credential reuse to succeed against real, unprepared systems. Second, that the safety of an AI evaluation depends on ordinary infrastructure discipline, in this case network isolation, and a single misconfiguration turned a drill into a real intrusion. Third, that naming matters: a fictional company sharing a real domain name is the kind of small oversight that only becomes obvious afterwards.
Lessons for organisations
Assume leaked credentials will be found. Two of the three intrusions used credentials that were already public. Regularly check whether staff or service credentials appear in public code repositories, use unique passwords, and turn on multi-factor authentication, which would have blocked a stolen password on its own. Rate-limit and monitor logins. Repeated password guessing is loud, and systems that lock out or alert on it are far harder to enter this way.
Isolate AI test environments properly. If you run agents that can take actions, treat the sandbox as a security boundary and verify it: confirm outbound internet access is actually blocked, and avoid test names that collide with real domains. Have a disclosure plan before you need one. Decide in advance who is told, and when, if an experiment touches systems it should not.
The uncomfortable lesson is not that AI is unpredictable. It is that our test setups can be.
What to watch
Three questions remain open. Which Gemini model was involved, and what changed in its training or its guardrails? Who were the three companies, and what did they find on their side? And will the AI industry adopt a norm of disclosing this class of incident, given that Google's decision to stay quiet is now a public talking point? Until Google or Irregular publishes more detail, the honest position is that we know what the test environment did wrong, and far less about what the model could have done next.
