- Gemini in Workspace presents unique opportunities for fraud, researchers warn
- The AI tool can be tricked to display fake security warnings
- Businesses should make sure invisible text is not processed by the AI
Cybercriminals have found a creative new way to abuse Google’s Generative Artificial Intelligence (GenAI) to steal people’s Gmail accounts.
Google introduced Gemini, its AI-powered chatbot assistant into its Workspace suite of productivity apps some time ago, and one of the things Gemini can do is summarize incoming emails – so when a person receives an email, they can bring up a vertical pane on the right-hand side of the screen, asking Gemini for assistance with different things, such as bringing up vital email information, adding calendar entries, and more.
However experts have warned this also opens up the Gmail accounts for so-called “prompt-injection” attacks – so if the incoming email message contains a hidden prompt for Gemini, it can be executed in the pane.
Is Gemini phishing for your password?
According to security researcher Marco Figueroa, this is exactly what the email provider is now susceptible to.
By using HTML and CSS, threat actors can add a prompt for Gemini, with its font size set to zero, and its color to white. Therefore, the victim will not be able to see it, but Gemini will act on it. If that prompt makes Gemini display a phishing message, it will do just that, and since the message would come from a trusted source, it increases the chances of success.
Figueroa showed how a malicious prompt could notify the victim that their email account has been compromised, and that they need to “call” Google on a phone number displayed in the message to resolve the issue.
To protect against future prompt injection attacks, companies should make sure their email clients remove, neutralize, or ignore content that is styled to be hidden in the body text. Furthermore, they could include a post-processing filter that scans the inbox for “urgent messages”, URLs, or phone numbers.
Finally, businesses should educate their employees that summaries provided by the Gemini tool should not be a replacement for security alerts.
Even though there is no evidence that prompt-injection attacks are being used in the wild, Google acknowledged their existence and is working on mitigations.
In an emailed statement shared with TechRadar Pro, a company spokesperson confirmed that defending against prompt injection attacks is a priority at Google:
“We are constantly hardening our already robust defenses through red-teaming exercises that train our models to defend against these types of adversarial attacks,” they said.
Google has also recently published a blog outlining the specific defenses it is currently deploying for prompt injection-style attacks. These defenses are currently mid-deployment, t was added.
Via BleepingComputer
You might also like
