Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

this.weekinsecurity.com

Post reply on HN
Print view View original

31–40 of 287 posts

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#31

Post by trehalose »

theptip wrote: post #21
nkrisc wrote: post #16

Earlier quoted context omitted.

The tool worked correctly and as intended, but due to a bug it did not work correctly nor as intended.

Sounds like they are saying the agent did not malfunction, and this vuln could have been triggered by a human support agent too.

It probably could have been, but how likely is that compared to with the AI agent? I'd assume (and I'm ready to look like an idiot if I'm wrong) that the humans are trained to send the verification code to the email address on file, rather than any address the client asks them to. I'd certainly assume most of them are more afraid of the consequences than the AI is.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#32

Post by joshuat »

jhhh wrote: post #26

Why was 'can a user request a different email' not literally the first test that comes to mind when making something like this? Do they not test anything because the scale is too big?

In their defense, they asked the LLM to make no mistakes

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#33

Post by TZubiri »

nkrisc wrote: post #16
Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

The tool worked correctly and as intended, but due to a bug it did not work correctly nor as intended.

I get the joke, but it's a relevant nuance that the new code, the chatbot, did not have 'the bug'. I still think that the mistake and head that should roll should be the one that published the chatbot.

But it's important to acknowledge that there was a 'bug' in an underlying tool and not in the chatbot, and still PIP/fire those responsible for publishing the chatbot and exposed an otherwise internal tool to the public, and not those that introduced the 'bug' to an internal tool.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#34

Post by TZubiri »

ludwik wrote: post #6
Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

I like to dunk on Meta as much as the next guy, but I think this makes sense: deterministic verification like this is not, and should never be, the LLM’s job. The tools it has access to should enforce the permissions layer, ensuring that the LLM can never perform actions the user themselves should not be allowed to perform. In this case, the tool failed to do that.

>deterministic verification like this is not, and should never be, the LLM’s job.

But when humans handled it, this was not as much as a problem. That is, the humans did the job, because they recognized the need to do that job.

Sure sometimes accounts could get recovered if a human was tricked, but evidently it was easier to trick the LLM in masse than humans.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#36

Post by TZubiri »

webbdev wrote: post #29

Meanwhile an account I created for a new product was permanently disabled by an automated system with no path for me to appeal to a human. (If anyone at Meta/Instagram sees this I wrote a brief blog post with the details. Please help! https://addisonwebb.com/blog/2026-06-05-Can%20Someone%20at%2... )

Did you create the account separately? Or as an asset of your main Meta account (like Meta Business Suite)?

I'm creating the accounts in Meta Business Suite, so I would have a recourse with my main personal account which can be linked to some adspend, so I'm assuming it will have better support channels than accounts created through an end-user interface.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#38

Post by nvme0n1p1 »

thih9 wrote: post #19
nkrisc wrote: post #16

Earlier quoted context omitted.

The tool worked correctly and as intended, but due to a bug it did not work correctly nor as intended.

To be fair, that quote in the original article could have more context. By "The tool" they meant "AI-assisted support tool"[1]; perhaps they meant that the issue was not an AI hallucination inherent of the tool, but a fixable bug. [1]: https://www.documentcloud.org/documents/28202858-meta-ai-ag-...

In that case, the statement is so meaningless as to be useless. Why should we care how Meta splits up their microservices? The tool still failed. They just want to redefine the "tool" as something else, anything else, to avoid having to admit something negative about their precious AI.

> The LLM correctly generated tokens according to user input, however due to a bug in a separate code path, the system did not properly verify the email address

> Nginx correctly handled the user requests according to the HTTP standard, however due to a bug in a separate code path, the system did not properly verify the email address

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#39

Post by embedding-shape »

vb-8448 wrote: post #30
Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

In italian we say "l'operazione è riuscita perfettamente, ma il paziente è morto" -> "the surgery was a complete success, but the patient died"

Both this and what Meta said reminds me of "Clarke and Dawe - The Front Fell Off" (https://www.youtube.com/watch?v=3m5qxZm_JqM)

I also can't believe the people who were involved with writing this response from Meta, didn't realize how obviously bad it sounds. It's like there is no humans working and writing there anymore.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#40

Post by [deleted] »

vb-8448 wrote: post #30
Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

In italian we say "l'operazione è riuscita perfettamente, ma il paziente è morto" -> "the surgery was a complete success, but the patient died"

[deleted]
Post reply on HN