Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

this.weekinsecurity.com

Post reply on HN
Print view View original

11–20 of 287 posts

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#11

Post by nico »

Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

That sounds a lot like the justifications Claude and ChatGPT give when confronted about something they did wrong, or when asked to provide a customer support response about software issues

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#12

Post by jgalt212 »

Cpoll wrote: post #10
Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

The argument here is that the AI is a glorified input page. The input field asks for your username and email and sends it to a backend function. Such an input page is working as intended. The problem is when the backend function doesn't verify that the email matches the username.

Fair enough. Never trust client-submitted browser form, but always trust LLM-submitted form.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#13

Post by rvz »

If this was a bank that had zero humans and the AI chatbot was abused to hand over sensitive information about their customers which led to this disaster, people would never trust their bank ever again and leave.

Meta believes that they can vibe-code their reputation down the drain by removing humans in the loop.

Applying a technical solution to a social problem almost always ends in disasters like this.

Reputation can’t be vibe-coded.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#14

Post by totetsu »

Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

Then ‘ The tool itself’ was not appropriate to the job in the first place

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#15

Post by laweijfmvo »

Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

so how long was the bug there? was there a way to access it before/without the support agent? it feels like Meta will throw anything under the bus to redirect blame from the AI, because that would be the end of their $600B (depending on “which number you want to go with”) experiment

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#16

Post by nkrisc »

Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

The tool worked correctly and as intended, but due to a bug it did not work correctly nor as intended.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#17

Post by CivBase »

rvz wrote: post #13

If this was a bank that had zero humans and the AI chatbot was abused to hand over sensitive information about their customers which led to this disaster, people would never trust their bank ever again and leave. Meta believes that they can vibe-code their reputation down the drain by removing humans in the loop. Applying a technical solution to a social problem almost always ends in disasters like this. Reputation c…

Meta's brand is already toxic. Idk if there's much to lose there.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#18

Post by dgoldstein0 »

Cpoll wrote: post #10
Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

The argument here is that the AI is a glorified input page. The input field asks for your username and email and sends it to a backend function. Such an input page is working as intended. The problem is when the backend function doesn't verify that the email matches the username.

Why on earth would the backend function even take an email?

Or perhaps said different: use the submitted info to identify the account; send any sensitive messages (recovery codes, password resets whatever) to only the contact info on file. If the chat bot can send such email it should do so via an API that sends only to contact info on file for the associated account and not to an email that's provided by the bot.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#19

Post by thih9 »

nkrisc wrote: post #16
Cyan488 wrote: post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

The tool worked correctly and as intended, but due to a bug it did not work correctly nor as intended.

To be fair, that quote in the original article could have more context. By "The tool" they meant "AI-assisted support tool"[1]; perhaps they meant that the issue was not an AI hallucination inherent of the tool, but a fixable bug.

[1]: https://www.documentcloud.org/documents/28202858-meta-ai-ag-...

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#20

Post by cyanydeez »

"abusing" by using it's built in insecurity to do insecure things.

It's like, people abusing an open door. "Guys, just because we left the door open to your bedroom doesn't mean we're responsible".

God can only hope this is a business ending lawsuit.

Post reply on HN