Live data from Hacker News

Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

this.weekinsecurity.com

191–200 of 287 posts

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#191
post #52

I really hope this accelerates meta's decline. The world will adapt just fine without social media.

Somehow this company still earns over a billion dollars a week in net profits, which I find puzzling.

Why? They clearly are not spending money on employee salaries, taxes, or benefits.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#192

> as well as the ability to access the person's posts, direct messages god dang!! we are going to see some juicy stuff

Will we, or will we read a bunch of crypto requests to not see that juicy stuff and a lot of people paying?

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#193

Earlier quoted context omitted.

This is not how liability works, anywhere. So I write a piece of code that "makes your screen do cool things" and it causes the power supply to fail on those screens. Someone reports that bug to me and I check it out and say "Oh, shit it does break power supplies." Then I immediately put a notice on and in the code that says "WARNING: This code will break the power supply of your montitor." And I put that warning in…

I broadly agree with you but TBF to the earlier comment consider what would happen if a FOSS author did something wrong and was found to be liable. How about curl for example? That sees use in car infotainment systems among other things and cars can be pretty expensive and there sure are an awful lot of them. The point is that we should be able to accommodate someone pushing a hobby project to github under a permissi…

I realize this is drifting off topic, and happy to talk more in email (address in profile), in the interest of sharing a bit more, consider this statement you paraphrase:

"a FOSS author did something wrong and was found to be liable"

In fairness, I not sure the earlier commentator really understood what they were saying, at least not as far as legal liability is concerned.

The FOSS author simply wrote some code and shared it right? That is their 'action' can you think of ways that does direct harm, which is to say they published their code, and with nothing else happening someone got harmed? One way that can cause harm is the FOSS author publishes a trade secret[1] or access credentials of a third party. In both cases they could (and would) be sued by that third party. But absent that, I'm having a hard time coming up where simply the existence of most code causes someone else harm.

So to get to harm we have to add another person, that person somehow applies the code, and in that application harms another person. Our FOSS author might be sued as being contributory because the person who caused harm might not have done so if they didn't have access to the code. To prove that, the plaintiff would have to prove that the FOSS author knew that the code could cause harm if used in this way, and encouraged or otherwise abetted the person who did harm to use it in doing the harm. That can be a hard standard to reach[2].

In your car example, it would be challenging to prove that Daniel Stenberg wrote curl so that you could use it to brick car infotainment systems. But it would be easier to prove that a manufacturer that incorporated FOSS code and didn't check their system for risks like this should be found liable.

Liability accrues first to the party that did the action. Secondary liability can reach out to suppliers[3] of things used in that action. This is also civil law rather than criminal law and so it works a bit differently in terms of evidence standards and penalties.

[1] We can make a joke here about badly formatted code, but hopefully we're in a agreement so far. A real example was the DVD decoding software that included the key for decoding encrypted DVDs.

[2] Not that people might not try, its too easy to sue. There have been cases where someone wrote some code that was later used in a weapon (and example might be Ardupilot software in drones used to kill Russians). But even in that case, the courts in the US at least have consistently found that if it is not the primary purpose of the software to do harm, then the author is not liable.

[3] Unless you're a gun company as Gun companies have managed to keep themselves from being found liable for people using their guns to do harm. But there is also lots of interesting case law there too which might help inform.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#194
post #30
post #3

> "The tool itself worked properly and functioned as intended; however due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user’s Instagram account," said Meta in its breach notice. I'm not sure "worked properly" and "as intended" accurately describe this situation.

In italian we say "l'operazione è riuscita perfettamente, ma il paziente è morto" -> "the surgery was a complete success, but the patient died"

"Operation succesful, patient dead" is a common saying in India.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#195

Earlier quoted context omitted.

After the GDPR every website added an option to export your personal data and to delete your account. Something most were missing at the time. It was an immediate and massive win.

Right, but nothing stops companies from refusing SARs on baloney grounds. Complain to a DPA? They tell you to go through ADR or outright ignore you. Complain to Ombudsman? They'll tell you the same. (In my experience, the Dutch do this) Company ignores ADR? Sure, now you can go through the legal route and spend copious amounts of money all because a multi billion dollar company knows the game and how to navigate the…

This. In reality, GDPR isn't preventative, nor punitive enough for any meaningful user protection. We get cookie banners everywhere and user data harvesting companies happily pay the negligible fines

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#196
post #129

Earlier quoted context omitted.

Incidents like this show how unenforceable GDPR is, and how it's been a net negative for users since its inception. It's idealogical back-patting, toothless when it matters.

After the GDPR every website added an option to export your personal data and to delete your account. Something most were missing at the time. It was an immediate and massive win.

You don't have to have a fb account for meta to fingerprint every little page you visit, perfectly legally.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#197
post #185
post #6

Earlier quoted context omitted.

I like to dunk on Meta as much as the next guy, but I think this makes sense: deterministic verification like this is not, and should never be, the LLM’s job. The tools it has access to should enforce the permissions layer, ensuring that the LLM can never perform actions the user themselves should not be allowed to perform. In this case, the tool failed to do that.

The overall system that allowed this implementation is accountable. So why put such a fine point on it so as to exculpate the LLM?

It helps set expectations for the fix. "The bug was in an external system that has now been fixed" means we it's probably fine going forward. "The LLM got tricked but we are gonna train it super hard not to do that again" means it will break again and again as people find new angles to convince it.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#198
I'll never understand using AI/bot for customer support. IG is a well know platform. If I have an issue I feel pressed to connect with a support agent about it very likely is something a bot would struggle with, otherwise I'd just google. I understand there some grandmas who can do a google search, but the vast majority of folks reaching out for support are doing so because they have a real issue that can't be simply automated.

Furthermore, having a bot handle a hacked account is support ticket is just insane. Why tf would you put a bot there and give it permission to take action?

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#199
post #34
post #6

Earlier quoted context omitted.

I like to dunk on Meta as much as the next guy, but I think this makes sense: deterministic verification like this is not, and should never be, the LLM’s job. The tools it has access to should enforce the permissions layer, ensuring that the LLM can never perform actions the user themselves should not be allowed to perform. In this case, the tool failed to do that.

>deterministic verification like this is not, and should never be, the LLM’s job. But when humans handled it, this was not as much as a problem. That is, the humans did the job, because they recognized the need to do that job. Sure sometimes accounts could get recovered if a human was tricked, but evidently it was easier to trick the LLM in masse than humans.

Maybe that’s because I work with agentic AI in my day job, but this seems utterly obvious to me: no reasonable person would ever claim that LLMs are better at keeping secrets or enforcing rules than human employees.

This notice is not about comparing humans and LLMs. It seems that the system was designed in the only reasonable way: with a deterministic permissions layer separate from the agent. But that layer failed to work properly.

So the notice is comparing the difference between how the system was supposed to work and how it actually worked in reality. Normal post-mortem stuff.

Re: Meta confirms 1000s of Instagram accounts were hacked by abusing its AI chatbot

#200
post #60

Earlier quoted context omitted.

> It's like there is no humans working and writing there anymore. Don't know if AI is to blame, but I've used to see these kinds of nonsense post-mortems even in the pre-llm era, and it's always due to some internal fighting ongoing between various departments.

Where do you think the LLMs learned it from...

"Who taught you how to do this stuff?"

"You, alright! I learned it by watching you!"

Post reply on HN