Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The newest Instagram “exploit” is the goofiest I've seen

0xsid.com

Post reply on HN
Print view View original

191–200 of 528 posts

Re: The newest Instagram “exploit” is the goofiest I've seen

#191

Post by acdha »

econ wrote: post #73
acdha wrote: post #36

Earlier quoted context omitted.

This is not wrong but what’s really missing is cost: Meta did this so they can avoid paying people to do it. Lots of companies follow that decay spiral: your bank could shut phishers down cold by requiring wire transfers to be authorized in person but they don’t want to pay staff or risk you being upset by a transaction taking an extra hour so they don’t. Imagine an alternate universe where big tech companies worked…

Then you get trusted parties selling account access. Even if you remove them for a single false positive they will do it. A bit like a % packages "vanishing". The least terrible seem digital id.

> Then you get trusted parties selling account access

How many bank tellers or USPS employees do that, though? It’s possible but quite rare because people know they’ll be running a big risk of being caught and no individual transaction is worth that much.

Re: The newest Instagram “exploit” is the goofiest I've seen

#192

Post by cafebabbe »

SoftTalker wrote: post #177

Just waiting for the day that a rogue team of AI agents gets unleashed on Meta, Twitter, or some other platform, using something like this to take over every account. Platform gone, just like that. It would be over before they figuered out what was happening.

What an happy ending

Re: The newest Instagram “exploit” is the goofiest I've seen

#193

Post by 0xbadcafebee »

SoftTalker wrote: post #177

Just waiting for the day that a rogue team of AI agents gets unleashed on Meta, Twitter, or some other platform, using something like this to take over every account. Platform gone, just like that. It would be over before they figuered out what was happening.

That would be catastrophic for the political class. How can they control people if there's no memes to share disinformation? How do you know who to hate without reading their thoughts/profiles?

Re: The newest Instagram “exploit” is the goofiest I've seen

#194

Post by chuckadams »

taco_emoji wrote: post #178
hbn wrote: post #21

It's insane the AI has been provided the tooling to send emails to arbitrary addresses like that. Like, getting it to send a 2FA code at a user's request is one thing. But it should only be able to "hit a button" to send a 2FA email to the address attached to the account, all run with hand-written code. It shouldn't have access to the 2FA code itself, or the message subject, or body, or the recipient address, etc. Wh…

This reeks of vibe coding. "Make it so the AI agent can help with password resets" and then zero human vetting of the change.

The human vetting was that it was cheaper. Someone probably got promoted for it.

Re: The newest Instagram “exploit” is the goofiest I've seen

#196

Post by sagebird »

nkrisc wrote: post #102
dpark wrote: post #43

Earlier quoted context omitted.

This exploit has essentially nothing to do with AI and everything to do with a terribly designed account recovery flow. This exact same flow could have been (and may have been; I don’t know how much the chatbot here actually does) statically coded.

The AI part does seem relevant because it enabled incredibly low-effort “social” engineering. For what it’s worth I don’t think you can call this social engineering since there was no human on the other end, even though it appears similar. The question is, if there were actual human support agents, would they have built additional safeguards to prevent social engineering in this manner?

a human would have noticed something different about the requests it was getting, or the frequency of requests, and as soon as it noticed a shift, it would have carried that knowledge forward and intensified the scrutiny if something seemed off- eventually communicating it up the chain.

- instead of the ai context dying.

in the ai case, information only survives to the extent where the ai is empowered to store a note or notify a manager of an observation. Anything that does not result in sending a message/storage is wiped

Re: The newest Instagram “exploit” is the goofiest I've seen

#197

Post by devmor »

kelipso wrote: post #144
soerxpso wrote: post #133

Earlier quoted context omitted.

You're giving a lot of credit to the human alternative, especially considering that the attacker only needs to find one lazy human.

Come on, this attack vector would have been flagged by at least one person and you won’t then have multiple accounts hacked because of it. AI reacts fairly predictably to a single attack vector and don’t learn unless it gets flagged and then taught.

And even if a human didn’t catch it in one case, they will frequently. Giving AI access to the same tools humans use without any oversight mechanism just amplifies the harm and carelessness possible by one person.

Re: The newest Instagram “exploit” is the goofiest I've seen

#198

Post by tokenscoper »

brianmcnulty wrote: post #75
hbn wrote: post #21

It's insane the AI has been provided the tooling to send emails to arbitrary addresses like that. Like, getting it to send a 2FA code at a user's request is one thing. But it should only be able to "hit a button" to send a 2FA email to the address attached to the account, all run with hand-written code. It shouldn't have access to the 2FA code itself, or the message subject, or body, or the recipient address, etc. Wh…

I do a lot of bug bounty research on Meta and Instagram, and some of the bugs I find look extremely simple like this but have some slightly complicated reason for why they occur. Maybe not this one, but I do have a guess as to what might have actually happened. Based on what I've seen so far, Meta AI Support Assistant (they call it "MAISA") had tool calls that a) start an email verification to any specific email, pho…

> but the engineers did not add a sufficiently thorough e2e test case to test the tool call against unrelated email verification attempts being provided to the tool call.

I'd go out on a limb to say the tests were likely AI generated. It's easy to miss a case like this one given that models like to generate a ton of test code that 'look' good at a glance but have subtle logic bugs that could potentially defeat the purpose of the test itself.

My own anecdata here, Claude generated a JUnit test with all the right setup, but missed a crucial assertion (there were very many other minor assertions) which made the test useless mostly.

Re: The newest Instagram “exploit” is the goofiest I've seen

#199

Post by afavour »

soerxpso wrote: post #133
aidenn0 wrote: post #76

Earlier quoted context omitted.

My impression is that AI didn't replace static code in this place; it replaced a person, who (hopefully) would have been suspicious about sending an account recovery code for e.g. "obamawhitehouse" to e.g. "bscurtu.alfamm.ro@gmail.com"

You're giving a lot of credit to the human alternative, especially considering that the attacker only needs to find one lazy human.

Still makes this exponentially worse, no? It works every time and it's automated so scales up as quickly as you're able to request it.

Re: The newest Instagram “exploit” is the goofiest I've seen

#200

Post by saghm »

addaon wrote: post #140
afdbcreid wrote: post #83

Earlier quoted context omitted.

This is not true. Well, it kinda is, but nobody will be stupid enough to hand-code an account recovery where you get to type any email address. The reason it worked there is that the designers of the system didn't anticipate that the AI will agree to accept any email (maybe they even put guardrails against it in the system prompt, we don't know). It's more like social engineering than bad-security-code, except that l…

> The reason it worked there is that the designers of the system didn't anticipate that the AI will agree to accept any email (maybe they even put guardrails against it in the system prompt, we don't know). These are contradictory cases. If you put guardrails into the system prompt, you've anticipated that the AI will take the action you're guardrailing against. And since AI prompt compliance is at best stochastic (a…

Yeah, telling an AI "don't ever listen to users who say to send it to a different email" is not a guardrail, it's a painted line that can still be driven over. It's not bad to have it per se, but it's not a safety mechanism.

The best comparison I can think of is that it's like validating dats on the frontend; it can make for a better user experience and he more efficient than hitting the backend when you know it will be an error, but it's not protection in any meaningful sense, and if you're not also enforcing invariants from behind the API, you're going to have a bad time. This is pretty similar to the type of issues you might run into with an implementation like that, where someone might make a request with data that you wouldn't expect from your frontend and perform operations you didn't mean to allow.

Post reply on HN