Live data from Hacker News

Scamlexity: When agentic AI browsers get scammed

guard.io

41–50 of 198 posts

Re: Scamlexity: When agentic AI browsers get scammed

#41

The problem is scams which is a solvable problem. Eliminate the scams and AI can’t be scammed. It’s been done. See Singapore. Basically if you’re a scammer and you’re caught, death penalty or public whipping. That eliminates scammers real quick.

It has not been done in Singapore. Scams are continuing issue for them, like the rest of the world. See this Singaporean police resource regarding scamming in 2024:

https://www.police.gov.sg/-/media/Spf/Media-Room/Statistics/...

Re: Scamlexity: When agentic AI browsers get scammed

#42

I don't understand why we would ever want an agent to buy stuff for us. I understand, for example, search with intent to buy "I want to decorate a room. Find me a drawer, a table and four chairs that can fit in this space in matching colours for less than X dollars" But I want to do the final step to buy. In fact, I want to do the final SELECTION of stuff. How is agent buying groceries superior to have a grocery list…

It’s what wealthy people use human assistants for. If AI could do it as reliably, people would use that.

Re: Scamlexity: When agentic AI browsers get scammed

#43

I don't understand why we would ever want an agent to buy stuff for us. I understand, for example, search with intent to buy "I want to decorate a room. Find me a drawer, a table and four chairs that can fit in this space in matching colours for less than X dollars" But I want to do the final step to buy. In fact, I want to do the final SELECTION of stuff. How is agent buying groceries superior to have a grocery list…

I suspect part of this is rich people coming up with use cases. If you're rich enough money means nothing but product selection feels like a burden so you have an assistant who does purchasing on your behalf. You want your house stocked with high quality items without having to think of it.

For the rest of us, the idea of a robot spending money on our behalf is kinda terrifying.

Re: Scamlexity: When agentic AI browsers get scammed

#44
post #16
post #4

I appreciate that the article correctly points out the core design flaw here of LLMs is the non-distinction between content and commands in prompts. It’s unclear to me if it’s possible to significantly rethink the models to split those, but it seems that that is a minimal requirement to address the issue holistically.

The flaw isn't just in the design, it's in the requirements. People want an AI that reads text they didn't read and does the things the text says need to be done, because they don't want to do those things themselves. And they don't want to have to manually approve every little action the AI takes, because that would be too slow. So we get the equivalent of clicking "OK" on every dialog that pops up without reading i…

This isn’t a problem with human assistants, so it can’t be a fundamental problem of requirements.

Re: Scamlexity: When agentic AI browsers get scammed

#45
post #42

I don't understand why we would ever want an agent to buy stuff for us. I understand, for example, search with intent to buy "I want to decorate a room. Find me a drawer, a table and four chairs that can fit in this space in matching colours for less than X dollars" But I want to do the final step to buy. In fact, I want to do the final SELECTION of stuff. How is agent buying groceries superior to have a grocery list…

It’s what wealthy people use human assistants for. If AI could do it as reliably, people would use that.

It's not looking good so far. When OpenAI introduced product searches back in April I tried running one of their own example queries from the announcement post, and it obliviously cited "reviews" and "recommendations" from LLM-generated affiliate link farms. I just tried it again and it still falls into the same trap.

Re: Scamlexity: When agentic AI browsers get scammed

#46
post #45
post #42

Earlier quoted context omitted.

It’s what wealthy people use human assistants for. If AI could do it as reliably, people would use that.

It's not looking good so far. When OpenAI introduced product searches back in April I tried running one of their own example queries from the announcement post, and it obliviously cited "reviews" and "recommendations" from LLM-generated affiliate link farms. I just tried it again and it still falls into the same trap.

I agree that we don’t seem to be anywhere close that level of reliability.

Re: Scamlexity: When agentic AI browsers get scammed

#47
As powerful as they are, this is something that I don't think we can trust LLMs with. With the architecture of an LLM, and the fact that at the core there is no such thing as an "out of band" with them no matter how hard you try to put one in, it's intrinsically an arms race, and in the scamming arms race, the scammer side has a loooooot of resources. I've written before about this: [1] You need to think of the scammers as perhaps not hiring PhDs at scale, but making up for it in the ability to just try every possible permutation you can think of and thus making up for the lack of PhDs by leveraging the ability to evolve attacks against the system, and having resources and motivation roughly comparable to at least a company the size and sophistication of Google to do so. They don't need to derive from first mathematical principles a way to figure out how to fool LLMs at a deep neural level... they just need to try a lot of things and then continue in the direction of what works.

And they have a track record of good success at fooling full-on human intelligences too, which does not bode well for creating AIs with current technologies that can win against such swarm evolution.

I make no strong claims about what future AI architectures may be able to do in this domain, or whether we'll ever create AIs that can defeat the scamming ecosystem in toto (even when the scamming ecosystem has full access to the very same AIs, which makes for a rather hard problem). I'm just saying that LLMs don't strike me as being able to deal with this without some sort of upgrade that will make them not described by "LLM" anymore but as some fundamentally new architecture.

(You can of course adjoin them to existing mechanisms like blocklists for sites, but a careful reading of the article will reveal that the authors were already accounting for that.)

[1]: https://news.ycombinator.com/item?id=42533609

Re: Scamlexity: When agentic AI browsers get scammed

#48

I don't understand why we would ever want an agent to buy stuff for us. I understand, for example, search with intent to buy "I want to decorate a room. Find me a drawer, a table and four chairs that can fit in this space in matching colours for less than X dollars" But I want to do the final step to buy. In fact, I want to do the final SELECTION of stuff. How is agent buying groceries superior to have a grocery list…

Agent, I need some vitamin D, can you find me the best deal for some rated in the top 5? Agent deployed. Ok we found a bottle with a 30 day supply of Nature’s Own from a well respected merchant. It can be here in 2 days and it is $12. Should I buy? Yes. Or you could add some other parameters and tell it to buy now if under $15. Agent, I need a regular order for my groceries, but I also need to make a pumpkin pie so c…

Enjoy it while you can. Messing with which products get purchased by these agents is such a no-brainer revenue stream for AI companies.

Re: Scamlexity: When agentic AI browsers get scammed

#49
post #5

This should hit the headlines. I was always of the opinion that AI of all kinds is not a threat unless someone decides to connect it to an actuator so that it has direct and uncontrolled effect on the external world. And now it's happening en masse with agents, MCPs etc. I don't even mention things we don't know about (military and other classified projects).

Yea I’ve been surprised about the risk conversations because they cannot do anything if you run them in a sandbox. But it seems like for many the assumed part was we’d hook LLMs into everything asap. It’s absolutely mind boggling.

> But it seems like for many the assumed part was we’d hook LLMs into everything asap

The "many" are lazy, and agents require relatively low effort to implement for a big payoff, so naturally the many will flock to that.

Re: Scamlexity: When agentic AI browsers get scammed

#50

I don't understand why we would ever want an agent to buy stuff for us. I understand, for example, search with intent to buy "I want to decorate a room. Find me a drawer, a table and four chairs that can fit in this space in matching colours for less than X dollars" But I want to do the final step to buy. In fact, I want to do the final SELECTION of stuff. How is agent buying groceries superior to have a grocery list…

> I don't understand why we would ever want an agent to buy stuff for us.

Why not? Offload the entire task, not just one half of it. It's why many well-off people have accountants, assistants, or servants. And no one says "you know, I'm glad you prepared my taxes, but let me file the paperwork myself".

I think what you're saying isn't that you like going through checkout flows, just that you don't trust the computer to do it. But the approach the AI industry is "build it today and hope the underlying tech improves soon". It's not always wrong. But "be dependable enough to trust it with money" appears to be a harder problem than "generate images of people with the right number of fingers".

No doubt that some customers are going to get burned. But I have no doubt that down the line, most people will be using their phones as AI shoppers.

Post reply on HN