Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking leading AI agents against Google reCAPTCHA v2

research.roundtable.ai

Post reply on HN
Print view View original

41–50 of 104 posts

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#41

Post by felixfurtak »

Xenoamorphous wrote: post #4

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

I always assume that people are lazy and try and click the least amount of squares as possible to get broadly the correct answer. Therefore, if it says motorbikes just click on the body of the bike and leave out rider and tiles with hardly any bike in them.

If it says traffic lights just click on the ones you can see lit and not the posts and ignore them if they are too far in the distance. Seems to work for me.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#42

Post by daveguy »

Wow. Cross-tile performance was 0-2%. That's the challenge where you select all of the tiles containing an item where the single item is in a subset of tiles. As opposed to all the tiles that contain the item type (static - 60% max) and the reload version (21% max). Seems to really highlight how far these things are from reasoning or human level intelligence. Although to be fair, the cross-tile is the one I perform worst on too (but more like 90+% rather than 2%).

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#43

Post by Legend2440 »

mehdibl wrote: post #30

Ok and then? Those models were not trained for this purpose. It's like the last hype over using generative AI for trading. You might use it for sentiment analysis, summarization and data pre-processing. But classic forecast models will outperform them if you feed them the right metrics.

It is relevant because they are trained for the purpose of browser use and completing tasks on websites. Being able to bypass captchas is important for using many websites.

It would be nice to see comparisons to some special-purpose CAPTCHA solvers though.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#44

Post by daveguy »

mehdibl wrote: post #30

Ok and then? Those models were not trained for this purpose. It's like the last hype over using generative AI for trading. You might use it for sentiment analysis, summarization and data pre-processing. But classic forecast models will outperform them if you feed them the right metrics.

These are all multi-modal models, right? And the vision capabilities are particularly touted in Gemini.

https://ai.google.dev/gemini-api/docs/image-understanding

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#45

Post by RobertDeNiro »

daveguy wrote: post #42

Wow. Cross-tile performance was 0-2%. That's the challenge where you select all of the tiles containing an item where the single item is in a subset of tiles. As opposed to all the tiles that contain the item type (static - 60% max) and the reload version (21% max). Seems to really highlight how far these things are from reasoning or human level intelligence. Although to be fair, the cross-tile is the one I perform w…

I think the prompt is probably at fault here. You can use LLMs for object segmentation and they do fairly well, less than 1% seems too low.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#46

Post by viccis »

Sayrus wrote: post #17
Xenoamorphous wrote: post #4

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

Testing those same captcha on Google Chrome improved my accuracy by at least an order of magnitude. Either that or it was never about the buses and fire hydrants.

That's because Chrome tracks so much telemetry about you that Google is satisfied with how well it has you surveilled. If you install a ton of privacy extensions like Privacy Badger, uBlock, VPN extensions with information leakage protections, etc., watch that "accuracy" plummet again as it makes you click 20 traffic signals to pass one check.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#48

Post by criddell »

flakiness wrote: post #11

To be honest I'm surprised how well it holds. I expected close-to-total collapse. It'll be a matter of time I guess, but still.

I wonder if any of the agents hit the audio button and listened to the instructions? In my experience, that can be pretty helpful.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#50

Post by timshell »

rkagerer wrote: post #49

Forget whether humans can't distinguish your AI from another human. The real Turing test is whether your AI passes all the various flavors of captcha checks.

One of the writers here. We believe the real Turing Test is whether your AI performs a CAPTCHA like a human would/does.
Post reply on HN