Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking leading AI agents against Google reCAPTCHA v2

research.roundtable.ai

Post reply on HN
Print view View original

91–100 of 104 posts

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#91

Post by bagacrap »

Legend2440 wrote: post #43
mehdibl wrote: post #30

Ok and then? Those models were not trained for this purpose. It's like the last hype over using generative AI for trading. You might use it for sentiment analysis, summarization and data pre-processing. But classic forecast models will outperform them if you feed them the right metrics.

It is relevant because they are trained for the purpose of browser use and completing tasks on websites. Being able to bypass captchas is important for using many websites. It would be nice to see comparisons to some special-purpose CAPTCHA solvers though.

And more broadly, if an agent is supposed to do everything a human can on the web, its ability to solve a captcha is likely a decent litmus test.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#93

Post by plingbang »

terminalshort wrote: post #67
Xenoamorphous wrote: post #4

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

There is a doom loop mode where it doesn't matter how many you solve or even if you get them correct. My source for this works on this product at Google.

I have a recording of me trying to pass the captcha for straight 5 minutes and giving up. To be fair, this has only happened once.

What is the purpose of such loop? Bots can simply switch to another residential proxy when the captcha success rate gets low. For normal humans, it is literally "computer says no".

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#94

Post by terminalshort »

Tostino wrote: post #69
terminalshort wrote: post #67

Earlier quoted context omitted.

There is a doom loop mode where it doesn't matter how many you solve or even if you get them correct. My source for this works on this product at Google.

Tell them I hate them.

Oh, don't worry. That's just v2. V3 uses the Google panopticon to watch your every move and decide if you're human or not that way without ever making you click on images. I'm sure you'll love it!

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#95

Post by nfRfqX5n »

plingbang wrote: post #93
terminalshort wrote: post #67

Earlier quoted context omitted.

There is a doom loop mode where it doesn't matter how many you solve or even if you get them correct. My source for this works on this product at Google.

I have a recording of me trying to pass the captcha for straight 5 minutes and giving up. To be fair, this has only happened once. What is the purpose of such loop? Bots can simply switch to another residential proxy when the captcha success rate gets low. For normal humans, it is literally "computer says no".

It’s not just IP. The score is linked to your google account as well and tracked across google properties

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#96

Post by imcritic »

varenc wrote: post #83
michaelt wrote: post #68

Earlier quoted context omitted.

Of course, if you have 3rd party cookies disabled, Google would never link your recaptcha activity to your Google account. They just link it to your IP address, browser, operating system, screen resolution, set of fonts, plugins, timezone, mouse movements, GPU, number of CPU cores, and of course the fact you've got third party cookies disabled.

Isn't Chrome shifting to blocking 3rd party cookies by default? If that's the new default than the default behavior would be that being logged into Google isn't used as a signal for recaptcha

Do you really think they won't make a hidden whitelist for their own domains?

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#97

Post by alexnewman »

amirhirsch wrote: post #65
alexnewman wrote: post #47

Hcaptcha cofounder here. Enterprise users have a lot of fancy configuration behind the scenes. I wonder if they coordinated with recaptcha or just assume there sitekey in the same as others

This was done on the re-captcha demo page no invisible fingerprinting, behavioral test, or user classification.

Ah yea probably not a good test then. Good point

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#99

Post by varenc »

imcritic wrote: post #96
varenc wrote: post #83

Earlier quoted context omitted.

Isn't Chrome shifting to blocking 3rd party cookies by default? If that's the new default than the default behavior would be that being logged into Google isn't used as a signal for recaptcha

Do you really think they won't make a hidden whitelist for their own domains?

There'd be no way to hide this. If 3rd party cookies are disabled it's trivial to observe if an embedded google.com iframe is sending my full google.com 1st party cookies in violation of the 3rd party cookie settings. There's no pinky promises involved, you can just check what it's sending with a MITM proxy.

I'm sure they're doing other sketchy things but wouldn't make sense to lie in such a blindingly obvious way. (I just tested it, and indeed, it works as expected)

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#100

Post by max51 »

fooker wrote: post #86
max51 wrote: post #71

Earlier quoted context omitted.

That doesn't surprise me. I find it hard to believe it's a pure coincidence that I would get stuck in the loop regularly when I'm on the university wifi but it would never happen anywhere else ever. After a dozen try, I would remote connect to my home pc and it would magically work on the first try every single time.

Someone from your university tried to scrape data from Google. I know it's still not justified, but it's the easy solution that works for preventing DOS attacks.

>Someone from your university tried to scrape data from Google.

Kinda wild that someone scraping google's data would prevent me from getting into my PAID (>90$/yr) Dropbox account. That experience is a big part of why I pay extra to host my host data on my own server now.

Post reply on HN