Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking leading AI agents against Google reCAPTCHA v2

research.roundtable.ai

Post reply on HN
Print view View original

61–70 of 104 posts

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#61

Post by timshell »

cm2187 wrote: post #60
charcircuit wrote: post #59

Earlier quoted context omitted.

It doesn't matter. Select it of you think other people would select it too.

That's the thing, you could go either way. I am not sure I can answer the question "what would a resonable person click?".

The 'Process Turing Test' extends the CAPTCHA from 'What would a reasonable person click' to 'How would a reasonable person click'.

For example, hesitation/confusion patterns in CAPTCHAs are different between humans and bots and those can actually be used to validate humans

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#62

Post by jameslk »

Xenoamorphous wrote: post #4

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

> Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

This type of captcha is too infuriating so I always skip it until I get the ones where I’m just selecting an entire image, not parts of an image

Google’s captchas are too ambiguous and might as well be answered philosophically with an essay-length textbox

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#63

Post by consp »

viccis wrote: post #46
Sayrus wrote: post #17

Earlier quoted context omitted.

Testing those same captcha on Google Chrome improved my accuracy by at least an order of magnitude. Either that or it was never about the buses and fire hydrants.

That's because Chrome tracks so much telemetry about you that Google is satisfied with how well it has you surveilled. If you install a ton of privacy extensions like Privacy Badger, uBlock, VPN extensions with information leakage protections, etc., watch that "accuracy" plummet again as it makes you click 20 traffic signals to pass one check.

I stop going to sites using that method due to this. I have no intention of proving I'm a human it I have to click several dubious images 3-4 times in a row.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#64

Post by jameslk »

Is it assumed that humans perform 100% against this captcha? Because being one of those humans it’s been closer to 50% for me

I’m guessing Google is evaluating more than whether the answer was correct enough (ie does my browser and behavior look like a bot?), so that may be a factor

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#65

Post by amirhirsch »

alexnewman wrote: post #47

Hcaptcha cofounder here. Enterprise users have a lot of fancy configuration behind the scenes. I wonder if they coordinated with recaptcha or just assume there sitekey in the same as others

This was done on the re-captcha demo page no invisible fingerprinting, behavioral test, or user classification.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#66

Post by abacadaba »

cm2187 wrote: post #60
charcircuit wrote: post #59

Earlier quoted context omitted.

It doesn't matter. Select it of you think other people would select it too.

That's the thing, you could go either way. I am not sure I can answer the question "what would a resonable person click?".

Answer how you'd want a Waymo to react

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#67

Post by terminalshort »

Xenoamorphous wrote: post #4

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

There is a doom loop mode where it doesn't matter how many you solve or even if you get them correct. My source for this works on this product at Google.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#68

Post by michaelt »

varenc wrote: post #56
Alex2037 wrote: post #40

Earlier quoted context omitted.

stock Chrome logged into a Google account = definitely not a bot. here, click a few fire hydrants and come on in :^) I sincerely wish all the folx at Google directly responsible for this particular user acquisition strategy to get every cancer available in California.

I would think that when you're viewing recaptcha on a site, if you have 3rd party cookies disabled the embedded recaptcha script won't have anyway of connecting you with your Google account, even if you're logged in. At least that's how disabling 3rd party cookies is supposed to work.

Of course, if you have 3rd party cookies disabled, Google would never link your recaptcha activity to your Google account.

They just link it to your IP address, browser, operating system, screen resolution, set of fonts, plugins, timezone, mouse movements, GPU, number of CPU cores, and of course the fact you've got third party cookies disabled.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#69

Post by Tostino »

terminalshort wrote: post #67
Xenoamorphous wrote: post #4

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

There is a doom loop mode where it doesn't matter how many you solve or even if you get them correct. My source for this works on this product at Google.

Tell them I hate them.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#70

Post by johnbatch »

pants2 wrote: post #52
hnburnsy wrote: post #25

Earlier quoted context omitted.

Pro tip, select a section you know is wrong, then de select it before submitting. Seems to help prove you are not a bot.

Another pro tip: the audio version of the captcha is usually much easier / faster to solve if you're in a quiet environment

I wonder how well the AI models would do on the audio version.
Post reply on HN