Live data from Hacker News

Benchmarking leading AI agents against Google reCAPTCHA v2

research.roundtable.ai

51–60 of 104 posts

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#51
I’ve used LLMs to solve captchas for shits and giggles, just taking a screenshot and pasting it into ChatGPT and having it tell me what squares to click and I think it solves them better than I do.

Can we just get rid of them now, they are so annoying and basically useless.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#52

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

Pro tip, select a section you know is wrong, then de select it before submitting. Seems to help prove you are not a bot.

Another pro tip: the audio version of the captcha is usually much easier / faster to solve if you're in a quiet environment

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#53
post #38
post #36

Earlier quoted context omitted.

The solvers are a problem but they give themselves away when they incorrectly fake devices or run out of context. I run a bot detection SaaS and we've had some success blocking them. Their advertised solve times are also wildly inaccurate. They take ages to return a successful token, if at all. The number of companies providing bot mitigation is also growing rapidly, making it difficult for the solvers to stay on top…

> when they incorrectly fake devices And how often does this happen? Do you have any proof? Most YC companies building browser agents have built-in captcha solvers.

That's a good question. I haven't checked the stats to see how often it happens but I will make a note to return with some info. We're dealing with the entire internet, not just YC companies, and many scrapers / solvers will pass up a user agent that doesn't quite match the JS capabilities you would expect of the browser version. Some solving companies allow you to pass up user agent , which causes inconsistencies as they're not changing their stack to match the user agent you supply. Under the hood they're running whatever version of headless Chrome they're currently pinned to.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#54
post #42

Wow. Cross-tile performance was 0-2%. That's the challenge where you select all of the tiles containing an item where the single item is in a subset of tiles. As opposed to all the tiles that contain the item type (static - 60% max) and the reload version (21% max). Seems to really highlight how far these things are from reasoning or human level intelligence. Although to be fair, the cross-tile is the one I perform w…

I think the prompt is probably at fault here. You can use LLMs for object segmentation and they do fairly well, less than 1% seems too low.

The cross-tile challenges were quite robust - every model struggled with them, and we tried with several iterations of the prompt. I'm sure you could improve with specialized systems, but the models out-of-the-box definitely struggle with segmentation

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#55
post #17

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

Testing those same captcha on Google Chrome improved my accuracy by at least an order of magnitude. Either that or it was never about the buses and fire hydrants.

Yeah, we've looked at it in the context of reCAPTCHA v3 and 'invisible behavioral analysis': https://www.youtube.com/watch?v=UeTpCdUc4Ls

It doesn't catch OpenAI even though the mouse/click behavior is clearly pretty botlike. One hypothesis is that Google reCAPTCHA is overindexing on browser patterns rather than behavioral movement

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#56

Earlier quoted context omitted.

It's a known "issue" of reCaptcha, and many other systems like it. If it thinks you're a bot, it will "fail" the first few correct solves before it lets you through. The worst offenders will just loop you forever, no matter how many solves you get right.

stock Chrome logged into a Google account = definitely not a bot. here, click a few fire hydrants and come on in :^) I sincerely wish all the folx at Google directly responsible for this particular user acquisition strategy to get every cancer available in California.

I would think that when you're viewing recaptcha on a site, if you have 3rd party cookies disabled the embedded recaptcha script won't have anyway of connecting you with your Google account, even if you're logged in. At least that's how disabling 3rd party cookies is supposed to work.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#57

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

The buses and fire hydrants are easy. It is the bicycles. If it goes a pixel over the next box, do I select the next box? Is the pole part of the traffic light? And the guy as you say. There is a special place in hell for the inventor of reCaptcha (and for all of Cloudflare staff as fas as I am concerned!)

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#59
post #57

I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?

The buses and fire hydrants are easy. It is the bicycles. If it goes a pixel over the next box, do I select the next box? Is the pole part of the traffic light? And the guy as you say. There is a special place in hell for the inventor of reCaptcha (and for all of Cloudflare staff as fas as I am concerned!)

It doesn't matter. Select it of you think other people would select it too.

Re: Benchmarking leading AI agents against Google reCAPTCHA v2

#60
post #57

Earlier quoted context omitted.

The buses and fire hydrants are easy. It is the bicycles. If it goes a pixel over the next box, do I select the next box? Is the pole part of the traffic light? And the guy as you say. There is a special place in hell for the inventor of reCaptcha (and for all of Cloudflare staff as fas as I am concerned!)

It doesn't matter. Select it of you think other people would select it too.

That's the thing, you could go either way. I am not sure I can answer the question "what would a resonable person click?".
Post reply on HN