Would performance improve if the tiles were stitched together and fed to a vision model, and then tiles are selected based on a bounding box?
Benchmarking leading AI agents against Google reCAPTCHA v2
31–40 of 104 posts
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#32interesting results. why does reload/cross-tile have worse results? would be nice to see some examples of failed results (how close did it to solving?)
I'm also curious what the success rates are for humans. Personally I find those two the most bothersome as well. Cross-tile because it's not always clear which parts of the object count and reload because it's so damn slow.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#33I hypothesize that these AI agents are all likely higher than human performance now.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#34I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?
Pro tip, select a section you know is wrong, then de select it before submitting. Seems to help prove you are not a bot.
Shhh, you're not supposed to tell people. Now they'll patch it and I'll have to select stairs and goddamn motorcycles 4 times in a row.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#35To be honest I'm surprised how well it holds. I expected close-to-total collapse. It'll be a matter of time I guess, but still.
i mean did you see the cross-tile numbers
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#36If not today, models will get better at solving captchas in the near future. IMHO, the real concern, however, is cheap captcha solving services.
The solvers are a problem but they give themselves away when they incorrectly fake devices or run out of context. I run a bot detection SaaS and we've had some success blocking them. Their advertised solve times are also wildly inaccurate. They take ages to return a successful token, if at all. The number of companies providing bot mitigation is also growing rapidly, making it difficult for the solvers to stay on top of reverse engineering etc.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#37I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?
Not sure it is your case but I think I sometimes had to solve many of them when I am in my daily task rush. My hypothesis is that I solve them too fast for "average human resolving duration" recaptcha seems to expect (I think solving it too fast triggers bot fingerprint). More recently when I fall on a recaptcha to solve, I consciently do not rush it and feel have no more to solve more than one anymore. I don't think…
that, and VPN.
Yes.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#38If not today, models will get better at solving captchas in the near future. IMHO, the real concern, however, is cheap captcha solving services.
The solvers are a problem but they give themselves away when they incorrectly fake devices or run out of context. I run a bot detection SaaS and we've had some success blocking them. Their advertised solve times are also wildly inaccurate. They take ages to return a successful token, if at all. The number of companies providing bot mitigation is also growing rapidly, making it difficult for the solvers to stay on top…
> when they incorrectly fake devices
And how often does this happen? Do you have any proof? Most YC companies building browser agents have built-in captcha solvers.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#39Would performance improve if the tiles were stitched together and fed to a vision model, and then tiles are selected based on a bounding box?
That's a cool idea. I bet it would work better.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#40Earlier quoted context omitted.
Testing those same captcha on Google Chrome improved my accuracy by at least an order of magnitude. Either that or it was never about the buses and fire hydrants.
It's a known "issue" of reCaptcha, and many other systems like it. If it thinks you're a bot, it will "fail" the first few correct solves before it lets you through. The worst offenders will just loop you forever, no matter how many solves you get right.
stock Chrome logged into a Google account = definitely not a bot. here, click a few fire hydrants and come on in :^)
I sincerely wish all the folx at Google directly responsible for this particular user acquisition strategy to get every cancer available in California.