To be honest I'm surprised how well it holds. I expected close-to-total collapse. It'll be a matter of time I guess, but still.
Benchmarking leading AI agents against Google reCAPTCHA v2
11–20 of 104 posts
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#12I know people were solving CAPTCHAS with neural nets (with PHP no less!) back in 2009.
You could definitely do better than we do here - this was just a test of how well these general-purpose systems are out-of-the-box
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#13I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?
While running this I looked at hundreds and hundreds of captchas. And I still get rejected on like 20% of them when I do them. I truly don't understand their algorithm lol
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#14To be honest I'm surprised how well it holds. I expected close-to-total collapse. It'll be a matter of time I guess, but still.
Same! As we talk about in the article, the failures were less from raw model intelligence/ability than from challenges with timing and dynamic interfaces
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#153 models only, can we really call that a benchmark?
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#16interesting results. why does reload/cross-tile have worse results? would be nice to see some examples of failed results (how close did it to solving?)
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#17I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?
Testing those same captcha on Google Chrome improved my accuracy by at least an order of magnitude.
Either that or it was never about the buses and fire hydrants.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#18interesting results. why does reload/cross-tile have worse results? would be nice to see some examples of failed results (how close did it to solving?)
We have an example of a failed cross-tile result in the article - the models seem like they're much better at detecting whether something is in an image vs. identifying the boundaries of those items. This probably has to do with how they're trained - if you train on descriptions/image pairs, I'm not sure how well that does at learning boundaries.
Reload are challenging because of how the agent-action loop works. But the models were pretty good at identifying when a tile contained an item.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#19Seems like Google Gemini is tied for the best and is the cheapest way to solve Google's reCAPTCHA. Will be interesting to see how Gemini 3 does later this year.
After watching hundreds of these runs, Gemini was by far the least frustrating model to observe.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#20I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?
Not sure it is your case but I think I sometimes had to solve many of them when I am in my daily task rush. My hypothesis is that I solve them too fast for "average human resolving duration" recaptcha seems to expect (I think solving it too fast triggers bot fingerprint). More recently when I fall on a recaptcha to solve, I consciently do not rush it and feel have no more to solve more than one anymore. I don't think I have super powers, but as tech guy I do a lot a computing things mechanically.