3 models only, can we really call that a benchmark?
yes
Benchmarking leading AI agents against Google reCAPTCHA v2
21–30 of 104 posts
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#22So, when do we reach a level where AI is better than humans and we remove captcha from pages alltogether? If you don't want bots to read content, don't put it online, you're just inconveniencing real people now.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#23I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?
Testing those same captcha on Google Chrome improved my accuracy by at least an order of magnitude. Either that or it was never about the buses and fire hydrants.
It's a known "issue" of reCaptcha, and many other systems like it. If it thinks you're a bot, it will "fail" the first few correct solves before it lets you through.
The worst offenders will just loop you forever, no matter how many solves you get right.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#24[deleted]
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#25I’m sure they do better than me. Sometimes I get stuck on an endless loop of buses and fire hydrants. Also, when they ask you to identify traffic lights, do you select the post? And when it’s motor/bycicles, do you select the guy riding it?
Pro tip, select a section you know is wrong, then de select it before submitting. Seems to help prove you are not a bot.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#26If not today, models will get better at solving captchas in the near future. IMHO, the real concern, however, is cheap captcha solving services.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#27So, when do we reach a level where AI is better than humans and we remove captcha from pages alltogether? If you don't want bots to read content, don't put it online, you're just inconveniencing real people now.
They can also sign up and post spam/scams. There are a lot of those spam bots on YouTube, and there probably would be a lot more without any bot protection. Another issue is aggressive scrapers effectively DOSing a website. Some defense against bots is necessary.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#28Seems like Google Gemini is tied for the best and is the cheapest way to solve Google's reCAPTCHA. Will be interesting to see how Gemini 3 does later this year.
After watching hundreds of these runs, Gemini was by far the least frustrating model to observe.
In my admittedly limited-domain tests, Gemini did _far_ better at image recognition tasks than any of the other models. (This was about 9 months ago, though, so who knows what the current state of things). Google has one of the best internal labeled image datasets, if not the best, and I suspect this is all related.
Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#29Re: Benchmarking leading AI agents against Google reCAPTCHA v2
#30Ok and then? Those models were not trained for this purpose.
It's like the last hype over using generative AI for trading.
You might use it for sentiment analysis, summarization and data pre-processing. But classic forecast models will outperform them if you feed them the right metrics.