Subtle bug in Google can get you banned • Hacker News Board

Re: Subtle bug in Google can get you banned

#11

Post by jacquesm » Sun, Oct 03, 2010, 2:54 AM UTC

po wrote: post #10

jacquesm wrote: post #9

Earlier quoted context omitted.

So, are you seriously telling me that google can't tell the difference between their own toolbar used by a logged in user and a bot? How about changing the toolbar code so it paces the requests to something that sits below the frequency of the 'ban for bot use' trigger? That would seem to me to be an obvious fix.

Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different. I'm assuming the toolbars can't communicate between each other. On toolbar launch, it should pick a random number between 1 and x and wait that many ms before contacting google. Pick x by looking at the number of req/sec that trigger a ban and the high-end number of tabs a power user might restart…

Of course they can communicate with each other, they're extensions, not web pages. The first one could act as the 'master', and proxy all the requests.

It's obvious the limiting is rate based, otherwise this would never have happened, so if it is rate based then the toolbar could pace itself to below that rate. Of course that would 'give away' the rate to observers of the toolbar during a browser restart but they could observe that just the same by checking when they get blocked, so that's no loss.

The toolbar knows I'm logged in, knows that a browser has just restarted and presumably can see how many instances/tabs are open (after all that's what it provides the info on) so it has all the data at it's disposal to make the right decision. This seems like a simple oversight to me (that a user installing the toolbar on a machine with a large number of tabs open would land in this situation).

Re: Subtle bug in Google can get you banned

#12

Post by abraham » Sun, Oct 03, 2010, 3:00 AM UTC

po wrote: post #10

jacquesm wrote: post #9

Earlier quoted context omitted.

So, are you seriously telling me that google can't tell the difference between their own toolbar used by a logged in user and a bot? How about changing the toolbar code so it paces the requests to something that sits below the frequency of the 'ban for bot use' trigger? That would seem to me to be an obvious fix.

Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different. I'm assuming the toolbars can't communicate between each other. On toolbar launch, it should pick a random number between 1 and x and wait that many ms before contacting google. Pick x by looking at the number of req/sec that trigger a ban and the high-end number of tabs a power user might restart…

The toolbar should really only pull the page rank for active tabs. This would effectively only require a single request initially.

Re: Subtle bug in Google can get you banned

#13

Post by gregable » Sun, Oct 03, 2010, 3:26 AM UTC

Generally when this type of thing happens, Google will reply with a captcha that, if you correctly solve it, will let you keep going for a while. I guess toolbar requests might be a little different than web requests.

Re: Subtle bug in Google can get you banned

#14

Post by darrenkopp » Sun, Oct 03, 2010, 3:45 AM UTC

same thing happened to me with auto pagerize extension while incrementally refining my search because none of the results i was getting back were meaningful. i moved to duck duck go and bing. if they don't want me to use them for search, then fine. plenty of alternatives. haven't missed google so far.

Re: Subtle bug in Google can get you banned

#15

Post by proexploit » Sun, Oct 03, 2010, 4:37 AM UTC

po wrote: post #10

jacquesm wrote: post #9

Earlier quoted context omitted.

So, are you seriously telling me that google can't tell the difference between their own toolbar used by a logged in user and a bot? How about changing the toolbar code so it paces the requests to something that sits below the frequency of the 'ban for bot use' trigger? That would seem to me to be an obvious fix.

Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different. I'm assuming the toolbars can't communicate between each other. On toolbar launch, it should pick a random number between 1 and x and wait that many ms before contacting google. Pick x by looking at the number of req/sec that trigger a ban and the high-end number of tabs a power user might restart…

Wouldn't it be simpler to rate limit the toolbar? E.g. it will only try x requests/second? Then bots wouldn't emulate it because it wouldn't be able to provide a high enough rate to be really useful.

Of course, that solution is so simple, I'm sure there's a reason it's not possible.

Re: Subtle bug in Google can get you banned

#16

Post by po » Sun, Oct 03, 2010, 5:00 AM UTC

jacquesm wrote: post #11

po wrote: post #10

Earlier quoted context omitted.

Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different. I'm assuming the toolbars can't communicate between each other. On toolbar launch, it should pick a random number between 1 and x and wait that many ms before contacting google. Pick x by looking at the number of req/sec that trigger a ban and the high-end number of tabs a power user might restart…

Of course they can communicate with each other, they're extensions, not web pages. The first one could act as the 'master', and proxy all the requests. It's obvious the limiting is rate based, otherwise this would never have happened, so if it is rate based then the toolbar could pace itself to below that rate. Of course that would 'give away' the rate to observers of the toolbar during a browser restart but they cou…

Of course they can communicate with each other, they're extensions, not web pages.

Firefox extensions are Javascript CSS and XUL so I don't think that's obvious. I think it's entirely reasonable to assume that they might be sand boxed and have no awareness of each other. Is it one instance of the toolbar per "page-opened" event? Is it one instance per window? What I was describing was a way to stay under the limit without having centralized state-aware rate-limiting code. If that's possible, then yeah sure, do it that way.

It's obvious the limiting is rate based, otherwise this would never have happened, so if it is rate based then the toolbar could pace itself to below that rate.

It's not obvious to me. I think the issue is that the OP opens 50 tabs simultaneously after a crash and each window opens a connection to google without a rate limit of any kind. My idea was a way to do it without a centralized state.

Re: Subtle bug in Google can get you banned

#17

Post by houseabsolute » Sun, Oct 03, 2010, 6:12 AM UTC

jacquesm wrote: post #11

po wrote: post #10

Earlier quoted context omitted.

Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different. I'm assuming the toolbars can't communicate between each other. On toolbar launch, it should pick a random number between 1 and x and wait that many ms before contacting google. Pick x by looking at the number of req/sec that trigger a ban and the high-end number of tabs a power user might restart…

Of course they can communicate with each other, they're extensions, not web pages. The first one could act as the 'master', and proxy all the requests. It's obvious the limiting is rate based, otherwise this would never have happened, so if it is rate based then the toolbar could pace itself to below that rate. Of course that would 'give away' the rate to observers of the toolbar during a browser restart but they cou…

> Of course that would 'give away' . . .

Meh, I have trouble believing that spammers cannot experiment to find this number out themselves. The binary search on rate would require only a handful of IPs before you acquire it to a sufficient resolution for working purposes.

Re: Subtle bug in Google can get you banned

#18

Post by regularfry » Sun, Oct 03, 2010, 6:30 AM UTC

po wrote: post #10

jacquesm wrote: post #9

Earlier quoted context omitted.

So, are you seriously telling me that google can't tell the difference between their own toolbar used by a logged in user and a bot? How about changing the toolbar code so it paces the requests to something that sits below the frequency of the 'ban for bot use' trigger? That would seem to me to be an obvious fix.

Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different. I'm assuming the toolbars can't communicate between each other. On toolbar launch, it should pick a random number between 1 and x and wait that many ms before contacting google. Pick x by looking at the number of req/sec that trigger a ban and the high-end number of tabs a power user might restart…

> Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different.

If they do, isn't that precisely what Google would want? Isn't it only the rate and volume of requests that are a problem?

Re: Subtle bug in Google can get you banned

#19

Post by mfukar » Sun, Oct 03, 2010, 6:33 AM UTC

jacquesm wrote: post #9

Matt_Cutts wrote: post #8

There are a lot of people that scrape Google pretty badly, so we do need to have protection against bots, including ones that look like the Google toolbar. If you're resuming ~50 tabs, I can believe that might look like a scraper to us for while. I'm glad you could do regular Google searches after 15 minutes or so.

So, are you seriously telling me that google can't tell the difference between their own toolbar used by a logged in user and a bot? How about changing the toolbar code so it paces the requests to something that sits below the frequency of the 'ban for bot use' trigger? That would seem to me to be an obvious fix.

You're oversimplifying a real problem. Would you "authenticate" your own toolbar somehow? That's probably excess effort for a temporary 15 minute ban. Would you rate limit it? Well that's a lost cause if I ever saw one; bots can rate limit themselves too.

It's not an easy problem to solve.

Re: Subtle bug in Google can get you banned

#20

Post by dmoney » Sun, Oct 03, 2010, 6:44 AM UTC

regularfry wrote: post #18

po wrote: post #10

Earlier quoted context omitted.

Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different. I'm assuming the toolbars can't communicate between each other. On toolbar launch, it should pick a random number between 1 and x and wait that many ms before contacting google. Pick x by looking at the number of req/sec that trigger a ban and the high-end number of tabs a power user might restart…

> Bots can probably perfectly duplicate the behavior of a toolbar. Only the rate and volume of requests would be different. If they do, isn't that precisely what Google would want? Isn't it only the rate and volume of requests that are a problem?

They might not want someone to build a large database of pageranks.