Live data from Hacker News

Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

news.ycombinator.com

11–19 of 19 posts

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#11
post #6

Sounds more like an issue on cloud flare or whatever server Google is hitting. It should return 429 so Google can reduce the requests. I'm not even sure why you would rate limit in the first place? IPS are not unique. One company gateway or university for example has plenty of users. Rate limit requests from users you know and make sure every public API is properly cached.

Cloudflare doesn't offer rate limiting by user. Caching is completely useless as a replacement for rate limiting. Simply add a bunch of nonsense to the url, or change query parameters, and you will bypass the cache.

Cloudflare does offer rate limiting - it's in Security / WAF / Rate limiting rules.

On the free plan it is perhaps a bit limited in the rules you can create (match based on bot or URL regex, limit by # requests per IP per 10 seconds). But still pretty useful for slowing stupid requests.

https://developers.cloudflare.com/waf/rate-limiting-rules/cr...

And / or you can use Transform Rules to better normalize the Origin URLs so that the query string or other path info doesn't create new origin requests. Or more simply, enable Ignore Query String in the Cache Settings.

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#12
post #9

Is there any possibility there is a problem with the 429 response? So something is converting to say a bad gateway error because it doesn’t like the 429? Not trying to criticize if this was already checked. Just something I’d try to double check out of being overly cautious.

with spammed curl -i, I get Too many requests, please try again later.HTTP/2 429 date: Wed, 14 Aug 2024 22:09:40 GMT content-type: text/html;charset=UTF-8 expires: Thu, 19 Nov 1981 08:52:00 GMT cache-control: no-store, no-cache, must-revalidate pragma: no-cache retry-after: 3600 vary: Accept-Encoding set-cookie: PHPSESSID=b25ece07c8923fa6b66b14599e1ed545; path=/ cf-cache-status: DYNAMIC report-to: {"endpoints":[{"url…

It's usually used to prevent caching from older caching systems.

Fun fact: this specific value of Expires is the birthday of Sasha Schumann, the person who added this code to PHP[1][2].

[1]: https://stackoverflow.com/a/8194500/2805120

[2]: https://github.com/php/php-src/blob/master/ext/session/sessi...

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#13

Is there any possibility there is a problem with the 429 response? So something is converting to say a bad gateway error because it doesn’t like the 429? Not trying to criticize if this was already checked. Just something I’d try to double check out of being overly cautious.

I can't find a problem with the 429 ...

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#14

I have gotten these emails as well for a Cloudflare-hosted site. I was surprised when I got them because they seemed to crop up out of the blue (a month or so ago?) and AFAIK I couldn't think of much that could actually cause a 500 status code on my end. I'll take a look for 429s. Cheers.

Cool. Let me know if it turns out it's the same thing.

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#15

Sounds more like an issue on cloud flare or whatever server Google is hitting. It should return 429 so Google can reduce the requests. I'm not even sure why you would rate limit in the first place? IPS are not unique. One company gateway or university for example has plenty of users. Rate limit requests from users you know and make sure every public API is properly cached.

I rate limit because of aggressive bots (scrapers and ones trying random injection attacks etc).

And yes, it's returning 429 ...

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#16
post #15

Sounds more like an issue on cloud flare or whatever server Google is hitting. It should return 429 so Google can reduce the requests. I'm not even sure why you would rate limit in the first place? IPS are not unique. One company gateway or university for example has plenty of users. Rate limit requests from users you know and make sure every public API is properly cached.

I rate limit because of aggressive bots (scrapers and ones trying random injection attacks etc). And yes, it's returning 429 ...

[dead]

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#17
post #9

Is there any possibility there is a problem with the 429 response? So something is converting to say a bad gateway error because it doesn’t like the 429? Not trying to criticize if this was already checked. Just something I’d try to double check out of being overly cautious.

with spammed curl -i, I get Too many requests, please try again later.HTTP/2 429 date: Wed, 14 Aug 2024 22:09:40 GMT content-type: text/html;charset=UTF-8 expires: Thu, 19 Nov 1981 08:52:00 GMT cache-control: no-store, no-cache, must-revalidate pragma: no-cache retry-after: 3600 vary: Accept-Encoding set-cookie: PHPSESSID=b25ece07c8923fa6b66b14599e1ed545; path=/ cf-cache-status: DYNAMIC report-to: {"endpoints":[{"url…

Lower case headers are required for http/2 requests and responses.

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#18
How do you rate limit unidentified users? If you are treating all unidentified users, or at least all Google bots as 1 user and they consider each Google bot instances with a unique ip and no credentials a unique user and are hitting a 429 with a single request they may be treating that as a 503 instead.

Re: Tell HN: Server error (5xx) in Google Search Console may not be 5xx at all

#19

How do you rate limit unidentified users? If you are treating all unidentified users, or at least all Google bots as 1 user and they consider each Google bot instances with a unique ip and no credentials a unique user and are hitting a 429 with a single request they may be treating that as a 503 instead.

Simply identifying them by IP so that each Google bot instance is a unique user ...
Post reply on HN