Live data from Hacker News

Ask HN: Google Search down?

news.ycombinator.com

241–250 of 286 posts

Re: Ask HN: Google Search down?

#241

Earlier quoted context omitted.

Google doesn't have HSTS (on google.com).

The irony. Google understands that HSTS negatively impacts availability massively, and shouldn't be used for production websites, but recommends it to others.

Don't forget enforcing it on certain TLDs, even for internal DNS records used for testing.

Re: Ask HN: Google Search down?

#242

Earlier quoted context omitted.

Google doesn't have HSTS (on google.com).

The irony. Google understands that HSTS negatively impacts availability massively, and shouldn't be used for production websites, but recommends it to others.

But HSTS absolutely should be used for production websites.

Re: Ask HN: Google Search down?

#243

Earlier quoted context omitted.

10 hours ago we had a flood of network failures in us-central1 and saw no GCP status changes. We blindly attempted to mitigate in various ways (freezing HPAs because we thought that we were making excessive calls to external infra and getting throttled) and it resolved itself eventually. Maybe we were at fault the entire time but not seeing this issue exposed up on the GCP dashboard is infuriating.

At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board. Maybe it’s the same issue with GCP, or maybe concern for their injured peers made everyone forget to update the status. I really hope the later.

> At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board.

This can't be true, can it? What's the reason to lie, when the lie would be so incredibly obvious?

Re: Ask HN: Google Search down?

#244

Earlier quoted context omitted.

The irony. Google understands that HSTS negatively impacts availability massively, and shouldn't be used for production websites, but recommends it to others.

But HSTS absolutely should be used for production websites.

It really shouldn't. HSTS is a harmful specification which both robs user agency and prohibits good service uptime. In 100% of all cases HSTS is in play, it's pretty much a cert that expired yesterday (aka, not actually a security risk).

In exchange for this problem, HSTS demands that browsers stop acting as user agents and refuse to operate, no matter what the user says. It allows the server to dictate what the browser is supposed to do, without any bypass.

Finally, the number one person who ends up wasting a lot of time trying to bypass HSTS: The dude trying to fix the certificate. A hilarious number of poorly-considered services have admins administrate the certificate from the web interface... which it won't allow the admin to access because of HSTS. This ends up exacerbating a simple problem into a complex one, and adds significantly to downtime.

HSTS should be considered hostile and not recommended until amended with the removal of the "no user agency" clause. Honestly, a real world practical improvement for the entire HTTPS stack would be a more gentle expiration curve (browsers loading sites with expired certs for a week or so with a yellow address bar or something seems like a good idea, it's not like sites publish their private keys the day a cert expires).

Unfortunately, the authors of HSTS did not consider the real world when developing their spec, they considered a virtually non-existent security case they wanted to address, and built a solution to an imaginary problem with significant real world negative impact.

Re: Ask HN: Google Search down?

#245

Earlier quoted context omitted.

I know many people’s immediate reaction is that 500GB seems small, but remember, they’re not storing the whole internet — just an index of it. The whole internet would need a much larger RAID, with literally dozens of hard disks.

They kind of do store the internet though, don't they? They store a cached version of most pages.

Yeah but pages are HTML and HTML compresses extremely well. With the latest algorithms you could probably get as low as one byte per page. Probably even better with a decent middle-out compression algorithm.

(Also yes, you are correct within the realm of reality, but not within the realm of comedy.)

https://en.m.wikipedia.org/wiki/Yes,_and

Re: Ask HN: Google Search down?

#246
post #140

https://www.sfgate.com/news/article/google-electrical-incide... Three people in critical condition after Google data center 'electrical incident' in the Council Bluffs, Iowa datacenter (us-central region, I believe)

not at all relevant to this outage

Do you have anything to add about the root cause?

Re: Ask HN: Google Search down?

#249

Earlier quoted context omitted.

10 hours ago we had a flood of network failures in us-central1 and saw no GCP status changes. We blindly attempted to mitigate in various ways (freezing HPAs because we thought that we were making excessive calls to external infra and getting throttled) and it resolved itself eventually. Maybe we were at fault the entire time but not seeing this issue exposed up on the GCP dashboard is infuriating.

At AWS, if the status changes then someone somewhere gets fired, so a lot of time incidents happen without being recorded on the status board. Maybe it’s the same issue with GCP, or maybe concern for their injured peers made everyone forget to update the status. I really hope the later.

It used to be that one of the best things about working at Google was the "blameless postmortem". As long as you are able to learn from an incident and weren't attempting to look at private data, then you could write up a postmortem document and actually use that as part of a promotion packet. Google would loose a key part of its soul if they were to change that.

Re: Ask HN: Google Search down?

#250

Earlier quoted context omitted.

OK, so in round numbers, Google makes roughly $65 billion in profit per quarter. Dividing by 90, is around $722 million per day, which is roughly $200,000 per second. 2 minutes is quite the expensive fuckup.

I think you're looking at revenue. Year end 2021 was $257,637M, divide by 4 = $64.4B per quarter. Net income was $76B or about $19B per quarter. https://abc.xyz/investor/static/pdf/2021Q4_alphabet_earnings...

I'm so confused. I honestly think we said the same thing. But I'll go with whatever. It wasn't really a serious point.
Post reply on HN