Live data from Hacker News

GCP Incidents

blog.railway.app

31–40 of 167 posts

Re: GCP Incidents

#31

Earlier quoted context omitted.

I always wonder whether an NDA can prevent you speaking/whistleblow to a Regulator, Police, DA, or some (truly) state authority. I would like to assume, 'no, you can always report a crime'.

Not in the US or EU. However, they can still sue you in the US and due to the broken legal system there, it will cost you decent money even if they are bound to lose from day 1.

In Holland too. Even if someone sues you maliciously, you still have to pay the state a fee to be heard. Otherwise the judgement will fall to the enemy party by default.

You can recuperate this cost from them when you win but you're still out of pocket for your time and the money until you manage to cash it from them which can be hard. And they can keep doing it. The system is very unfairly biased in favour of people with lots of money

Re: GCP Incidents

#32

Earlier quoted context omitted.

As someone who has moved from native app dev to web dev I just feel my productivity and satisfaction has plummeted. The stack (html, js, css, browser functionality) that makes up the web is just not fit for the purpose of rich client applications. The development of the thick leaky abstractions that make up web frameworks have consumed millenia of human man hours, and yet the experience of developing & using the resu…

To be fair, you probably should expect to be less proficient and satisfied embracing a stack that you weren't using before.

I’m not the GP but have been building websites off and on since the 1994.

Back in the 90s it was hacky, but the extent of those hacks were small enough that even hobbyists could memorise those edge cases. Whereas these days there are so many footguns and edge cases that even seasoned professionals find it impossible to memorise everything. The amount of trial and error it takes to build a modern site is immense. With native applications it is a lot easier to write unit tests to catch this stuff but how do you unit a CSS property alignment across 3 browser engines? The answer is almost always to hire QA experts because setting up automated tests has become so complicated and burdensome that it’s now a profession in its own right.

It doesn’t help that “web” is really a plethora of different technologies: CSS, HTML, JS, HTTP, TLS, and image libraries like SVG, JPEG, PNG and GIF. And a few of those have entire subcategories of crap to wade through, like cross site origin headers, subtle incompatibilities in ECMAScript, support of different web extensions, different HTTP protocols, CSS incompatibilities, differing viewport sizes, TLS ciphers, etc.

And that’s just user facing code. What about backend? SQL or No-SQL? Table indexes, web server solutions, load balancing, caching, CDNs, backend server side code, how you cluster your web services, where you host it…etc.

Then you have to secure the damn thing because it’s open to the public.

And all this just to produce interactive documents!!

But the worst thing of all is once you’ve finally wrapped your beard around everything, it’s all out of date again because the latest batch of 20-somethings got so fed up learning this shit as well, that they’ve just gone and rewritten half of the stack from scratch again.

It’s all massively over engineered because we keep on trying to polish this turd rather than accepting we have outgrown the web spec and need something more tailored.

And it’s only getting worse. With more and more software switching to Electron and a growing trend of native applications embracing WASM for portable modules, we keep doubling down on this turd — acting like if we spend enough time polishing it then one day it might turn to gold.

So as someone who’s multi-disciplined, I too find web development the least satisfying of all the software development domains I’ve worked on. It’s down right frustrating at times.

Re: GCP Incidents

#33
post #22

> In 2022, we experienced continual networking blips from Google’s cloud products. After escalating to Google on multiple occasions, we got frustrated. So we built our own networking stack — a resilient eBPF/IPv6 Wireguard network that now powers all our deployments. Suddenly, no more networking issues. My understanding is that the network is a VLAN programed via switches for VMs so when you create VPC, you're creati…

It's not creating a VLAN by programming switches. It's all done in an overlay network.

This is out of date but gives you the idea https://www.usenix.org/conference/nsdi18/presentation/dalton

Re: GCP Incidents

#34

You should've migrated many months ago, if a cloud provider forces you to build your own networking or registry, you shouldn't use that cloud provider.

That was the first thing that struck me, the 'workarounds' stagger belief, but they seem to be casually dropped in (?).

If I were in a situation where my company was contemplating implementing building our own registry/network stack, then the benefits of using a cloud provider are gone, and I would have considered moving to another provider... not saying "I can fix him". This feels like a sunken cost perhaps that is the right term.

Re: GCP Incidents

#35
post #24

Interesting, I’m starting to think undocumented thresholds are quite common in GCP. I experienced something similar with Clod Run: inexplicable scaling events based on CPU utilization and concurrent requests (the two metrics that regulate scaling according to their docs). After a lot of back and forth with their (premium) support it turns out there are additional criteria, smthg related to request duration, but of co…

Unnanounced changes too, there was a Firefox outage in 2022 due to GCP:

https://hacks.mozilla.org/2022/02/retrospective-and-technica...

Re: GCP Incidents

#36
post #6

As someone who's into virtual worlds, and a user of Second Life, it's impressive to see how well those systems stay up. There hasn't been a total outage of Second Life in 5-10 years. Once Amazon's networking went down in a way that prevented new logins for a whole day, but existing logins remained. The 3D world, which has a lot of stuff going on even with no users around, continued to work. This is an extremely compl…

> The quality of web software is underwhelming I hate where "web scale" has brought us. Because some 0.01% of giants have tremendous scalability problems, every small project needs an overly complicated architecture that consists of layers of services. In the end, nobody understands the monster, and the complexity brings more issues than it solves. But still, this is somehow a standard today. I have a lot of love for…

Yep. If you keep off the “latest trends frameworks” and just keep it as vanilla and simple as possible web development can be productive, scalable and pleasant.

Re: GCP Incidents

#37

Earlier quoted context omitted.

Not in the US or EU. However, they can still sue you in the US and due to the broken legal system there, it will cost you decent money even if they are bound to lose from day 1.

In Holland too. Even if someone sues you maliciously, you still have to pay the state a fee to be heard. Otherwise the judgement will fall to the enemy party by default. You can recuperate this cost from them when you win but you're still out of pocket for your time and the money until you manage to cash it from them which can be hard. And they can keep doing it. The system is very unfairly biased in favour of people…

This is pretty absurd to me.

I'm not a lawyer so don't quote me on this, but here in Brazil, it seems the losing side has to pay the winning side's lawyers and other expenses. It's not clear cut how much will be paid and there are different rates but the message seems to be clear: don't sue willy-nilly because there will be consequences, so make sure you have a case.

Re: GCP Incidents

#38

You should've migrated many months ago, if a cloud provider forces you to build your own networking or registry, you shouldn't use that cloud provider.

Well for folks building out cloud infrastructure, building your own networking stack and registry is a good way to achieve platform independence, without which you'll be left at a disadvantage and vulnerable to the whims of cloud providers who may or may not extend volume discounts, thus indirectly harming your ability to compete.

Re: GCP Incidents

#39
post #6

As someone who's into virtual worlds, and a user of Second Life, it's impressive to see how well those systems stay up. There hasn't been a total outage of Second Life in 5-10 years. Once Amazon's networking went down in a way that prevented new logins for a whole day, but existing logins remained. The 3D world, which has a lot of stuff going on even with no users around, continued to work. This is an extremely compl…

As someone who has moved from native app dev to web dev I just feel my productivity and satisfaction has plummeted. The stack (html, js, css, browser functionality) that makes up the web is just not fit for the purpose of rich client applications. The development of the thick leaky abstractions that make up web frameworks have consumed millenia of human man hours, and yet the experience of developing & using the resu…

> The stack (html, js, css, browser functionality) that makes up the web is just not fit for the purpose of rich client applications.

you get much better mileage from web pages than rich client apps. one thing i’ve learned and applied everywhere i worked is to let the web be the web, don’t force things that come from other platforms.

Re: GCP Incidents

#40
>We have automated systems in place to detect and resolve this. We’re notified in Discord

Isn't Discord hosted on GCP, too? If it goes down, monitoring also goes down?

Post reply on HN