Live data from Hacker News

GCP Incidents

blog.railway.app

151–160 of 167 posts

Re: GCP Incidents

#151
post #111

Earlier quoted context omitted.

I have a direct cross connection to Google in a colocation facility (aka Dedicated Interconnect). One issue I found is that Google would randomly shuffle around their BGP routers which would cause BGP to flap and briefly losing all connectivity. When I raised this issue with support their answer was that this is expected behavior and we need to purchase a redundant connection. Mind you this isn’t cheap, we’re talking…

Google doesn't do any 'black magic' even though they do presentations and publish papers. Their edge infra is very boring, and yes, they shuffle edge a log and their edge routers like everybody's' else - off the shelf Juniper/Cisco without any tcp session preservation.

Migrating an in-use TCP session from one host to another is far from easy.

I did it for a project and the number of corner cases is insane - both in the TCP protocol (what of the window has holes in? What if the connection is half closed?), but also in the OS's handling of the TCP state and interactions with userspace (will be correctly wake up a process poll()'ing a socket if we migrate the socket after a packet is received but before the kernel wakes the poll()ING process?)

Re: GCP Incidents

#152
post #131

It sounds like if you deploy on Railway they don't automatically handle a box dying (e.g. with K8s or other) -- "half the company was called in to go through runbooks." When they move to their own hardware, how will they handle that?

GCP is pretty reliable - for a smallish deployment you could probably go a couple of years before seeing a machine die.

So they probably never built in health checks and auto fail over.

Re: GCP Incidents

#153
post #2

When they say moving are off Google Cloud services to bare metal, where do they plan to move?

Many data centers provide colo/hardware renting facilities, such as Equinix, Coresite, Digital Realty etc. (Even AWS got started off those, though they mostly build their own data centers now.)

When small companies get big, there must sometimes be legacy compute jobs still left on the original infrastructure right? Ie. Jeff's original php script to trigger some automation that never got adopted by any team.

Do all the big tech companies still have a box somewhere full of legacy stuff that 'probably isn't important, but not worth turning off just incase it is'?

Re: GCP Incidents

#154

“ We paid them multiple millions of dollars per year” Never heard of railway but paying this many $$$ per year should give you a dedicated support rep. But google doesn’t do support for anything lol

Oh - they give you a support rep. Just the support rep is powerless to do anything.

Re: GCP Incidents

#155
post #107

Earlier quoted context omitted.

GCP is the ugly stepchild of Google. They prioritize their own infra which unlike Aws doesn’t even run on gcp! It’s a joke. They don’t dogfood anything. All Google infra runs on separate systems (both) or dedicated deployment like their own spanner clusters. Google employees look down on gcp employees like second class citizens

> They prioritize their own infra which unlike Aws doesn’t even run on gcp! I don't believe that to be the case, last I heard a year or two back the vast majority of it -does- run on a google GCP tenant account and what didn't was largely at least in the process of migration planning. (my source here is "pillow talk with a senior GCP engineer" and I don't believe she had any reason to lie to me)

Sorry, but that's not true.

I wish it was. But running anything that bridges google3 and GCP is a nightmare. Outside of acquisitions and OSSish stuff like chrome, it's really rare to see GCP used. Oddly enough their corp eng team does quite a bit with making GCP accessible to the rest of the company.

Source: former Google SRE who actually worked on one of those teams.

Re: GCP Incidents

#156
post #105
post #63

Earlier quoted context omitted.

sorry but the blame here was 100% on Mozilla. No matter which http version, headers should always be treated as case-insensitive. Blanking anything on google here is just stupid. The problem was nih-syndrome and ignored the http spec.

Mozilla are entirely clear that this was their bug. However, GCP changing the default under their infrastructure without prior warning was still unacceptable. Operations work should (IMO must) be conducted with the expectation that any major change like that will expose existing bugs in deployed code. (I've done enough ops work in my life that I'd love to say 'will potentially expose' but in practice there's always -…

GCP does send mails when you abo‘d them. GCP is not to blame if they used auto. Heck if your loadbalancer sends you headers lowercase with a new http version it should not result in a bug. GCP‘s change was fine. Their software had a bug that would‘ve led to request smuggling.

Re: GCP Incidents

#157
post #150

Earlier quoted context omitted.

> I think you're misremembering then. There _was_ no web development to speak of in the early 90s. The web was largely a niche technology until the mid-90s. Mosaic released in January '93, Netscape in October '94, and IE in August '95. By the end of '93, there were a total of 130 websites[1], most of them from universities and research centers. By the end of '94, a whopping 2,278 websites. JavaScript first appeared i…

> My first website went public in 1994. So yours was one of the first 2,278 websites? Congrats. I don't see how any of your accomplishments are relevant, but thanks for sharing. So your point is that the web when JavaScript and CSS were in their infancy, before web standards existed and were widely adopted, before AJAX and when you had to use "a lot of hacks" to implement streaming... that _that_ web was somehow easi…

> So your point is that the web when JavaScript and CSS were in their infancy, before web standards existed and were widely adopted, before AJAX and when you had to use "a lot of hacks" to implement streaming... that _that_ web was somehow easier to work with than the modern web? That sounds delusional.

My point was that the amount of hacks required these days has grown exponentially.

> VRML, along with Java applets, ActiveX, Flash, and a myriad other technologies around that time were decidedly not web-native

Ofcourse they weren't. I never implied otherwise.

> Nowadays all of these technologies are dead, replaced by native web alternatives.

Indeed. Technologies that are exponentially harder to write the same code in. Hence my point: modern web tech is a shitshow.

> Which particular standards are you referring to? Native HTML5/CSS3/ES2015+ are stable and well supported standards, and you've been able to target them for nearly a decade now. Their capabilities are obviously much greater compared to the early web, but this is what happens when platforms evolve. If you dislike using them, then I can't convince you otherwise, but I'm arguing against your point that the state of the web was somehow better in the 90s.

You're fixated on that point and it's not what I said. I said it was easier to grok in the 90s and has just gotten worse over time. Which is a fact.

I also said the current web is an unfit clusterfuck that people are Stockholm syndromed into believing is good. Everything you've posted thus far reinforces that Stockholm syndrome point.

> > React isn't popular any more.

> It's in the top 10 most popular JS libraries. And how come you're judging based on popularity anyhow? Above you were criticizing choosing technologies based on what's "in vogue at the moment" over "what's mature". React is a _mature_ UI library, and is a safe choice in 2023, unless you're chasing the latest hype train.

I haven't worked with a single engineer, how hasn't bitched and moaned about React. And I've managed a lot of engineering teams over the years.

Vue is a different matter.

> JavaScript frameworks as a concept are barely a decade old. React isn't a framework, it's a library.

It's both. The term "framework" has an pretty meaning in software and React falls under that heading quite comfortably. What's happened, and why you're confused, is that kids have overloaded the term with "web framework" to mean something more specific. React on its own isn't a "web framework" in the trendy web sense but it's still 100% a "framework" in the stricter software development sense.

This is actually another great example of the lack of consistency in the web ecosystem.

That all said, React can certainly fall under the "web framework" umbrella. Wikipedia when used in real world systems. Hence why wikipedia lists it: https://en.wikipedia.org/wiki/Comparison_of_JavaScript-based...

> But my point is that _today_ you don't strictly need any of them to build advanced interactive experiences.

You never had to. You're making another strawman argument because you're not only claiming I'm saying you need these frameworks (you don't) but also making it sound like this is something that's only come about because of the modern web (which isn't true).

> I thought the fact that you're reading and typing this on a forum built with simple HTML, CSS and minimal amounts of JS would make this self-evident. (The fact it uses a bespoke backend is irrelevant; this could just as well be served by a mainstream backend stack.)

HN is far from your typical website. lol

> Nice humblebrag again, but if you'd be willing to accept that the web has grown exponentially since the days you were building websites before JavaScript and CSS existed, that there are orders of magnitude more web developers and software now than back then, and that the core web technologies are more mature and stable than they've ever been, then you'd be able to see that the status quo is not so bad.

It's not a "humblebrag", it's an illustration that my opinion comes from years of experience using a multitude of different technologies. Honestly, I think you need to diversify your experience too because your comments fall firmly into the Stockholm syndrome bracket I described by the fact that seem completely unwilling to accept that we could have all the same power of the current web but massively more simplified and elegant if we were to redesign things from the ground up. There are so many footguns that developers need to learn simply because of the way how the web has evolved. And all you keep harping on about is that "its powerful" -- sure. But so is assembly. Yet literally no-one advocates writing commercial desktop software in assembly.

The problem here is trying to convince someone that the domain which they earn their living from is a shitshow, is simply always going to be met with opposition because you have no impartiality. Whereas people like myself and the OP do. And that's why we make the comments we do when we say that the web is unsatisfying to develop against.

Re: GCP Incidents

#158

It's hilarious people are bashing GCP for having one compute instance go down and the author acknowledges it's a rare event. On AWS I've got instances getting forced stopped or even straight disappearing all the time. 99.95% durability vs 99.999% is way different. If they had the same architecture on AWS it would go down all the time IME. AWS primitives are way less reliable than GCP, according to AWS' docs and my ow…

In general in Cloud and as somebody said, you should Architect assuming everything fails all the time.

Re: GCP Incidents

#159
post #150

Earlier quoted context omitted.

> My first website went public in 1994. So yours was one of the first 2,278 websites? Congrats. I don't see how any of your accomplishments are relevant, but thanks for sharing. So your point is that the web when JavaScript and CSS were in their infancy, before web standards existed and were widely adopted, before AJAX and when you had to use "a lot of hacks" to implement streaming... that _that_ web was somehow easi…

> So your point is that the web when JavaScript and CSS were in their infancy, before web standards existed and were widely adopted, before AJAX and when you had to use "a lot of hacks" to implement streaming... that _that_ web was somehow easier to work with than the modern web? That sounds delusional. My point was that the amount of hacks required these days has grown exponentially. > VRML, along with Java applets,…

[flagged]

Re: GCP Incidents

#160
post #158

It's hilarious people are bashing GCP for having one compute instance go down and the author acknowledges it's a rare event. On AWS I've got instances getting forced stopped or even straight disappearing all the time. 99.95% durability vs 99.999% is way different. If they had the same architecture on AWS it would go down all the time IME. AWS primitives are way less reliable than GCP, according to AWS' docs and my ow…

In general in Cloud and as somebody said, you should Architect assuming everything fails all the time.

So why not have EC2s have a 50% SLA. Have them all force quit at some random interval between 2 hours and 200 hours, guaranteed. Have EBS volumes just corrupt your data every week. Why bother with SLAs at all when the solution is buy more redundant resources?

Or how about having actually reliable primitives?

I don't disagree, if you need extreme reliability build your infra to handle multi-az, even multi-region outages. But sometimes I'd rather just have an instance just stay online instead of having to pay for it three times over and still have it reasonably be expected to not corrupt itself. Hypervisor and storage technology could make that happen, as it's true on other clouds and has been true in the data center for decades.

I can have an instance on GCP with it's block storage having 99.9999% durability. I can't do the same with gp3 on AWS without having to deal with the complexity of clustering and all it's headaches and costs, the volume has a durability of 99.95%. Why is that an unreasonable ask?

Post reply on HN