Live data from Hacker News

GCP Incidents

blog.railway.app

141–150 of 167 posts

Re: GCP Incidents

#141

It's hilarious people are bashing GCP for having one compute instance go down and the author acknowledges it's a rare event. On AWS I've got instances getting forced stopped or even straight disappearing all the time. 99.95% durability vs 99.999% is way different. If they had the same architecture on AWS it would go down all the time IME. AWS primitives are way less reliable than GCP, according to AWS' docs and my ow…

EC2 [0] and GCP Compute [1] have the exact same SLAs, which is 99.99%, dipping below which gets you a 10% refund. Dipping below 95% gets you a 100% refund.

[0] https://aws.amazon.com/compute/sla/

[1] https://cloud.google.com/compute/sla

Re: GCP Incidents

#142
post #71
post #64

Earlier quoted context omitted.

It’s sad to see people rediscovering that GCP is not a serious product over and over again.

What's your personal experience with it? We've been on the platform for almost eight years. Three clusters, hundreds of compute VMs, four or five public and private DNS zones, 10+ cloud sql dbs, about the same number of memorystore and firebase instances. We just don't see these issues as related in the OP, and when we do have problems support has been fast and helpful. Not to gainsay their experience, since it was o…

Not everyone get cancer from cigarettes you know? See the avalanche of horror stories. I had mine.

Re: GCP Incidents

#143

It's hilarious people are bashing GCP for having one compute instance go down and the author acknowledges it's a rare event. On AWS I've got instances getting forced stopped or even straight disappearing all the time. 99.95% durability vs 99.999% is way different. If they had the same architecture on AWS it would go down all the time IME. AWS primitives are way less reliable than GCP, according to AWS' docs and my ow…

This is very different from my experience. In my years with AWS I’ve only had an instance get stopped once for a reason that was weird AWS background stuff that had nothing to do with my application. I don’t think I’ve ever had or even heard of an instance just disappearing.

Re: GCP Incidents

#144
“ We paid them multiple millions of dollars per year”

Never heard of railway but paying this many $$$ per year should give you a dedicated support rep. But google doesn’t do support for anything lol

Re: GCP Incidents

#145

It's hilarious people are bashing GCP for having one compute instance go down and the author acknowledges it's a rare event. On AWS I've got instances getting forced stopped or even straight disappearing all the time. 99.95% durability vs 99.999% is way different. If they had the same architecture on AWS it would go down all the time IME. AWS primitives are way less reliable than GCP, according to AWS' docs and my ow…

EC2 [0] and GCP Compute [1] have the exact same SLAs, which is 99.99%, dipping below which gets you a 10% refund. Dipping below 95% gets you a 100% refund. [0] https://aws.amazon.com/compute/sla/ [1] https://cloud.google.com/compute/sla

By the links you shared instance level SLA on AWS is 99.5%. GCP instance level is 99.99%. That's not the same.

> For each individual Amazon EC2 instance (“Single EC2 Instance”), AWS will use commercially reasonable efforts to make the Single EC2 Instance available with an Instance-Level Uptime Percentage of at least 99.5%

The underlying storage isn't the same as well, and that matters more. EBS is 99.95% durable. Even standard zonal PD's on GCP are >99.99%, balanced are >99.999%, SSDs are >99.9999%.

Even if it was 99.99% (it's not on AWS) what's the point of having your instance be 99.99% if the underlying disks might disappear? That's something I've seen happen multiple times on AWS, never once on GCP.

Re: GCP Incidents

#146

It's hilarious people are bashing GCP for having one compute instance go down and the author acknowledges it's a rare event. On AWS I've got instances getting forced stopped or even straight disappearing all the time. 99.95% durability vs 99.999% is way different. If they had the same architecture on AWS it would go down all the time IME. AWS primitives are way less reliable than GCP, according to AWS' docs and my ow…

This is very different from my experience. In my years with AWS I’ve only had an instance get stopped once for a reason that was weird AWS background stuff that had nothing to do with my application. I don’t think I’ve ever had or even heard of an instance just disappearing.

By "disappear" I mean the instance failed hard and couldn't be restarted. It's just gone. Usually related to the EBS volume dying.

But yeah, usually when they die they can just be relaunched. Still they die way more often on AWS than in GCP, and will just end up staying stopped. Until very recently they couldn't even migrate the instances when the underlying hardware had some maintenance, you had to stop and relaunch it on your own. FFS most decent hypervisors have had live migrations for decades and yet I still get notifications of "this instance will stop on x day..." emails. I should never see that. The cloud provider should keep the instance running forever. There's no excuse.

Re: GCP Incidents

#147
post #2

When they say moving are off Google Cloud services to bare metal, where do they plan to move?

Any business can rent space in a colo pretty easily. The constraint is mostly hiring engineers with experience racking and stacking boxes, and willing to drive to the colo when on call.

Re: GCP Incidents

#148
post #78

Earlier quoted context omitted.

That was the first thing that struck me, the 'workarounds' stagger belief, but they seem to be casually dropped in (?). If I were in a situation where my company was contemplating implementing building our own registry/network stack, then the benefits of using a cloud provider are gone, and I would have considered moving to another provider... not saying "I can fix him". This feels like a sunken cost perhaps that is…

I would bet that they were thinking about colocating and would need to have secured intra-service communication anyhow. In Google this is transparent but at a (probably yet identified facility) it'd be up to them to provide.

(Blogpost Author) Yup. We've been thinking about colocation for a while, so we've just been building these up. Basically all that's left is to make our volume storage bulletproof. We'll do that as we're moving stateless workloads to bare metal early next year, and ideally be off GCP EOY latest

Re: GCP Incidents

#149
post #40

>We have automated systems in place to detect and resolve this. We’re notified in Discord Isn't Discord hosted on GCP, too? If it goes down, monitoring also goes down?

(Blogpost Author). We use Discord to notify. Our monitoring runs directly to PagerDuty for anything we actually need to action on.

Re: GCP Incidents

#150
post #134

Earlier quoted context omitted.

> Or we are talking about different eras. I'm on about early 90s. I think you're misremembering then. There _was_ no web development to speak of in the early 90s. The web was largely a niche technology until the mid-90s. Mosaic released in January '93, Netscape in October '94, and IE in August '95. By the end of '93, there were a total of 130 websites[1], most of them from universities and research centers. By the en…

> I think you're misremembering then. There _was_ no web development to speak of in the early 90s. The web was largely a niche technology until the mid-90s. Mosaic released in January '93, Netscape in October '94, and IE in August '95. By the end of '93, there were a total of 130 websites[1], most of them from universities and research centers. By the end of '94, a whopping 2,278 websites. JavaScript first appeared i…

> My first website went public in 1994.

So yours was one of the first 2,278 websites? Congrats.

I don't see how any of your accomplishments are relevant, but thanks for sharing.

So your point is that the web when JavaScript and CSS were in their infancy, before web standards existed and were widely adopted, before AJAX and when you had to use "a lot of hacks" to implement streaming... that _that_ web was somehow easier to work with than the modern web? That sounds delusional.

VRML, along with Java applets, ActiveX, Flash, and a myriad other technologies around that time were decidedly not web-native (i.e. a W3C standard, implemented by all browsers). They only existed because the primitive state of the early web was incapable of delivering advanced interactive UIs, so there were competing proposals from all sides. Nowadays all of these technologies are dead, replaced by native web alternatives.

> Better capabilities doesn't always equate to something being objectively better. Particularly if those capabilities are a complete clusterfuck to code for, as the current web standards are.

Which particular standards are you referring to? Native HTML5/CSS3/ES2015+ are stable and well supported standards, and you've been able to target them for nearly a decade now. Their capabilities are obviously much greater compared to the early web, but this is what happens when platforms evolve. If you dislike using them, then I can't convince you otherwise, but I'm arguing against your point that the state of the web was somehow better in the 90s.

> The problem isn't the choice. The problem is that "the right technology to use" is more about what's in vogue at the moment than it is about that's mature.

That's a problem caused by the surrounding ecosystem, not the web. How is this different from VRML being replaced by X3D in 3 years? The good thing is that today you can safely rely on native web technologies without fearing that they'll disappear in a few years. (For the most part. Standards still evolve, but once they're widely adopted by browsers, backwards compatibility is kept for a long time. E.g. HTML4/CSS2/ES5 are still supported.)

If you're talking about frontend frameworks and libraries, again: they're not a standard part of the web, and you don't have to use them. If you do, it's on you to manage whatever complexity and difficulty they bring to your workflow.

> True elegance of an ecosystem isn't about raw capabilities, else we'd still be writing everything in assembly. Its about the ease of which it is to accomplish a task.

I fail to see how all the improvements of the past 20 years made things more difficult. The capabilities have evolved because user expectations have grown, and complexity arises from that. But if you were to build the same web sites you were building in the 90s with modern technologies, like your streaming HTML chatrooms site, you would find the experience vastly easier and more productive. This is an objective improvement.

> jQuery isn't recommended any more.

Because it's not needed anymore, because JS has evolved leaps and bounds since 2006, and implementations in all browsers are standardized. It's still the most popular JS library by far, and used by 77.3% of all websites[1].

> React isn't popular any more.

It's in the top 10 most popular JS libraries. And how come you're judging based on popularity anyhow? Above you were criticizing choosing technologies based on what's "in vogue at the moment" over "what's mature". React is a _mature_ UI library, and is a safe choice in 2023, unless you're chasing the latest hype train.

> You talk about "decades" and cannot list a single framework that is still in widespread use and more than 10 years old.

JavaScript frameworks as a concept are barely a decade old. React isn't a framework, it's a library. Like I said, jQuery is the most popular library and is 17 years old. Underscore (2009), Bootstrap (2011), Lodash (2012), and many more, are still in widespread use today.

But my point is that _today_ you don't strictly need any of them to build advanced interactive experiences. If you do want to, though, there are many to choose from that simplify development of modern UIs, without being a "clusterfuck" to work with IME. htmx, Lit and Tailwind are all lightweight, well maintained, and help with quickly iterating without resorting to full-blown frameworks. If you do want a framework, Svelte is now 7 years old, so quite mature, and is very pleasant to use.

> Yes it is. Simply saying it isn't doesn't disprove my point.

I thought the fact that you're reading and typing this on a forum built with simple HTML, CSS and minimal amounts of JS would make this self-evident. (The fact it uses a bespoke backend is irrelevant; this could just as well be served by a mainstream backend stack.)

But to save you a web search, here are other examples courtesy of ChatGPT[2].

> As someone who's written software in more than a dozen different languages for well over 3 decades, every time I come back to writing websites I always feel disappointed that this is what we've decided to standardise on.

Nice humblebrag again, but if you'd be willing to accept that the web has grown exponentially since the days you were building websites before JavaScript and CSS existed, that there are orders of magnitude more web developers and software now than back then, and that the core web technologies are more mature and stable than they've ever been, then you'd be able to see that the status quo is not so bad.

I have more issues with the modern state of centralized mega-corporations and advertising ruining the web than anything I can complain about the technology itself. But that's a separate topic.

[1]: https://w3techs.com/technologies/overview/javascript_library

[2]: https://chat.openai.com/share/6b15c659-e47c-4b64-aace-f8d6e9...

Post reply on HN