Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCP Incidents

blog.railway.app

Post reply on HN
Print view View original

111–120 of 167 posts

Re: GCP Incidents

#111

Post by betaby »

Bluecobra wrote: post #90
devsda wrote: post #29

Earlier quoted context omitted.

My guess is that whatever clever network optimizations that Google has are probably interfering with their traffic. By building their own network stack, they are skipping them and also wireguard might be better equipped to dealt with occasional faults as it built on udp which is inherently unreliable.

I have a direct cross connection to Google in a colocation facility (aka Dedicated Interconnect). One issue I found is that Google would randomly shuffle around their BGP routers which would cause BGP to flap and briefly losing all connectivity. When I raised this issue with support their answer was that this is expected behavior and we need to purchase a redundant connection. Mind you this isn’t cheap, we’re talking…

Google doesn't do any 'black magic' even though they do presentations and publish papers. Their edge infra is very boring, and yes, they shuffle edge a log and their edge routers like everybody's' else - off the shelf Juniper/Cisco without any tcp session preservation.

Re: GCP Incidents

#112

Post by xyzzy_plugh »

wavemode wrote: post #93
ransom1538 wrote: post #80

"On December 1st, at 8:52am PST, a box dropped offline; inaccessible. And then, instead of automatically coming back after failover — it didn’t. Our primary on-call engineer was alerted for this and dug in. While digging in, another box fell offline and didn’t come back" This makes no sense. A machine restarted and you had catastrophic failure? VMs reboot time to time. But if you design your setup to completely destr…

Read the article more carefully. The article (the text you quoted, even) clearly states that the machine didn't "restart". It crashed and didn't come back online. And nowhere in the article do they state that this was a "catastrophic failure" - Railway itself didn't go down entirely. But Railway is a deployment company, so they are re-selling these compute resources to their customers to deploy applications. So when…

TFA is a bit too light on details. Boxes due, it's a fact of life. I don't really follow what "didn't come back online" is supposed to mean. Nodes aren't 100% durable. A lot depends on your particular configuration.

In any case, there's no world where all VM failures trigger automatic reboots. Expecting that to be the case just makes no sense. Automatically failing over should be handled at another layer, for which there a many possibilities.

Manually restoring nodes sounds like a "pets, not cattle" problem.

Long ago, we used to run into this on AWS all the time before we started automatically aging them out.

Re: GCP Incidents

#113

Post by lokar »

bushbaba wrote: post #101
Bluecobra wrote: post #90

Earlier quoted context omitted.

I have a direct cross connection to Google in a colocation facility (aka Dedicated Interconnect). One issue I found is that Google would randomly shuffle around their BGP routers which would cause BGP to flap and briefly losing all connectivity. When I raised this issue with support their answer was that this is expected behavior and we need to purchase a redundant connection. Mind you this isn’t cheap, we’re talking…

It’s about scale. Google was built for an order of magnitude greater scale where such reliability of a single link would be cost prohibitive. However in general if you need high uptime, you’ll need multiple peering links. AWS and azure also recommend the same.

It’s more that a single link/router/host/vm/switch/etc can never be reliable enough, so don’t waste time and money chasing that. Build your software to tolerate it. This approach is pervasive throughout all of Googles systems.

Re: GCP Incidents

#114

Post by Wuzado »

vel0city wrote: post #95

It's hilarious people are bashing GCP for having one compute instance go down and the author acknowledges it's a rare event. On AWS I've got instances getting forced stopped or even straight disappearing all the time. 99.95% durability vs 99.999% is way different. If they had the same architecture on AWS it would go down all the time IME. AWS primitives are way less reliable than GCP, according to AWS' docs and my ow…

The article doesn't seem to mention AWS, really. I also feel like the primary issue is the lack of communication and support, even for a large corporate partner.

Seems like they're moving to bare-metal, which has an obvious benefit of being able to tell your on-call engineer to fix the issue or die trying.

Re: GCP Incidents

#115

Post by davidgerard »

monlockandkey wrote: post #72
diamondfist25 wrote: post #66

Earlier quoted context omitted.

What about DO?

You should always reach out to use Digital Ocean, Linode, Vultr as your starting point. Aws and the gang are mega mega expensive compared to what you pay for a vps. If you require services beyond compute, database and storage, then use the big names. Otherwise save yourself headache with complexity, unpredictable and absurd costs. Please don't use AWS especially as a startup, you are going to kill yourself paying for…

my personal site is on Hetzner, fwiw. they are extremely good IME

Re: GCP Incidents

#116

Post by m_sahaf »

chrismorgan wrote: post #99
m_sahaf wrote: post #67

Earlier quoted context omitted.

For web, Flutter has 2 renderers and offers 3 options: - html: Uses HTML elements and CSS - canvaskit: which, as you mentioned, uses canvas to own the full drawing process - auto: defaults to canvaskit on web but html on mobile. Source: https://docs.flutter.dev/platform-integration/web/renderers

Curious, I haven’t heard of the HTML renderer. Is it any good?

I don't know what metric to use of "good", but the throwaway app I've used it for worked flawlessly with both, so good enough for me ¯\_(ツ)_/¯

Re: GCP Incidents

#117

Post by niuzeta »

I wonder how many of these stories it would take before it starts affecting Google's bottom line. I've tinkered with GCP on small side projects, sure - but after exposure of these stories for over a decade in HN, I can never recommend GCP as a serious cloud alternative. I can't imagine I'm the only one in this boat.

Re: GCP Incidents

#118

Post by yibers »

wavemode wrote: post #86
supermatt wrote: post #21

No doubt all cloud providers have their problems. For my day job, over the last 2 years we have discovered and reported multiple issues with Keyspaces, Amazon Aurora, and App Runner. In all cases these issues have resulted in performance degradation, and AWS support wasting our time sending us chasing our tails. After many weeks of escalation, we eventually ended up with project leads who confirmed the issues (some o…

Oh, for goddamn sure. Half the services on AWS, probably, are very poorly designed or very poorly run (or both). CloudWatch stands out to me as one that is mind-bogglingly buggy and slow. To the point of basically being a "newbie trap" - when I see companies using it for all their logging, I assume it's due to inexperience with the many alternatives. At least the compute services are reliable.

I actually use cloudwatch quite a lot. I didn't notice many bugs or slowness, but I assume I am missing something. Can you perhaps point to some specific issues you had with Cloudwatch?

Re: GCP Incidents

#120

Post by wmf »

supriyo-biswas wrote: post #38
Kwpolska wrote: post #30

You should've migrated many months ago, if a cloud provider forces you to build your own networking or registry, you shouldn't use that cloud provider.

Well for folks building out cloud infrastructure, building your own networking stack and registry is a good way to achieve platform independence, without which you'll be left at a disadvantage and vulnerable to the whims of cloud providers who may or may not extend volume discounts, thus indirectly harming your ability to compete.

An over-overlay is almost never the right solution. If you want platform-independent networking you should use an API shim layer that configures the underlying VPCs the way you want.
Post reply on HN