Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCP Incidents

blog.railway.app

Post reply on HN
Print view View original

21–30 of 167 posts

Re: GCP Incidents

#21

Post by supermatt »

No doubt all cloud providers have their problems.

For my day job, over the last 2 years we have discovered and reported multiple issues with Keyspaces, Amazon Aurora, and App Runner. In all cases these issues have resulted in performance degradation, and AWS support wasting our time sending us chasing our tails. After many weeks of escalation, we eventually ended up with project leads who confirmed the issues (some of which they were already aware of, yet the support teams had wasted our time anyway!) and (some of them) have since been resolved.

We are stuck with Keyspaces for the time being, but now refuse to use any non core services (EC2, EBS, S3). As soon as you venture away from those there be dragons.

Re: GCP Incidents

#22

Post by wg0 »

> In 2022, we experienced continual networking blips from Google’s cloud products. After escalating to Google on multiple occasions, we got frustrated. So we built our own networking stack — a resilient eBPF/IPv6 Wireguard network that now powers all our deployments. Suddenly, no more networking issues.

My understanding is that the network is a VLAN programed via switches for VMs so when you create VPC, you're creating a VLAN probably.

So how can an overlay (UDP/Wire guard) be more reliable if the underlaying network isn't stable?

PS: Had even 1/10th of issues have happened on AWS with such a customer, their army of solution architects would be camping in conference rooms every other week reviewing architecture, taking support engineers on call and what not.

Re: GCP Incidents

#23

Post by ur-whale »

esafak wrote: post #2

When they say moving are off Google Cloud services to bare metal, where do they plan to move?

> where do they plan to move?

Basement of their office?

We reached the same conclusion they did a while back and went back to good-old self-hosted.

Reliability has been as good as cloud and TCO is divided by a factor of 10.

Re: GCP Incidents

#24

Post by simo7 »

Interesting, I’m starting to think undocumented thresholds are quite common in GCP.

I experienced something similar with Clod Run: inexplicable scaling events based on CPU utilization and concurrent requests (the two metrics that regulate scaling according to their docs).

After a lot of back and forth with their (premium) support it turns out there are additional criteria, smthg related to request duration, but of course nobody was able to explain in details.

Re: GCP Incidents

#25

Post by Hammershaft »

Animats wrote: post #6

As someone who's into virtual worlds, and a user of Second Life, it's impressive to see how well those systems stay up. There hasn't been a total outage of Second Life in 5-10 years. Once Amazon's networking went down in a way that prevented new logins for a whole day, but existing logins remained. The 3D world, which has a lot of stuff going on even with no users around, continued to work. This is an extremely compl…

As someone who has moved from native app dev to web dev I just feel my productivity and satisfaction has plummeted. The stack (html, js, css, browser functionality) that makes up the web is just not fit for the purpose of rich client applications.

The development of the thick leaky abstractions that make up web frameworks have consumed millenia of human man hours, and yet the experience of developing & using the resulting web applications is still trash.

Re: GCP Incidents

#26

Post by pseg134 »

dilyevsky wrote: post #8
b112 wrote: post #3

Earlier quoted context omitted.

My response to this, is that there are endless ways, and places to do this. There are far more colos, people that will rent you a rack, and bandwidth, than VPS types. And you can rent servers too, instead of buying your own. Colo is literally 10000x cheaper than many AWS deployments. I've seen million dollar bills drop to tens of thousands per year. And of course, you can always deploy in house, in your own server ro…

I’ve done some modeling and for our high cpu, high egress, medium storage multi-million $ a year deploy it was 70-90% lower cost than cloud when you factor in amortized cost of boxes, remote hands, transit etc. Pretty substantial but not 10000x ;)

[flagged]

Re: GCP Incidents

#27

Post by curtisblaine »

Hammershaft wrote: post #25
Animats wrote: post #6

As someone who's into virtual worlds, and a user of Second Life, it's impressive to see how well those systems stay up. There hasn't been a total outage of Second Life in 5-10 years. Once Amazon's networking went down in a way that prevented new logins for a whole day, but existing logins remained. The 3D world, which has a lot of stuff going on even with no users around, continued to work. This is an extremely compl…

As someone who has moved from native app dev to web dev I just feel my productivity and satisfaction has plummeted. The stack (html, js, css, browser functionality) that makes up the web is just not fit for the purpose of rich client applications. The development of the thick leaky abstractions that make up web frameworks have consumed millenia of human man hours, and yet the experience of developing & using the resu…

To be fair, you probably should expect to be less proficient and satisfied embracing a stack that you weren't using before.

Re: GCP Incidents

#28

Post by StopHammoTime »

I have a lot of interaction with Google Cloud Support, mostly around their managed services. I am genuinely not over-impressed with their service, considering with similar employers of size on AWS the support experience was always wonderful.

However, I will say if you are on Google Cloud and you have a positive interaction, make a big deal about someone helping you. Given the rarity it occurs, it’s not a big deal to really go out of your way to reward someone with some emphatic positive feedback. I’ve had four genuinely fantastic experiences and there’s always a message to a TAM that flows soon after. I hope more people like those I interacted with get rewarded and promoted.

Re: GCP Incidents

#29

Post by devsda »

wg0 wrote: post #22

> In 2022, we experienced continual networking blips from Google’s cloud products. After escalating to Google on multiple occasions, we got frustrated. So we built our own networking stack — a resilient eBPF/IPv6 Wireguard network that now powers all our deployments. Suddenly, no more networking issues. My understanding is that the network is a VLAN programed via switches for VMs so when you create VPC, you're creati…

My guess is that whatever clever network optimizations that Google has are probably interfering with their traffic.

By building their own network stack, they are skipping them and also wireguard might be better equipped to dealt with occasional faults as it built on udp which is inherently unreliable.

Re: GCP Incidents

#30

Post by Kwpolska »

You should've migrated many months ago, if a cloud provider forces you to build your own networking or registry, you shouldn't use that cloud provider.
Post reply on HN