Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCP Incidents

blog.railway.app

Post reply on HN
Print view View original

121–130 of 167 posts

Re: GCP Incidents

#121

Post by rubiquity »

yibers wrote: post #118
wavemode wrote: post #86

Earlier quoted context omitted.

Oh, for goddamn sure. Half the services on AWS, probably, are very poorly designed or very poorly run (or both). CloudWatch stands out to me as one that is mind-bogglingly buggy and slow. To the point of basically being a "newbie trap" - when I see companies using it for all their logging, I assume it's due to inexperience with the many alternatives. At least the compute services are reliable.

I actually use cloudwatch quite a lot. I didn't notice many bugs or slowness, but I assume I am missing something. Can you perhaps point to some specific issues you had with Cloudwatch?

$$$ and a strange API. The internal metrics service that Amazon has used for ages works much better for power users. CW is slowly becoming like it.

Re: GCP Incidents

#122

Post by mbStavola »

asylteltine wrote: post #85
doubloon wrote: post #48

"reasons why Oxide has a business #12390"

An oxide rack has a minimum cost of something like 600k not including all the infra you need to run a rack, maintenance, and then needing to upgrade

In the post they say they pay Google "multiple millions" of dollars already. Depending on their needs, the TCO of Oxide racks may end up being less than what they pay GCP.

Re: GCP Incidents

#123

Post by asylteltine »

mst wrote: post #108
asylteltine wrote: post #85

Earlier quoted context omitted.

An oxide rack has a minimum cost of something like 600k not including all the infra you need to run a rack, maintenance, and then needing to upgrade

Railway's bill was into the multiple millions per year at the very least so that doesn't necessarily rule it out.

That’s one misconception about leaving cloud people think it’s a one time cost compared to opex, but in reality you are just moving the spending. Devops for your now custom on prem workflows, toil due to inferior tooling compared to cloud, physical costs like electricity and cooling, space for the racks, physical security, high availability, etc I mean there’s so much downside.

Re: GCP Incidents

#124

Post by asylteltine »

mst wrote: post #107
asylteltine wrote: post #83

Earlier quoted context omitted.

GCP is the ugly stepchild of Google. They prioritize their own infra which unlike Aws doesn’t even run on gcp! It’s a joke. They don’t dogfood anything. All Google infra runs on separate systems (both) or dedicated deployment like their own spanner clusters. Google employees look down on gcp employees like second class citizens

> They prioritize their own infra which unlike Aws doesn’t even run on gcp! I don't believe that to be the case, last I heard a year or two back the vast majority of it -does- run on a google GCP tenant account and what didn't was largely at least in the process of migration planning. (my source here is "pillow talk with a senior GCP engineer" and I don't believe she had any reason to lie to me)

That’s not been my experience having done consulting for Google. They may have some stuff on gcp but they don’t dog food much like Aws does (literally running Amazon on top of vanilla dynamo and kinesis). Google has a lot of custom infra

Re: GCP Incidents

#125

Post by hnlmorg »

civilitty wrote: post #98
hnlmorg wrote: post #94

Earlier quoted context omitted.

I was writing resizeable desktop applications in the 90s and it was easy then too. Differing screen resolutions is not a recent thing. The reason for Electron's popularity is because: 1. web developers are cheaper and easier to hire than people with desktop software development experience 2. Electron is cross platform Point 2 is a compelling reason. But it's not like other cross platform toolkits haven't existed. The…

I was there too with a Sony GDM-FW900 monitor with 2300x1400 resolution and my experience was very negative. Both as a user and as a developer (though I was just a kid getting started in ‘99). Any software that wasn’t made by a major company was a crapshoot and Delphi layout was brittle for my own apps. Anything that got deployed to university computers with their bulk purchased monitors was mostly fine except for th…

> I was there too with a Sony GDM-FW900 monitor

I’m talking about the early to mid 90s of web development and you reply with a monitor released around 2000. That’s a completely different era.

> with 2300x1400 resolution and my experience was very negative.

That’s more an issue with OSs not doing scaling back then than it is a problem with desktop software.

You’d still have the same issue today if you run a super high resolution with zero scaling.

> There are more web developers because it was a lot easier

It’s has a lower barrier for entry but I wouldn’t call it easier.

> even before […] frontend frameworks went out of control.

So you’re basically agreeing with me then. The current state of affairs is out of control.

> Electron was really late on the scene as far as frontend tech goes. It was originally made to support an IDE and not regular business apps and still won.

Its origins doesn’t disprove my point. Lots of things start out as one thing and evolve into something else. That isn’t unique to the web nor Electron.

Re: GCP Incidents

#126

Post by mst »

asylteltine wrote: post #123
mst wrote: post #108

Earlier quoted context omitted.

Railway's bill was into the multiple millions per year at the very least so that doesn't necessarily rule it out.

That’s one misconception about leaving cloud people think it’s a one time cost compared to opex, but in reality you are just moving the spending. Devops for your now custom on prem workflows, toil due to inferior tooling compared to cloud, physical costs like electricity and cooling, space for the racks, physical security, high availability, etc I mean there’s so much downside.

I said "doesn't necessarily rule it out" rather than a stronger claim advisedly.

You're entirely correct that there are an unfortunate number of people who hold that misconception, but (a) I'm not one of them (b) that wasn't my point.

Re: GCP Incidents

#127

Post by vel0city »

Wuzado wrote: post #114
vel0city wrote: post #95

It's hilarious people are bashing GCP for having one compute instance go down and the author acknowledges it's a rare event. On AWS I've got instances getting forced stopped or even straight disappearing all the time. 99.95% durability vs 99.999% is way different. If they had the same architecture on AWS it would go down all the time IME. AWS primitives are way less reliable than GCP, according to AWS' docs and my ow…

The article doesn't seem to mention AWS, really. I also feel like the primary issue is the lack of communication and support, even for a large corporate partner. Seems like they're moving to bare-metal, which has an obvious benefit of being able to tell your on-call engineer to fix the issue or die trying.

But in this case the answer from AWS would have been that's their SLA and you need to just be ready to handle an instance getting messed up from time to time, because it's guaranteed to happen.

Re: GCP Incidents

#128

Post by wavemode »

yibers wrote: post #118
wavemode wrote: post #86

Earlier quoted context omitted.

Oh, for goddamn sure. Half the services on AWS, probably, are very poorly designed or very poorly run (or both). CloudWatch stands out to me as one that is mind-bogglingly buggy and slow. To the point of basically being a "newbie trap" - when I see companies using it for all their logging, I assume it's due to inexperience with the many alternatives. At least the compute services are reliable.

I actually use cloudwatch quite a lot. I didn't notice many bugs or slowness, but I assume I am missing something. Can you perhaps point to some specific issues you had with Cloudwatch?

The user interface is unintuitive, text search is slow, querying is even slower, refining the metric graphs to a time period is really annoying, the graph controls are consistently buggy in my experience... those are just off the top of my head

Re: GCP Incidents

#129

Post by civilitty »

hnlmorg wrote: post #125
civilitty wrote: post #98

Earlier quoted context omitted.

I was there too with a Sony GDM-FW900 monitor with 2300x1400 resolution and my experience was very negative. Both as a user and as a developer (though I was just a kid getting started in ‘99). Any software that wasn’t made by a major company was a crapshoot and Delphi layout was brittle for my own apps. Anything that got deployed to university computers with their bulk purchased monitors was mostly fine except for th…

> I was there too with a Sony GDM-FW900 monitor I’m talking about the early to mid 90s of web development and you reply with a monitor released around 2000. That’s a completely different era. > with 2300x1400 resolution and my experience was very negative. That’s more an issue with OSs not doing scaling back then than it is a problem with desktop software. You’d still have the same issue today if you run a super high…

I thought we were talking about "resizeable desktop applications in the 90s"? The reason I bring up that monitor is that I had the misfortune of using dozens of desktop applications written in the 90s to control lab equipment and spent copious amounts of time manually tiling them in all kinds of positions and sizes on a high resolution monitor. Scaling was definitely not the issue thanks to great eyesight. I was lucky if they supported a non 4:3 aspect ratio. Anything that wasn't a graphical or CAD or other app that did its own rendering or was developed by a large dev team was a crapshoot.

Lots of absolutely positioned buttons clipped by a resize, toolbars that didn't hide their buttons behind a menu when the window was too small, uncollapsible panels with minimum widths that exceeded the main content, and so on. Most of them were about as resizable as a water balloon when you smash it on the ground.

Re: GCP Incidents

#130

Post by M_bara »

tlogan wrote: post #104

All these cloud service providers have bugs and issues. But the problem with Google is that their support seems somehow disconnected from the real world. There is support, and they do respond to chats, calls, or emails. However, it often feels like I'm talking to someone who doesn't genuinely care about my concerns or do understand what I’m talking about. Good support is hard to come by and hard to implement. So I re…

The mantra at aws is customer experience. Any time there’s an outage or impact, the first numbers to be stated are customer impact related. In fact, as an engineer you might be empowered enough to recommend a refund for a customer (even though the customer may be at fault) and the refund will go through.

Full disclosure: worked for aws devops for a couple of years

Post reply on HN