[deleted]
GCP Incidents
81–90 of 167 posts
Re: GCP Incidents
#82Earlier quoted context omitted.
> Back in the 90s it was hacky, but the extent of those hacks were small enough that even hobbyists could memorise those edge cases. You're either misremembering, or have some thick rose-tinted glasses on. The late 90s were times of the Wild West Web. Every browser had custom behaviour and rendered pages differently. JavaScript and CSS were still new, and it took years for the implementations to be standardized. Webs…
Agreed. The thing that has changed about the web is the expectations have increased. There is no way that people could make the websites users expect today with only technology from the 90s.
The expectations of apps have skyrocketed too which is why everyone is using Electron to cope. If GUI development were still as easy as the drag and drop builders in Delphi 7 and VB6 with a bunch of static 800x600 monitors and double-click to edit component code-behind, we’d be shoehorning fewer apps into HTML and making more desktop apps.
I remember how big of a pain in the ass Windows distribution (alone) was in the 2000s with entire businesses like InstallShield built around it. Tack compatibility testing on top of that and a desktop app for a single OS because a massive effort. With the web you could just upload PHP and off to the race. That slowly evolved into what we had today via jquery and electron and friends but what won was distribution and ease of development, despite all the handwringing people do about frontend. The grass isn’t greener on the other side and hasn’t been for decades.
It’s not even remotely competitive anymore. My favorite example is GoldenLayout: it takes me less than a few hours to read the docs and implement a full blown tabbed and split screen interface using a third party library and combine it with any framework I want like React, Svelte, or just vanilla JS. Each desktop framework have their own solutions to the problem usually in the form of commercial components but even in the best case, they take lots more time to wire together.
Re: GCP Incidents
#83> In our experience, Google isn’t the place for reliable cloud compute, and it’s sure as heck not the place for reliable customer support. Always was, always will be. For them customers are always the last
GCP is the ugly stepchild of Google. They prioritize their own infra which unlike Aws doesn’t even run on gcp! It’s a joke. They don’t dogfood anything. All Google infra runs on separate systems (both) or dedicated deployment like their own spanner clusters. Google employees look down on gcp employees like second class citizens
Re: GCP Incidents
#84Earlier quoted context omitted.
Yep. If you keep off the “latest trends frameworks” and just keep it as vanilla and simple as possible web development can be productive, scalable and pleasant.
HN does that, and yet has outages and is so non performant that it can't handle when a post gets a couple thousand comments.
Well HN is probably in the top 1% of forums
Re: GCP Incidents
#85"reasons why Oxide has a business #12390"
An oxide rack has a minimum cost of something like 600k not including all the infra you need to run a rack, maintenance, and then needing to upgrade
Re: GCP Incidents
#86No doubt all cloud providers have their problems. For my day job, over the last 2 years we have discovered and reported multiple issues with Keyspaces, Amazon Aurora, and App Runner. In all cases these issues have resulted in performance degradation, and AWS support wasting our time sending us chasing our tails. After many weeks of escalation, we eventually ended up with project leads who confirmed the issues (some o…
Oh, for goddamn sure. Half the services on AWS, probably, are very poorly designed or very poorly run (or both). CloudWatch stands out to me as one that is mind-bogglingly buggy and slow. To the point of basically being a "newbie trap" - when I see companies using it for all their logging, I assume it's due to inexperience with the many alternatives.
At least the compute services are reliable.
Re: GCP Incidents
#87When they say moving are off Google Cloud services to bare metal, where do they plan to move?
A data centre or (less likely) their own office. This was the way things were done not that long ago ;)
And they are terrible for ops and security. People also used incandescent lights not that long ago
Re: GCP Incidents
#88> In our experience, Google isn’t the place for reliable cloud compute In the early days of cloud computing unreliability was understandable, but for Google to be frustrating its large customers in 2023 is a pretty bad look. Curious to know if others have had similar experiences, or if the author was simply unlucky?
[deleted]
Re: GCP Incidents
#89Generally, I think over the last few years, GCP has lost its way.
There was a time several years ago where they were a meaningfully better option when looking at price / performance for compute / storage / bandwidth when compare to AWS. At the time, we did detailed performance testing and cost modeling to prove this for our workload (hundreds of compute engine instances etc).
Support back then was also excellent. One of our early tickets was an obscure networking issue. The request was quickly escalated then passed from engineers in different regions around the world until it was resolved. We were very impressed. It was a change on the GCP end that ended up being reverted. We quickly got to real engineers who competently worked the problem with us to resolution.
The sales team interactions were also better back then. We had a great sales rep who would quickly connect us with any internal resources we needed. The sales rep was a net positive and made our experience with GCP better.
Since then, AWS has certainly caught up and is every bit as good from a cost / performance standpoint. They remain years ahead on many managed services.
The GCP support experience has degraded significantly at this point. Most cases seem to go to outsourced providers who don’t seem able to see any data about the actual underlying GCP infrastructure. We too have detected networking issues that GCP does not acknowledge. The support folks we are dealing with don’t seem to have any greater visibility than we do. It’s pathetic and deeply frustrating. I’m sure it’s just as frustrating for them.
The sales experience is also significantly worse. Our current rep is a significant net negative.
We’ve made significant investments in GCP and we hate seeing this happen. While we would love to see things improve, we don’t see any signs of that actually happening. We are actively working to reduce our GCP spend.
A few years ago, I was a vocal GCP advocate. At this point, I’d have a hard time suggesting anyone build anything new on GCP.
Re: GCP Incidents
#90> In 2022, we experienced continual networking blips from Google’s cloud products. After escalating to Google on multiple occasions, we got frustrated. So we built our own networking stack — a resilient eBPF/IPv6 Wireguard network that now powers all our deployments. Suddenly, no more networking issues. My understanding is that the network is a VLAN programed via switches for VMs so when you create VPC, you're creati…
My guess is that whatever clever network optimizations that Google has are probably interfering with their traffic. By building their own network stack, they are skipping them and also wireguard might be better equipped to dealt with occasional faults as it built on udp which is inherently unreliable.
I have a direct cross connection to Google in a colocation facility (aka Dedicated Interconnect). One issue I found is that Google would randomly shuffle around their BGP routers which would cause BGP to flap and briefly losing all connectivity. When I raised this issue with support their answer was that this is expected behavior and we need to purchase a redundant connection. Mind you this isn’t cheap, we’re talking around $2,000 per month for a 10G connection when you add up all the GCP/colo fees.
It’s pretty laughable that they can’t preserve TCP connections when they migrate their cloud routers around. I have had BGP uptimes on direct cross connects for over a year with other vendors on bare metal.