Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GCP Incidents

blog.railway.app

Post reply on HN
Print view View original

11–20 of 167 posts

Re: GCP Incidents

#11

Post by HenryBemis »

363082a9-58a7 wrote: post #4

I've had an experience with GCP that involved a very enterprise-y feature breaking in a way that clearly showed the feature never worked properly up until this point (aside from causing downtime when they tried to quietly fix it). GCP reps proceeded to remind everyone in the call in which they were supposed to explain what happened they were under NDA, because admitting to the above would've been a nightmare for regu…

I always wonder whether an NDA can prevent you speaking/whistleblow to a Regulator, Police, DA, or some (truly) state authority.

I would like to assume, 'no, you can always report a crime'.

Re: GCP Incidents

#12

Post by supriyo-biswas »

esafak wrote: post #2

When they say moving are off Google Cloud services to bare metal, where do they plan to move?

Many data centers provide colo/hardware renting facilities, such as Equinix, Coresite, Digital Realty etc. (Even AWS got started off those, though they mostly build their own data centers now.)

Re: GCP Incidents

#13

Post by supriyo-biswas »

HenryBemis wrote: post #11
363082a9-58a7 wrote: post #4

I've had an experience with GCP that involved a very enterprise-y feature breaking in a way that clearly showed the feature never worked properly up until this point (aside from causing downtime when they tried to quietly fix it). GCP reps proceeded to remind everyone in the call in which they were supposed to explain what happened they were under NDA, because admitting to the above would've been a nightmare for regu…

I always wonder whether an NDA can prevent you speaking/whistleblow to a Regulator, Police, DA, or some (truly) state authority. I would like to assume, 'no, you can always report a crime'.

Company lawyers like flexing their muscles regardless of the actual legality of such agreements or clauses.

As an example, in India, non-competes are outright illegal since the Indian Contracts Act directly states that any clauses in a contract restraining a lawful profession or trade will be disregarded, and yet most companies out there will add a non-compete clause.

Re: GCP Incidents

#14

Post by supriyo-biswas »

Animats wrote: post #6

As someone who's into virtual worlds, and a user of Second Life, it's impressive to see how well those systems stay up. There hasn't been a total outage of Second Life in 5-10 years. Once Amazon's networking went down in a way that prevented new logins for a whole day, but existing logins remained. The 3D world, which has a lot of stuff going on even with no users around, continued to work. This is an extremely compl…

> Amazon's outgoing HTTPS proxy.

Is this ELB or Cloudfront we’re talking about?

Re: GCP Incidents

#15

Post by iot_devs »

dilyevsky wrote: post #8
b112 wrote: post #3

Earlier quoted context omitted.

My response to this, is that there are endless ways, and places to do this. There are far more colos, people that will rent you a rack, and bandwidth, than VPS types. And you can rent servers too, instead of buying your own. Colo is literally 10000x cheaper than many AWS deployments. I've seen million dollar bills drop to tens of thousands per year. And of course, you can always deploy in house, in your own server ro…

I’ve done some modeling and for our high cpu, high egress, medium storage multi-million $ a year deploy it was 70-90% lower cost than cloud when you factor in amortized cost of boxes, remote hands, transit etc. Pretty substantial but not 10000x ;)

Was this comparing AWS on demand pricing or the 3 years plan?

Re: GCP Incidents

#16

Post by dilyevsky »

iot_devs wrote: post #15
dilyevsky wrote: post #8

Earlier quoted context omitted.

I’ve done some modeling and for our high cpu, high egress, medium storage multi-million $ a year deploy it was 70-90% lower cost than cloud when you factor in amortized cost of boxes, remote hands, transit etc. Pretty substantial but not 10000x ;)

Was this comparing AWS on demand pricing or the 3 years plan?

3 year plan. Basically for colos you also get it cheaper if you sign for 3y so that’s more apples to apples. Equipment “annual” cost was calculated over 5-7 year lifetime - I didnt go as far as calculating how much you could recover if you pawned it after 3 years…

Re: GCP Incidents

#17

Post by hermitcrab »

We are a small software company (2 people) and we've also had plenty of issues with Google over the years. Mostly related to Google Adwords. For example:

https://successfulsoftware.net/2015/03/04/google-bans-hyperl...

https://successfulsoftware.net/2016/12/05/google-cpa-bidding...

https://successfulsoftware.net/2020/08/21/google-ads-can-cha...

https://successfulsoftware.net/2021/05/04/wtf-google-ads/

If Google have no interest in providing decent support to the author of the original article, who are paying megabucks to Google, what hope do small businesses like mine have?

Re: GCP Incidents

#18

Post by phartenfeller »

Animats wrote: post #6

As someone who's into virtual worlds, and a user of Second Life, it's impressive to see how well those systems stay up. There hasn't been a total outage of Second Life in 5-10 years. Once Amazon's networking went down in a way that prevented new logins for a whole day, but existing logins remained. The 3D world, which has a lot of stuff going on even with no users around, continued to work. This is an extremely compl…

> The quality of web software is underwhelming

I hate where "web scale" has brought us. Because some 0.01% of giants have tremendous scalability problems, every small project needs an overly complicated architecture that consists of layers of services. In the end, nobody understands the monster, and the complexity brings more issues than it solves. But still, this is somehow a standard today.

I have a lot of love for static HTML sites or simple backend/frontend solutions. The web is great, but current development trends are not.

Re: GCP Incidents

#19

Post by dragoncrab »

HenryBemis wrote: post #11
363082a9-58a7 wrote: post #4

I've had an experience with GCP that involved a very enterprise-y feature breaking in a way that clearly showed the feature never worked properly up until this point (aside from causing downtime when they tried to quietly fix it). GCP reps proceeded to remind everyone in the call in which they were supposed to explain what happened they were under NDA, because admitting to the above would've been a nightmare for regu…

I always wonder whether an NDA can prevent you speaking/whistleblow to a Regulator, Police, DA, or some (truly) state authority. I would like to assume, 'no, you can always report a crime'.

Not in the US or EU. However, they can still sue you in the US and due to the broken legal system there, it will cost you decent money even if they are bound to lose from day 1.

Re: GCP Incidents

#20

Post by strstr »

Sounds like a genuinely frustrating experience.

Bit confused about why nested virt has anything to do with their problems given that they aren’t using virt inside the VMs. Softlocks are a generic indication of a lack of forward progress.

Same confusion with the MMIO instructions comment. If that’s about instruction emulation, not sure why it matters where it happens? It’s both slow and bound for userspace anyway. If it’s supposed to be fast it should basically never be exiting the guest, let alone be emulated.

Sounds like the author is a bit frustrated and (understandably) grasping at whatever straws they can for that most recent incident.

Post reply on HN