Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Google Cloud suspended our production projects at 1am on Saturday

news.ycombinator.com

Post reply on HN
Print view View original

511–520 of 531 posts

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#511

Post by rglullis »

ufmace wrote: post #494
rglullis wrote: post #175

Earlier quoted context omitted.

The point is not to limit the size of the industry. It is to limit the size of a company . Think of a oil refinery as a monolith, and that such a mandate would enforce that you'd have to build the whole system a set of independent microservices. The industry as a whole would still employ a large number of people, but now they would be forced to coordinate through specific interfaces (the smaller business units) inste…

I know you meant a general idea, but for oil refineries in particular, and probably a bunch of others, this sounds disastrously bad from a safety perspective. A lot of accidents that are tremendously costly in terms of human lives and health, equipment destroyed, and operational uptime eventually trace back to poor communication between groups. There needs to be some top-level department with stop-work authority over…

> There needs to be some top-level department with stop-work authority over everything to coordinate who's doing what maintenance when, equipment replacements and switchovers, and the overall plan for what's being produced when.

I find it very hard to believe that this can not be automated to the point of requiring much more than a dozen controllers (4 groups of 3 people, on a 12-on/36-off work schedule). And there is nothing stopping this "top-level" department to be a company in itself.

But anyway, let's say that I'm underestimating the amount of human brainpower that is needed to get this work done safely. Not to be a Luddite, but shouldn't we be asking ourselves if there really is any type of (isolated) economical activity that is so fundamental for us that can only be done with more than 150 people involved?

It's not like we can not refine oil with less than 150 people. It's just that we can not do it at the scale and efficiency that we are used to, right? Instead of having one "giant" refinery that requires thousands of people, perhaps the limitation would force us to have smaller plants spread out around the world, or to have the downstream industries actually doing the refinement themselves, etc.

So, how about instead of looking for maximizing the efficient usage of resources in order to keep the economy growing at an accelerated pace, we start to focus on a constant growth rate and better distribution?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#512

Post by sumedh »

rglullis wrote: post #243
sumedh wrote: post #229

Earlier quoted context omitted.

Capitalism would not allow such a company to exists. There would be too much bottlenecks and the left hand not knowing what the right hand is doing scenario and the end product would be more expensive compared to a company with no such limitation. Buyers would prefer buying the product at much cheaper rates and then just transporting it to their locations.

I could easily argue the opposite: in a scenario where all companies are reduced in size, they would be forced to become a lot more efficient in their communication and they would be a lot more open and agile. Also, a lot of the work done by people in Big Corp is something that is not outsourced merely because they don't want to let their competitors to have access it. A lot of these redundancies would be eliminated…

> in a scenario where all companies are reduced in size

What does all companies mean, are you proposing the entire world follow this approach or just the US. If its just the US then other refineries would just offer cheaper oil and US consumers would buy it from overseas while the US refining goes bankrupt.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#513

Post by michaelcampbell »

baobabKoodaa wrote: post #133
intellectronica wrote: post #115

Generally, all cloud providers have two ways of engaging with and billing customers. One is using a credit card, which is very easy for getting started. The other one is having an established relationship with a sales / account team and setting up billing by invoice. Smaller companies sometimes don't understand that and stick with credit card billing for too long. Once you move to production and have your own custome…

Ah yes, it was the customer's fault. Stupid customer using method recommended by service provider.

I'll counterargue with a saying from the driving context; "The graveyard is full of people who were `right`."

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#514

Post by notyourday »

aseipp wrote: post #446
MOARDONGZPLZ wrote: post #427

Earlier quoted context omitted.

Same experiences here. AWS was awesome. They did crazy things I wouldn’t expect like spend days in person and join us at equinix colo to get the direct connect working. We didn’t even spend that much relatively, under $100k/mo.

At a prior gig we were using their FPGA instances and had a way of causing hard faults on the system that required reboots, with specific FPGA bitstreams. Their systems monitored this and when we reached out about it they scheduled at least 2-3 different calls with the engineers/PIs who were directly responsible so we could coordinate with them and help pin down the issue to get things working. We were literally rent…

The funniest thing was that we got about a million in CAF bucks from them to go contract and move more stuff from AWS ( our monthly was low triple digits thousands ) and their AMs still messed everything up.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#515

Post by prirun »

alyandon wrote: post #459
rurp wrote: post #444

Earlier quoted context omitted.

The last time I tried to make a new Google account it required a valid phone number, so this might require getting a second phone line these days. I wouldn't trust Google to only disable one account if there is a related one using the same number.

Even that won't protect you from being banned if Google ever infers that the two accounts are related. I just refuse to use Google for anything other than mail at this point.

I even got off gmail because of these horror stories. If they banned my gmail account it would be a major pain and I'd have no recourse.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#516

Post by splix »

Same happened to us the last Thursday, but in our case they suspended our GCP account without a prior notification. Just learned about this from our external monitoring.

Turns out they wanted to check my IDs, and it's after I had been their customer for 14 years. And it took them 5 days to verify that ID. An hour ago they activated it back.

I've been running a few projects on GCP, with active customers, which I lost now. Those 5 days of outrage is basically a catastrophe for us.

And no apology was given by Google. There was barely any communication from their side.

Never use GCP for production.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#517

Post by deelowe »

rglullis wrote: post #72
deelowe wrote: post #62

Earlier quoted context omitted.

Because it’s regressive. We need to fix the issues, not throw the whole thing out.

These "issues" only exist because of the scale and reach of Corporations. You can not fix "too big to fail" issues unless you stop letting these institutions to grow too big in the first place. I'm yet to hear a single convincing argument of why we need to have organizations of such a large scale nowadays. Could you make a case of what we would be missing if we just "threw the whole thing out"?

Because scale breeds efficiencies. It should be obvious why WalMart is a better business model than the previous experience of shopping at 15 different stores to get the same stuff.

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#518

Post by rglullis »

sumedh wrote: post #512
rglullis wrote: post #243

Earlier quoted context omitted.

I could easily argue the opposite: in a scenario where all companies are reduced in size, they would be forced to become a lot more efficient in their communication and they would be a lot more open and agile. Also, a lot of the work done by people in Big Corp is something that is not outsourced merely because they don't want to let their competitors to have access it. A lot of these redundancies would be eliminated…

> in a scenario where all companies are reduced in size What does all companies mean, are you proposing the entire world follow this approach or just the US. If its just the US then other refineries would just offer cheaper oil and US consumers would buy it from overseas while the US refining goes bankrupt.

Ok. Let's think it through.

1) The price of oil (and basically everything downstream of crude) has little to do with the cost to produce it, and a lot more to do with current market conditions. So even if their productions costs were (slightly) higher, I don't see why a local, smaller, refinery would have too much trouble to keep themselves profitable even if they had to have lower margins to stay competitive.

2) Smaller companies would mean that you'd be consuming a lot less primary resources and your business relationships would be on a much more local scale. You wouldn't be getting a networks of branded gas stations that receive their gas from the same mega-distributor, you'd be more likely than many different gas station owners, each of them too small to be interesting to a foreign refinery.

3) Even if that is not enough and "US refining goes bankrupt"... so what? Why do Americans need to have their own refining industry, if they can be served by others?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#519

Post by rglullis »

deelowe wrote: post #517
rglullis wrote: post #72

Earlier quoted context omitted.

These "issues" only exist because of the scale and reach of Corporations. You can not fix "too big to fail" issues unless you stop letting these institutions to grow too big in the first place. I'm yet to hear a single convincing argument of why we need to have organizations of such a large scale nowadays. Could you make a case of what we would be missing if we just "threw the whole thing out"?

Because scale breeds efficiencies. It should be obvious why WalMart is a better business model than the previous experience of shopping at 15 different stores to get the same stuff.

"Efficiency" is not a goal in itself, and it does not pass as an argument here.

1) Walmart may be a better "business model", but is it a better model for a society that we want to live? I surely don't want to live in a place where big box stores are the norm, and the only alternatives are crazy expensive "niche" stores. And I surely don't want to live in a world where the quality of my food is determined by Walmart's weight, who will favor products that can last longer, are cheaper to produce and are easier to store and transport - i.e, ultra processed crap. If the price to pay to have better groceries is the "inconvenience" that I will have to go to 2-3 separate shops, it's absolutely worth it and I'll gladly accept the "inefficiency".

2) Optimizing for efficiency is a recipe for systemic, catastrophic failures. Just as an example: think of the semi-weekly github outages that are happening. Everyone is convinced that the cost of paying for a SaaS is negligible compared to the cost of operating your own, so they don't even try to set up their own CI and code repository and have to pray everyday to make sure that will be able to work with it. Meanwhile, my self-hosted gitea/drone/docker repo has been running for almost 4 years already with no issues and it requires minimal maintenance. It took me some time initially to set things up in a way that I was satisfied, and it probably doesn't save me any money compared with a off-the-shelf solution, but thanks to my initial investment and willingness to accept these costs I am more resilient than any competitor.

3) Taken to an extreme, focusing on "efficiency" could be used to justify authoritarian governments and the most dystopian worlds. I really do not share this techno-utilitarian worldview that thinks that maximizing economic output can justify the existence of corporations that reduce us to nothing but consumers that can be placed in a segmented box. It's this worldview that is brought us Big Data, the invasion of our privacy, the "gamification" of everything, and so on. Google/Apple/Microsoft/Amazon may all be trillion-dollar companies and may have built incredible products, but the societal/environmental/civil cost is just too much to be worth it. If I knew in 2004 that by accepting the invite to 1GB of Gmail I would be contributing to the emergence of Surveillance Capitalism, I'd never had done it.

Anything else you'd like to try?

Re: Tell HN: Google Cloud suspended our production projects at 1am on Saturday

#520

Post by theshrike79 »

omniglottal wrote: post #500
theshrike79 wrote: post #428

Earlier quoted context omitted.

Alphabet doesn't care, as far as they are concerned, someone messed with their Holy Money Machine and doesn't deserve to exist.

Again, the "doesn't deserve to exist" refers clearly to a different entity than the paying customer who was punished for zero action of their own.

And again, Alphabet doesn't care =)

"This credit card messed with our money machine, it and everyone connected to it are now permanently blacklisted everywhere on our ecosystem."

They genuinely do not care who did what and whose number it was and who got hurt and whether it was used with or without permission. That CC tried to exploit the mighty money machine and that's it.

Post reply on HN