Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

191–200 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#191
post #52

Earlier quoted context omitted.

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

So don't waste your money on it then! Why in the world would you buy one "Enterprise" disk instead of three "consumer" disks?

Because the enterprise disks are rated for years of continuous service and have things like firmware which doesn’t lie about whether data has been committed durably for the sake of benchmarks on review sites.

None of this means that you should trust any particular disk enough not to need redundancy, backups, etc. Companies can and do make trade offs based on their needs and management competency and people have been shifting software for a generation to rely less on the hardware – back when Sun announced ZFS, one of the major appeals was that you could drop expensive hardware RAID controller dependencies in favor of cheap boxes of disks – but there isn’t a single global optimum point. A lot of enterprise purchases are driven by being able to satisfy your most demanding users with the same service as everyone else so you can avoid needing your admins to be trained and experienced with dozens of different storage systems. That last part especially extends to testing: for example, does your rack of consumer drives with software redundancy come back up cleanly after a kernel panic or power outage, especially a nasty one like a fluctuating brownout? Depending on your budget, needs, and technical bench depth you might reasonably conclude that the savings are worth the ops work, or that it’s safer to pay an enterprise storage vendor who’ll certify that they’ve done that testing and will have tech support on-sight within an hour, or that you’ll use AWS/Azure/GCP because they do even more of that tedious but important work. All of those can be right, but I’ve typically found that people in the first two categories think they’re doing better than they are and would be paying less for better service in the cloud.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#192
post #62

Earlier quoted context omitted.

Often this is intentional administrative backpressure. Finance tells IT to minimize costs. IT knows that the majority of users are poor at housekeeping, so tight control over storage allocation provides an incentive to delete-first-expand-later. Unfortunately, this strategy frustrates users who have a genuine need to expand storage and who are unable to efficiently obtain an exception to the process.

Being slow to provide resources when I need them also means that I'm less likely to relenquish them when I'm done.

Also more likely to request far more than you need so you don’t have to deal with them as frequently.

Everywhere I’ve seen that strategy tried had massive overspending and outages due to it. What’s worked best is cloud style usage billing because that aligns the incentives with the people making the decisions, but it’s really non-trivial to get the accurate full cost.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#193
post #90
post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

dealing with the gate keeping often costs more in dev time then just approving. Especially when the DevOps think they know better - thank goodness a tech director can step in and bust the impasse.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#194
post #52

Earlier quoted context omitted.

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

So don't waste your money on it then! Why in the world would you buy one "Enterprise" disk instead of three "consumer" disks?

It’s because all consumer disks are garbage. The way it works is that disks are tested. Disks without failures are sold as “Enterprise.” Those that have failures are labeled and sold as “consumer”, not thrown in the bin.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#195

As we move to more abstract systems I wonder how well we (as in companies) are keeping basic systems management capabilities in place at a personnel level. At $DAY_JOB we recently scuttled most development efforts for a week for our teams. Our nightly backup job that sanitizes PHI ballooned overtime to, say, 20GB+1Byte and ran out of disk space. Because we are running Kubernetes on Fargate we don’t need a full time o…

If it suits your setup, I find that offloading stateful services to managed cloud providers, then running the stateless stuff in k8s is a nice combo. It's not k8s or anything else that makes it hard/easy. Maintaining and keeping stateful services running the long run is what's a bit tricky.

That’s been my preferred model since the 2000s, too. I’ve seen a lot of projects where someone wanted to do something which they couldn’t do with RDS, etc. and every time the results didn’t match the early optimism.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#196

Earlier quoted context omitted.

> If you have a good reason So, you are not expecting that your co-workers have good reasons for what they are doing? Maybe the hiring bar at your place is too low then. I prefer to work at places where my default assumption is that everybody around me is smart and responsible. Lifts lots of worries off my shoulders (and tends to benefit the stock price over time too and thereby my income).

Generally? No. Not because they are not smart, but because in a large company, each individual have different goals and priorities - that's why we have e.g. SREs as dedicated roles - and it takes a bit of effort to find the intersection between all these. Let's say I work in DevOps and want to optimize cloud costs. In that case, I would challenge the size of everything, the use of higher-costs services, the number of…

> It's a tug of war,

Yeah, any workplace in which the word "war" was used in the context of colleague interaction saw me leave within a few months.

I like to plan those things ahead of time with all stakeholders involved, then we work together instead of against each other.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#197
post #91

Earlier quoted context omitted.

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

A few years ago I joined a company aggressively trying to reduce their AWS costs. My jaw got the floor when I realized they were spending over a million a month in AWS fees. I couldn't understand how they got there with what they were actually doing. I feel like this comment perfectly demonstrates how that happens.

AWS also purposefully makes it easy to shoot yourself in the foot. Case in point that we were burned on recently:

- set up some service that talks to a s3 bucket

- set up that bucket in the same region/datacenter

- send a decent but not insane amount of traffic through there (several hundred Gb per day)

- assume that you won’t get billed any data transfer fees since you’re talking to a bucket in the same data center

- receive massive bill under “EC2-Other” line item for NAT data transfer fees

- realize that AWS routes all traffic through NAT gateway by default even though it’s just turning around and going back into the data center it came from and billing exorbitant fees for that

- come to the conclusion that this is obviously a racket designed to extract money from unsuspecting people because there is almost no situation where you would want to do that by default and discover that hundreds to thousands of other people have been screwed in the exact same way for years (and there is a documented trail of it[1])

1: https://www.lastweekinaws.com/blog/the-aws-managed-nat-gatew...

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#198

Earlier quoted context omitted.

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

You have to fill out a load chart to fly a plane, they should have to fill out something like a storage chart to get a production allocation. What size are your objects? How many per unit of time and served entity? What is the lifetime of those objects? How is that lifetime managed?

You missed the part where I said they are "notoriously bad at estimating". We really do suck at estimating everything... storage, work estimates, etc. Why can't we just say "it'll be done when it's done and I'll use ALL the storage until I'm done"?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#199

Earlier quoted context omitted.

Generally? No. Not because they are not smart, but because in a large company, each individual have different goals and priorities - that's why we have e.g. SREs as dedicated roles - and it takes a bit of effort to find the intersection between all these. Let's say I work in DevOps and want to optimize cloud costs. In that case, I would challenge the size of everything, the use of higher-costs services, the number of…

> It's a tug of war, Yeah, any workplace in which the word "war" was used in the context of colleague interaction saw me leave within a few months. I like to plan those things ahead of time with all stakeholders involved, then we work together instead of against each other.

I believe you are intentionally misunderstanding. The term "tug of war" is not used to indicate armed conflict or even a problem. It indicates balancing forces that you want to maintain - pull the rope too far to one side, and you end up in a suboptimal extreme.

Unless you work with clones of yourself, there will always be differences in opinions and priorities, and not every feature and bug fix can be a company-wide stakeholder meeting, and you certainly will not get any social points for trying to micro-manage other teams.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#200

Earlier quoted context omitted.

Wouldn’t you expected to have to provide some level of justification if you were, say, requesting a new development machine?

The meeting to bring in the relevant stakeholders and discuss that reasoning literally costs more than just fucking buying some cloud space.

Had a boss that would swoop into “suspicious” meetings.

“There’s 10 people here whose time I bill out at $250/hr each, spending an hour discussing whether to buy a $1,000 software license? Why?”

Post reply on HN