Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

181–190 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#181

Post by solatic »

stuff4ben wrote: post #59
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

> You can't give them carte blanche to the storage because they'll waste it

So what? Just buy more. Storage is cheap.

It's hard to have a discussion here without understanding the scales involved. Is the problem that they're wasting 100 GB or 100 TB? And if the issue is truly that they're wasting 100 TB, then clamp down on it as part of cost reduction efforts. The truth in most organizations is you get rewarded for eliminating mountains of waste, but trying to prevent the waste in the first place brands you as someone difficult to work with who is standing in the way. Why not lean into that?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#182

Post by ilyt »

ido wrote: post #166
akira2501 wrote: post #161

Earlier quoted context omitted.

You have to fill out a load chart to fly a plane, they should have to fill out something like a storage chart to get a production allocation. What size are your objects? How many per unit of time and served entity? What is the lifetime of those objects? How is that lifetime managed?

If you agree to add a few months of development time and reduce future velocity to make sure these limits are enforced, sure. Usually adding storage costs about as much as 1 developer’s salary cost for what, an hour? A day?

It's not for saving storage. It's for making sure it will actually not overflow.

End number doesn't matter, what matters is developers thinking about how long data should be stored and what data should be stored.

Not doing that analysis and overprovisioning 4x will just cause disaster in 2 years instead of 6 months.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#183

Post by ilyt »

0cf8612b2e1e wrote: post #105
JackFr wrote: post #96

Earlier quoted context omitted.

And my mother sets all her clocks ahead 10 minutes so she's never late.

It is not an entirely unreasonable idea. If a system runs out of disk space, an unexpectedly large number of operations will fail. Which can make recovery more problematic than you would assume. If you can immediately recover some disk space and have breathing room, it could make the difference in restoring service.

Or, you know, have alert on disk space like adults.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#184

Post by ilyt »

floorballchamp wrote: post #103
xaerise wrote: post #90

Earlier quoted context omitted.

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

> If you have a good reason So, you are not expecting that your co-workers have good reasons for what they are doing? Maybe the hiring bar at your place is too low then. I prefer to work at places where my default assumption is that everybody around me is smart and responsible. Lifts lots of worries off my shoulders (and tends to benefit the stock price over time too and thereby my income).

If your smart colleagues can't write a sentence like "we need extra 1TB for next 3 years of growth", they are not smart and you're not either...

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#185

Post by ilyt »

lovehashbrowns wrote: post #89
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I work for a retailer where the service I’m responsible for is used by every cash register around the world for certain operations. When I came in, the RDS DB for this service had 60GB allocated to it, had literally just run out of space and caused an outage. The last team just gave it an additional 20GB. A month later, I was put in charge of it and it was already 5GB away from running out of space again. I put an en…

Does nobody have space alerts ?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#186

Post by ilyt »

jbverschoor wrote: post #40
Ekaros wrote: post #39

Earlier quoted context omitted.

I just had much more fun experience on Debian. The VM simply froze after startup as disk was absolutely full... And figuring out how to actually extend disk through all of the indirection layers took a bit of time.

Never happened to me.. You'll still be able to login, free up some space to get the services up again, and then resize your disk if that's needed.

If you use SSH agent, key-based auth will fail and fallback to password, because that requires creating file in /tmp that you can't do if partition with ssh ran out of space.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#187

Post by ilyt »

codetrotter wrote: post #63
jbverschoor wrote: post #40

Earlier quoted context omitted.

Never happened to me.. You'll still be able to login, free up some space to get the services up again, and then resize your disk if that's needed.

Could it depend on what file system you use, and OS settings you have? For example, Google says that by default ext4 will reserve 5% of disk space. So depending on your system, a “100% full” drive might still have a (sometimes significant) bit of space left.

Most people turn that off coz it is near-useless feature and I don't need OS reserving 5% of 2TB drive

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#188

Post by arghwhat »

floorballchamp wrote: post #103
xaerise wrote: post #90

Earlier quoted context omitted.

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

> If you have a good reason So, you are not expecting that your co-workers have good reasons for what they are doing? Maybe the hiring bar at your place is too low then. I prefer to work at places where my default assumption is that everybody around me is smart and responsible. Lifts lots of worries off my shoulders (and tends to benefit the stock price over time too and thereby my income).

Generally? No. Not because they are not smart, but because in a large company, each individual have different goals and priorities - that's why we have e.g. SREs as dedicated roles - and it takes a bit of effort to find the intersection between all these.

Let's say I work in DevOps and want to optimize cloud costs. In that case, I would challenge the size of everything, the use of higher-costs services, the number of regions, all that - but the team might want more regions and bigger resources to improve latency and performance, and use more high-cost services for developer experience, and ship features without having to think about utilization.

It's a tug of war, and only works when you have forces on both sides to balance out. Being too conservative might stall innovation or make things too slow to save a buck, not being conservative enough might drain funds or make things impossible to scale.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#189

Post by greedo »

thomastjeffery wrote: post #154
greedo wrote: post #150

Earlier quoted context omitted.

Because if you're providing services to a large business, you're typically not managing servers with physical disks attached. You're using a SAN with fiber connections or NFS mounts. Most SANs require specific drives sold by the vendor with firmware they've tested, mounted to custom sleds etc. You can't just connect a WD Mybook drive. Mom and Pop businesses using single servers can do all they want in regards to driv…

If you are in that situation, then obviously you have to do what you have to do. Absolutely none of that has to apply to the sort of situation we are talking about here. The only real reason to spend more per disk is when you know all your disks are going to fail, and extending the lifespan per average disk will definitely save you more than the enterprise markup costs. So you better have dozens or hundreds of disks…

You try telling a VP that their business unit can't function because you decided to purchase the cheaper drive.

All disks fail eventually. Outliers may run longer than the MTBF for that drive model, but they all fail eventually.

And backups are fine for restoring data, but they don't help provide access to that data in a timely fashion. That's why people use SANs, AWS etc.

The cost of a SAN storage array is a nit for a business making cars (like Toyota) or selling insurance (like my company).

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#190

Post by bestouff »

randmeerkat wrote: post #163
BowBun wrote: post #83

Earlier quoted context omitted.

I worry about the same thing - cloud provider and especially specialty hosted solutions (Heroku) are getting more expensive as dependency on them grows and smaller shops basically have to use them to get off the ground. In the end what you have is a stack of technology that the company cannot actually maintain on their own. Now I know that 'shoulders of giants' is a thing and that we shouldn't expect everyone to be a…

> Part of the original promise from these providers was that there would be a 90%+ reduction in supporting work, when we all know the reality is that you become specialized in solving issues with Heroku/AWS/providerX For all but the most simple setups, providerX solutions are still better than maintaining your own bare metal solutions with their associated hardware / software / datacenter vendor relationships. Also,…

I have heard this many times but I've never seen convincing proof of that.
Post reply on HN