Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

41–50 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#41

Post by londons_explore »

atoav wrote: post #28
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

This is why one should generally be using network/cloud storage, with soft/hard limits.

As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#42

Post by davedx »

archildress wrote: post #19

Just-in-time will cost you far more than it will save you. The problem is that when it fails, you judge those failures as "backwards looking one offs" as opposed to a cost that is part and parcel to being JIT. Factory downtime is the single biggest cost a manufacturing shop can incur. No amount of working capital savings from carrying fewer screws, widgets, and bolts can offset the costs of taking 14 plants down due…

It would be nice if you could back that up with some facts, considering Toyota has been using some variant of this system since 1948.

Please provide a citation for how JIT manufacturing cost TM "far more than it saved them"?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#43

Post by Ekaros »

jbverschoor wrote: post #40
Ekaros wrote: post #39

Earlier quoted context omitted.

I just had much more fun experience on Debian. The VM simply froze after startup as disk was absolutely full... And figuring out how to actually extend disk through all of the indirection layers took a bit of time.

Never happened to me.. You'll still be able to login, free up some space to get the services up again, and then resize your disk if that's needed.

In this case I had to use rescue mode. As even different tty was too slow to respond.

Also just removing some stuff up wasn't working.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#44

Post by kenoh »

xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I'm just here to admire the fact that a Google search for "Toyototian" results this 1 hour old comment.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#45

Post by alanbernstein »

londons_explore wrote: post #41
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#46

Post by sct202 »

xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

We had an on-prem server at a new factory I was working at and less than a year in the whole thing shut down because we ran out of disk space for logs so none of the automation could successfully report back that they completed tasks to move to the next task. It turns out it was apparently terabytes of unallocated disk space on the server, so it was a quick fix but just felt so avoidable.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#47

Post by atoav »

londons_explore wrote: post #41
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

This was a VM with a ceph cluster.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#48

Post by wil421 »

sjdfbnionio wrote: post #38
willcipriano wrote: post #24

Earlier quoted context omitted.

There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.

Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering p…

Are you talking consumer prices, enterprise redundant SSDs in a Data Center, or enterprise cloud storage? In my experience the latter two could add another digit to your price.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#49

Post by Consultant32452 »

willcipriano wrote: post #24
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.

The audit department at my current gig says all records must be retained for eternity because of regulatory reasons but cannot direct anyone to the relevant regulation.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#50

Post by delecti »

BoxFour wrote: post #35
picadores wrote: post #14

Earlier quoted context omitted.

Why is that a joke? Factory equipment is ordered and has a lifetime of 20 years. Of course it has old systems. These are grownup industries. They have no time for development hipster nonsense and extraction games for unwanted features to stave of unemployment in some glass-towers.

> They have no time for development hipster nonsense https://support.microsoft.com/en-us/windows/windows-xp-suppo... > extraction games for unwanted features https://www.theverge.com/2022/7/12/23204950/bmw-subscription...

> If you continue to use Windows XP now that support has ended, your computer will still work but it might become more vulnerable to security risks and viruses

It only matters if the machine is online, and they often aren't.

Post reply on HN