Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

51–60 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#51

Post by wil421 »

archildress wrote: post #19

Just-in-time will cost you far more than it will save you. The problem is that when it fails, you judge those failures as "backwards looking one offs" as opposed to a cost that is part and parcel to being JIT. Factory downtime is the single biggest cost a manufacturing shop can incur. No amount of working capital savings from carrying fewer screws, widgets, and bolts can offset the costs of taking 14 plants down due…

A family member worked for a large military aerospace company. In the 80s they were building a system to order parts and tell the factory what to produce and complete daily so airplane production was maximized. He said they had so many parts employees said they used JIC or Just In Case.

Reducing the amount of parts on hand saved them money and made them more efficient.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#52

Post by greedo »

sjdfbnionio wrote: post #38
willcipriano wrote: post #24

Earlier quoted context omitted.

There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.

Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering p…

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#53

Post by dncornholio »

BoxFour wrote: post #35
picadores wrote: post #14

Earlier quoted context omitted.

Why is that a joke? Factory equipment is ordered and has a lifetime of 20 years. Of course it has old systems. These are grownup industries. They have no time for development hipster nonsense and extraction games for unwanted features to stave of unemployment in some glass-towers.

> They have no time for development hipster nonsense https://support.microsoft.com/en-us/windows/windows-xp-suppo... > extraction games for unwanted features https://www.theverge.com/2022/7/12/23204950/bmw-subscription...

Funny when you never touched a computer in 20 years and as soon as microsoft posts "Support ended" it suddenly is a problem, when in fact, nothing changed. Computer still send signals and recieves signals just fine.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#54

Post by hhhhhtime »

Aaah, plant shutdowns. I worked in IT in an automotive assembly plant at one point. Once, I was out on the plant floor with a colleague. We were diagnosing a new network drop and they plugged in a ping testing device into the cable, hit the button, and within seconds the entire assembly line went down. Cue lots of radio chatter and people driving up to us in their carts trying to figure out what was going on.

Turns out the ping testing device was last configured with the IP of a critical application server that talks to every device in the plant. When the colleague pressed the button, every plant device stopped contacting the apps server and the plant went down. Unplugging the device eventually fixed it. The line was stopped for 7 minutes. I believe stoppages were usually quoted at tens of thousands of dollars per minute and billed to the responsible department.

I remember how stressed everyone was after and meetings for root cause analysis in the following days. I was always surprised at this considering how frequently the line went down for small amounts of time, and they never seemed as big of a deal as that one specific time, though for the others we were not responsible.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#55

Post by albertgoeswoof »

Wow, my company (mailpace.com) recently had an outage for basically the same reason:

https://blog.mailpace.com/blog/postgres-outage-post-mortem/

Luckily, we only went offline for about 2 hours, but glad to see behemoths like these suffer from similar issues...

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#56

Post by CamJN »

alanbernstein wrote: post #45
londons_explore wrote: post #41

Earlier quoted context omitted.

This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.

Since apfs, “large” empty files can also take only the 4K or so for the fs entry on disk, gotta make sure it’s not a sparse file, and doing so is a bit tricky, better to fill it with junk that doesn’t compress well.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#57

Post by tetha »

atoav wrote: post #28
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

Be stingy in a smart way. I'd like a description why you need the storage, an estimation how much and a projection of growth over the next 6 - 12 month, though the latter one can wait for a month or three for something new. Beyond a certain scale we'd need a PO or a project to write up the cost too. And yes, we'll start bugging you again once it starts filling up to 80% or more, because we don't want your systems to fail :)

But this way, I directly get an idea of the increase in storage we will need over the next year in order to plan the next hardware expansion.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#58

Post by pgt »

atoav wrote: post #28
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

"It is difficult to get a man to understand something, when his salary depends upon his not understanding it." – Upton Sinclair

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#59

Post by stuff4ben »

atoav wrote: post #28
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave me a direct view in how hard it is to manage storage for developers. And as a developer using Artifactory, I don't want to worry about storage, I just want my builds and CI pipelines to complete.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#60

Post by willcipriano »

greedo wrote: post #52
sjdfbnionio wrote: post #38

Earlier quoted context omitted.

Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering p…

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

I'd bet the salary of the guy who is making this a pain could buy enough storage that they don't have to worry about it anymore.
Post reply on HN