[deleted]
Toyota blames factory shutdown in Japan on ‘insufficient disk space’
21–30 of 223 posts
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#22I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
Anecdotes from bank-employed chums have reported similar things - including where it's been quicker and easier for them to nip to the shops and buy an external drive to get some space to relieve pressure whilst the 800 requisition forms are slowly working their way through the systems.
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#23[deleted]
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#24I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#25I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
Anecdotes from bank-employed chums have reported similar things - including where it's been quicker and easier for them to nip to the shops and buy an external drive to get some space to relieve pressure whilst the 800 requisition forms are slowly working their way through the systems.
It’s year ago now, but my father bought a second hand hard drive and found it full of banking data with customer names and all.
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#26I worked at a phone factory once. One day, an app stopped working. It was running on an old Linux server. Nobody had access to it, or so I thought. I noticed the pings were showing high failure rates. I asked to check switch logs or the server's ifconfig output and got weird excuses why they couldn't check anything.
One week goes by, people are living with the error and adjusting the factory to work around that. Many meetings, etc.
Network engineer pulls me to the side and says "you seem like a nice guy, stop worrying about this issue. we won't fix it because if we do, this server becomes ours and we are responsible for this app.. so nobody will touch it. yes, the cable to that server needs to be redone. yeah we won't do that"
I stayed there for 10 months and quit.
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#27Earlier quoted context omitted.
Anecdotes from bank-employed chums have reported similar things - including where it's been quicker and easier for them to nip to the shops and buy an external drive to get some space to relieve pressure whilst the 800 requisition forms are slowly working their way through the systems.
It’s year ago now, but my father bought a second hand hard drive and found it full of banking data with customer names and all.
[deleted]
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#28I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org.
And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the disk 40 Gb. He could have easily just allocated 100 Gb and call it a day, this way we had to go to him 3 times to scale it up in 10 Gb steps each time, including a the stress of figuring out why things are failing (something that the admin should have seen on his monitoring system).
Admins are my heros, but please if you allocate disk space, just take the biggest expected backup and multiply it by π. And if you need to be stingy with storage for some reason, be stingy, but decide when and where to be stingy — and at least keep an eye on the monitoring and upsize the storage before it is too late.
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#29I worked at a phone factory once. One day, an app stopped working. It was running on an old Linux server. Nobody had access to it, or so I thought. I noticed the pings were showing high failure rates. I asked to check switch logs or the server's ifconfig output and got weird excuses why they couldn't check anything. One week goes by, people are living with the error and adjusting the factory to work around that. Many…
Typical.
Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’
#30Why is the article tying just-in-time to this issue?
JIT in context of Toyota refers to their philosophical concept of keeping as many parts in transit. I'm not completely sure but I think runtime compilation in software is named after that than the other way around.
Many things in lean/agile software development processes come from Toyota Manufacturing - kanban is essentially just-in-time.