Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

91–100 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#91

Post by Veserv »

geerlingguy wrote: post #70
stuff4ben wrote: post #59

Earlier quoted context omitted.

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day.

Seems pretty reasonable to me.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#93

Post by JackFr »

xaerise wrote: post #90
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

Why wouldn't you assume that if I'm asking for it I have a good reason?

Are you going to rearchitect my system for me?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#94

Post by oasisbob »

NKosmatos wrote: post #77

If it’s not an excuse for something more serious they definitely need to rethink their infrastructure. It’s not allowed to have a single server with no disk space, taking offline 14 factories. That should’ve used the old trick. Create a big empty file inside the server (i.e. delete_me_in_case_of_need.txt with size a few GBs) and delete that in case of emergency. This will buy you some time to take the necessary actio…

Classic scaling hack.

There's a talk from someone at a high-volume adult entertainment site which used this trick too - can't remember which one.

As told, the caching functionality in a key service had an artificial 10ms delay added. When things got tough, it was an easy change to disable the delay.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#95

Post by JackFr »

doubled112 wrote: post #88
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

I agree with most of your points, but with system resources, sometimes it is simply that you can give them, but you can never take them back.

If cost accounting is done correctly, business units will give them back willingly.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#96

Post by JackFr »

alanbernstein wrote: post #45
londons_explore wrote: post #41

Earlier quoted context omitted.

This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.

And my mother sets all her clocks ahead 10 minutes so she's never late.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#97

Post by oasisbob »

hhhhhtime wrote: post #54

Aaah, plant shutdowns. I worked in IT in an automotive assembly plant at one point. Once, I was out on the plant floor with a colleague. We were diagnosing a new network drop and they plugged in a ping testing device into the cable, hit the button, and within seconds the entire assembly line went down. Cue lots of radio chatter and people driving up to us in their carts trying to figure out what was going on. Turns o…

Hope the RCA highlighted the risks of having a critical network server share the L2 segment with all its clients.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#98

Post by tcbawo »

geerlingguy wrote: post #70
stuff4ben wrote: post #59

Earlier quoted context omitted.

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

This doesn’t seem terrible if the business benefits justify the costs. There is a cost/benefit to this, presumably.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#99

Post by floorballchamp »

lovehashbrowns wrote: post #89
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I work for a retailer where the service I’m responsible for is used by every cash register around the world for certain operations. When I came in, the RDS DB for this service had 60GB allocated to it, had literally just run out of space and caused an outage. The last team just gave it an additional 20GB. A month later, I was put in charge of it and it was already 5GB away from running out of space again. I put an en…

I'd guess the worry is that once you increase the storage, you never decrease it again. Ever. It's a one-way street. So, once everything is 5x over-provisioned, then the services tend to fill that space anyway (cause why not be wasteful if it doesn't cost anything) and a year later you are in the same seat again.

I'm not saying this is real, but the worry certainly is.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#100

Post by xeromal »

lovehashbrowns wrote: post #89
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I work for a retailer where the service I’m responsible for is used by every cash register around the world for certain operations. When I came in, the RDS DB for this service had 60GB allocated to it, had literally just run out of space and caused an outage. The last team just gave it an additional 20GB. A month later, I was put in charge of it and it was already 5GB away from running out of space again. I put an en…

In my experience itd just control for the sake of control
Post reply on HN