Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

101–110 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#101

Post by marcosdumay »

greedo wrote: post #52
sjdfbnionio wrote: post #38

Earlier quoted context omitted.

Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering p…

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

Well, that's a choice your ops people make.

There is absolutely no technical reason to provide all of your storage needs with a single quality of disks. You may do that to get some economies of scale, but if the impact is that high, you should rethink it. You can just as well have two different solutions, one with a huge amount of disk space, little redundancy and low performance, and one with a limited amount of space, plenty of redundancy and high performance.

And yes, I know that goes against every common ops procedure, because disks are so cheap. But the next thing you hear is always that disks are not cheap at all, and you really can't have it both ways.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#102

Post by floorballchamp »

Veserv wrote: post #91
geerlingguy wrote: post #70

Earlier quoted context omitted.

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

Plus, logs have enormous compression potential since their entropy is so low. That's the property exploited by every logging-as-a-service out there.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#103

Post by floorballchamp »

xaerise wrote: post #90
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

> If you have a good reason

So, you are not expecting that your co-workers have good reasons for what they are doing? Maybe the hiring bar at your place is too low then.

I prefer to work at places where my default assumption is that everybody around me is smart and responsible. Lifts lots of worries off my shoulders (and tends to benefit the stock price over time too and thereby my income).

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#104

Post by marcosdumay »

Yes, when you fully optimize something by one metric, like cost, other metrics, like reliability, get worse and worse.

Funny thing that if you get any book about Toyotism and Lean, this warning is around the first things you see. You only optimize until you start seeing problems, then you either stop or improve things so the problems don't happen anymore and you can move further. Well, looks like the management at the company that led this culture is forgetting the lesson.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#105

Post by 0cf8612b2e1e »

JackFr wrote: post #96
alanbernstein wrote: post #45

Earlier quoted context omitted.

On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.

And my mother sets all her clocks ahead 10 minutes so she's never late.

It is not an entirely unreasonable idea. If a system runs out of disk space, an unexpectedly large number of operations will fail. Which can make recovery more problematic than you would assume. If you can immediately recover some disk space and have breathing room, it could make the difference in restoring service.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#106

Post by ddoolin »

xeromal wrote: post #68
kenoh wrote: post #44

Earlier quoted context omitted.

I'm just here to admire the fact that a Google search for "Toyototian" results this 1 hour old comment.

I was considering what to call a Toyota employee and thought Toyototian sounded right. Accidentally added an additional to* but oh well. It's pretty incredible how fast Google adds stuff to their index this quickly!

I work for Toyota but I've never pondered what our demonym is; I think Toyototian (Toyotian?) has something going for it though. :)

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#108

Post by COM9 »

delecti wrote: post #50
BoxFour wrote: post #35

Earlier quoted context omitted.

> They have no time for development hipster nonsense https://support.microsoft.com/en-us/windows/windows-xp-suppo... > extraction games for unwanted features https://www.theverge.com/2022/7/12/23204950/bmw-subscription...

> If you continue to use Windows XP now that support has ended, your computer will still work but it might become more vulnerable to security risks and viruses It only matters if the machine is online, and they often aren't.

> It only matters if the machine is online, and they often aren't.

No.

Really, you shouldn't think that. It's really bad security practice.

https://en.wikipedia.org/wiki/Stuxnet

It is typically introduced to the target environment via an infected USB flash drive, thus crossing any air gap.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#109

Post by fillipvt »

atoav wrote: post #28
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

Just wondering, why pi?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#110

Post by conductr »

willcipriano wrote: post #24
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.

“No problem, just put in a ticket”
Post reply on HN