Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

141–150 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#141

Post by thomastjeffery »

greedo wrote: post #52
sjdfbnionio wrote: post #38

Earlier quoted context omitted.

Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering p…

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

So don't waste your money on it then!

Why in the world would you buy one "Enterprise" disk instead of three "consumer" disks?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#142

Post by rawgabbit »

They didn't say which database software Toyota uses. But in general, if you are performing DML and you have configured backup retention to "normal".... that means the transactions file will balloon in size until the database is BACKEDUP. Because this caused an outage, I assume there was no DBA assigned and management assumes it will run faithfully like it always did.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#143

Post by mcny »

BoxFour wrote: post #67
delecti wrote: post #50

Earlier quoted context omitted.

> If you continue to use Windows XP now that support has ended, your computer will still work but it might become more vulnerable to security risks and viruses It only matters if the machine is online, and they often aren't.

> they often aren't. This was NOT my experience while working in the automotive industry.

> This was NOT my experience while working in the automotive industry.

Exactly. The only people who can climb the corporate ladder are people who care about business "impact" more than anything else.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#144

Post by Veserv »

breakwaterlabs wrote: post #139
Veserv wrote: post #91

Earlier quoted context omitted.

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

In what world is 2160TB $100k? Current single disk solutions are around $25/TB for HDDs and ~$100/TB for NVMe. At a minimum you're looking at $54k just for raw capacity-- assuming no backup, no chassis, no networking, and no redundancy. More reasonable estimations would be in excess of $400/TB.

Sure, whatever, a factor of 10 here or there hardly matters. I literally misinterpreted “multiple gigabytes per hour” as 999 GB/hr, not a much more reasonable 10 GB/hr. I literally overestimated data rates by a factor of 10,000% and the number still comes out “reasonable” i.e. a cost that can be paid if the cost/benefit is there.

Unless you want to claim storage costs $5,000/TB for 3 MB/s of I/O “multiple gigabytes per hour” with 90 day retention for a team worth of logging is not stupid on its face. Not to say that is a efficient or smart solution, but certainly not a “look at this insane request by developers” the person I was originally responding to was making it out to be.

Personally, I would probably question the competence of the team if they had that sort of logging rate with manual logging statements, but I am merely pointing out that “multiple gigabytes per hour” for 90 days is not crazy on its face and a plausible business case could be made for it even with a relatively modest engineering team.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#145

Post by justinclift »

Veserv wrote: post #91
geerlingguy wrote: post #70

Earlier quoted context omitted.

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

For their use case it sounds like they wanted to index the heck out of it for near instant lookups and similar too. So probably need to double the data size (rough guess) to include the indexes. And it may need some dedicated server nodes just for processing/ingestion/indexing, etc.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#146

Post by justinclift »

thomastjeffery wrote: post #141
greedo wrote: post #52

Earlier quoted context omitted.

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

So don't waste your money on it then! Why in the world would you buy one "Enterprise" disk instead of three "consumer" disks?

Well, "enterprise storage" can actually be multiple, redundant copies of the data behind the scenes.

So, internally the storage team might quote say $1000/TB (simple numbers for an example) for a given storage quantity. And behind the scenes they'll likely have at least redundant storage arrays, plus backups and 24/7 monitoring for all of the data, etc.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#147

Post by thomastjeffery »

justinclift wrote: post #146
thomastjeffery wrote: post #141

Earlier quoted context omitted.

So don't waste your money on it then! Why in the world would you buy one "Enterprise" disk instead of three "consumer" disks?

Well, "enterprise storage" can actually be multiple, redundant copies of the data behind the scenes. So, internally the storage team might quote say $1000/TB (simple numbers for an example) for a given storage quantity. And behind the scenes they'll likely have at least redundant storage arrays, plus backups and 24/7 monitoring for all of the data, etc.

Yes, but that still refutes the point all the same.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#148

Post by justinclift »

dncornholio wrote: post #53
BoxFour wrote: post #35

Earlier quoted context omitted.

> They have no time for development hipster nonsense https://support.microsoft.com/en-us/windows/windows-xp-suppo... > extraction games for unwanted features https://www.theverge.com/2022/7/12/23204950/bmw-subscription...

Funny when you never touched a computer in 20 years and as soon as microsoft posts "Support ended" it suddenly is a problem, when in fact, nothing changed. Computer still send signals and recieves signals just fine.

Mostly works fine until some internal limit is reached. For example, the Y2K bug at the turn of the century. Good thing that was mostly dealt with ok. :)

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#149

Post by mrguyorama »

bandyaboot wrote: post #127
JackFr wrote: post #93

Earlier quoted context omitted.

Why wouldn't you assume that if I'm asking for it I have a good reason? Are you going to rearchitect my system for me?

Wouldn’t you expected to have to provide some level of justification if you were, say, requesting a new development machine?

The meeting to bring in the relevant stakeholders and discuss that reasoning literally costs more than just fucking buying some cloud space.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#150

Post by greedo »

thomastjeffery wrote: post #141
greedo wrote: post #52

Earlier quoted context omitted.

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

So don't waste your money on it then! Why in the world would you buy one "Enterprise" disk instead of three "consumer" disks?

Because if you're providing services to a large business, you're typically not managing servers with physical disks attached. You're using a SAN with fiber connections or NFS mounts. Most SANs require specific drives sold by the vendor with firmware they've tested, mounted to custom sleds etc. You can't just connect a WD Mybook drive.

Mom and Pop businesses using single servers can do all they want in regards to drives, I don't care. I would argue they shouldn't have servers, but if you do have servers, you should at minimum use RAID etc, not a drive plugged into your USB port.

Post reply on HN