Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

161–170 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#161

Post by akira2501 »

stuff4ben wrote: post #59
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

You have to fill out a load chart to fly a plane, they should have to fill out something like a storage chart to get a production allocation. What size are your objects? How many per unit of time and served entity? What is the lifetime of those objects? How is that lifetime managed?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#162

Post by xwdv »

ndhfkashdben wrote: post #26

I worked at a phone factory once. One day, an app stopped working. It was running on an old Linux server. Nobody had access to it, or so I thought. I noticed the pings were showing high failure rates. I asked to check switch logs or the server's ifconfig output and got weird excuses why they couldn't check anything. One week goes by, people are living with the error and adjusting the factory to work around that. Many…

Well if I knew I was leaving anyway then I would have just quietly fixed it on my own. And then immediately put in a two weeks notice and use my vacation time.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#163

Post by randmeerkat »

BowBun wrote: post #83
wizerdrobe wrote: post #11

As we move to more abstract systems I wonder how well we (as in companies) are keeping basic systems management capabilities in place at a personnel level. At $DAY_JOB we recently scuttled most development efforts for a week for our teams. Our nightly backup job that sanitizes PHI ballooned overtime to, say, 20GB+1Byte and ran out of disk space. Because we are running Kubernetes on Fargate we don’t need a full time o…

I worry about the same thing - cloud provider and especially specialty hosted solutions (Heroku) are getting more expensive as dependency on them grows and smaller shops basically have to use them to get off the ground. In the end what you have is a stack of technology that the company cannot actually maintain on their own. Now I know that 'shoulders of giants' is a thing and that we shouldn't expect everyone to be a…

> Part of the original promise from these providers was that there would be a 90%+ reduction in supporting work, when we all know the reality is that you become specialized in solving issues with Heroku/AWS/providerX

For all but the most simple setups, providerX solutions are still better than maintaining your own bare metal solutions with their associated hardware / software / datacenter vendor relationships.

Also, it’s not hard figuring out how to stand up a datacenter if it comes down to it and you want to pivot at some point. What’s actually hard is _growing_ and _distributing_ your datacenter to meet demand / legal requirements.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#164

Post by sgerenser »

0cf8612b2e1e wrote: post #160
LeifCarrotson wrote: post #64

Earlier quoted context omitted.

In some (American) automotive plants I've worked in, JIT lifetimes are about 10 hours. The order for a seat or mirror or door handle doesn't exist at the tier 1 suppliers where my equipment gets installed until 10 hours before it is needed. At that point, my PLC gets a request to induct a particular part number and paint color into the manifest, inducts it and acknowledges that request, the machine builds it, laser-m…

Do the plants have capacity to store more if they wanted? I am thinking of foreseeable short term disruptions which could break the 10 hour windows. Hurricanes, blizzards, train/truck worker strikes, cargo ship stuck in a canal, etc. For a multinational, those are not black swan events, something of that nature is going to break the pipeline on a regular basis. Potentially for days of backlog. Clearly it must mostly…

Yeah, think of what would happen if they were hit with a global pandemic!

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#165

Post by 10000truths »

breakwaterlabs wrote: post #139
Veserv wrote: post #91

Earlier quoted context omitted.

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

In what world is 2160TB $100k? Current single disk solutions are around $25/TB for HDDs and ~$100/TB for NVMe. At a minimum you're looking at $54k just for raw capacity-- assuming no backup, no chassis, no networking, and no redundancy. More reasonable estimations would be in excess of $400/TB.

> Current single disk solutions are around $25/TB for HDDs

More like $15/TB. $100K for 2 PB of storage with redundancy and backups is quite reasonable.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#166

Post by ido »

akira2501 wrote: post #161
stuff4ben wrote: post #59

Earlier quoted context omitted.

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

You have to fill out a load chart to fly a plane, they should have to fill out something like a storage chart to get a production allocation. What size are your objects? How many per unit of time and served entity? What is the lifetime of those objects? How is that lifetime managed?

If you agree to add a few months of development time and reduce future velocity to make sure these limits are enforced, sure. Usually adding storage costs about as much as 1 developer’s salary cost for what, an hour? A day?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#167

Post by grecy »

hhhhhtime wrote: post #54

Aaah, plant shutdowns. I worked in IT in an automotive assembly plant at one point. Once, I was out on the plant floor with a colleague. We were diagnosing a new network drop and they plugged in a ping testing device into the cable, hit the button, and within seconds the entire assembly line went down. Cue lots of radio chatter and people driving up to us in their carts trying to figure out what was going on. Turns o…

A friend was a process engineer and studied and worked in a bunch of car assembly plants optimizing stuff.

Every single time the line stopped, no matter why, it was estimated to cost, at absolutely minimum, $1 million dollars.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#168

Post by FreshStart »

A well oiled car factory drops one of the belt ever n minutes. On a new car there are y% of profit. n x y > price of Harddisk is were the yelling and firing of administration began. Power trips and savings are nice. But factory stops have a price and when it turns out to be thiefdoms, those chiefthiefdomtains are goners. Can't run "the rules stated.." by the shareholders who know that the rules are made up for operations not against it.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#169

Post by PeterStuer »

alanbernstein wrote: post #45
londons_explore wrote: post #41

Earlier quoted context omitted.

This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.

Can't you set up overprovisioning so the storage controller can do something usefull with it while you don't need the space?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#170

Post by PeterStuer »

JackFr wrote: post #96
alanbernstein wrote: post #45

Earlier quoted context omitted.

On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.

And my mother sets all her clocks ahead 10 minutes so she's never late.

In my experience this makes the problem worse. People either compensate for it, or stop trusting clocks at all. Usually a mix of both of those resulting in even less punctuality.
Post reply on HN