Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

71–80 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#71

Post by Bluecobra »

xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I can say the same thing but as a sysadmin working at a place where it was pulling teeth to get money to buy additional storage. Upper management was so daft/cheap and they can't see past six months so they decide to buy the cheapest enterprise solution with the least amount of upgrade capacity.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#72

Post by pnutjam »

hhhhhtime wrote: post #54

Aaah, plant shutdowns. I worked in IT in an automotive assembly plant at one point. Once, I was out on the plant floor with a colleague. We were diagnosing a new network drop and they plugged in a ping testing device into the cable, hit the button, and within seconds the entire assembly line went down. Cue lots of radio chatter and people driving up to us in their carts trying to figure out what was going on. Turns o…

I once plugged a toner into a network patch panel to locate a drop and crashed the AIX system at a medical office. :(

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#73

Post by froh »

archildress wrote: post #19

Just-in-time will cost you far more than it will save you. The problem is that when it fails, you judge those failures as "backwards looking one offs" as opposed to a cost that is part and parcel to being JIT. Factory downtime is the single biggest cost a manufacturing shop can incur. No amount of working capital savings from carrying fewer screws, widgets, and bolts can offset the costs of taking 14 plants down due…

the benefit of JIT is to detect problems early. If you minimize the number of parts "in flight" you minimize the time between creating defective parts and detecting the defects and taking countermeasures. any stockpile hides systematic production issues in the stockpiled parts until the part has percolated through the stockpile.

that's how it saves you far more than it can cost you through such a quite unique event as the one of this news article.

controlling JIT was originally done with little "kanban" paper slips, which pulled in the next batch of work. those are the origin of software "kanban", too, the software engineering "pull" workflow.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#75

Post by bluedino »

Had a production system crash because of a similar issue somewhat recently.

The cause? / filled. It was configured at somewhere around 80GB. There was a scratch volume on the VM as well, also 80GB.

Recently the machine was upgraded to a newer version of Ubuntu, that used a Snap for Chromium. tmp on the Snap wasn't on the scratch drive like on the old version, it was in /tmp

For some reason, ops never set a watch/alert on that disk. That would have prevented the issue.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#76

Post by giancarlostoro »

thelastparadise wrote: post #36
giancarlostoro wrote: post #33

Earlier quoted context omitted.

You're likely not wrong, but also the fact that Windows tells you you're running out of storage going as far back as XP if not further back. I wont be surprised if Mac and Linux do the same. My guess is, they kept ignoring it until it became impossible to ignore. I wonder if years passed until it was unavoidable.

A lot of server OS's/configurations will just silently fill up and take down any services depending on that volume with it.

Well yeah, if your swap partitions on the same as the rest of your OS, eventually it will just make itself known as an issue.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#77

Post by NKosmatos »

If it’s not an excuse for something more serious they definitely need to rethink their infrastructure. It’s not allowed to have a single server with no disk space, taking offline 14 factories.

That should’ve used the old trick. Create a big empty file inside the server (i.e. delete_me_in_case_of_need.txt with size a few GBs) and delete that in case of emergency. This will buy you some time to take the necessary actions ;-)

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#78

Post by doublepg23 »

greedo wrote: post #52
sjdfbnionio wrote: post #38

Earlier quoted context omitted.

Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering p…

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

I’ve been getting organic and cage-free hard drives nowadays.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#79

Post by b112 »

geerlingguy wrote: post #70
stuff4ben wrote: post #59

Earlier quoted context omitted.

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

With silliness like that, you can bet it was the cost of their logging provider, as the feature they didn't like.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#80

Post by throw0101c »

codetrotter wrote: post #63
jbverschoor wrote: post #40

Earlier quoted context omitted.

Never happened to me.. You'll still be able to login, free up some space to get the services up again, and then resize your disk if that's needed.

Could it depend on what file system you use, and OS settings you have? For example, Google says that by default ext4 will reserve 5% of disk space. So depending on your system, a “100% full” drive might still have a (sometimes significant) bit of space left.

> For example, Google says that by default ext4 will reserve 5% of disk space.

Reserved 5% for root, but if the files/logs that are taking up space are written by the root user then it's a moot point:

> Specify the percentage of the file system blocks reserved for the super-user. This avoids fragmentation, and allows root-owned daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the file system. The default percentage is 5%.

* https://man.archlinux.org/man/mke2fs.8

One trick possible with ZFS is to take a pool (e.g., rpool) and create an 'extra' data set in it (rpool/reservation) and set a reservation for it, so that even if all the other areas are filled (rpool/root, rpool/var, rpool/home) you can simply lower the reservation to get some space back quickly.

Post reply on HN