Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

131–140 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#131

Post by SAI_Peregrinus »

0cf8612b2e1e wrote: post #105
JackFr wrote: post #96

Earlier quoted context omitted.

And my mother sets all her clocks ahead 10 minutes so she's never late.

It is not an entirely unreasonable idea. If a system runs out of disk space, an unexpectedly large number of operations will fail. Which can make recovery more problematic than you would assume. If you can immediately recover some disk space and have breathing room, it could make the difference in restoring service.

Keeping a bit of disk reserved for recovery is extremely common with copy-on-write filesystems like ZFS & BTRFS. Even deletion takes some extra space, so without a reservation it's effectively impossible to delete any files from a full disk.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#132

Post by hnbear »

zimpenfish wrote: post #22
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

Anecdotes from bank-employed chums have reported similar things - including where it's been quicker and easier for them to nip to the shops and buy an external drive to get some space to relieve pressure whilst the 800 requisition forms are slowly working their way through the systems.

As an ex-bank employee one time I needed space and was told I couldn’t get it because all the filers were full.

When would we get new filers?

Dunno, because the datacenter was also full.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#133

Post by Veserv »

giovannibonetti wrote: post #130
floorballchamp wrote: post #102

Earlier quoted context omitted.

Plus, logs have enormous compression potential since their entropy is so low. That's the property exploited by every logging-as-a-service out there.

Related to that, last year Uber's engineering blog mentioned very interesting results with their internal log service [1]. I wonder if there's anything as good in the open-source world. The closest thing I can think of is Clickhouse's "new" JSON type, which is backed by columnar storage with dynamic columns [2]. [1] https://www.uber.com/en-BR/blog/reducing-logging-cost-by-two... [2] https://clickhouse.com/docs/en/int…

https://messagetemplates.org/

The design described there is what Uber should be logging in the first place. Instead they are logging the fully resolved message and then compressing back into the templated form.

However, the compression back into the templated form is a good idea if you have third party logs that you want to store where you can not rewrite the logging to generate the correct form in the first place.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#134

Post by Volundr »

Veserv wrote: post #91
geerlingguy wrote: post #70

Earlier quoted context omitted.

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

A few years ago I joined a company aggressively trying to reduce their AWS costs. My jaw got the floor when I realized they were spending over a million a month in AWS fees. I couldn't understand how they got there with what they were actually doing. I feel like this comment perfectly demonstrates how that happens.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#135

Post by giovannibonetti »

Veserv wrote: post #133
giovannibonetti wrote: post #130

Earlier quoted context omitted.

Related to that, last year Uber's engineering blog mentioned very interesting results with their internal log service [1]. I wonder if there's anything as good in the open-source world. The closest thing I can think of is Clickhouse's "new" JSON type, which is backed by columnar storage with dynamic columns [2]. [1] https://www.uber.com/en-BR/blog/reducing-logging-cost-by-two... [2] https://clickhouse.com/docs/en/int…

https://messagetemplates.org/ The design described there is what Uber should be logging in the first place. Instead they are logging the fully resolved message and then compressing back into the templated form. However, the compression back into the templated form is a good idea if you have third party logs that you want to store where you can not rewrite the logging to generate the correct form in the first place.

Neat! The only downside of this approach is having to force the developers to use the library, which can work in some companies. On the other hand, other approaches discussed previously like Uber's don't require any change in the application code, which should make adoption way simpler.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#136

Post by ballenf »

stuff4ben wrote: post #59
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

Allocated storage should come directly from the consuming team's budget. Divvy up the total storage cost and allocate in proportion to requested limits.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#137

Post by danielfoster »

I don’t see how a one-day factory stoppage aligns with the article’s teaser, “ Just in time’ production system minimises costs but technical glitch highlights risks.”

It seems like Toyota identified the issue and got things running again quickly enough. This does not at all seem like a good case study for exploring the risks of JIT.

I read The Guardian regularly and will continue to, but they sometimes go out of their way to find an anti-business angle.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#138

Post by codetrotter »

throw0101c wrote: post #80
codetrotter wrote: post #63

Earlier quoted context omitted.

Could it depend on what file system you use, and OS settings you have? For example, Google says that by default ext4 will reserve 5% of disk space. So depending on your system, a “100% full” drive might still have a (sometimes significant) bit of space left.

> For example, Google says that by default ext4 will reserve 5% of disk space. Reserved 5% for root, but if the files/logs that are taking up space are written by the root user then it's a moot point: > Specify the percentage of the file system blocks reserved for the super-user. This avoids fragmentation, and allows root-owned daemons, such as syslogd(8), to continue to function correctly after non-privileged proces…

> if the files/logs that are taking up space are written by the root user then it's a moot point

Yeah. Sorry if my comment was a bit unclear but what I meant to say is that because of this reserved amount I could see it being the case that one machine is still able to finish booting even though the hard drive is “full”, because it is using the reserve capacity to allow processes to write to disk.

Whereas the other person that was unable to boot may have either been using a different file system without reserved amounts, or have settings that made it so reserve was set to 0, or their machine could have been actually physically out of space.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#139

Post by breakwaterlabs »

Veserv wrote: post #91
geerlingguy wrote: post #70

Earlier quoted context omitted.

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

In what world is 2160TB $100k?

Current single disk solutions are around $25/TB for HDDs and ~$100/TB for NVMe.

At a minimum you're looking at $54k just for raw capacity-- assuming no backup, no chassis, no networking, and no redundancy.

More reasonable estimations would be in excess of $400/TB.

Post reply on HN