Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

121–130 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#121

Post by ooterness »

atoav wrote: post #28
xeromal wrote: post #13

I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

Nitpick: "b" is for bits, "B" is for bytes. Please don't casually mislabel units.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#122

Post by floorballchamp »

lovehashbrowns wrote: post #114
floorballchamp wrote: post #99

Earlier quoted context omitted.

I'd guess the worry is that once you increase the storage, you never decrease it again. Ever. It's a one-way street. So, once everything is 5x over-provisioned, then the services tend to fill that space anyway (cause why not be wasteful if it doesn't cost anything) and a year later you are in the same seat again. I'm not saying this is real, but the worry certainly is.

That's certainly real and something to consider when provisioning systems. I'm fully on board with that. The problem is when the cost of the cost-savings solution vastly outweighs the cost of over-provisioning infrastructure. Like this Jenkins issue bubbling up ~2-4 times a month vs just giving the worker nodes more storage space. There's been times where it happened during the night and people got paged. Or comparin…

Yeah. I suppose the tricky thing is:

> compared to just giving the DB 250GB of space so this never comes up again?

As long as there is reasonable confidence in that this is actually the case, then just provision the space and be done with it. That requires a certain understanding of future space requirements/expectations, and anything even just so slightly running away / leaking space will hit any limit given enough time. So, due diligence requires looking at whether it's actually needed.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#123

Post by jbverschoor »

codetrotter wrote: post #63
jbverschoor wrote: post #40

Earlier quoted context omitted.

Never happened to me.. You'll still be able to login, free up some space to get the services up again, and then resize your disk if that's needed.

Could it depend on what file system you use, and OS settings you have? For example, Google says that by default ext4 will reserve 5% of disk space. So depending on your system, a “100% full” drive might still have a (sometimes significant) bit of space left.

In general both ext3 and ext4.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#124

Post by jbverschoor »

Ekaros wrote: post #43
jbverschoor wrote: post #40

Earlier quoted context omitted.

Never happened to me.. You'll still be able to login, free up some space to get the services up again, and then resize your disk if that's needed.

In this case I had to use rescue mode. As even different tty was too slow to respond. Also just removing some stuff up wasn't working.

Sounds like swapping/trashing + an exception loop taking all resources

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#125

Post by lovehashbrowns »

floorballchamp wrote: post #122
lovehashbrowns wrote: post #114

Earlier quoted context omitted.

That's certainly real and something to consider when provisioning systems. I'm fully on board with that. The problem is when the cost of the cost-savings solution vastly outweighs the cost of over-provisioning infrastructure. Like this Jenkins issue bubbling up ~2-4 times a month vs just giving the worker nodes more storage space. There's been times where it happened during the night and people got paged. Or comparin…

Yeah. I suppose the tricky thing is: > compared to just giving the DB 250GB of space so this never comes up again? As long as there is reasonable confidence in that this is actually the case, then just provision the space and be done with it. That requires a certain understanding of future space requirements/expectations, and anything even just so slightly running away / leaking space will hit any limit given enough…

Yup, I implemented a bunch of graphs and alerts. Right now it's at 100GB of usage so it's still growing but at a fairly predictable rate. Another nice thing to know is if it's possible to reduce that usage. I haven't been able to look into that but I know one of the causes of the usage increase. The service uses the DB to store some indexing data. There's a team forcing it to re-index and I can tell when they deploy because the storage spikes a little bit every time they do a deployment. Nothing I can do about that, sadly.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#126

Post by panarky »

fillipvt wrote: post #109
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

Just wondering, why pi?

I've tested both π and g, and while they both work well, g results in far fewer disk full errors. I've heard c works even better, though I haven't tried it yet.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#127

Post by bandyaboot »

JackFr wrote: post #93
xaerise wrote: post #90

Earlier quoted context omitted.

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

Why wouldn't you assume that if I'm asking for it I have a good reason? Are you going to rearchitect my system for me?

Wouldn’t you expected to have to provide some level of justification if you were, say, requesting a new development machine?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#128

Post by yjftsjthsd-h »

Veserv wrote: post #91
geerlingguy wrote: post #70

Earlier quoted context omitted.

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

... only 2PB? You might be using a different scale than some of us.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#129

Post by Gud »

bandyaboot wrote: post #127
JackFr wrote: post #93

Earlier quoted context omitted.

Why wouldn't you assume that if I'm asking for it I have a good reason? Are you going to rearchitect my system for me?

Wouldn’t you expected to have to provide some level of justification if you were, say, requesting a new development machine?

There is a difference between spending $2000+ on a new computer and $10, which is about what a terabyte costs. Probably just having the discussion itself would waste more resources than just giving the storage space.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#130

Post by giovannibonetti »

floorballchamp wrote: post #102
Veserv wrote: post #91

Earlier quoted context omitted.

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

Plus, logs have enormous compression potential since their entropy is so low. That's the property exploited by every logging-as-a-service out there.

Related to that, last year Uber's engineering blog mentioned very interesting results with their internal log service [1].

I wonder if there's anything as good in the open-source world. The closest thing I can think of is Clickhouse's "new" JSON type, which is backed by columnar storage with dynamic columns [2].

[1] https://www.uber.com/en-BR/blog/reducing-logging-cost-by-two... [2] https://clickhouse.com/docs/en/integrations/data-formats/jso...

Post reply on HN