Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

111–120 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#111

Post by gameman144 »

floorballchamp wrote: post #103
xaerise wrote: post #90

Earlier quoted context omitted.

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

> If you have a good reason So, you are not expecting that your co-workers have good reasons for what they are doing? Maybe the hiring bar at your place is too low then. I prefer to work at places where my default assumption is that everybody around me is smart and responsible. Lifts lots of worries off my shoulders (and tends to benefit the stock price over time too and thereby my income).

My coworkers have called out gaps in my thinking thousands of times when I have explained perceived needs to them, that's one of the main value-adds one gets from working in a team.

If I wanted unquestioned control, I'd run my own shop. If I want the best product, then I hope that people question my assumptions.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#112

Post by justsomehnguy »

doublepg23 wrote: post #78
greedo wrote: post #52

Earlier quoted context omitted.

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

I’ve been getting organic and cage-free hard drives nowadays.

> and cage-free hard drives nowadays.

SOmewhat recently I toyed with an idea of HDDs with an 'interposer' which would be just a dumb ATA2iSCSI interface, with Ethernet or WiFi connectivity.

That would allow you to place those drives literally anywhere and with a minimal footprint.

Sadly it would be too costly for the home usage (I assume about $30/unit at best) and for the enterprise usage... 1Gbit is too slow, 10Gbit is too hot and bulky and then you can't sell a $3 plastic case for $100 'vendor approved with light-path(r)(tm) diagnostic indication' for each drive.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#113

Post by xeromal »

ddoolin wrote: post #106
xeromal wrote: post #68

Earlier quoted context omitted.

I was considering what to call a Toyota employee and thought Toyototian sounded right. Accidentally added an additional to* but oh well. It's pretty incredible how fast Google adds stuff to their index this quickly!

I work for Toyota but I've never pondered what our demonym is; I think Toyototian (Toyotian?) has something going for it though. :)

Yeah, I think toyotian works. Haha

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#114

Post by lovehashbrowns »

floorballchamp wrote: post #99
lovehashbrowns wrote: post #89

Earlier quoted context omitted.

I work for a retailer where the service I’m responsible for is used by every cash register around the world for certain operations. When I came in, the RDS DB for this service had 60GB allocated to it, had literally just run out of space and caused an outage. The last team just gave it an additional 20GB. A month later, I was put in charge of it and it was already 5GB away from running out of space again. I put an en…

I'd guess the worry is that once you increase the storage, you never decrease it again. Ever. It's a one-way street. So, once everything is 5x over-provisioned, then the services tend to fill that space anyway (cause why not be wasteful if it doesn't cost anything) and a year later you are in the same seat again. I'm not saying this is real, but the worry certainly is.

That's certainly real and something to consider when provisioning systems. I'm fully on board with that. The problem is when the cost of the cost-savings solution vastly outweighs the cost of over-provisioning infrastructure. Like this Jenkins issue bubbling up ~2-4 times a month vs just giving the worker nodes more storage space. There's been times where it happened during the night and people got paged.

Or comparing the cost of one store not being able to open on time because the RDS database's space ran out. VPs and directors start yelling and there's suddenly like 20+ people involved in figuring out why this one store didn't open on time. What's the cost of that compared to just giving the DB 250GB of space so this never comes up again?

But you are also 100% correct and I've seen that happen here, too. There's some instances I'm responsible for that were using EFS for their local storage. Costing thousands of dollars every month for absolutely no reason. I switched those to reasonably-sized EBS volumes and that alone was half of my annual savings goal.

I was completely flabbergasted seeing these instances using EFS while others were stuck on 8GB EBS volumes. Backups on the EFS drives had ballooned to the many TBs. And the backups were worthless! Instances themselves are ephemeral. They use S3 for long-term storage & metadata is on a database. Those are the things that should be backed up & their cost compared to EFS is minuscule.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#115

Post by datadrivenangel »

marcosdumay wrote: post #101
greedo wrote: post #52

Earlier quoted context omitted.

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

Well, that's a choice your ops people make. There is absolutely no technical reason to provide all of your storage needs with a single quality of disks. You may do that to get some economies of scale, but if the impact is that high, you should rethink it. You can just as well have two different solutions, one with a huge amount of disk space, little redundancy and low performance, and one with a limited amount of spa…

When you start requiring redundancy, you start needing 2-6x the number of disks, minimum.

Still cheap, but not that cheap.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#116

Post by syndicatedjelly »

Veserv wrote: post #91
geerlingguy wrote: post #70

Earlier quoted context omitted.

This reminds me of a time we were helping a dev team bring logging in house because they weren't liking the features of their logs-as-a-service provider. They set all applications to "debug" level logs in production and were generating multiple gigabytes of logs per hour. They wanted 90 days retention, and the ability to do advanced searching through the live log data so they could debug in production (they didn't re…

90 days retention is only 2,160 hours. Even at 999 GB/hr that is only ~2160 TB of storage. So, if we stretch the definition of “multiple gigabytes”, is maybe $100k in storage which is around 3-6 developer-months. If we use a more reasonable definition like 10 GB/hr, then that is 20 TB, so maybe $1k in storage which is around 1 developer-day. Seems pretty reasonable to me.

…what? Without any other context on what they’re working on or the size of the company, an extra developer of cost is automatically reasonable?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#117

Post by syndicatedjelly »

JackFr wrote: post #93
xaerise wrote: post #90

Earlier quoted context omitted.

I don't get it... If you have a good reason to use 2 TB, i'm happy to allocate it for you. If you just "I want 20 GB of storage", i'm not going to give it to you. Storage is cheap in relation to other things. Just have a good reason to why you need it.

Why wouldn't you assume that if I'm asking for it I have a good reason? Are you going to rearchitect my system for me?

Accountability? Is that attitude any different from just asking for money and refusing to explain what it’s for?

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#118

Post by syndicatedjelly »

alanbernstein wrote: post #45
londons_explore wrote: post #41

Earlier quoted context omitted.

This is why one should generally be using network/cloud storage, with soft/hard limits. As soon as the soft limit is hit, fire off an alert. Have the hard limit set at double or more.

On my MacBook, I like to keep a few giant blank files that I can delete in a disk space emergency.

How about a cron job that checks disk usage once per day and prints the top 3 culprits by file type if du exceeds 90%

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#119

Post by syndicatedjelly »

ndhfkashdben wrote: post #26

I worked at a phone factory once. One day, an app stopped working. It was running on an old Linux server. Nobody had access to it, or so I thought. I noticed the pings were showing high failure rates. I asked to check switch logs or the server's ifconfig output and got weird excuses why they couldn't check anything. One week goes by, people are living with the error and adjusting the factory to work around that. Many…

He was low key telling you to quit and gtfo

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#120

Post by floorballchamp »

gameman144 wrote: post #111
floorballchamp wrote: post #103

Earlier quoted context omitted.

> If you have a good reason So, you are not expecting that your co-workers have good reasons for what they are doing? Maybe the hiring bar at your place is too low then. I prefer to work at places where my default assumption is that everybody around me is smart and responsible. Lifts lots of worries off my shoulders (and tends to benefit the stock price over time too and thereby my income).

My coworkers have called out gaps in my thinking thousands of times when I have explained perceived needs to them, that's one of the main value-adds one gets from working in a team. If I wanted unquestioned control, I'd run my own shop. If I want the best product, then I hope that people question my assumptions.

We are not in disagreement here. Bouncing off ideas and thoughts is a good thing.

The way this was phrased was more from the angle "who knows what these guys were thinking; if they can't give me a good reason, no way they will get storage space as I don't trust that they make good decisions on their own".

Post reply on HN