Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Toyota blames factory shutdown in Japan on ‘insufficient disk space’

theguardian.com

Post reply on HN
Print view View original

201–210 of 223 posts

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#201

Post by floorballchamp »

arghwhat wrote: post #199
floorballchamp wrote: post #196

Earlier quoted context omitted.

> It's a tug of war, Yeah, any workplace in which the word "war" was used in the context of colleague interaction saw me leave within a few months. I like to plan those things ahead of time with all stakeholders involved, then we work together instead of against each other.

I believe you are intentionally misunderstanding. The term "tug of war" is not used to indicate armed conflict or even a problem. It indicates balancing forces that you want to maintain - pull the rope too far to one side, and you end up in a suboptimal extreme. Unless you work with clones of yourself, there will always be differences in opinions and priorities, and not every feature and bug fix can be a company-wide…

Of course there will be differences. That's why you sit down and plan things together, pulling in and coordinating with all _relevant_ stakeholders. Of course not the whole company.

But the attitude needs to be "let's put the requirements on the table and see what we can do" instead of "you don't get what you want unless you give me a good reason". The latter comes from an angle of distrust which I'm arguing against. The former comes from an angle of collaborative problem solving.

In a company in which I go to a team relevant to a project and like to engage in a discussion and am met with an attitude of "unless you give us a good reason we'll stop talking to you", the atmosphere is not one that will keep me personally for long. YMMV.

> I believe you are intentionally misunderstanding.

You are free to believe what you like. Opening a reply with such a sentence is pretty sad though. It does not foster a healthy atmosphere, nor does it match reality, I might add.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#202

Post by atoav »

akira2501 wrote: post #161
stuff4ben wrote: post #59

Earlier quoted context omitted.

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

You have to fill out a load chart to fly a plane, they should have to fill out something like a storage chart to get a production allocation. What size are your objects? How many per unit of time and served entity? What is the lifetime of those objects? How is that lifetime managed?

I mean in my case it was literally a database file filled with the number of (dummy) people who are currently in our org. So that database size was the size of the project. He just didn't plan for the size of backups (backups were his job, not ours).

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#203

Post by atoav »

ballenf wrote: post #136
stuff4ben wrote: post #59

Earlier quoted context omitted.

Having been on both sides (admin and developer), developers are notoriously bad at estimating how much space they need. You can't give them carte blanche to the storage because they'll waste it and consume as much as they're given without a thought to conserving it. And then when you put limits in, they'll whine and complain until they get what they want. Being an Artifactory service provider for a large IT dept gave…

Allocated storage should come directly from the consuming team's budget. Divvy up the total storage cost and allocate in proportion to requested limits.

Sure, what are you going to bill me for a 30GB VM on a 100TB cluster? Whether I want 30GB or 100GB for an absolute central service for the whole org shouldn't matter. If we are talking about personal pet projects or user accounts — sure — but that wasn't my complaint here.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#204

Post by thomastjeffery »

greedo wrote: post #189
thomastjeffery wrote: post #154

Earlier quoted context omitted.

If you are in that situation, then obviously you have to do what you have to do. Absolutely none of that has to apply to the sort of situation we are talking about here. The only real reason to spend more per disk is when you know all your disks are going to fail, and extending the lifespan per average disk will definitely save you more than the enterprise markup costs. So you better have dozens or hundreds of disks…

You try telling a VP that their business unit can't function because you decided to purchase the cheaper drive. All disks fail eventually. Outliers may run longer than the MTBF for that drive model, but they all fail eventually. And backups are fine for restoring data, but they don't help provide access to that data in a timely fashion. That's why people use SANs, AWS etc. The cost of a SAN storage array is a nit for…

It's very unlikely for all of your disks to fail at the same time. Even if they do, that's why you have an offsite backup. The name of the game is redundancy, not longevity.

In practically every use case, two consumer disks will be better than one enterprise disk. Once you start failing enough disks often enough, longevity can be worth the additional cost. Until then, it just isn't.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#205

Post by greedo »

thomastjeffery wrote: post #204
greedo wrote: post #189

Earlier quoted context omitted.

You try telling a VP that their business unit can't function because you decided to purchase the cheaper drive. All disks fail eventually. Outliers may run longer than the MTBF for that drive model, but they all fail eventually. And backups are fine for restoring data, but they don't help provide access to that data in a timely fashion. That's why people use SANs, AWS etc. The cost of a SAN storage array is a nit for…

It's very unlikely for all of your disks to fail at the same time. Even if they do, that's why you have an offsite backup. The name of the game is redundancy , not longevity. In practically every use case, two consumer disks will be better than one enterprise disk. Once you start failing enough disks often enough, longevity can be worth the additional cost. Until then, it just isn't.

Actually, it's not uncommon for a batch of disk from a vendor (the same lot #) to have failures.

And again, a backup is fine for deleted data, or fire/ransomware. But for day to day operations, no one is really willing to wait for you to restore from a local backup, much less an offsite backup.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#206

Post by atoav »

ilyt wrote: post #176
atoav wrote: post #28

Earlier quoted context omitted.

I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…

We tried. Devs when give more space just didn't bother to clean old crap and exact same thing happened but with few months of delay. But then we generally just ask how much do they need and bill project for it so that's generally also on them. But, unlike Toyota, we do have disk space alerts. Sometimes the problem is also entirely political, the management needs to tell client and charge them for more storage and won…

In my case the data was a dummy database of more dummy users than are at our org (maybe 50% more). So once this goes to production it would likely get smaller.

The problem in this case was twofold:

- admin had the job to implement database backups. He didn't factor in backup size when allocating disk space. So this was wntirely his own fault.

- the database does store certain transactions for a certain period, so this grew initially until it it setteled at a certain level. Because the margin of storage was slim, this caused the problem

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#207

Post by WirelessGigabit »

SOLAR_FIELDS wrote: post #197
Volundr wrote: post #134

Earlier quoted context omitted.

A few years ago I joined a company aggressively trying to reduce their AWS costs. My jaw got the floor when I realized they were spending over a million a month in AWS fees. I couldn't understand how they got there with what they were actually doing. I feel like this comment perfectly demonstrates how that happens.

AWS also purposefully makes it easy to shoot yourself in the foot. Case in point that we were burned on recently: - set up some service that talks to a s3 bucket - set up that bucket in the same region/datacenter - send a decent but not insane amount of traffic through there (several hundred Gb per day) - assume that you won’t get billed any data transfer fees since you’re talking to a bucket in the same data center…

If you're up for it... https://github.com/AndrewGuenther/fck-nat

Even has ha mode.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#208

Post by tremon »

marcosdumay wrote: post #101
greedo wrote: post #52

Earlier quoted context omitted.

This is such a uninformative comment. Business/Enterprise storage is far, far more expensive than a drive you pick up at Best Buy or Fry's.

Well, that's a choice your ops people make. There is absolutely no technical reason to provide all of your storage needs with a single quality of disks. You may do that to get some economies of scale, but if the impact is that high, you should rethink it. You can just as well have two different solutions, one with a huge amount of disk space, little redundancy and low performance, and one with a limited amount of spa…

Single disks are cheap. Multi-disk storage systems are not. Multi-tiered storage systems are even more expensive. I don't see an inherent conflict there.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#209

Post by arghwhat »

floorballchamp wrote: post #201
arghwhat wrote: post #199

Earlier quoted context omitted.

I believe you are intentionally misunderstanding. The term "tug of war" is not used to indicate armed conflict or even a problem. It indicates balancing forces that you want to maintain - pull the rope too far to one side, and you end up in a suboptimal extreme. Unless you work with clones of yourself, there will always be differences in opinions and priorities, and not every feature and bug fix can be a company-wide…

Of course there will be differences. That's why you sit down and plan things together, pulling in and coordinating with all _relevant_ stakeholders. Of course not the whole company. But the attitude needs to be "let's put the requirements on the table and see what we can do" instead of "you don't get what you want unless you give me a good reason". The latter comes from an angle of distrust which I'm arguing against.…

> Opening a reply with such a sentence is pretty sad though. It does not foster a healthy atmosphere, nor does it match reality, I might add.

Your response hitched on a single word ("war") within a common phrase ("tug of war", a game). While it might have been accidental, such answers mislead from the actual discussion (and tends to be used as distractions when no good answer is present).

> Of course there will be differences. That's why you sit down and plan things together, pulling in and coordinating with all _relevant_ stakeholders.

When you discuss new architectures or large projects, this is a given, but this covers only a small portion of company operation - the rest is organic day-to-day work, which slowly but surely distorts initial assumptions. Slowly boiling the frog, so to speak. Think one team making changes that affect request patterns, another team making something that is accidentally quadratic, and a third team suddenly asking for a large number of cloud resources to carry this that should absolutely be challenged.

And at the same time, teams are under different organization units with different budgets, schedules, leaderships and priorities - and most certainly don't care about daily scrum work of other teams.

> In a company in which I go to a team relevant to a project and like to engage in a discussion and am met with an attitude of "unless you give us a good reason we'll stop talking to you", the atmosphere is not one that will keep me personally for long. YMMV.

No one said "we'll stop talking to you", but "you get what can be justified". If you take offense to be challenged and would rather work somewhere else, you do you, but if you can't justify your request I'd argue that you are not doing your job properly in the first place.

Re: Toyota blames factory shutdown in Japan on ‘insufficient disk space’

#210

Post by randmeerkat »

bestouff wrote: post #190
randmeerkat wrote: post #163

Earlier quoted context omitted.

> Part of the original promise from these providers was that there would be a 90%+ reduction in supporting work, when we all know the reality is that you become specialized in solving issues with Heroku/AWS/providerX For all but the most simple setups, providerX solutions are still better than maintaining your own bare metal solutions with their associated hardware / software / datacenter vendor relationships. Also,…

I have heard this many times but I've never seen convincing proof of that.

> I have heard this many times but I've never seen convincing proof of that.

What kind of proof are you looking for?

Post reply on HN