81–90 of 223 posts
If it’s not an excuse for something more serious they definitely need to rethink their infrastructure. It’s not allowed to have a single server with no disk space, taking offline 14 factories. That should’ve used the old trick. Create a big empty file inside the server (i.e. delete_me_in_case_of_need.txt with size a few GBs) and delete that in case of emergency. This will buy you some time to take the necessary actio…
I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.
As we move to more abstract systems I wonder how well we (as in companies) are keeping basic systems management capabilities in place at a personnel level. At $DAY_JOB we recently scuttled most development efforts for a week for our teams. Our nightly backup job that sanitizes PHI ballooned overtime to, say, 20GB+1Byte and ran out of disk space. Because we are running Kubernetes on Fargate we don’t need a full time o…
In the end what you have is a stack of technology that the company cannot actually maintain on their own. Now I know that 'shoulders of giants' is a thing and that we shouldn't expect everyone to be able to do every part of work that came before them, but as your example shows it is a real business risk when things go south. Part of the original promise from these providers was that there would be a 90%+ reduction in supporting work, when we all know the reality is that you become specialized in solving issues with Heroku/AWS/providerX
Earlier quoted context omitted.
Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering p…
Are you talking consumer prices, enterprise redundant SSDs in a Data Center, or enterprise cloud storage? In my experience the latter two could add another digit to your price.
More disk space isn't one drive, but redundancy too And backups. And even network bandwidth.
This thread sort of highlights the problems. DEVs don't get it. Nor does management.
Earlier quoted context omitted.
There was probably an array with a few dozen terabytes to spare and the guy made you run back to him once a month for some misguided job security purpose.
Note that a few dozen terabytes is also nothing. 12 TiB is about $800 on an SSD and about $250 on a HDD. Plus some overhead for the enclosure and redundancy, of course. It costs on the order of a day's pay for an engineer, at most. I don't excuse wanton waste of storage because it's easy for sloppy practices to balloon to massive confusion and inefficiency. But that discipline should be enforced by good engineering p…
All Flash arrays will get even more $$$$
Price out a Nimble or PureStorage array can see if they will sell you 12TB for $800... I will wait.
Earlier quoted context omitted.
Anecdotes from bank-employed chums have reported similar things - including where it's been quicker and easier for them to nip to the shops and buy an external drive to get some space to relieve pressure whilst the 800 requisition forms are slowly working their way through the systems.
It’s year ago now, but my father bought a second hand hard drive and found it full of banking data with customer names and all.
Earlier quoted context omitted.
I wonder, wouldn't it be more of a question of how much later to shut down. Or in other words, how big is the time buffer you get for not being "just in time"? The article suggests (the way I read it, at least) that it is some fundamental time span, but I feel like it's more like half a day?
In some (American) automotive plants I've worked in, JIT lifetimes are about 10 hours. The order for a seat or mirror or door handle doesn't exist at the tier 1 suppliers where my equipment gets installed until 10 hours before it is needed. At that point, my PLC gets a request to induct a particular part number and paint color into the manifest, inducts it and acknowledges that request, the machine builds it, laser-m…
I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…
I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
The instances for the service itself had 20GB of EBS allocated to them. Luckily they don’t need much local storage. But that’s typical here. There’s a Jenkins instance that is even more of a pain. I’m not responsible for it but every week or two one of the worker nodes runs out of space because they’re given 8GB of storage space. I’m just watching the disaster unfold over the course of a year and a half as I’m constantly telling that team to just up the storage space on the worker nodes instead of constantly having to fiddle with cron jobs.
It’s not even an expense thing. They just… don’t want to increase the storage space. It drives me insane.
I was responsible for some dev ops stuff at a state's health department and one of the more infuriating things about working at that place was that it was like pulling teeth getting more storage allocated. Our backups would be running out of disk and they'd allocate me 50 or 100 GB at a time. I'm sure someone Toyototian was yelling that this was going to happen for the past 6 months.
I worked with an admin like that. We had a huge cluster, but he was greedy with the storage space for a service that was critical for the operation of the org. And I get that this is a good mindset for not wasting space overall, but if a single backup fills 90% of your storage space in test use, that machine is not ready for production. And we are not talking about a lot of space here. The backup was maybe 30 Gb the…
If you just "I want 20 GB of storage", i'm not going to give it to you.
Storage is cheap in relation to other things. Just have a good reason to why you need it.