51–60 of 281 posts
Earlier quoted context omitted.
someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph
So the drives are never going to fail? PSUs are never going to burn out? You are never going to need to procure new parts? Negotiate with vendors?
Ignoring failed hdds week likely mean very little maintenance.
Nice writeup. All of the technical detail is great! I'm curious about the process of getting colo space. Did you use a broker? Did you negotiate, and if so, how large was the difference in price between what you initially were quoted and what you ended up paying?
Earlier quoted context omitted.
Not included is overhead of dealing with maintenance. S3/R2 generally don’t require OPS type dedicated to care and feeding. This type of setup will likely require someone to spend 5 hours a week dealing with it.
Why 5h a week? Just for hardware?
Earlier quoted context omitted.
someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph
You are under the assumption that only Ceph (and similar complex software) requires staff, whereas plain 30 PB can be operated basically just by rebooting from time to time. I think that anyone with actual experience of operating thousands of physical disks in datacenters would challenge this assumption.
Earlier quoted context omitted.
someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph
So the drives are never going to fail? PSUs are never going to burn out? You are never going to need to procure new parts? Negotiate with vendors?
Yes, drives are going to fail. Yes, power supplies are going to burn out. Yes, god, you’re going to get new parts. Yes, you will have to actually talk to vendors.
Big. Deal. This shit is -not- hard.
For the amount of money you save by doing it like that, you should be clamoring to do it yourself. The concern trolling doesn’t make any sort of argument against it, it just makes you look lazy.
Earlier quoted context omitted.
You are under the assumption that only Ceph (and similar complex software) requires staff, whereas plain 30 PB can be operated basically just by rebooting from time to time. I think that anyone with actual experience of operating thousands of physical disks in datacenters would challenge this assumption.
we have 6 months of experience operating thousands of physical disks in datacenters now! it's about a couple hours a month of employee time in steady-state.
Don't get me wrong, I buy the actual numbers regarding hardware costs, but in addition to that presenting the rest as basically a one man show in terms of maintenance hours is the point where I'm very sceptical.
The biggest part that is always missing in such comparisons is the employee salaries. In the calculation they give $354k/year of total cost per year. But now add the cost of staff in SF to operate that thing.
someone has to go and power-cycle the machines every couple months it's chill, that's the point of not using ceph
The biggest part that is always missing in such comparisons is the employee salaries. In the calculation they give $354k/year of total cost per year. But now add the cost of staff in SF to operate that thing.
The reality nowadays is: the on-prem staff is covered in the colo fees, which is split between everyone coloing in the location and reasonably affordable. The software-level work above that has massively simplified over the past 15 years, and effectively rivals the volume of work it would take to run workloads in the cloud (do you think managing IAM and Terraform is free?)