311–320 of 374 posts
Earlier quoted context omitted.
Here is Brendan Gregg showing how vibrations can affect disk latency: https://www.youtube.com/watch?v=tDacjrSCeq4
I'm an absolute noob here, but is using SSD racks for storage a feasible option cost wise and for this issue in particular?
Earlier quoted context omitted.
You realize you can't fit 10 appliances of 4U in a rack? (A rack is 42U) There's network equipment and power equipment that requires space in the rack. There's power limitations and weight limitations on the rack that prevents to fill it to the brim.
I've put 39U of drives in a rack before. You only need 1U for a network switch, and you can get power that attaches vertically to the back, so it doesn't take up any space. If you have a cabinet with rack in front and back and all the servers have rails, the weight shouldn't be an issue. The biggest issue will be cooling depending on how hot your servers run. Specifically, it was a rack full of Xserve RAIDs, which ar…
* A typical rack is rated for somewhere between 450 and 900 kg (your mileage may vary).
* A disk is about 720g.
* A 4U quanta enclosure is 36 kg empty.
* With 10 enclosures of 60 disks, that's a total of 792 kg inside the rack.
You will want to check what rack you have exactly and weight things up.
The rack itself is another 100 to 200 kg. You will want to double check whether the floor was designed to carry 1 ton per square meter. It might not be.
My personal tip. Definitely do NOT put a row of that in an improvised room in a average office building. You might have a bad surprise with the floor. ;)
Anyway. The project will probably be abandoned after the OP tries to assemble the first enclosure (80kg fully loaded) and realize he's not going to move that.
Earlier quoted context omitted.
> Let's do the math. Offers no math. At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1). I think your conclusion that S3 is "pretty good" needs a lot more math to support. (1) https://twitter.com/eastdakota/status/1371252709836263425
The math should also include the price of the staff who babysit 625 spinning metal disks, who likely drive to a data center multiple times a week to swap failed drives. I shudder to think if this job fell in my lap!
Earlier quoted context omitted.
The math should also include the price of the staff who babysit 625 spinning metal disks, who likely drive to a data center multiple times a week to swap failed drives. I shudder to think if this job fell in my lap!
Sure, but you actually have to go through the steps. The very first step, back of the napkin, indicates savings. More than enough to warrant a more detailed evaluation. Then you can start considering the more complicated factors (redundancy, staffing, power, ...). My response was in the context of someone who didn't do any of that. Also, my tongue-in-cheek response to you is: the price will be offset by the SRE engin…
I did and you ignored all of it -_-
* To store 10+ PB of data.
* You need 15 PB of storage (running at 66% capacity)
* You need 30 PB of raw disks (twice for redundancy).
>>> At retail, 625 16TB drives is $400000.
That's only 10 PB of disks. That's about one third of the actual need.
Please triple your number and we will start talking. That's about 2000 disks and well above a million dollar.
You can't just call a supplier and get for a thousand 16 TB disks (or even a hundred). They don't have that in stock now. The lead time might be 6 months to get a few hundreds. They might not have 16 TB disks for sale at all, the closest might be a 12 or 14TB.
Handling large amount of hardware is a logistic problem. Not a cost problem.
As for my own storage I use 1TB SanDisc SD cards in a raspberry 2 cluster for write once data (user) and 8x64GB 50nm SATA drives from 2011 on 2xAtom 8-core for data that changes all the time! Xo
People say that content is king, I think that final technology (systems that don't need rewriting ever) is king and content has peaked! ;)
There is S3-like interface, so you may just change access key, and region host: https://www.backblaze.com/b2/docs/s3_compatible_api.html
Earlier quoted context omitted.
> 625 16TB drives is $400000 how much is the real estate cost of 625 drives and associated machinery to run it? At a guess, AWS has an operating margin of about 30%, so you can approximate their cost of hardware, bandwidth, and other fixed costs as 70% of their sticker price. As a start up, can you actually get this price to be lower? I actually dont think you can, unless your operation is very small and can be done…
Their margin on bandwidth is literally over 1000%. Quick google says that S3 costs 320% more than Backblaze (which, presumably, isn't running at a loss). > At a guess The comments in this discussion that try to provide actual numbers show a fairly lopsided argument against S3. The comments that are advocating for S3 aren't as detailed. You can look at this at the macro level, as on comment did, and see that one 1.2PB…
protip: $11k is the cost of the empty enclosure. disks are sold separately.
Way cheaper than AWS, and a lot less headache than trying to run it all yourself.
At that kind of scale, S3 makes zero sense. You should definitely be rolling your own. 10PB costs more than $210,000 per month at S3, or more than $12M after five years. RackMountPro offers a 4U server with 102 bays, similar to the BackBlaze servers, which fully configured with 12GB drives is around $11k total and stores 1.2 PB per server. ( https://www.rackmountpro.com/product.php?pid=3154 ) That means that you coul…
After a certain size, AWS et al simply don't make sense, unless you have infinitely deep pockets. For storage that you pull from, AWS et al charge bandwidth costs. These costs are non-trivial for non-trivial IO. I worked up financial operational models for one of my previous employers, when were were looking at costs of remaining on S3 and using it, versus rolling it into our own DCs. The download costs, the DC space…
The important point to understand in all of this is, is that there are cross-over points in the economics for which one becomes better than the other. Part of the economics is the speed of standing up new bits (opportunity cost of not having those new bits instantly available). This flexibility and velocity is where cloud generally wins on design, for small projects (well below 10PB).
This said, if your use case appears to be rapidly blasting through these cross-over points, the economics usually dictates a hybrid strategy (best case) or a migration strategy (worst case).
And while your use case may be rapidly approaching these limits (you need to determine where they are if you are growing/shrinking), there are things you can do to risk and cost reduce transitions ahead of this.
Hybrid as a strategy can work well, as long as your hot tier is outside of the cloud. Hybrid makes sense also if you have to include the possibility of deplatforming from cloud providers (which, sadly, appears to be a real, and significant, risk to some business models and people).
None of this analysis is trivial. You may not even need to do it, if you are below 1PB, and your cloud bills are reasonable. This is the approach that works best for many folks, though as you grow, it is as if you are a frog in ever increasing temperature water (with regard to costs). Figuring out the pain point where you need to make changes to get spending on a different (better) trajectory for your business is important then.
Earlier quoted context omitted.
Wow that’s up to $500,000 just to export 10PB (depending on region).
According to https://aws.amazon.com/snowball/pricing/ , egress fees depends on the region, which can range from $0.03/GB (North America & parts of Europe) to $0.05/GB (parts of Asia and Africa). So US$300K to US$500K for egress fees + cost of Snowball devices. The major downside of Snowball in this export case is the size limit of 80TB per device - from https://aws.amazon.com/snowball/features/ : "Snowball Edge Stora…
I remember asking an Amazon executive in London when AWS was very new and they were evangelising it to developers; I asked him what is the cost of getting data out of AWS if I want to move it to other service provider, or how easy it will be? And he avoided giving a straight simple answer. I realised then than the business model from the start was to lock-in developer/startups/companies in to the AWS ecosystem.