I really love these, and I desperately wanted to be a Backblaze customer, but I was put off by the abysmal desktop software :(
Storage execution seems top-notch. End-user software deserves some love.
Backblaze Storage Pod 5.0
111–120 of 222 posts
Re: Backblaze Storage Pod 5.0
#112It's wonderful what you guys are doing. Thank you for open sourcing your work.
Re: Backblaze Storage Pod 5.0
#113Earlier quoted context omitted.
You can do a lot more than 60, and you can get a lot bigger than 6T. > It's disingenuous to call this not "state of the art" when it's really quite close, and obviously is meeting a backblaze design goal. So, I never said it didn't meet their design goal; it looks like quite a nice system, and it's better than their previous gen. Looks like they're doing things right, looks like solid engineering. But I stand by the…
> you can get a lot bigger than 6T Come on - most companies reasonably can't. There's a WD 10T smr/helium drive, and seagate is selling that crap 8T video drive. Unless you're talking ssd 3d/TLC density (more $), or the vapor-ware 20T drives that some storage vendors hint at and don't let me file a PO for, I literally don't know what you're referring to (now I'm getting where I can't name names) The reason I'm compla…
> You merely state that it's just not state of the art, and to come work at dropbox.
That's about all I can do. However...
> This is what I mean by being disingenuous.
I'm don't think disingenuous is what I'm being. Maybe I'm not being as forthcoming as you'd like. Maybe my comment wasn't useful without details I'm not providing. But I'm not lying or misrepresenting anything.
So let me try to be helpful without using proprietary information. Using information people have unearthed in this thread, we can do some math to prove I'm not completely bonkers:
96 unit chassis * 10T SMR = 960T, which is in the > 5x the density in 4U
If you could manage that 960T with the same compute resources, you'd save a lot on compute, since that's currently a ~30% of their cost.
With custom enclosures and hardware, you can do more still. You can have deep relationships with vendors and have them customize firmware for your particular use case. You can take their discards that didn't quite meet spec. You can buy crazy cheap stock with high failure rates b/c your distributed system is insanely good at repairs. You get access to stuff that's not being sold yet.
However, let me state again--this would take a lot of work. You need to mold your software around your hardware profile with very small tolerances. You might need to move some I/O stuff in-kernel, and/or move the IP stack out of kernel. You might need to reinforce your DC's raised floors to handle the weight (spoiler: you definitely do unless you planned for these types of machines ahead of time). You need great operations, and really thorough hardware quals b/c your failure domain is huge. Network traffic will go through the roof if a box fails, since you're repairing so much data.
Dozens of people on a handfuls of teams will spend time attacking this problem from different angles.
So, again--I don't necessarily think Backblaze did anything wrong here. Doing all this might be a crazy idea for them and not worth it. My statement was: it's not state of the art in terms of $/GB or density.
But for some shops, they spend so much on storage they can afford invest really, really deeply on optimizing storage cost. That's where the state of the art is.
Re: Backblaze Storage Pod 5.0
#114Earlier quoted context omitted.
The 60 HDDs aren't single points of failure. {Edit: I mean for the server, not for the whole system.} And a raid1 pair of HDDs for the system disk is more expensive than a small SSD, more fussy, and the SSD is still less likely to totally fail.
It doesn't sound like the boot drive is a single point of failure since data is stored Reed-Solomon coded in chunks across many pods. If one data drive fails, the whole pod has to go down for maintenance for it to be replaced. The only difference is that you get to choose when to take a pod down for maintenance to replace a data drive.
That's the situation I was talking about, yes. It's much cheaper to go into the datacenter once a month to replace failed data disks than to have to go in to promptly replace any system disk HDD that fails, in order to not have 60 idle disks.
Re: Backblaze Storage Pod 5.0
#115Worth pointing out (for those of you not in the storage game) that these appliances, while cool, are pretty far from state of the art. We're (Dropbox) packing up to an order of magnitude more storage into 4U. And costs aren't close, let alone the addressable throughput given that you guys just went to 10G. Serious kudos though are due for the Backblaze blog. You're all at least talking about what you're up to, which…
That statement carries 0 value without any proof. Backblaze is doing an excellent job documenting their process and shouting 'we are doing 10 times better', 'costs aren't close' and 'far from the state of the art' is pretty lame if you're not willing to back those claims up. I'm a big fan of dropbox but this was below the belt.
Re: Backblaze Storage Pod 5.0
#116I just posted a link to an interesting blog post detailing the risks of building a storage pod [1] [1] https://news.ycombinator.com/item?id=10541055
Isn't that basically long-form restatement that you should know what you need before spending money? The storage pods are pretty openly targeted at large-scale operations where the lower storage costs pay for a dedicated ops team and things like redundancy are handled at the software layer. Dinging them for not having hardware RAID is like buying a semi-truck and complaining that it doesn't fit in your garage.
The post itself makes the same points.
Re: Backblaze Storage Pod 5.0
#117Earlier quoted context omitted.
It doesn't sound like the boot drive is a single point of failure since data is stored Reed-Solomon coded in chunks across many pods. If one data drive fails, the whole pod has to go down for maintenance for it to be replaced. The only difference is that you get to choose when to take a pod down for maintenance to replace a data drive.
That's the situation I was talking about, yes. It's much cheaper to go into the datacenter once a month to replace failed data disks than to have to go in to promptly replace any system disk HDD that fails, in order to not have 60 idle disks.
Sure, that would make sense if they only had personnel in the datacenter once a month, but: "we replace about 10 drives every day" [1].
[1] https://www.backblaze.com/blog/vault-cloud-storage-architect...
Re: Backblaze Storage Pod 5.0
#118Worth pointing out (for those of you not in the storage game) that these appliances, while cool, are pretty far from state of the art. We're (Dropbox) packing up to an order of magnitude more storage into 4U. And costs aren't close, let alone the addressable throughput given that you guys just went to 10G. Serious kudos though are due for the Backblaze blog. You're all at least talking about what you're up to, which…
That statement carries 0 value without any proof. Backblaze is doing an excellent job documenting their process and shouting 'we are doing 10 times better', 'costs aren't close' and 'far from the state of the art' is pretty lame if you're not willing to back those claims up. I'm a big fan of dropbox but this was below the belt.
Hmm, maybe I was being tone deaf?
I've repeatedly tried to state in subsequent children in this thread that it seems as though Backblaze made the right set of tradeoffs given their previous gen, and their time to recoup their investment in this, etc.
I was also (somewhat selfishly, I suppose?) trying to tap into the excitement of a group of readers who where thinking about big storage, and let them know that the possibilities were even great given greater investment.
The responses I've been getting (including yours) lead me to believe that my comment is being seen as a slight against the set of tradeoffs that Backblaze has made, and that really wasn't how I intended it.
Edit: I do disagree about it providing zero value, though. We've had a good subsequent exploratory discussion about ways to optimize storage density and (to some degree) cost. And I hope we're not at the point where everything said on the internet demands "proof"; I'm not sure a blog post would provide that, either!
Re: Backblaze Storage Pod 5.0
#119Earlier quoted context omitted.
That statement carries 0 value without any proof. Backblaze is doing an excellent job documenting their process and shouting 'we are doing 10 times better', 'costs aren't close' and 'far from the state of the art' is pretty lame if you're not willing to back those claims up. I'm a big fan of dropbox but this was below the belt.
Hmm, maybe I was being tone deaf? I've repeatedly tried to state in subsequent children in this thread that it seems as though Backblaze made the right set of tradeoffs given their previous gen, and their time to recoup their investment in this, etc. I was also (somewhat selfishly, I suppose?) trying to tap into the excitement of a group of readers who where thinking about big storage, and let them know that the poss…
The way to tap into that excitement is by making a blog post of your own highlighting what makes DB unique and special and how you do your thing, not by making disparaging comments about companies active in the same space. I'm sure your post would get just as much or even more attention as this one.
It's very easy to do what you just did and quite hard to do what Backblaze did. Essentially, if you're not willing to do what Backblaze did here the graceful thing would be to either stay quiet, simply compliment them on their achievements or, and most useful, to be specific about what could be improved.
Re: Backblaze Storage Pod 5.0
#120I hope the memory is ECC, but don't see a mention in the article. Is it? At least the CPU (and I think motherboard) support it.