Live data from Hacker News

832 TB – ZFS on Linux

jonkensy.com

161–165 of 165 posts

Re: 832 TB – ZFS on Linux

#161
post #121

Earlier quoted context omitted.

https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/160... Unpatched on 16.04, referenced as supported in the article

That was strictly speaking an openzfs bug that hit all the ports. There's another one, somewhat related: https://github.com/zfsonlinux/zfs/issues/6224 which (so far) seems tied to using recordsize > 128k without either of the -L or -c flags on the zfs send side, with the result that the sendstream is corrupted in such a way that the receiver cannot detect the corruption. As with the filled-holes problem, the problem…

ZoL or OpenZFS is irrelevant, to be honest. The point is it's an experimental filesystem (at least on Linux), and there's NO REASON you should have 800TB on one single server/filesystem, because it opens you up to bugs like these.

If, instead, they had sharded+replicated it across 4020TB(replication factor) systems, they'd pay a lot more in power, but they'd be able to tolerate a single FS bug unless it somehow it all of the replicas.

Re: 832 TB – ZFS on Linux

#162
post #148
post #147

>It’s hard – if not impossible – to beat the $/GB and durability that Amazon is able to provide with their object storage offering. what the actual fuck?? AWS S3 is a abominable rip off. After I rented to my own dedicated server, I am paying several times less.

You're running three geographically separated servers with 24x7 monitoring & security, automatic rebuilds, and active bit-rot scrubbing? If not, you're doing a lot less than S3. It's possible to beat S3 pricing but you either need to be buying a lot of storage or cutting corners to do it. The most common mistake I've seen when people make those comparisons is excluding staff time, followed by presenting a system with…

just buy/rent a dedicated box and put RAID1, that's all most stratups need. Your points would be valid if S3 would be allowing to disable `geographically separated servers`, `bit-rot scrubbing` (no idea wtf is this). But those who say S3 is a cheap solution for more than few GB are fools or shills

Re: 832 TB – ZFS on Linux

#163
post #162
post #148

Earlier quoted context omitted.

You're running three geographically separated servers with 24x7 monitoring & security, automatic rebuilds, and active bit-rot scrubbing? If not, you're doing a lot less than S3. It's possible to beat S3 pricing but you either need to be buying a lot of storage or cutting corners to do it. The most common mistake I've seen when people make those comparisons is excluding staff time, followed by presenting a system with…

just buy/rent a dedicated box and put RAID1, that's all most stratups need. Your points would be valid if S3 would be allowing to disable `geographically separated servers`, `bit-rot scrubbing` (no idea wtf is this). But those who say S3 is a cheap solution for more than few GB are fools or shills

So … that box is run by a volunteer sysadmin who doesn't charge you? … and doesn't mind getting up at 3am to replace a drive?

That server has perfectly reliable power and environmental setup so you never have prolonged downtime or a double disk failure?

You're okay losing everything if someone makes a mistake running that server since backups cost too much?

You have higher-level software which tells you when data on that RAID array is corrupted? Your free sysadmin periodically runs an audit to make sure that the data stored on disk is what you originally stored? That's what I was referring to with scrubbing: even with RAID corruption happens and most storage admins have stories about the time they found out it'd happened after the only good disk failed, been written to tape, etc. The best solution is to actively scan every copy and verify it against the stored hashes for what you originally stored, which also protects against cases where a bug or human error meant that e.g. your RAID array faithfully protected a truncated file because the original write failed and nobody noticed in time. S3 provides a strong guarantee that you will get back the original data you stored or an error but never a corrupt copy and that you can prevent storing a partial or corrupted upload. If you roll your own, you need to provide those same protections for the full stack or accept a higher level of risk and perhaps mitigate it in other ways (e.g. Git-style distributed full copies with integrity checks).

Again, I'm not saying that it's impossible to pay less than S3 but your response is a bingo card for the corners people cut until something breaks and they learn the hard way why raw storage costs less than a supported storage service. Doing this for real adds support cost for the OS, your software, security, monitoring, backups & other DR planning, etc. If you use S3, Google, etc. you get all of that built into a price which is known in advance, which is a significant draw for anyone who wants to spend their mental capacity on other issues.

Many places don't have enough storage demand for that overhead to pay off in less than years and startups in particular should be extremely careful about spending their limited staff time on commodity functions rather than something which furthers their actual business. If you're Dropbox, sure, invest in a capable storage team because that's a core function but if your business is different it's time to look long and close at whether it makes any sense to devote staff time to saving a few grand a year.

Re: 832 TB – ZFS on Linux

#164
post #161

Earlier quoted context omitted.

That was strictly speaking an openzfs bug that hit all the ports. There's another one, somewhat related: https://github.com/zfsonlinux/zfs/issues/6224 which (so far) seems tied to using recordsize > 128k without either of the -L or -c flags on the zfs send side, with the result that the sendstream is corrupted in such a way that the receiver cannot detect the corruption. As with the filled-holes problem, the problem…

ZoL or OpenZFS is irrelevant, to be honest. The point is it's an experimental filesystem (at least on Linux), and there's NO REASON you should have 800TB on one single server/filesystem, because it opens you up to bugs like these. If, instead, they had sharded+replicated it across 40 20TB (replication factor) systems, they'd pay a lot more in power, but they'd be able to tolerate a single FS bug unless it somehow it…

Oh, Sean and Jeff. But for back ups, OK, right?

Re: 832 TB – ZFS on Linux

#165
post #131
post #78

Earlier quoted context omitted.

I think your argument lies more in the fact that linux is non standard to Unix. They have gone their own way and made it difficult to transfer knowledge. Ask anyone who has ported a Linux application to any other Unix. It's at best a PITA. At worst a nightmare. Can you do it? Sure. But it isn't pleasant. The Linux community is off in the weeds imo. Doing their own poor re-implementations of tech others have already d…

That's not really a fair statement. First of all, Linux started as Linus just reimplementing the Unix semantics, so edge cases and subtle semantics should be expected to be different. Secondly, most Unix-based OSes are barely compatible in their facilities. Sure, you have DTrace (which only macOS, illumos and FreeBSD have) and ZFS (which only illumos and FreeBSD have) but the rest is similarly incompatible. Solairs/i…

event ports is not an NIH reimplementation. It was a framework intentionally developed to meet specific business and technical needs within the Solaris threading model at that time.

It's far closer to Windows IOCP than FreeBSD kqueues IMHO.

It's also been one of the most successful features added to Solaris and is used throughout the system.

Post reply on HN