Live data from Hacker News

832 TB – ZFS on Linux

jonkensy.com

151–160 of 165 posts

Re: 832 TB – ZFS on Linux

#151
post #137
post #14

Earlier quoted context omitted.

Yes. $30-$50 each. Multiply that by 60 or 90 and multiply that by (however many you put in a rack). That adds up to five figure premiums on drives we're going to burn in anyway . We know how to read an HCL - it's not rocket science.

Ok 60X50 is 3K even with extra 3K Supermicro is still a lot cheeper

When drives are never replaced, yes. In the case where the chassis lasts longer than the drives (which I'd imagine is often), extra cost in drives adds up.

Re: 832 TB – ZFS on Linux

#152
post #66
post #59

Earlier quoted context omitted.

But obviously Canonical's lawyers don't think shipping it separately is the same as shipping it in the kernel, because if they did they would not ship it separately to begin with.

Granted. But Canonical's lawyers are not canonical. There's legal disagreement, and saying "Ubuntu did it" doesn't make those problems disappear. (Sadly)

I haven't seen any legal opinion since Canonical's press release, which states that they consulted with counsel who believed that they were in compliance with both licenses.

I -have- seen a lot of argument online, opinions and otherwise that disagree with this, but not anything from a lawyer - mainly "philosophical" or opining on the "intent" of the GPL, rather than "based on clauses x, y, and z, this is impermissible".

Which is not to say, if something of an opinion exists in response to Canonical's release of ZFS, I would love to see it - a legal opinion, not armchair lawyering or opinionizing.

Re: 832 TB – ZFS on Linux

#153
We bought our initial 2 TrueNAS servers from IX Systems (SuperMicro) back in 2011, have been upgrading over the years and they have been very reliable servers.

Currently each server has 63 drives (4TB HGST NL-SAS) with 1 hot spare, configured as RAIDZ.

Right now there is 200TB of usable storage, we initially started with 29TB and have been expanding as needed when it hits about 79%, I buy 18 drives roughly every 6-8 months, 9 drives per server and expand the pool.

To say that we never had issues is lying, we did have some major issues when upgrading from versions, but this was early on, now it is a rock solid storage system.

Although there is less than 300 active users connecting to the primary server, there is a lot of very important pre & post production high dev videos.

Reboot with 63 drives is around 10 minutes or less.

Resilvering could take 24-48 hours, depending on load, depending on how much data the failed drive contained.

Performance has been great, reliability has been great, support has been great.

Sadly IX Systems can no longer provide support after the end of this year, they've extended support beyond the expected lifetime of the hardware.

Re: 832 TB – ZFS on Linux

#154

Earlier quoted context omitted.

Been running an 8 drive array for nearly 5 years now myself and never once had an issue through 3 PCs and 3 different operating systems. What was your issue?

With utmost respect: this isn't a super valuable data point. A "production deployment" with lots of users and/or workloads will see issues you will never encounter in a moderate setting. FWIW, I use FreeNAS in a similar small but diverse setting with nary a problem, but when I sat it up for a 30+ organization, issue arose that I hadn't expected (not data loss, but usability and performance issues).

Does anyone have any proof of this? Seriously I'd love to see some. Bug reports or anything?

Re: 832 TB – ZFS on Linux

#155
post #114

Earlier quoted context omitted.

Aren't Synology NAS devices also using Brtfs?

Yep, although they don't use Btrfs RAID but instead run it on top of LVM which provides the necessary RAID functionality.

Does LVM usage facilitates bitrot self-healing on btrfs part?

Re: 832 TB – ZFS on Linux

#156
post #93

There are a couple needful tweaks to this BOM for anyone wanting to follow this.. Only populate one CPU socket. Zone allocation between two NUMA nodes is kind of hard, especially since Ubuntu 16.04 zfs is pre- OpenZFS ABD where memory fragmentation is reality. I would recommend better NICs like a Chelsio T5 or T6. Aside from better drivers and a responsive vendor, you can experiment with some of the iscsi offloads or…

Why FreeBSD?

Goldilocks zone for this kind of stuff.. it's a bit simpler, most of what you need is in the base system, and a lot of people are doing the same type of builds.

And for self-support, the freebsd-scsi and freebsd-net mailing lists will get you in touch right away with the people that understand the firmware, protocols, drivers, and systems engineering. You will probably get a response from a kernel and operations aware developers from Dell Isilon, SpectraLogic, or Netflix within the day that commit the code you are using. You probably wont transcend customer support at other organizations in that timeframe.

Re: 832 TB – ZFS on Linux

#158
post #134
post #110

Earlier quoted context omitted.

I disagree with the use of the word "deprecating", and everyone has been parroting around the news as though Red Hat announced that btrfs causes machines to catch on fire. Red Hat never provided enterprise support for btrfs, it was a technical preview that didn't pan out to become fully supported as part of their distribution. It's barely a story (there are plenty of other filesystems Red Hat doesn't support), but it…

> I disagree with the use of the word "deprecating" Then you disagree (literally) with Red Hat's official statement as a matter of fact (not opinion). From Chapter 53, RHEL 7.4 release notes: "Btrfs has been deprecated ... Red Hat will not be moving Btrfs to a fully supported feature and it will be removed in a future major release of Red Hat Enterprise Linux."[0] Pretty definitive. [0] https://access.redhat.com/docu…

The context that deprecated is used is "as a technical preview" which is different than saying that "in our opinion, btrfs is a deprecated technology and people shouldn't use it". So, while Red Hat did use that word, if you don't follow it up by explaining in what way support was deprecated then you're misleading people who aren't aware of the whole situation.

Re: 832 TB – ZFS on Linux

#159
Biggest question: why?

At that scale something like Ceph would be more reasonable. Just because ZFS can handle those filesystem sizes doesn't necessarily mean that it's the best tool for the job. There's a reason why all big players like Google, Amazon and Facebook go for the horizontal scaling approach.

Re: 832 TB – ZFS on Linux

#160
post #144
post #131

Earlier quoted context omitted.

That's not really a fair statement. First of all, Linux started as Linus just reimplementing the Unix semantics, so edge cases and subtle semantics should be expected to be different. Secondly, most Unix-based OSes are barely compatible in their facilities. Sure, you have DTrace (which only macOS, illumos and FreeBSD have) and ZFS (which only illumos and FreeBSD have) but the rest is similarly incompatible. Solairs/i…

Sun's event ports are so similar to kqueue cantrill has said they should have just adopted kqueue. And there are not different views on zones and jails. They are the same thing. Sun just took the idea of jails and flushed them out further adding a separate network stack for each zone, which jails now also have. But they operate on the same principle and ideas. Jails was bare bones at inception. Jails shared their IP…

I also have watched Cantrill's talks. But I think you're intentionally ignoring my point, in two ways:

* You were discussing porting a Linux application, not a kernel feature like DTrace, ZFS, Zones, kqueue, etc. Obviously porting a kernel feature between two OSes that share a kernel history is going to be easier than porting to an entirely different kernel. It's almost tautological. Porting an application has more to do with whether the syscall/libc interfaces are compatible and if you take kqueue/eventports as an example you still need to do standard porting work. glibc provides BSD-like interfaces so it's not like you have to switch away from bzero or whatever -- that's not the hard part of porting code.

* You specifically stated that Linux is "off in the weeds", "doing their own poor re-implementations of tech others have already done". Ignoring how disrespectful that is, your response to me saying "the whole Unix family re-implements each others ideas all the time -- DTrace and ZFS are the exception and only three members of the family use them" isn't helping your original point.

Also this whole section is just a non-sequitur:

> And there are not different views on zones and jails. They are the same thing. [Long description of how they are different and were developed separately.] [Random aside about Linux containers and how they're a mess.] So jail and zones are similar.

I am aware of the similarities and differences between Jails and Zones, and I'm also very painfully familiar with Linux containers. Not sure why you're bringing them up in a discussion about porting applications between different Unix-like operating systems. Sounds like you just have an axe to grind.

Post reply on HN