Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenZFS 2.0

github.com

Post reply on HN
Print view View original

141–150 of 151 posts

Re: OpenZFS 2.0

#141

Post by tpetry »

rcthompson wrote: post #132
tpetry wrote: post #112

Zstd compression with configurable levels is really interesting: You could write every block first with a level comparable to lz4 for very fast performance. And if a block has not been rewritten for some time you recompress them with a compression level allowing more compression and comparable decompression performance. So cold data (cold write, cold/hot read) will take less and less space over time while still havin…

My understanding is that for ZFS, things like this would require a mythical feature called "block pointer rewrite", the same feature required to implement out-of-band deduplication.

The block could still be rewritten from the view of zfs as long as it does not update the last-written timestamp (does zfs have this?). I was just describing how it would look like from the birds eye.

Re: OpenZFS 2.0

#142

Post by yjftsjthsd-h »

scaladev wrote: post #93
yjftsjthsd-h wrote: post #92

Earlier quoted context omitted.

> It's still a big pain if you like to keep your kernel relatively up to date. Replacing a core system component with an out-of-repo version is always going to hurt, yes. > I switched to btrfs; it just working is worth the few extra warts over ZFS. I'm not sure I'd call "catastrophic failure and data loss" a "wart". In all my years of distro hopping, I've had 3 root filesystems become unbootable: 1 F2FS system early…

>and 2 on an openSUSE tubleweed system using BTRFS as root. How long ago was that? and have you been using other fully checksummed filesystems (like ZFS) on that hardware since then? I'm asking because if you're using btrfs without any raid features (or with simple RAID modes like 1/0) for the past several years and it breaks, if you dig deep enough into the problem, often the hardware is found to be at fault. And ex…

Letsee, this root filesystem says it was installed on 2019-05-11, so what, a year and a half ago? ish? I just wiped it and reinstalled since only the root filesystem was hosed (separate home filesystem thankfully wasn't affected) and this box was already fully managed by ansible so I just rebuilt an exact replica of the same system in place. (In hindsight, no, I don't know why I didn't use that opportunity to switch to XFS.)

Also, I'm going to somewhat mirror sibling comments: Even if the hardware is faulty, that should produce a filesystem with explicit checksum errors, not an unreadable filesystem. There is certainly an upper limit to what it could catch, but you'll have to forgive my skepticism that only one of the 2 filesystems on the system was affected and only after months of use, and then the corruption was so complete that it couldn't even tell me what was wrong and try to fix it.

Re: OpenZFS 2.0

#143

Post by justinclift »

jclulow wrote: post #88
justinclift wrote: post #86

Slightly off topic, but it seems like GitHub can't/won't display the user profile page for one of the OpenZFS developers: https://github.com/behlendorf For me, that gives a unicorn 100% of the time (tried across several minutes), instead of showing the developer profile. Anyone else seeing that?

It does, indeed, report that "This page is taking too long to load."!

Yeah, it's still unicorning for me, about a day later. :(

Re: OpenZFS 2.0

#144

Post by justinclift »

rincebrain wrote: post #101
justinclift wrote: post #86

Slightly off topic, but it seems like GitHub can't/won't display the user profile page for one of the OpenZFS developers: https://github.com/behlendorf For me, that gives a unicorn 100% of the time (tried across several minutes), instead of showing the developer profile. Anyone else seeing that?

Loaded in under 5 seconds flat for me, perhaps it's something strange with whatever edge server you're hitting?

Could be, but if so it's persistent. It's about a day later now, and the page still won't load.

Re: OpenZFS 2.0

#145

Post by johnisgood »

linsomniac wrote: post #51
anderspitman wrote: post #8

I'd love to get rid of my FreeNAS VM and run ZFS directly on my Linux desktop, but having to mess with the kernel has kept me from attempting it so far. Maybe I'm worrying about nothing. btrfs seems like the main alternative if you want native kernel support, but when I checked a couple years ago there seemed to be a lot of concerns about the stability. Is that still the case?

I've been running encrypted ZFS on 20.04 on my main workstation since it came out and it's worked great. Wrote up details here, it's a slight hack for encryption, no hack if you don't want crypto. https://linsomniac.gitlab.io/post/2020-04-09-ubuntu-2004-enc... A friend did a video based on my blog: https://www.youtube.com/watch?v=PILrUcXYwmc

Does it offer plausible deniability like dm-crypt?

Re: OpenZFS 2.0

#146

Post by takeda »

aseipp wrote: post #136
mook wrote: post #106

Earlier quoted context omitted.

Interesting; when I tried it a while back it seemed like you needed to know the language to manage your configurations. Was that impression incorrect? I may have tried it far enough back that I pretty much immediately encountered packages I wished it had and tried (and failed) to package it myself, though, and got the experience mixed up…

No, you're 100% correct. Despite the prosthelytization you'll get from many died-in-the-wool users who forgot what it was like in the beginning, in order to use NixOS effectively you either need to learn Nix, or be willing to spend a lot of time on IRC asking questions, which will end up with... you learning Nix. That's the reality. I think I spent something like 3 weeks porting my server configuration from Ubuntu to…

I don't think what you said necessarily contradicts what I said.

NixOS is easy to use as long as as what you're trying to do is contained within the configuration, then set up is pretty much just editing that configuration file (which essentially is just series of dictionaries and lists). I wouldn't for example have with installing NixOS for my grandparents.

Here's example config when that's true: https://github.com/areina/nixos-config/blob/master/thinkpad-...

If you want to do something that's not covered then you'll have to learn Nix, and that part is indeed hard, because it's like configuring linux through use of saltstack/chef/puppet/ansible through a functional language (which many people don't have experience with), but as you said it pays off.

I think the hardest part is the paradigm shift where everything you do is no longer imperative but declarative. It also doesn't help that documentation is always behind what nixpkgs can do and Nix functionality would cover multiple books.

Re: OpenZFS 2.0

#147

Post by rdc12 »

tpetry wrote: post #141
rcthompson wrote: post #132

Earlier quoted context omitted.

My understanding is that for ZFS, things like this would require a mythical feature called "block pointer rewrite", the same feature required to implement out-of-band deduplication.

The block could still be rewritten from the view of zfs as long as it does not update the last-written timestamp (does zfs have this?). I was just describing how it would look like from the birds eye.

Directly no, but if you moved the data to a new dataset, with a command that preserves the timestamp that would work (rsync -a or zfs send/recv), which could be run from a cronjob.

Compression settings are set at a per dataset level, so applying this to only some files in a dataset isn't practical.

Re: OpenZFS 2.0

#148

Post by justinclift »

justinclift wrote: post #144
rincebrain wrote: post #101

Earlier quoted context omitted.

Loaded in under 5 seconds flat for me, perhaps it's something strange with whatever edge server you're hitting?

Could be, but if so it's persistent. It's about a day later now, and the page still won't load.

Testing now, the page is finally loading. Page load time of ~2 days... that's different. ;)

Re: OpenZFS 2.0

#149

Post by hrez »

stubish wrote: post #102
phil21 wrote: post #84

Earlier quoted context omitted.

I wanted to say "as someone who tends to follow the unix philosophy" and realized the irony of saying that regarding ZFS... That said, I generally agree with you in that do one thing and do it well is a laudable design goal. However, I also am very excited about encrypted ZFS for one main reason: backups. Okay two. Snapshots and backups! ZFS is absolutely amazing to use as a home NAS that does daily (or more) snapsho…

https://zfs.rent/ is very interesting. I also recall someone working years ago on a way to push snapshots to S3 or similar, but I never heard if that idea got off the ground (downside is of course the snapshots need to be recovered before they can be mounted, but the dollar cost would be rock bottom). What would be more interesting is a backup application for Desktop Linux that assumes a ZFS root; all the problems th…

> I also recall someone working years ago on a way to push snapshots to S3

You can pipe zfs send to gof3r.

Re: OpenZFS 2.0

#150

Post by rleigh »

nix23 wrote: post #130
rleigh wrote: post #123

Earlier quoted context omitted.

The reason is they hired the XFS developers from SGI. And they bought Sistina for LVM. As a result, they have been wedded to both XFS and LVM for many years now, because there is likely a combination of wanting to maximise their investment into these technologies and developing the in-house expertise to support them very well, and also in having a number of staff who are deeply committed to them and don't want to cha…

>because there is likely a combination of wanting to maximise their investment into these technologies and developing the in-house expertise to support them very well, and also in having a number of staff who are deeply committed to them and don't want to change. Not sure if you would risk your customers data just because of that. I never had any problems with XFS. >At some level, they must understand that both XFS a…

Absolutely agreed, the customer's data is paramount, and I think from the perspective of supporting that with their well established in-house expertise, it makes sense.

However, XFS isn't perfect. As I wrote in a separate reply in this thread, my team in a previous position suffered catastrophic dataloss when a power cut took out some massive storage arrays. XFS does not handle power loss gracefully, and in two cases, the whole storage array was unrecoverable and required restoring from tape.

I use ZFS on FreeBSD (and Linux) too, and while it dates back to 2006 and was designed around ~2000, LVM and XFS date back at least a decade prior to that. They are a generation apart, and ZFS builds upon the knowledge of that previous generation, and its successes and its flaws.

Regarding competitive stuff, that's a mystery to me as well. My organisation went with some proprietary IBM storage array kit, but it was a real pain. Required hand compiling kernel modules against the RHEL kernel. And it still resulted in the above dataloss issues.

Post reply on HN