Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The State of ZFS on Linux

clusterhq.com

Post reply on HN
Print view View original

21–30 of 125 posts

Re: The State of ZFS on Linux

#21

Post by prakashsurya »

ownedthx wrote: post #9

At a previous job, we built a proof-of-concept Sinatra service (i.e., HTTP/RESTful service) that would, on a certain API call, clone from a specified snapshot, and also create an iscsi target to that new clone. This was on OpenIndiana initially, then some other variant of that OS as a second attempt. The client making the HTTP request was IPXE; so, every time the machine booted, you'd get yourself a flesh clone + isc…

I can't recall the exact detail from memory, but I believe #1 has to do with the fact that zfs creates/clones/snapshots/etc are done in "syncing" context. Thus, each command has to wait for a full pool sync to complete, limiting the rate at which these can be done.

This is a known problem, and likely to be fixed in the not too distant future.

Re: The State of ZFS on Linux

#22

Post by xioxox »

sp332 wrote: post #10
a2743906 wrote: post #5

I'm using ZFS right now, because I need something that cares for data integrity, but the fact that it will never be included in Linux is a very big issue for me. Every time you upgrade your kernel, you have to upgrade the separate modules as well - this is the point where bad things can happen. I will definitely be looking into Btrfs once it is more reliable. For now I'm having a bit of a problem with SSD caching and…

From what I understand, aside from certain RAID levels, btrfs is production-ready. RAID5 and RAID6 don't have recovery code finished yet, but RAID0, RAID1, "dup" which just keeps 2 copies of each chunk, and "single" mode all work fine.

I've setup btrfs on software mdraid (raid 6) as a backup system (not the only backup!). You still get the checksums and snapshotting, but not the flexibility of the btrfs raid system. It has the advantage of being easy to grow, unlike zfs, which can't be resized once created. We've encountered no problems, even though it has been running for around three years of rsyncing and snapshotting.

Re: The State of ZFS on Linux

#23

Post by prakashsurya »

ryao wrote: post #11
agapon wrote: post #4

Great blog post! Something from personal experience. OpenZFS on FreeBSD feels mostly like a port of illumos ZFS where most of the non-FreeBSD-specific changes happen in illumos and then get ported downstream. On the other hand, OpenZFS on Linux feels like a fork. There is certainly a stream of changes from illumos, but there's a rather non-trivial amount of changes to the core code that happen in ZoL.

This is because Martin Matuška of FreeBSD has been focused on upstreaming changes made in FreeBSD's ZFS port into Illumos. At present, the ZFSOnLinux project has had no one dedicated to that task and code changes mostly flow from Illumos to Linux. This is starting to change. A small change went upstream to Illumos earlier this year and more should follow in the future. That being said, there are commonalities between…

FWIW, if there's anybody interested in learning about the ZFS code base, we'd love help porting patches from ZoL into Illumos and vice versa. That's a good way to get a new developer integrated with the code and process surrounding each platform.

Re: The State of ZFS on Linux

#24

Post by ryao »

bussiere wrote: post #20

I may have read the article too fast , but what about cryptography in zol ? is there a way to crypt data on zol ? regards and thks for the article

At present, you need to either encrypt the block devices beneath ZFS via LUKS or the filesystem on top of ZFS via ecryptfs. There are some guides on how to do this for each distribution.

There is an open issue for integrating encryption into ZoL itself:

https://github.com/zfsonlinux/zfs/issues/494

This will likely be added to ZoL in the future, but no one is actively working on it at this time.

Re: The State of ZFS on Linux

#25

Post by DiabloD3 »

I've used ZoL since it was created, and zfs-fuse before that. I ran it on my workstation for a few years (managing a 4x750gb RAID-Z (= ZFS's RAID-5 impl), with ext3 on mdadm RAID 1 2x400gb root), and then swapped to BTRFS for 2x2TB BTRFS native RAID 1 (which was Oracle's ZFS competitor that seems to be largely abandoned although I see commits in the kernel changelog periodically), and now back to ZFS on a dedicated file server using 2x128GB Crucial M550 SSD + 2x2TB, setup as mdadm RAID 1 + XFS for the first 16GB of the SSDs for root[2], 256MB on each for ZIL[1], and the rest as L2ARC[3], and the 2x2TB as ZFS mirror. I honestly see no reason to use any other FS for a storage pool, and if I could reliably use ZFS as root on Debian, I wouldn't even need that XFS root in there.

All of this said, I get RAID 0'ed SSD-like performance with very high data reliability and without having to shell out the money for 2TB of SSD. And before someone says "what about bcache/flashcache/etc", ZFS had SSD caching before those existed, and ZFS imo does it better due to all the strict data reliability features.

[1]: ZFS treats multiple ZIL devs as round robin (RAID 0 speed without increased device failure taking down all your RAID 0'ed devices). You need to write multiple files concurrently to get the full RAID 0-like performance out of that because it blocks on writing consecutive inodes, allowing no more than one in flight per file at a time. ZIL is only used for O_SYNC writes, and it is concurrently writing to both ZIL and the storage pool, ie, ZIL is not a write-through cache but a true journal.

The failure of a ZIL device is only "fatal" if the machine also dies before ZFS can write to the storage pool, and the mode of the failure cannot leave the filesystem in an inconsistent state. ZFS does not currently support RAID for ZIL devices internally, nor is it recommended to hijack this and use mdadm to force it. It only exists to make O_SYNC work at SSD speeds.

[2]: /tank and /home are on ZFS, the rest of the OS takes up about 2GB of that 16GB. I oversized it a tad, I think. If I ever rebuild the system, I'm going for 4GB.

[3]: L2ARC is a second level storage for ZFS's in memory cache, called ARC. ARC is a highly advanced caching system that is designed to increase performance by caching often used data obsessively instead of being just a blind inode cache like the OS's usual cache is, and is independent of the OS's disk cache. L2ARC is sort of like a write through cache, but is more advanced by making a persistent version of ARC that survive reboots and is much larger than system memory. L2ARC is implicitly round robin (like how I described ZIL above), and survives the loss of any L2ARC dev with zero issues (it just disables the device, no unwritten data is stored here). L2ARC does not suffer from the non-concurrent writing issue that ZIL "suffers" (by design) from.

Re: The State of ZFS on Linux

#26

Post by IgorPartola »

ryao wrote: post #19
IgorPartola wrote: post #7

Earlier quoted context omitted.

Seconded. ZFS is the only filesystem I trust with my children's baby pictures, as well as to store the git repo's for my personal projects (stuff I don't want on GitHub for a variety of reasons).

I am happy to hear that. While I certainly think ZFS is the best filesystem available for storing this kind of data, i would like to add a word of caution that ZFS is not a replacement for backups. I elaborated on this in one of the supplementary blog posts: https://clusterhq.com/blog/file-systems-data-loss-zfs/#disk-...

Absolutely. I have a regular backups strategy that also backs up to a ZFS system :).

Re: The State of ZFS on Linux

#27

Post by gnu8 »

ryao wrote: post #11
agapon wrote: post #4

Great blog post! Something from personal experience. OpenZFS on FreeBSD feels mostly like a port of illumos ZFS where most of the non-FreeBSD-specific changes happen in illumos and then get ported downstream. On the other hand, OpenZFS on Linux feels like a fork. There is certainly a stream of changes from illumos, but there's a rather non-trivial amount of changes to the core code that happen in ZoL.

This is because Martin Matuška of FreeBSD has been focused on upstreaming changes made in FreeBSD's ZFS port into Illumos. At present, the ZFSOnLinux project has had no one dedicated to that task and code changes mostly flow from Illumos to Linux. This is starting to change. A small change went upstream to Illumos earlier this year and more should follow in the future. That being said, there are commonalities between…

I'm interested in point 2, can you clarify how and why Linux in-kernel virtual memory is crippled or provide a link?

Re: The State of ZFS on Linux

#28

Post by bussiere »

ryao wrote: post #24
bussiere wrote: post #20

I may have read the article too fast , but what about cryptography in zol ? is there a way to crypt data on zol ? regards and thks for the article

At present, you need to either encrypt the block devices beneath ZFS via LUKS or the filesystem on top of ZFS via ecryptfs. There are some guides on how to do this for each distribution. There is an open issue for integrating encryption into ZoL itself: https://github.com/zfsonlinux/zfs/issues/494 This will likely be added to ZoL in the future, but no one is actively working on it at this time.

thks

Re: The State of ZFS on Linux

#29

Post by barrkel »

xioxox wrote: post #22
sp332 wrote: post #10

Earlier quoted context omitted.

From what I understand, aside from certain RAID levels, btrfs is production-ready. RAID5 and RAID6 don't have recovery code finished yet, but RAID0, RAID1, "dup" which just keeps 2 copies of each chunk, and "single" mode all work fine.

I've setup btrfs on software mdraid (raid 6) as a backup system (not the only backup!). You still get the checksums and snapshotting, but not the flexibility of the btrfs raid system. It has the advantage of being easy to grow, unlike zfs, which can't be resized once created. We've encountered no problems, even though it has been running for around three years of rsyncing and snapshotting.

zfs can grow - new vdevs can be added, and existing vdevs can have their disks replaced one at a time with larger disks.

A bigger downside of ZFS, IMO, is lack of defragmentation and similar larger scale pool management. If you ever push a ZFS pool close to its space limit, you can end up with fragmentation that never really goes away, even if you delete lots of files. The recommended solution is to recreate the pool and restore from backup, or create a new pool and stream a snapshot across with zfs send | zfs receive. Not terribly practical for most home users.

Re: The State of ZFS on Linux

#30

Post by JeremyNT »

ryao wrote: post #17
JeremyNT wrote: post #14

Earlier quoted context omitted.

I very much adore ZoL. Thank you for your efforts. Everything critical works and works very well. While I get the sense that this is probably not the focus of your own work, do you have any thoughts on the maturity of the "share" facilities when using ZoL, and as the project matures will these become more of a priority? These are "nice to have" features that are obviously of relatively low importance. You mentioned s…

You can obtain the commands that you used to massage your sharenfs and sharesmb settings from `zpool history`. Please file issues with them in the issue tracker: https://github.com/zfsonlinux/zfs/issues/new Please include information describing your distribution, the distribution release, your kernel version, the ZoL release and also the Samba version.

I guess I am curious about how this functionality is viewed more generally / philosophically and what state is considered to be in. I wonder whether it is perceived to be good enough for production use, and whether it's expected to work as smoothly as it does under Illumos.

In my case I encountered situations where valid configurations from OI weren't supported in ZoL, but I remember coming to the conclusion that these limitations were already known and addressing them simply wasn't a priority at that time. Since this was some months ago the situation very may well have changed already!

Post reply on HN