Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is ZFS a suitable replacement for other Linux filesystems?

clusterhq.com

Post reply on HN
Print view View original

51–60 of 86 posts

Re: Is ZFS a suitable replacement for other Linux filesystems?

#51

Post by nske »

prakashsurya wrote: post #27
ChrisInEdmonton wrote: post #22

Earlier quoted context omitted.

Ah ha! Thanks. Data deduplication. Well, yeah, that makes sense to me. Thank you very much for clearing that up.

No problem. It's really a shame to have such a high bar for using dedup. It either fits a given workload extremely well, or can be extremely detrimental. There's been talk in the developer community about ways to address the usability of dedup, but so far nothing has gone further than small prototypes.

I think for many or most use-cases, it would make more sense to have "off-line" deduplication (like, I believe, BTRFS does), as you can free-up space on-demand, when you judge it would yield you the most benefit and the system is the least busy.

I'm not sure how much benefit compared to the "real time" deduplication this approach can provide however, as the mapping table would still need to exist in memory, but I think there should be an increase to the write performance of non-duplicate data.

PS. My use case is that I have a few dozens of (linux-vserver) gentoo containers that obviously share many files and, unfortunately, trying to maintain a shared read-only mount of the core system doesn't seem to be practical/viable (as it does i.e. for FreeBSD jails, due to mostly the clear system seperation). The waste is not significant enough to really bother me, it would just be nice to avoid. The solutions that I am aware of are (not sure if I'm missing any, I'd be happy to be pointed to something else, if there is):

- integrated FS-level deduplication (ZoL, BTRFS)

- higher-level deduplication (lessfs and opendedup)

- hard-linking scripts (obviously at the file-level)

Re: Is ZFS a suitable replacement for other Linux filesystems?

#52

Post by tehmaco »

beagle3 wrote: post #44
prakashsurya wrote: post #19

Earlier quoted context omitted.

No, that's flat out not true. I've seen that metric thrown around when talking about the "dedup" feature of ZFS, but honestly, don't use dedup unless you know what you're doing. It's way to easy for things to go wrong otherwise.

How much memory does ZFS (and/or ZoL) need per 1TB of storage when dedup is off? Also, bup ("it backs things up!") efficiently dedups across an ssh connection (using bloom filters) Scales are differrent, but it might work for ZFS as well.

The recommendations I've read say you need 1GB RAM for system use (assuming a dedicated file server), and then as much RAM (ideally ECC RAM) as you want to give it for caching data.

If you're short on RAM (sub 4GB), you might need to change some of the default settings to avoid problems, but RAM's fairly cheap nowadays, so unless you've got an old machine, it's not likely to be a problem :)

Re: Is ZFS a suitable replacement for other Linux filesystems?

#53

Post by atoponce »

moe wrote: post #36
ryao wrote: post #8

Earlier quoted context omitted.

This is a myth started by someone who did not understand how filesystems work. Not having ECC memory is no more of a handicap for ZFS than it is for any other filesystem.

This is a myth started by someone who did not understand how filesystems work. You are wrong. Don't spread potentially dangerous maladvice on the internet when you have no idea what you're talking about. The official ZFS documentation[1] tells you to use ECC Ram and why. The first google hit for "zfs ecc ram"[2] further elaborates on the risks of using ZFS without ECC memory. [1] https://pthree.org/2013/12/10/zfs-adm…

The pthree.org article you linked to is my blog. It is not the official ZFS documentation nor the official ZFS on Linux documentation. It is only a storage administrator (me) explaining what I know of the inner workings of ZFS.

However, thanks for linking to my blog! :)

Re: Is ZFS a suitable replacement for other Linux filesystems?

#54

Post by ryao »

sp332 wrote: post #32
ryao wrote: post #8

Earlier quoted context omitted.

This is a myth started by someone who did not understand how filesystems work. Not having ECC memory is no more of a handicap for ZFS than it is for any other filesystem.

The trouble is, once you have a corrupted ZFS, there's no good way to recover it. That's why it's vital to make sure nothing bad happens in the first place.

The same can be said for other filesystems. The issues that fsck has been abused to automatically fix on them simply do not happen on ZFS. Failures so severe that they kill ZFS have should have analogous failure states on those filesystems too. People do not hear about such failures because they cannot be distinguished from the more typical issues that affect such filesystems.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#55

Post by nabla9 »

purplerails wrote: post #21
ryao wrote: post #2

I wrote the above article. I will try to answer any questions that I see here.

According to this article, ZFS is sub-optimal for databases due to fragmentation of the db files. Could you please comment? Thx. http://bartsjerps.wordpress.com/2013/02/26/zfs-ora-database-...

Oracle advice on how to configure ZFS and Oracle Database

http://www.oracle.com/technetwork/server-storage/solaris/con...

Some picks:

Free space is important 10-20%.

>In an environment with a high rate of data update (or churn), it is advisable to maintain a certain amount of free space. ZFS is a copy-on-write file system and relocates all writes to free disk space. Keeping a certain amount of free blocks allows ZFS to easily find space in big chunks, allows ZFS to aggregate writes and reduce the write IOPS demand on hard disks. Streaming aggregated writes can be 50-100 times less demanding on hard drives than doing small random writes. Therefore, the incentive for keeping free space is high, especially in a high churn environment, such as an active OLTP database.

>The number one rule for setting up an Oracle database on ZFS is to set ZFS recordsize equal to the database block size for the file systems that contain the Oracle data files. recordsize = db_block_size

Re: Is ZFS a suitable replacement for other Linux filesystems?

#56

Post by atoponce »

Thaxll wrote: post #42
ferrantim wrote: post #4

Earlier quoted context omitted.

Care to elaborate?

The article is missing many things that make a FS suitable for production, it's listing every features but lack the reliability / cons / waknesses ect... ZFS is nowhere near production ready for Linux. I know XFS and ext4 under heavy load with different scenarios, could you tell the same for ZoL?

Yes, I can. I am a system administrator who also knows a great deal about storage, and manages a good chuck of the storage servers at my employment. We use ZFS for our backup servers, both onsite, and offsite, and we use them for a couple generic storage servers as well, one of which is constantly under heavy stress, all the time. That server is http://mirrors.xmission.com.

I am personally using it on my workstation for a /home mount, and I have it on a highly available KVM 2-node cluster, replicated with GlusterFS using InfiniBand. While I have had networking issues with GlusterFS, which have since been ironed out, I have not had any stability, reliability, or data corruption issues with ZFS. At all. I've been running this cluster for 3 years straight, and it's the cluster that is housing my ZFS documentation at https://pthree.org/category/zfs

ZFS on Linux is absolutely "production ready". While it's true there are some ARC issues floating about, trim support is missing for SSDs, and some other things, remember that ZFS has been stable for a decade. It's just been brought into Linux kernel space with the help of the Solaris Porting Layer (spl) module.

Claiming that ZFS on Linux is not stable, and not production ready is nothing more than FUD.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#57

Post by ryao »

ryao wrote: post #8
valarauca1 wrote: post #5

Earlier quoted context omitted.

Zpool is a great feature. On top of the dozen(s) of amazing features in ZFS. The problem is, and ZFS admits this: If you don't run ECC ram Zpool can "accidentally" your whole hard drive (bit of a joke there, it can corrupt your drive as it attempts to correct bit rot that never happened). This happens a lot more often then we really care to think about. (Ram corruption) For most day-to-day linux users who are just us…

This is a myth started by someone who did not understand how filesystems work. Not having ECC memory is no more of a handicap for ZFS than it is for any other filesystem.

A person at the FreeNAS forums wrote a forum post explaining his mistaken believed that ZFS is somehow more prone to catastrophic data loss when bit flips occurred than other filesystems:

https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-...

The reality is that the worst case consequences for a bit flip is the complete loss of all data, regardless of the filesystem used. While the automated repair routined bundled in fsck utilities are often able to fix problems on other file systems, the problems that they do fix simply do not occur on ZFS. The kinds of problems that kill ZFS are not among those that an automated repair tool can fix. e.g. overwrite all ext4 superblocks with random data and then see if fsck.ext4 can fix it.

That said, I am usually able to resuscitate a pool that another person would have considered to have been killed by a bit flip. I not only find such failure modes to be incredibly rare, but I find those that I cannot fix to be a rarity among the cases where things did in fact go wrong.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#58

Post by atoponce »

dedward wrote: post #14

This should be fairly obvious... ZFS has it's own requirements... to work well it needs lots of resources (cpu/ram) and EEC memory. If you have those, then sure, it gives you many advantages. If you don't, then you are opening yourself up for struggle.

ZFS doesn't need any more "resources" than any other kernel space filesystem. Deduplication on the other hand, does need a good amount of RAM, if you want to prevent the deduplication table (DDT) from spilling to slow spinning rust. However, if you have a fast, and large SSD L2ARC, the DDT spilling over to the L2ARC might not have drastic consequences.

Regardless, unless using the DDT feature of ZFS, it doesn't need anymore resources than ext4 or XFS.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#59

Post by ryao »

beagle3 wrote: post #44
prakashsurya wrote: post #19

Earlier quoted context omitted.

No, that's flat out not true. I've seen that metric thrown around when talking about the "dedup" feature of ZFS, but honestly, don't use dedup unless you know what you're doing. It's way to easy for things to go wrong otherwise.

How much memory does ZFS (and/or ZoL) need per 1TB of storage when dedup is off? Also, bup ("it backs things up!") efficiently dedups across an ssh connection (using bloom filters) Scales are differrent, but it might work for ZFS as well.

I answered this question here:

https://news.ycombinator.com/item?id=8437921

Re: Is ZFS a suitable replacement for other Linux filesystems?

#60

Post by notacoward »

otterley wrote: post #49
notacoward wrote: post #6

Earlier quoted context omitted.

Nice article. I think you understate the importance of O_DIRECT, since it's the only way many kinds of software can work around the utter lameness of Linux local filesystems' durability semantics, but other than that I think it's extremely informative and useful.

O_DIRECT isn't about durability -- a broken device that falsely reports that data was committed will still be broken when O_DIRECT is used. O_DIRECT is about bypassing the kernel's buffer cache (and, presumably, zfs's ARC) so that the software can implement its own buffering.

Bypassing the kernel's page cache (not buffer cache any more BTW) is not a complete durability story, but it's an essential part of one. Ditto for bypassing filesystems' own not-always-appropriate queuing/syncing behavior. There are other steps that are also necessary to make sure data really made it to disk, and the failure to expose the block device layer's flush behavior other than through all those other layers is part of the lameness I mentioned. A real I/O system would provide more flexible support for ordering, flushing, completion notification, etc. instead of just forcing everything into a model designed for the workloads and machine characteristics of twenty years ago.
Post reply on HN