Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is ZFS a suitable replacement for other Linux filesystems?

clusterhq.com

Post reply on HN
Print view View original

31–40 of 86 posts

Re: Is ZFS a suitable replacement for other Linux filesystems?

#31

Post by prakashsurya »

FooBarWidget wrote: post #25
Qantourisc wrote: post #12

Earlier quoted context omitted.

Sure, COW(copy on write), snapshots, error checksum on files, compression, support for SSD's. You just don't have redundancy.

I thought the error checking (scrubbing) only works if you have multiple drives set up in a RAID.

Error detection can always be used, but error correction may or may not be available (it depends on the type of block). Metadata blocks are redundant even on a single drive pool; so if you just have a partial failure (e.g. overwrite a metadata block) it might be able to correct the block using another redundant copy on the same drive. Data blocks will require a redundant pool configuration, though, as these are not store redundantly by default (e.g. multiple drives in a raidz or mirror).

Re: Is ZFS a suitable replacement for other Linux filesystems?

#32

Post by sp332 »

ryao wrote: post #8
valarauca1 wrote: post #5

Earlier quoted context omitted.

Zpool is a great feature. On top of the dozen(s) of amazing features in ZFS. The problem is, and ZFS admits this: If you don't run ECC ram Zpool can "accidentally" your whole hard drive (bit of a joke there, it can corrupt your drive as it attempts to correct bit rot that never happened). This happens a lot more often then we really care to think about. (Ram corruption) For most day-to-day linux users who are just us…

This is a myth started by someone who did not understand how filesystems work. Not having ECC memory is no more of a handicap for ZFS than it is for any other filesystem.

The trouble is, once you have a corrupted ZFS, there's no good way to recover it. That's why it's vital to make sure nothing bad happens in the first place.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#33

Post by ryao »

ChrisInEdmonton wrote: post #13

I have heard that ZFS (at least, ZFS on Linux) requires about 1 GB of memory for every 1 TB of storage. Is this an accurate statement? That's certainly a critical flaw for many use cases, though completely irrelevant for many others. If accurate, what's behind this requirement?

On ZFS, all file data is stored using B-trees, where the leaves store X bytes, where X is the value of recordsize at file creation. When a write is done to a file in a dataset with dedup=on, a lookup is done on the data deduplication table. If the lookup finds an entry, it increments a reference. If it does not find an entry, it creates one. This involves 3 random seeks and consequentially your total write throughput is [average record size * IOPS / 3 random seeks]. If you have sufficient memory that the entire DDT can stay cached, then we avoid this limit.

The 1GB of RAM for every 1TB of storage is a rule of thumb for avoiding this limit that someone made a very long time ago. Unfortunately, that rule is wrong because it is impossible to estimate memory requirements by such a simple rule, but it has stuck with us for years despite being wrong.

The amount of memory needed to store a DDT is a function of the average record size. If your dataset has the default recorsize=128K (e.g. you are storing many >1MB files), then you can multiply your system memory by 153.6 to determine the total amount of unique data that you can store before hitting the limit I described. If you are storing many small The total amount of data that you can store on a pool before hitting the limit is [unique data * deduplication multipler], where unique data is what I described how to calculate and the deduplication multipler is a number that is either 1 or greater. A measure of the deduplication multipler is provided by `zpool list` as DEDUP, so you should be able to see that yourself. If your pool has data that was written without dedup=on, then any duplicates in that data will be counted as unique data for the purposes of that calculation. To provide a simple example of the deduplication multipler, imagine a pool with only two files that store the same data. The deduplication multipler for that pool would be 2, provided that both were written with dedup=on set on their dataset. If one or both were written with dedup=off, then the deduplication multiper would be 1. You can use zdb to calculate the theoretical deduplication statistics for an entire pool by running `zdb -D $POOLNAME`. Note that this will require significant memory because it constructs a full DDT in userland memory.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#34

Post by X-Istence »

FooBarWidget wrote: post #25
Qantourisc wrote: post #12

Earlier quoted context omitted.

Sure, COW(copy on write), snapshots, error checksum on files, compression, support for SSD's. You just don't have redundancy.

I thought the error checking (scrubbing) only works if you have multiple drives set up in a RAID.

It only works in the sense that if it finds an error it could recover when it is a multi-drive RAID, but a scrub can verify that the data on disk is still whole and matches what it expects.

Not only that, but a scrub can have ZFS tell you exactly what files are now "damaged".

Re: Is ZFS a suitable replacement for other Linux filesystems?

#35

Post by ryao »

prakashsurya wrote: post #27
ChrisInEdmonton wrote: post #22

Earlier quoted context omitted.

Ah ha! Thanks. Data deduplication. Well, yeah, that makes sense to me. Thank you very much for clearing that up.

No problem. It's really a shame to have such a high bar for using dedup. It either fits a given workload extremely well, or can be extremely detrimental. There's been talk in the developer community about ways to address the usability of dedup, but so far nothing has gone further than small prototypes.

Setting zfs_dedup_prefetch=0 has been found to help systems using ZFS data deduplication. There is a patch in ZoL HEAD for the next release that makes zfs_dedup_prefetch=0 by default:

https://github.com/zfsonlinux/zfs/commit/0dfc732416922e1dd59...

Aside from that, you are right that there has not been much done here. Making data deduplication more performant is a difficult task and so far, no one has proposed improvements beyond trivial changes.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#36

Post by moe »

ryao wrote: post #8
valarauca1 wrote: post #5

Earlier quoted context omitted.

Zpool is a great feature. On top of the dozen(s) of amazing features in ZFS. The problem is, and ZFS admits this: If you don't run ECC ram Zpool can "accidentally" your whole hard drive (bit of a joke there, it can corrupt your drive as it attempts to correct bit rot that never happened). This happens a lot more often then we really care to think about. (Ram corruption) For most day-to-day linux users who are just us…

This is a myth started by someone who did not understand how filesystems work. Not having ECC memory is no more of a handicap for ZFS than it is for any other filesystem.

This is a myth started by someone who did not understand how filesystems work.

You are wrong. Don't spread potentially dangerous maladvice on the internet when you have no idea what you're talking about.

The official ZFS documentation[1] tells you to use ECC Ram and why. The first google hit for "zfs ecc ram"[2] further elaborates on the risks of using ZFS without ECC memory.

[1] https://pthree.org/2013/12/10/zfs-administration-appendix-c-...

[2] http://louwrentius.com/please-use-zfs-with-ecc-memory.html

Re: Is ZFS a suitable replacement for other Linux filesystems?

#37

Post by ryao »

po wrote: post #20

I found an old ZFS disk laying around the other day (a few years old.. from when I was playing with zevo community edition ZFS and I had forgotten what I had stored on it) and I was interested in trying https://openzfsonosx.org so I installed it and mounted my disk up. It worked like a charm. I was pretty impressed. I kind of screwed myself over though because I to do a bunch of file cleanup that I was hoping to perh…

This issue was fixed in Illumos a few months ago, although the fix has not made its way to O3X yet:

https://github.com/illumos/illumos-gate/commit/4bb7380495217...

The workaround is to find a file that was written that is outside of a snapshot and truncate it. `zfs diff` can be used to do that. That allows you to delete an old snapshot.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#38

Post by robmccoll »

For private use, yes. I've been using ZoL for over a year on my home server with a simple ZFS raid pool and a number of volumes. Relatively stable. Snapshots every 15 minutes or so. Other than a few briefly scary kernel upgrades - nothing actually broke, just had to rebuild modules. Anytime your system comes up without pools mounted, it's scary. If only the ZFS license were compatible. Anyway, those weren't real breakages, so no problems.

Re: Is ZFS a suitable replacement for other Linux filesystems?

#40

Post by valarauca1 »

po wrote: post #30
valarauca1 wrote: post #5

Earlier quoted context omitted.

Zpool is a great feature. On top of the dozen(s) of amazing features in ZFS. The problem is, and ZFS admits this: If you don't run ECC ram Zpool can "accidentally" your whole hard drive (bit of a joke there, it can corrupt your drive as it attempts to correct bit rot that never happened). This happens a lot more often then we really care to think about. (Ram corruption) For most day-to-day linux users who are just us…

Are you referencing this thread? https://groups.google.com/forum/#!topic/zfs-macos/qguq6LCf1Q... I've heard this but I haven't actually confirmed anywhere official that ZFS will try to scrub your data due to a parity flip in ram values. Where in the manuals do they talk about it? (I'm actually curious about if it's just a rumor or if it's acknowledged)

Referencing this

https://pthree.org/2013/12/10/zfs-administration-appendix-c-...

http://louwrentius.com/please-use-zfs-with-ecc-memory.html

Post reply on HN