Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Five Years of Btrfs

markmcb.com

Post reply on HN
Print view View original

111–120 of 240 posts

Re: Five Years of Btrfs

#111

Post by tjoff »

clSTophEjUdRanu wrote: post #73
InTheArena wrote: post #31

I went on a quest a few years ago, thinking it would be good for the industry to standardize on a single next generation filesystem for UNIX. I started with ZFS on linux since that seemed to have the most vocal advocates. That lasted about a half year, until a bug in the code resulted in a completely corrupt disk, and I had to restore 4TB of data over a month from offside backups. That plus the licensing confusion ar…

I really don't understand the insane hype around ZFS. You can't read any thread that touches on filesystems without the ZFS zealots coming out.

The hype is quite easy to understand. Snapshots and checksums are two complete game-changers. ZFS has them both. And there are no real alternatives in many cases.

I've personally waited for BTRFS longer than a decade but my use-cases are yet to be considered stable (not something you really mess with in regard to filesystems).

Honestly, as sure as I have been on the success of BTRFS I now consider BTRFS dead on arrival - if it will ever even arrive. The pace of development is slower than the universe around it, that might be too harsh but really - no RAID6 yet? A decade ago the impression I got was "soon". And now 2-drive parity is becoming obsolete.

ZFS has tons of warts for home-use, I agree. So, for a home-user with high demands I don't see anything exciting in the future.

Re: Five Years of Btrfs

#112

Post by ValentineC »

izacus wrote: post #46

It's also worth noting that Synology uses btrfs as an option to do checksumming and snapshots on their NAS devices. They're still using their own RAID layer though.

> They're still using their own RAID layer though.

Synology's RAID implementation is largely mdadm + LVM.

Re: Five Years of Btrfs

#113

Post by seabrookmx »

louwrentius wrote: post #108
gravypod wrote: post #103

I've seen a lot of the hacker community focusing on btrfs and zfs but very little focusing on ceph. I think ceph has a lot of the features that we want in a file system and some things that aren't even possible on traditional file systems (per-file redundancy settings) with very little downsides. The setup is a little more complex involving a few daemons to manage disks, balance, monitor, etc. I wish there was someth…

I love Ceph, I even wrote an intro about it for those who are not familiar with it. https://louwrentius.com/understanding-ceph-open-source-scala... But Ceph is not designed to be a competitor to BTRFS or ZFS. The core vision of Ceph is scalability. If you need petabytes of storage and the performance to scale with it, take a look at Ceph. I may be totally wrong here, but from what I understand about Ceph, it's not me…

This.

In fact, it's really common to use a ZFS array on single nodes, and then create a SAN using multiple such machines by layering Ceph on top.

Re: Five Years of Btrfs

#114

Post by Teknoman117 »

jstrong wrote: post #91
iracic wrote: post #74

Earlier quoted context omitted.

It is possible to put database in state that is "ready" for snapshot, pushing changes to disk and sort of freezing I/O during snapshot.

this is generally not a matter of concern for a copy on write filesystem like zfs, since it's not possible for the file to be in an "in between" state. If a write were in progress, the filesystem would still be pointing to the previous state. Only when the data is written to disk is the pointer moved to the new location.

It is a matter of concern if said database systems leaves its filesystem contents in an inconsistent state at any point. ZFS, BTRFS, and others can only keep consistent what they have control over.

Re: Five Years of Btrfs

#115

Post by StreamBright »

KaiserPro wrote: post #82
StreamBright wrote: post #40

Earlier quoted context omitted.

>> Or, to put that another way: what are AWS and GCP using in their SANs (EBS; GCE PD) that allows them to take on-demand incremental snapshots of SAN volumes, and then ship those snapshots away from the origin node into safer out-of-cluster replicated storage (e.g. object storage)? As far as I know AWS does not use SANs because they consider it as anti-pattern. Most backups land on S3 because of reliability and pric…

SAN and s3 are different beasts. EBS is very much a SAN, if you read the docs, the Nitro HBA Controllers have dedicated bandwidth allocation for doing just EBS. As there is a dedicated network for just servicing block storage, that sounds suspiciously like a Storage Area Network to me. S3 for backup makes lots of sense, its ubiqutous, reliable and smeared over lots of regions. It also works well with large files. Its…

Sure thing. I was referring to the lack of SAN in the context of backups. Yes, EBS is a SAN in that sense.

Re: Five Years of Btrfs

#116

Post by RX14 »

starfallg wrote: post #109
InTheArena wrote: post #31

I went on a quest a few years ago, thinking it would be good for the industry to standardize on a single next generation filesystem for UNIX. I started with ZFS on linux since that seemed to have the most vocal advocates. That lasted about a half year, until a bug in the code resulted in a completely corrupt disk, and I had to restore 4TB of data over a month from offside backups. That plus the licensing confusion ar…

Btrfs is the only FS I used that resulted in complete FS corruption losing nearly all data on disk, not once, but 3 times. After that, none of the features like compression, snapshots, COW or checksums meant anything to me. I'm much happier with ext4 and xfs on lvm.

It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all).

I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. One possible cause is distro choice tends to be per person and how up to date said distro keeps it's kernel. But, I'm not sure.

Re: Five Years of Btrfs

#117

Post by Teknoman117 »

epx wrote: post #6

I have been using btrfs in my "NAS"/personal server for 3 years, changed disk configuration a couple times, I do snapshots every hour and prune them using a Fibonacci-like timeline, no problems yet.

My experience has been the same. Admittedly, I've not tried native BTRFS parity raid (I'm sitting the volume on top of mdraid). But, I ran the "mkfs.btrfs" 5 years ago at this point for my desktop and no data loss yet. I back things up religiously, so I'm not too worried about the volume failing, but it'll be nice if btrfs parity raid gets stabilized, because I could replace my current NAS storage config.

I used to use ZFS on my NAS, but after running it for a year and fiddling with it, I wasn't able to tune it in a way I liked. I always had random performance problems and zvols were super slow. It's now dm-integrity on all disks, an mdraid raid6 volume over those, with LVM2 on top of that and mirrored NVMe disks as a read and write cache.

I also wish BTRFS would add extents at some point so you could run virtual machine images from it without weird performance issues from time to time (although I imagine this is less of an issue on SSDs because they're "fragmented" inside anyways).

Re: Five Years of Btrfs

#118

Post by kiney »

I use BTRFS on several devices for years. The tooling is a bit rough, but no major problems. Just recently data checksumming saved me: In December I replace an old 2TB drive in my RAID1 (2+4+4+4) with an 8TB drive. The new drive had checksum errors after a few weeks which BTRFS handled gracefully. With "classical" RAID i might only have noticed when it's to late. (I RMAed the bad drive)

  [/dev/mapper/h4_crypt].write_io_errs    0
  [/dev/mapper/h4_crypt].read_io_errs     0
  [/dev/mapper/h4_crypt].flush_io_errs    0
  [/dev/mapper/h4_crypt].corruption_errs  0
  [/dev/mapper/h4_crypt].generation_errs  0
  [/dev/mapper/h2_crypt].write_io_errs    0
  [/dev/mapper/h2_crypt].read_io_errs     30
  [/dev/mapper/h2_crypt].flush_io_errs    0
  [/dev/mapper/h2_crypt].corruption_errs  0
  [/dev/mapper/h2_crypt].generation_errs  0
  [/dev/mapper/h1_crypt].write_io_errs    0
  [/dev/mapper/h1_crypt].read_io_errs     0
  [/dev/mapper/h1_crypt].flush_io_errs    0
  [/dev/mapper/h1_crypt].corruption_errs  0
  [/dev/mapper/h1_crypt].generation_errs  0
  [/dev/mapper/h3_crypt].write_io_errs    0
  [/dev/mapper/h3_crypt].read_io_errs     0
  [/dev/mapper/h3_crypt].flush_io_errs    0
  [/dev/mapper/h3_crypt].corruption_errs  0
  [/dev/mapper/h3_crypt].generation_errs  0
  [/dev/mapper/luks-e120f41e-9c8a-4808-876f-fa6665ee8bb8].write_io_errs    0
  [/dev/mapper/luks-e120f41e-9c8a-4808-876f-fa6665ee8bb8].read_io_errs     16
  [/dev/mapper/luks-e120f41e-9c8a-4808-876f-fa6665ee8bb8].flush_io_errs    0
  [/dev/mapper/luks-e120f41e-9c8a-4808-876f-fa6665ee8bb8].corruption_errs  20619
  [/dev/mapper/luks-e120f41e-9c8a-4808-876f-fa6665ee8bb8].generation_errs  0
edit: formatting

Re: Five Years of Btrfs

#119

Post by RX14 »

tjoff wrote: post #101
jstrong wrote: post #91

Earlier quoted context omitted.

this is generally not a matter of concern for a copy on write filesystem like zfs, since it's not possible for the file to be in an "in between" state. If a write were in progress, the filesystem would still be pointing to the previous state. Only when the data is written to disk is the pointer moved to the new location.

It very much is a concern. ZFS has no knowledge about the internals of a database, which parts of a file are related to each other etc.

DBMSes always keep their database in the file system in a consistent state to be able to recover from system crashes. Taking a file system snapshot is equivalent to pulling the power on the database server in terms of data recovery, but databases are designed to support this.

Re: Five Years of Btrfs

#120

Post by thfuran »

RX14 wrote: post #116
starfallg wrote: post #109

Earlier quoted context omitted.

Btrfs is the only FS I used that resulted in complete FS corruption losing nearly all data on disk, not once, but 3 times. After that, none of the features like compression, snapshots, COW or checksums meant anything to me. I'm much happier with ext4 and xfs on lvm.

It seems a lot of people have these stories, and then people like me and OP who have had btrfs survive the most fucked up situations (I've had a btrfs nas built on "random drives I've had lying around" and abused it for 5 years and had 0 bugs at all). I'm not sure what causes it, but there seems to be an effect where btrfs loves you or hates you and few people with mixed experiences regarding data loss. One possible…

I think the probable cause is that it's not common bugs that cause the corruption but uncommon ones. Most of the time, they work fine. But you really want a stronger guarantee than that out of your filesystem.
Post reply on HN