131–140 of 250 posts
I really wish we just had a completely different API as a filesystem. The API surface of filesystem on every OS is a complete disaster that we are locked into via backwards compatibility.
Why is it a disaster and what would you replace it with? Is the AWS S3 style API an improvement?
Earlier quoted context omitted.
> having to duplicate all data x2 Well I didn't suggest that. I said important files only for the extra backup, and I was talking about reallocating resources not getting new ones. The simplest version is the scenario where turning on dedup means you need one less drive of space. Convert that drive to parity and you'll be better off. Split that drive from the pool and use it to backup the most important files and you…
There was an implication in your statement that volume level was the level of granularity, yeah? I’m noting that during on volume wide dedup can have the con that you can’t choose (but it looks like you can!) to manually duplicate data.
> There was an implication in your statement that volume level was the level of granularity, yeah?
There was an implication that the volume level was the level of granularity for adding parity.
But that was not the implication for "another backup of your most important files".
> I’m noting that during on volume wide dedup can have the con that you can’t choose (but it looks like you can!) to manually duplicate data.
You can't choose just by copying files around, but it's pretty easy to set copies=2 on specific directories. And I'd say that's generally a better option, because it keeps your copies up to date at all times. Just make sure snapshots are happening, and files in there will be very safe.
Manual duplication is the worst kind of duplication, so while it's good to warn people that it won't work with dedup on, actually losing the ability is not a big deal when you look at the variety of alternatives. It only tips the balance in situations where dedup is near-useless to start with.
cp --reflink=auto
I wonder why they are having so much trouble getting this working properly with smaller RAM footprints. We have been using commercial storage appliances that have been able to do this for about a decade (at least) now, even on systems with "little" RAM (compared to the amount of disk storage attached). Just store fingerprints in a database and run through that at night and fixup the block pointers...
I've also heard there are some experimental branches that makes it possible to run Hammer2 on FreeBSD. But FreeBSD also lacks RDMA support. For FreeBSD 15, Chelsio has sponsored NVMe-oF target, and initiator support. I think this is just TCP though.
OT: does anyone have a good way to dedupe iCloud Photos. Or my Dropbox photos?
Earlier quoted context omitted.
yes inline dedupe has to fit in RAM. Perhaps enterprise storage arrays have spoiled me.
This array is a bit long-in-the-tooth and only has 192GB of RAM, but a bit over 40TB of net storage, which would be a 200GB dedup table size using the back-of-the-envelope estimate of 5GB/TB. A more precise calculation on my actual data shows that today's data would allow the dedup table to fit in RAM, but if I ever want to actually use most of the 40TB of storage, I'd need more RAM. I've had a ZFS system swap dedup…