Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zstandard RFC 8878

datatracker.ietf.org

Post reply on HN
Print view View original

41–45 of 45 posts

Re: Zstandard RFC 8878

#41

Post by thriftwy »

Taywee wrote: post #40
thriftwy wrote: post #14

Zstandard has very cool dictionary training feature, which allows to keep a separate dictionary and have a 50% ratio compression on very small (~100b) but repetitive data such as database records.

I've always thought it could be pretty cool to leverage that for transparent filesystem compression. For context, filesystem compression usually compresses blocks of data individually (for instance, every 64K block of a file will be individually compressed, and when you modify a file in the middle, that block needs to be recompressed entirely). This is usually good enough, and it has some pretty cool properties, like…

Compression dictionaries don't usually work that way, as far as my understanding goes they just look behind, like "take 15 bytes which were 63 bytes ago", so any data it also a dictionary.

Trained dictionary is just meaningless but very frequent bits of data which may be referenced in that fashion as if they preceded the real data.

Re: Zstandard RFC 8878

#42

Post by jeroenhd »

jhgb wrote: post #39
wolf550e wrote: post #20

Earlier quoted context omitted.

zstd is from 2015.

I was talking about the RFC. You can't just shove any random compression into a browser even if it had existed for years, or can you?

It worked for Brotli. As long as the server implements the proper behaviour (only offering it if the browser sends the corresponding Accept-Encoding header) I don't see why this would be a problem.

Sure, barely any server would ever use it, and it could introduce new bugs (or even security vulnerabilities), but there's nothing from a practical point of view that would block browser vendors from adding more compression algorithms.

It's not like Google, Microsoft and Apple are waiting for standardisation to happen. When Google at some point wants Web PCIe, it'll just appear in browsers once Google has finished it, same thing Apple did with WebGPU: as a PoC while the standard is getting finalised.

Mozilla isn't 100% standards compliant either, but it's introducing relatively few bleeding edge features. I think Mozilla is investing very much in areas other browser vendors aren't necessarily interested in, so I don't know any examples for them from the top of my head.

Re: Zstandard RFC 8878

#43

Post by ggm »

xoa wrote: post #38
ggm wrote: post #24

It's said to be a good fit for ZFS. I tend to lz4 because its baked into the older systems I use, but it may be at a point where my default should be zstd. bz2/gz still predominates for compressed objects in filestore from what I can see.

> It's said to be a good fit for ZFS. I tend to lz4 because its baked into the older systems I use, but it may be at a point where my default should be zstd. It is, and you should definitely at least give it a look. I posted a comment mentioning it the other day in the OpenZFS 2.0 thread [0], and it also came up recently on HN in a thread linked there, but there are some interesting performance graphs comparing diffe…

rpi4 8GB with the radxa USB-bridged 4 port card. 4x 2TB HDD. I couldn't afford SSD. Its Ubuntu ZFS. Not over-performant, but does the job. I run a single disk ZFS detachable for periodic snapshots. Really? I like the belt-and-braces aspect of this one. 4 disks should be reasonably safe for single disk failure and I have a snap monthly, albiet all on one platter. Not enterprise-grade safe, but good-enough.

There's enough grunt left over to be my plex headend as well as long as I avoid transcoding.

I actually wanted to run FreeBSD on it, but the rpi4 wasn't fully ported when I started. In the back of my mind, it should be "safe" to convert over because OpenZFS. Which is kind-of the point.

I avoided de-dup. It burns your CPU. But I think compression is worth it, even with a lot of mp4 and mp3 and jpg content (photo and live-TV PVR archives, aside from music)

There's a lot of FUD about ZFS on small devices, how much memory you "need" -I think something said of Solaris got conflated up into a ZFS "law" about minimum memory for the ARC which just isn't really true: it may not be performant, but it works fine on smaller memory (than 8gb) systems. I chose the 8GB pi4 because I could afford it. I would have been fine on 4GB.

Re: Zstandard RFC 8878

#45

Post by vermaden »

ggm wrote: post #43
xoa wrote: post #38

Earlier quoted context omitted.

> It's said to be a good fit for ZFS. I tend to lz4 because its baked into the older systems I use, but it may be at a point where my default should be zstd. It is, and you should definitely at least give it a look. I posted a comment mentioning it the other day in the OpenZFS 2.0 thread [0], and it also came up recently on HN in a thread linked there, but there are some interesting performance graphs comparing diffe…

rpi4 8GB with the radxa USB-bridged 4 port card. 4x 2TB HDD. I couldn't afford SSD. Its Ubuntu ZFS. Not over-performant, but does the job. I run a single disk ZFS detachable for periodic snapshots. Really? I like the belt-and-braces aspect of this one. 4 disks should be reasonably safe for single disk failure and I have a snap monthly, albiet all on one platter. Not enterprise-grade safe, but good-enough. There's eno…

I have run 2 x 2 TB ZFS mirror on 512 MB machine with FreeBSD for years along with compression enabled and also other services running like Samba/NFS/Syncthing/Nextcloud/... and it worked like a charm. The only time when this machine rebooted when I needed to apply security updates for the kernel or when there was power outage as it did not had UPS backup power.

The RAM requirement for ZFS is one of the biggest myths of ZFS.

If you have lots of RAM then great - ZFS can use it as cache with ARC.

Otherwise its just as fast as disks on which its running - like every other filesystem.

The other 'big' myth about ZFS is ECC requirement - which is of course not true.

ECC RAM is useful for ALL FILESYSTEMS - not just only for ZFS.

But from all filesystems ZFS works best of all when you do not have ECC RAM.

Regards.

Post reply on HN