Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

41–50 of 106 posts

Re: CDC File Transfer

#41

Post by EdSchouten »

Scaevolus wrote: post #27
EdSchouten wrote: post #23

I’ve also been doing lots of experimenting with Content Defined Chunking since last year (for https://bonanza.build/ ). One of the things I discovered is that the most commonly used algorithm FastCDC (also used by this project) can be improved significantly by looking ahead. An implementation of that can be found here: https://github.com/buildbarn/go-cdc

This lookahead is very similar to the "lazy matching" used in Lempel-Ziv compressors! https://fastcompression.blogspot.com/2010/12/parsing-level-1... Did you compare it to Buzhash? I assume gearhash is faster given the simpler per iteration structure. (also, rand/v2's seeded generators might be better for gear init than mt19937)

Yeah, GEAR hashing is simple enough that I haven't considered using anything else.

Regarding the RNG used to seed the GEAR table: I don't think it actually makes that much of a difference. You only use it once to generate 2 KB of data (256 64-bit constants). My suspicion is that using some nothing-up-my-sleeve numbers (e.g., the first 2048 binary digits of π) would work as well.

Re: CDC File Transfer

#42

Post by nolok »

_cenw wrote: post #38
nolok wrote: post #31

Earlier quoted context omitted.

For self hosted remote streaming of game look at Moonlight / Sunshine (Apollo) Stadia required special version of games, so it wouldn't be that useful

It's a shame that virtual / headless displays are such a mess on both Linux and Windows. I use a 32:9 ultrawide and stream to 16:9/16:10 devices, and even with hours of messing around with an HDMI dummy and kscreen-doctor[1] it was still an unreliable mess. Sometimes it wouldn't work when the machine was locked, and sometimes Sunshine wouldn't restore the resolution on the physical monitor (and there's no session tim…

Use Apollo (a fork of Sunshine) : https://github.com/ClassicOldSong/Apollo

> Built-in Virtual Display with HDR support that matches the resolution/framerate config of your client automatically

It includes a virtual screen driver, and it handles all the crap (it can disable your physical screen when streaming and re enable after, it can generate the virtual screen by client to match the client's needs, or do it by game, or ...)

I stream from my main pc to both my laptop and my steamdeck, and each get the screen that matches them without having to do anything more than connect to it with moonlight.

Re: CDC File Transfer

#43

Post by grodes »

laidoffamazon wrote: post #28

As I've gotten further in my career I've started to wonder - how many engineering quarters did it take to build this for their customers? How did they manage to get this on their own roadmap? This seems like a lot of code surface area for a fairly minimal optimization that would be redundant with a different development substrate (like running Windows on Stadia like how Amazon Luna worked...)

You are thinking like a manager, but this (as with most of the good things in life) has been built by doers, artisans, and engineers (developers).

This is a problem interesting enough, with huge potential benefits for humanity if it manages to improve anything, which it did.

Re: CDC File Transfer

#44

Post by _cenw »

nolok wrote: post #42
_cenw wrote: post #38

Earlier quoted context omitted.

It's a shame that virtual / headless displays are such a mess on both Linux and Windows. I use a 32:9 ultrawide and stream to 16:9/16:10 devices, and even with hours of messing around with an HDMI dummy and kscreen-doctor[1] it was still an unreliable mess. Sometimes it wouldn't work when the machine was locked, and sometimes Sunshine wouldn't restore the resolution on the physical monitor (and there's no session tim…

Use Apollo (a fork of Sunshine) : https://github.com/ClassicOldSong/Apollo > Built-in Virtual Display with HDR support that matches the resolution/framerate config of your client automatically It includes a virtual screen driver, and it handles all the crap (it can disable your physical screen when streaming and re enable after, it can generate the virtual screen by client to match the client's needs, or do it by gam…

Artemis/Apollo are mentioned in the post above - yeah they work better than the out of box experience, but you still have to configure your physical screen to be off for every virtual display. It unfortunately only runs on Windows and my machine usually doesn't. I also only have one dGPU and a Raphael iGPU (which are sensitive to memory overclocks) and I like the Linux gaming experience for the most part, so while I did have a working gaming VM, it wasn't for me (or I'd want another GPU).

Re: CDC File Transfer

#46

Post by pabs3 »

est wrote: post #32

I wonder if this could be applied to git. The git blob was hashed with a header of decimal length, and you change a slight bit of content, you have to calculate the hash from start again. Something like CDC would improve this alot.

Backup tools like restic/borg do this, I wonder if anyone has used them to replace git yet.

Re: CDC File Transfer

#47

Post by pbhjpbhj »

EdSchouten wrote: post #41
Scaevolus wrote: post #27

Earlier quoted context omitted.

This lookahead is very similar to the "lazy matching" used in Lempel-Ziv compressors! https://fastcompression.blogspot.com/2010/12/parsing-level-1... Did you compare it to Buzhash? I assume gearhash is faster given the simpler per iteration structure. (also, rand/v2's seeded generators might be better for gear init than mt19937)

Yeah, GEAR hashing is simple enough that I haven't considered using anything else. Regarding the RNG used to seed the GEAR table: I don't think it actually makes that much of a difference. You only use it once to generate 2 KB of data (256 64-bit constants). My suspicion is that using some nothing-up-my-sleeve numbers (e.g., the first 2048 binary digits of π) would work as well.

The random number generation could match the first 2048 digits of pi, so if it works with _any_ random number...

If it doesn't work with any random number, then some work better than others then intuitively you can find a (or a set of) best seed(s).

Re: CDC File Transfer

#48

Post by rincebrain »

AnonC wrote: post #30

Does anyone know if there’s work being done to integrate this into the standard rsync tool (even as an optional feature)? It seems like a very useful improvement that ought to be available widely. From this website it seems a bit disappointing that it’s not even available for Linux to Linux transfers.

You can find some thoughts on it not working for Linux to Linux, and more broad compatibility, here[1] and here[2].

[1] - https://github.com/google/cdc-file-transfer/issues/56#issuec...

[2] - https://github.com/librsync/librsync/issues/242

Re: CDC File Transfer

#49

Post by jve »

ur-whale wrote: post #22

Great initiative, especially the new sync algorithm, but giant hurdles to adoption: - only works on a weird combo of (src platform / dst platform). Why???? How hard is it to write platform-independent code to read/write bytes and send them over the wire in 2025? - uses bazel, an enormous, Java-based abomination, to build. Fingers crossed that these can be fixed, or this project is dead in the water.

Hey the repo is archived and as I read the tool was meant to solve one specific scenario. Not everything has to please the public.

The great thing is googlers could make such a tool and publish it in the first place. So you can improve it to use it in your scenario. Or become maintainer of such a tool.

Post reply on HN