Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

CDC File Transfer

github.com

Post reply on HN
Print view View original

91–100 of 106 posts

Re: CDC File Transfer

#91

Post by Scaevolus »

EdSchouten wrote: post #41
Scaevolus wrote: post #27

Earlier quoted context omitted.

This lookahead is very similar to the "lazy matching" used in Lempel-Ziv compressors! https://fastcompression.blogspot.com/2010/12/parsing-level-1... Did you compare it to Buzhash? I assume gearhash is faster given the simpler per iteration structure. (also, rand/v2's seeded generators might be better for gear init than mt19937)

Yeah, GEAR hashing is simple enough that I haven't considered using anything else. Regarding the RNG used to seed the GEAR table: I don't think it actually makes that much of a difference. You only use it once to generate 2 KB of data (256 64-bit constants). My suspicion is that using some nothing-up-my-sleeve numbers (e.g., the first 2048 binary digits of π) would work as well.

Right, just one fewer module dependency using the stdlib RNG.

Re: CDC File Transfer

#92

Post by TiredOfLife »

jakebasile wrote: post #21
sheepscreek wrote: post #20

Earlier quoted context omitted.

Probably wouldn’t have been feasible - I heard developers had to compile their games with Stadia support. Maybe it was an entirely different platform, with its own alternative to DirectX, or maybe had some kind of lightweight emulation (such as Proton) but I remember vaguely the few games I played had custom stadia key bindings (with stadia symbols). They would display like that within the game. So definitely some cu…

As I understand it, GeForce Now actually does require changes to the game to run in the standard and until recently only option of "Ready To Play". This is the supposed reason that new updates to games sometimes take time to get released on the service, since either the developers themselves or Nvidia needs to modify it to work correctly on the service. I have no idea if this is true, but it makes sense to me. They r…

GeForce Now doesn't require any changes.

Re: CDC File Transfer

#93

Post by petsfed »

NooneAtAll3 wrote: post #12
theamk wrote: post #3

This CDC is "Content Defined Chunking" - fast incremental file transfer. Use case is to copy file over slow net, but the previous version is already there, so one can save time by only sending changed parts of the file. Not to be confused with USB CDC ("communications device class"), an USB device protocol used to present serial ports and network cards. It can also be used to transfer files, the old PC-to-PC cables u…

not to be confused with Center of Disease Control

Especially in the context of recent (that is, last 10 years) removal of data from Center of Disease Control sources due to changing political winds.

Re: CDC File Transfer

#94

Post by xyzzy_plugh »

EdSchouten wrote: post #23

I’ve also been doing lots of experimenting with Content Defined Chunking since last year (for https://bonanza.build/ ). One of the things I discovered is that the most commonly used algorithm FastCDC (also used by this project) can be improved significantly by looking ahead. An implementation of that can be found here: https://github.com/buildbarn/go-cdc

> https://bonanza.build

I just wanted to let you know, this is really cool. Makes me wish I still used Bazel.

Re: CDC File Transfer

#95

Post by maxlin »

fsfod wrote: post #40
maxlin wrote: post #18

Having dabbled in trying to make a quick delta patch system like Steam's, which required me to understand delta patching methods and made small patches to big files in a 10gb+ installation in a few seconds, this is sure is quite interesting! I wonder if Steam ever decides to supercharge their content handling with some user-space filesystem stuff. With fast connections, there isn't really a reason they couldn't launc…

Someone already created that[1] using custom kernel driver and there own CDN, but they seem to of abandoned it[2], maybe because they would of attracted Valve's wrath trying to monetized it. [1] https://web.archive.org/web/20250517130138/https://venusoft.... [2] https://venusoft.net/#home

That's actually quite interesting. Not entirely what I had in mind but close! My version would have only the first boot be a bit slow, but the aspect of dynamically replacing local content there is cool.

This would be extra cool for LAN parties with good network hardware

Re: CDC File Transfer

#96

Post by maxlin »

Zekio wrote: post #71
maxlin wrote: post #18

Having dabbled in trying to make a quick delta patch system like Steam's, which required me to understand delta patching methods and made small patches to big files in a 10gb+ installation in a few seconds, this is sure is quite interesting! I wonder if Steam ever decides to supercharge their content handling with some user-space filesystem stuff. With fast connections, there isn't really a reason they couldn't launc…

steam game installs are bottlenecked by cpu speed these days due to the heavy compression, so doubt it be much faster

Well, the amount of compression isn't set in stone, obviously a system like this would run with a less compressed dataset to balance game boot time, time taken away from running the game by compression, and scale on available bandwidth.

With low bandwidth just downloading the whole thing while having enough compression to 80% saturate the local system would be optimal instead, sure.

Re: CDC File Transfer

#97

Post by mrguyorama »

rekttrader wrote: post #2

Nice to see Stadia had some long term benefit. It’s a shame they don’t make a self hosted version but if you did that it’s just piracy in today’s drm world.

I don't understand, "self hosted stadia" is just one of the myriad of services and tools that do literally that.

Steam has game streaming built in and works very well. Both Nvidia and AMD built this into their GPU drivers at one point or another (I think the AMD one was shut down?)

Those are just the solutions I accidentally have installed despite not using that functionality. You can even stream games from the steam deck!

Sony even has a system to let you stream your PS4 to your computer anywhere and play it. I think Microsoft built something similar for Xbox.

Re: CDC File Transfer

#98

Post by EdSchouten »

quotemstr wrote: post #70
EdSchouten wrote: post #23

I’ve also been doing lots of experimenting with Content Defined Chunking since last year (for https://bonanza.build/ ). One of the things I discovered is that the most commonly used algorithm FastCDC (also used by this project) can be improved significantly by looking ahead. An implementation of that can be found here: https://github.com/buildbarn/go-cdc

I wonder whether there's a role for AI here. (Please don't hurt me.) AI turns out to be useful for data compression ( https://statusneo.com/creating-lossless-compression-algorith... ) and RF modulation optimization ( https://www.arxiv.org/abs/2509.04805 ). Maybe it'd be useful to train a small model (probably of the SSM variety) to find optimal chunking boundaries.

Yeah, that's true. Having some kind of chunking algorithm that's content/file format aware could make it work even better. For example, it makes a lot of sense to chunk source files at function/scope boundaries.

In my case I need to ensure that all producers of data use exactly the same algorithm, as I need to look up build cache results based on Merkle tree hashes. That's why I'm intentionally focusing on having algorithms that are not only easy to implement, but also easy to implement consistently. I think that MaxCDC implementation that I shared strikes a good balance in that regard.

Re: CDC File Transfer

#99

Post by heavyset_go »

_cenw wrote: post #76
heavyset_go wrote: post #69

Earlier quoted context omitted.

On Linux with an AMD i/dGPU, you can set the `virtual_display` module parameter for `amdgpu`[1] and do what you want without the need for an HDMI dummy or weird software. It's also hardware accelerated. > virtual_display (charp) > Set to enable virtual display feature. This feature provides a virtual display hardware on headless boards or in virtualized environments. It will be set like xxxx:xx:xx.x,x;xxxx:xx:xx.x,x.…

Unfortunately this seems to disable physical outputs. https://bugzilla.kernel.org/show_bug.cgi?id=203339

I figure if you're using an HDMI dummy you're running headless anyway

edit: didn't realize you're the OP lol

Post reply on HN