Live data from Hacker News

CDC File Transfer

github.com

221–229 of 229 posts

Re: CDC File Transfer

#221
post #92

Earlier quoted context omitted.

If you’re distributing CSAM on your blob storage, and someone lets you know, you should probably remove it. This is independent of whether you distribute chunks or the whole file.

I think for piracy/DMCA it’s enough to simply remove it. As for CSAM or more serious stuff, I don’t know if that’s enough? Does section 230 cover that? Is there a difference between being a company and an individual?

[deleted]

Re: CDC File Transfer

#222
post #201

> Born from the ashes of Stadia Love it. Those were some very expensive ashes. I hope more comes from them too.

It's like how Google Docs' concurrent editing was born from the ashes of Google Wave.

I don’t believe that Google docs concurrent editing came from wave. I thought they were separate though I might be wrong.

Re: CDC File Transfer

#223
post #86

Earlier quoted context omitted.

I pay $130 AUD a month for 500/50 over HFC. Not ideal price wise but it works well enough, but man the NBN could’ve been so much better. I could go to 1000mbps, but then I’m limited to 25mbps upload which is just terrible

With who? I'm paying $200 for 250mb from Aussie BB.

Superloop! On the NBN for the infrastructure

Re: CDC File Transfer

#224

One of the biggest problems with extremely large files is how a simple insert or delete near the front of the file causes all bytes following it to be shifted and re-written to disk. Add 3 bytes to the beginning of a 50 GB file and you are writing 50 GB to disk. I have been implementing a file system replacement project for several years. It is designed to handle hundreds of millions of files within a single containe…

Every file system I know (even FAT32) supports file fragmentation and could do this (give or take block boundaries), but I don't know if there are any OS APIs to take advantage of that to actually let applications insert or remove data in the middle of a file. I'm assuming it's not in POSIX.

Re: CDC File Transfer

#225
post #87

I skimmed through the readme, which explains the concepts quite well, but am unclear on what needs to be installed on each machine (assuming Windows as the source and Linux as the destination). There’s a mention of copying the Linux build output, cdc_rsync_server, to the Windows machine. Why is this needed? And is there something on the Linux machine that needs to be (newly) added in the PATH?

Just uncompress the binaries on the Windows machine and run cdc_rsync. The Linux component, cdc_rsync_server, is deployed automatically on first run. It is scp'ed to ~/.cache/cdc-file-transfer/bin. So nothing has to be installed on the Linux machine.

Thank you so much for responding here, OP! Really appreciate it. Perhaps you could add this explanation in the readme?

Re: CDC File Transfer

#226

Earlier quoted context omitted.

I thought the USG did a similar thing but maybe didn’t require the installed fibre actually be used /connected to the cabinets?

Yeah. In my particular case the utility was already paid to deploy fiber, but they did “fiber to the node” which runs fiber to the DSL station. It’s still copper from there to the house, and in my case (end of the line in a cul de sac) it’s degraded down to only 30Mbps max bandwidth. But somehow that counts as a fiber deployment :/

Unless the local node is completely saturated, I would expect you should be able to get much in excess of that. If you're ever up at 2 am try running a speed test then.

I'd also try to rule out everything on your side, house wiring, routers, switches. Basically try to speed test it with the line they wired directly to the outside.

And upgrade any old network hardware.

Unless you're talking all they offer is 30Mbps. Then that's an ISP problem.

Re: CDC File Transfer

#229
post #73
post #58

Earlier quoted context omitted.

Isn't the main use of bittorrent for ML and research data? Academic torrents is a wonderful resource and what every developer should be using if they need to provide their neural network weights, training data, etc. How is there any legal problem using bittorrent? It's simply much more tailored for this problem than http. It doesn't make any sense to talk about 'Legal problems' for torrent protocols.

What planet have you been living on? Bittorrent is widely used to distribute copyrighted material - movies, TV shows, games, programs, porn... I'd imagine a large majority of bittorrent traffic worldwide is pirated material, with a small portion being datasets as you describe, and other legally-shared data like actual Linux distros, etc.

I suppose there could be many things happening on the internet that we are unaware of; however, torrents are very good and specifically tailored as a protocol for scientific data and ML.

It solves the link-rot issues that occur due to moving institutions, it allows huge storage for essentially free (ever tried to store 9 TB of training data or CERN data on Dropbox?), and it scales extremely beautifully.

It's really the absolute perfect solution for reproducible research in large data studies.

Post reply on HN