> Born from the ashes of Stadia Love it. Those were some very expensive ashes. I hope more comes from them too.
It's like how Google Docs' concurrent editing was born from the ashes of Google Wave.
CDC File Transfer
201–210 of 229 posts
Re: CDC File Transfer
#202I can't find any mention of rclone [1].
It would be intersting to know how this compares to rclone copy or sync, because it uses threading, rsync doesn't.
[1]: https://rclone.org/
Re: CDC File Transfer
#203Earlier quoted context omitted.
> I sort of took it for granted that people would spend the extra twenty bucks or so to make work from home a painless experience I think you took for granted the mere availability of fiber as an option. 30Mbps DSL is the best option I have, other than Starlink. And I live in San Jose!
I am sitting in a farming village in central EU and I have 900mbps fiber, as of last year. My family lives in old South Seattle neighborhoods and has excellent fiber. I had something like 100mpbs co-axial 21 years ago in West Seattle. What is the problem with San Jose?
My opinion- Internet providers running cables through public land are a natural monopoly but aren’t really regulated as such in most of the US, where it is believed that because DSL and an 5G cell phone plan exists there is competition.
Re: CDC File Transfer
#204One of the biggest problems with extremely large files is how a simple insert or delete near the front of the file causes all bytes following it to be shifted and re-written to disk. Add 3 bytes to the beginning of a 50 GB file and you are writing 50 GB to disk. I have been implementing a file system replacement project for several years. It is designed to handle hundreds of millions of files within a single containe…
I know that BitTorrent has to allocate all the space before it downloads but would that be a good base to start with?
This is also how Restic works, changing a few bytes in a 50 GB file won't reupload 50 GB of data unlike most backup solutions that work primarily with files.
Re: CDC File Transfer
#205Earlier quoted context omitted.
The Canadian government gives grants to telecoms to install fibre. And I think there is some relation to signing up end users too but I can't recall.
I thought the USG did a similar thing but maybe didn’t require the installed fibre actually be used /connected to the cabinets?
Yeah. In my particular case the utility was already paid to deploy fiber, but they did “fiber to the node” which runs fiber to the DSL station. It’s still copper from there to the house, and in my case (end of the line in a cul de sac) it’s degraded down to only 30Mbps max bandwidth.
But somehow that counts as a fiber deployment :/
Re: CDC File Transfer
#206Earlier quoted context omitted.
I know that BitTorrent has to allocate all the space before it downloads but would that be a good base to start with?
This is also how Restic works, changing a few bytes in a 50 GB file won't reupload 50 GB of data unlike most backup solutions that work primarily with files.
This change would not be just be for backups or file transfers. It would fundamentally change how big files are stored by the file system. Much like the way fragmentation is handled solely by the file system and an application accessing the file is completely unaware if the file is stored within a single fragment or multiple fragments; the file system would manage the individual 'chunks'.
For example, a 6 GB file might be made up of 3 separate 2 GB chunks. An application might delete 20 bytes from the front of the file. This causes the first chunk to now be 2 GB - 20 bytes. The other 2 chunks are unchanged.
Current file systems do not allow this where a file can have a block somewhere in its interior that is just a partial block.
Re: CDC File Transfer
#207Earlier quoted context omitted.
EMC had a disk based deduplication storage at the time. NetAppliance had a competing product. They had patents in the area. I believed that’s in the early 2000’s. One of the household name big techs had an internal product with similar design. ZFS has similar design. Mine was at the block device level. The advantage is you can format it to whatever file system of your choice, with read/write support and deduplication…
> Mine was at the block device level. Same! :) Originally I wrote it with an interface kinda similar to `tar` -- you add or extract huge blobs to/from what I called a coalesced archive. I could re-image a machine about 8x faster than Norton Ghost. After $WORK went under, I kept the code and toyed around with it, making it speak NBD so instead of extracting a huge blob from the archive to a destination block device yo…
Yes. Depending on how the claim languages are phrased, patents on the same idea can be approved.
For reclaiming deleted blocks, I just had a garbage collection phase to run from time to time. Like you've mentioned on refcount, I've considered it but it amplified writes 2X~3X and worse they were random access writes. Garbage collection was not so bad since it's only going through the virtual file control blocks containing the content-address-hash.
The storage layout was: file block -> virtual file control block -> dedup-data-block. The virtual file control block contained the dedup block hash entries where one control block hosted N file blocks. GC only needed to scan the control blocks to find out which dedup-data-blocks were in use.
Freed dedup-data-blocks remained in place and were linked to the free list; the first couple bytes of the free block were cooped to store the pointer to the next free block.
At the end, brand new file write performance degraded about 10% compared to normal file write, which I considered acceptable. M block writes -> M dedup block writes + M/N control block writes + M/K db index updates, where N was the number of hash entries hosted in the control block and K is the number of hashes stored in one db index page. Repeated file writes were much faster due to deduplication.
Re: CDC File Transfer
#208Earlier quoted context omitted.
This is also how Restic works, changing a few bytes in a 50 GB file won't reupload 50 GB of data unlike most backup solutions that work primarily with files.
This change would not be just be for backups or file transfers. It would fundamentally change how big files are stored by the file system. Much like the way fragmentation is handled solely by the file system and an application accessing the file is completely unaware if the file is stored within a single fragment or multiple fragments; the file system would manage the individual 'chunks'. For example, a 6 GB file mig…
> Current file systems do not allow this where a file can have a block somewhere in its interior that is just a partial block.
Because it would be costly for uncertain benefit?
Re: CDC File Transfer
#209How does it compare with Syncthing?
I was just wondering that, is the approach free, could Syncthing utilize it?
Re: CDC File Transfer
#210Earlier quoted context omitted.
There are many older and more venerable file transfer protocols, like FTP or TFTP. Add encryption and then you have SCP and SFTP. SFTP is the default protocol used by the scp command, and TFTP is still often used to communicate with routers and access points and similar equipment. And I'm not sure what that has anything to do with TCP not as reliable. TCP has real problems on the modern Internet but the first part do…
And none of those protocols keeps the connection alive if you unplug your network cable and plug it back on really quickly Might I say, even if you get the same IP back from the router
That's not a problem with TCP, that's more of a problem with IP itself. Connections are currently identified by the source/destination IP and port numbers, but that should not have been the case. Imagine we use a UUID for the connection and keep an updatable cache between UUID and IP.
Still nothing to do with TCP.
Also, this kind of partial file transfer protocol is still needed because what if a host crashes? Even if we completely solve the mobile IP problem we would still rely on both ends of the connection tracking state. That state is lost due to a crash or power failure.