Earlier quoted context omitted.
Low software performance
SHA-256 seems to score well on https://www.cryptopp.com/benchmarks.html .
Depends on the context. If there is a lot of hashing, then a faster alternative like BLAKE2b is better.
Casync – A tool for distributing file system images
21–30 of 36 posts
Re: Casync – A tool for distributing file system images
#22Earlier quoted context omitted.
both buzhash and SHA-256 are kinda poor choices for a new system Why?
Low software performance
Considering that it uses xz for compression, does the performance of SHA-256 matter? (Well, using faster hash function can speed up finding duplicate blocks, which were already packed.)
I'm more interested to hear about buzhash, though.
Re: Casync – A tool for distributing file system images
#23I'm not sure I buy the embedded/IoT use case; OSTree is a really good model there and is more featureful. The "well, if your filesystem image delta happens to be in the form of a lot of very small files it's not so great for CDNs" doesn't strike me as a terribly good reason to give up everything OSTree gives you (especially with stuff like the meta-updater [1] Yocto integration). [1] https://github.com/advancedtelema…
Well, I am pretty sure IoT devices should be designed with security in mind, and that means that they need to be protected against offline modification. And that's something OSTree can't really deliver, but dm-crypt can. And casync works pretty well for delivering dm-crypt enabled disk images. I think OSTree is great — but for embedded devices that are installed in the wild, humm, uh, I don't think so? I am pretty su…
Please elaborate on 'need to be protected against offline modification'?
Re: Casync – A tool for distributing file system images
#24Great. The chunked model (inspired by Borgbackup/Tarsnap) seems preferable to Docker layering, and diff-based approaches. As far as I can tell, the advantages compared to Borgbackup seem to be: * casync offers control over which FS metadata is included * casync, the server, exposes chunks over HTTP * casync, the library, is written in C so is more easily used by systems software. I'm betting we'll see machinectl inte…
Oops. Just realised my comment contains a mistake.
casync does not act as a server. Its on-disk representation and client behaviour is designed in such a way that the server need only serve static files. This makes deployment easy.
Re: Casync – A tool for distributing file system images
#25This sounds like a centralized BitTorrent.
Re: Casync – A tool for distributing file system images
#26Earlier quoted context omitted.
Well, I am pretty sure IoT devices should be designed with security in mind, and that means that they need to be protected against offline modification. And that's something OSTree can't really deliver, but dm-crypt can. And casync works pretty well for delivering dm-crypt enabled disk images. I think OSTree is great — but for embedded devices that are installed in the wild, humm, uh, I don't think so? I am pretty su…
Please elaborate on 'need to be protected against offline modification'?
He probably means modification by those who have physical access, which means often the users, but sometimes they are not the owners.
If you have devices like cable box or water meter, the real owners do not want you to modify the device. That's where mechanisms like dm-verity step in.
Re: Casync – A tool for distributing file system images
#27I'd be really interested in creating full disk images and then cloning to another disk.
If that is a use case (I think I skimmed the post correctly) it could be very useful and performant over dd and similar disk cloning tools.
Re: Casync – A tool for distributing file system images
#28Earlier quoted context omitted.
Well, I am pretty sure IoT devices should be designed with security in mind, and that means that they need to be protected against offline modification. And that's something OSTree can't really deliver, but dm-crypt can. And casync works pretty well for delivering dm-crypt enabled disk images. I think OSTree is great — but for embedded devices that are installed in the wild, humm, uh, I don't think so? I am pretty su…
Please elaborate on 'need to be protected against offline modification'?
Think of cell towers or wind power turbines: they both are primary hacking targets in today's world, and they are placed in the wild, in uncontrolled and unprotected locations. This means more or less anybody can just walk by, temporarily cut the power source, take the harddisk out, plug it into their hacking laptop, install an OS trojan on it, place it back into the original device and restore the power. From the PoV of the cell company or the power company this was just a short power cut, and nothing changed. I reality the system was just hacked. And in order to protect yourself against that OSTree can't help you, because disk accesses aren't validated. The only validation takes place during downloading. dm-verity OTOH will protect every single access, and if deployed properly then such "offline" modifications to the OS will result in the device not booting anymore, which is much preferable over accepting that the device was hacked with no scheme to detect it.
And it's not just cell towers or wind power turbines: pretty much any device which is around people not unconditionally trusted needs to be protected against such offline modifications. In fact, if people today build cars, TVs, surveillance cameras or anything else like that and do not deploy dm-verity in some form to make sure the devices cannot be modified offline without noticing are just participating in turning IoT into Internet of Shit.
Re: Casync – A tool for distributing file system images
#29Re: Casync – A tool for distributing file system images
#30Earlier quoted context omitted.
Low software performance
Considering that it uses xz for compression, does the performance of SHA-256 matter? (Well, using faster hash function can speed up finding duplicate blocks, which were already packed.) I'm more interested to hear about buzhash, though.
I assume™ that xz won't stay the only choice. I think it's important to understand that in deduplication, you'll pass all data through your hashes one to two times. Regarding buzhash, it can break with byte granularity, and it has a dependency chain that prohibits parallelization. You'll likely never see it go faster than 700-750 MB/s on a desktop CPU (~3.8 GHz Haswell) and it won't profit from non-clock improvements of CPUs. Giving up byte-granularity allows significant improvements in performance, but I don't think anyone comprehensively analysed the impact on deduplication performance. I didn't.
(OTOH if your storage is faster than ~200-300 MB/s (buzhash and a hash, naively combined) then there is likely no issue using higher degrees of I/O concurrency, so you can work around these problems).