Wikipedia-IPFS: An exploration to host Wikipedia in IPFS

Post by mirimir » Sun, May 10, 2020, 9:21 AM UTC

AnthonyMouse wrote: post #35

mirimir wrote: post #34

Earlier quoted context omitted.

> These are the stories fearmongers tell because it happened to one or two people many years ago. Here's a recent one:[0] > Gibson’s arrest grew out of an ongoing probe of the “Freenet” — an online network that allows users to anonymously share images, chat on message boards and access sites, the probable cause statement says. 0) https://eu.courierpostonline.com/story/news/2020/02/09/craig...

> also had some 900 images of suspected child pornography on the hard drive, says a criminal complaint. So not an example of someone being arrested just for operating a node, then.

Post by toomuchtodo » Sun, May 10, 2020, 6:51 PM UTC

imhoguy wrote: post #41

jokoon wrote: post #36

I recently tried to use IPFS to share 20 gigabytes of data on a local network, using ipfs-desktop. Apparently it hashes the entire file, meaning it will read it entirely, so I killed it because it took way too long. Apparently there are other methods for fingerprinting large files, like reading a limited set of chunks. I asked on IRC but I did not get a thorough answer.

To share files in a local network FTP or SMB/CIFS may be better suited. IPFS is content addressable storage by design on entire files. Reading limited set of chunks poses some problem regarding data integrity and deduplication colisions. One bit flip in JPG may destroy the entire image - thankfuly with currently used hash functions the right and broken files will have different "fingerprint". One way hash functions h…

Post by brokenmachine » Mon, May 11, 2020, 1:58 AM UTC

imhoguy wrote: post #45

toomuchtodo wrote: post #44

Earlier quoted context omitted.

It’s a shame modern file systems don’t generate and store common hash algorithms as file metadata.

Some file systems like ZFS, btrfs support checksum on block level. However file level checksum is not trivial with random write file system. Imagine 1 byte is changed in 20GB file - that would require full file scan to update the checksum. Recalculating a file checksum from all block checksums could be a solution however far from any standard.

Post by AnthonyMouse » Mon, May 11, 2020, 6:37 AM UTC

mirimir wrote: post #38

AnthonyMouse wrote: post #31

Earlier quoted context omitted.

> Me, I'll run my Freenet nodes in anonymously leased VPS, and access them via Tor. Some would say that's the way to do everything. But it doesn't exactly make it easy for the average Joe. And that's kind of the point too. Police behave badly more often when there is a network with 500 people on it because they don't really understand it and nobody is paying attention to anything. But when there are only 500 people o…

Hey, I respect Freenet hugely. It's almost 20 years old, and it's been ~well maintained throughout. People do hate on Java, but I've had no problems with it, using OpenJDK in Debian. But I don't buy the argument that it's targeted just because it's too small. No matter how many used Freenet, it would get targeted because it's way too laid back about child porn. Sure there's child porn on Tor onion sites, but it's not…

Post by imhoguy » Mon, May 11, 2020, 12:03 PM UTC

brokenmachine wrote: post #46

imhoguy wrote: post #45

Earlier quoted context omitted.

Some file systems like ZFS, btrfs support checksum on block level. However file level checksum is not trivial with random write file system. Imagine 1 byte is changed in 20GB file - that would require full file scan to update the checksum. Recalculating a file checksum from all block checksums could be a solution however far from any standard.

Wouldn't a merkle tree, like torrents or dat use, avoid this? You'd only have to generate a hash of the changed chunk, then it's really low cost to update the root hash. https://datprotocol.github.io/book/ch01-02-merkle-tree.html