141–150 of 376 posts
We need archives built on decentralized storage. Don't get me wrong, I really like and support the work Internet Archive is doing, but preserving history is too important to entrust it solely to singular entities, which means singular points of failure.
Yes, I was quite shocked when I found out that all their DCs are within driving distance.
> "It's dispiriting to see that even after being made aware of the breach weeks ago, IA has still not done the due diligence of rotating many of the API keys that were exposed in their gitlab secrets," reads an email from the threat actor. This is quite embarrassing. One of the first things you do when breached at this level is to rotate your keys. I seriously hope that they make some systemic changes, it seems that…
Earlier quoted context omitted.
IA is in bad need of a leadership change. The content of the archive is immensely valuable (largely thanks to volunteers) but the decisions and priorities of the org have been far off base for years.
I support archival of films, books, and music, but those items need to be write-only until copyright expires. The purpose of the Internet Archive is to achieve a wide-reaching, comprehensive archival, not provide easy and free read access to commercial works. Website caches can be handled differently, but bulk collection of commercial works can't have this same public access treatment. It's crazy to think this wouldn…
And it doesn't.
Earlier quoted context omitted.
To make the web distributed-archive-friendly I think we need to start referencing things by hash and not by a path which some server has implied it will serve consistently but which actually shows you different data at different times for a million different reasons. If different data always gets a different reference, it's easy to know if you have enough backups of it. If the same name gets you a pile of snapshots t…
Done. It is called IPFS. The IA already supports it. https://github.com/internetarchive/dweb-archive/blob/master/...
(this doc is 5-6 years old though, and I'm not sure what may have changed since then)
In my own (toy-scale) IPFS experiments a couple years ago it has been rather usable, but also the software has been utterly insane for operators and users, and if I were IA I would only consider it if I budgeted for a from-scratch rewrite (of the stuff in use). Nearly uncontrollable and unintrospectable and high resource use for no apparent reason.
Earlier quoted context omitted.
Done. It is called IPFS. The IA already supports it. https://github.com/internetarchive/dweb-archive/blob/master/...
IPFS has shown that the protocol is fundamentally broken at the level of growth they want to achieve and it is already extremely slow as it is. It often takes several minutes to locate a single file.
What's the point of using IPFS then? Others can still spread the file elsewhere and verify it's the correct one, by using the exact same ID of the file, although on two different networks. The beauty of content-addressing I guess.
Earlier quoted context omitted.
This seems to get brought at least once in the comments for every one of these articles that pops up. The IA has tried distributing their stores, but nowhere near enough people actually put their storage where their mouths are.
And it's guaranteed not to happen if the efforts don't continue.
Earlier quoted context omitted.
Lots of Copies Keeps Stuff Safe https://www.lockss.org/ This is a brilliant system relying on a randomised consensus protocol. I wanted to do my info sec dissertation on it, but its security model is extremely well thought out. There wasn't anything I felt I could add to it.
I wish IPFS wasn't so wasteful with respect to storage. I tried pinning a 200mb PDF on IPFS and doing so ended up taking almost a gigabyte of disk space altogether. It's also relatively slow. However its implementation of global deduplication is super cool – it means that I can host 5 pages and you can host 50, and any overlap between them means we can both help one another keep them available even if we don't know a…
Was that any file in particular? I just tried it myself with a 257mb PDF (as reported by `ls -lrth`) and doesn't seem to add that much overhead:
$ du -sh ~/.ipfs
84K /home/user/.ipfs
$ ipfs add ~/Downloads/large\ PDF\ File.pdf
added QmSvbEgCuRNZpkKyQm6nA5vz5RTHW1nxb6MJdR4cZUrnDj large PDF File.pdf
256.58 MiB / 256.58 MiB [============] 100.00%
$ du -sh ~/.ipfs
264M /home/user/.ipfsEarlier quoted context omitted.
Perhaps one idea is to let people choose what they want to protect. This way people wanting to support it can have their mission.
You already can, they have torrents for everything.
Including the index itself? That would be awesome.
Earlier quoted context omitted.
IPFS has shown that the protocol is fundamentally broken at the level of growth they want to achieve and it is already extremely slow as it is. It often takes several minutes to locate a single file.
The beauty is that IA could offer their own distribution of IPFS that uses their own DHT for example, and they could allow only public read access to it. This would solve the slow part of finding a file, for IA specifically. Then the actual transfers tend to be pretty quick with IPFS. What's the point of using IPFS then? Others can still spread the file elsewhere and verify it's the correct one, by using the exact sa…