This is interesting. I personally use Omnivore + backup to Obsidian for this purpose.
ArchiveBox is evolving: the future of self-hosted internet archives
141–150 of 166 posts
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#142Earlier quoted context omitted.
You could use a proxy. "Archiving is always lossy" No.
You're talking to the guy who built the best proxy recorder in the archiving industry ;) ikreymer created https://pywb.readthedocs.io/en/latest/ I think he has more context than any of us on the limits of proxy archiving vs browser based archiving. But also if you really need perfect packet-level replication, just wireshark it as he said. Why bother with WARCs at all?
pywb has WARC issues too, due to use of warcio:
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#143Earlier quoted context omitted.
Thanks, I'll have a look. My use for this is very different--I want to be able to use a specific subset of my archived pages (which is mostly reference documentation) to "chat" with, providing different LLM prompts depending on subset and fetching plaintext chunks as reference info for the LLM to summarize (and point me back to the archived pages if I need more info).
Ok that makes sense, I think archivebox works as the first step in a pipeline there, with some other tool doing the LLM analysis and query stuff.
Yep. That's what I've built for myself, I just can't really get at the data inside ArchiveBox until I upgrade.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#144I've been using an instance of https://readeck.org/ for personal archives of web pages and I really like it, but I might try out ArchiveBox at some point too. I also run an instance of ArchiveTeam Warrior which is constantly uploading things to archive.org, and I like the direction ArchiveBox is heading with the distributed/federated archiving on the roadmap, so I may end up setting up an instance like that even if I…
I've just tried Readeck and it doesn't save a good quality copy of the pages using the Firefox extension. SingleFile and WebScrapBook do a much better job.
I prefer WebScrapBook because it saves all the assets as files under the original names in a directory rather than a zip file. This means that I can use other tools such as find, grep, and file managers like Nemo to search the archive without needing to rely on the application that saved the page.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#145I've been using an instance of https://readeck.org/ for personal archives of web pages and I really like it, but I might try out ArchiveBox at some point too. I also run an instance of ArchiveTeam Warrior which is constantly uploading things to archive.org, and I like the direction ArchiveBox is heading with the distributed/federated archiving on the roadmap, so I may end up setting up an instance like that even if I…
I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution. https://addons.mozilla.org/en-US/firefox/addon/single-file/
Readeck saves a page as a zip file. It's not hard to open from the command line or file manager, just unzip and launch the index.html in the web browser.
But it strips out a lot of detail. Zipping it also means that it's hard to deduplicate. I use WebScrapBook and run rdfind to hardlink all the identical files.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#146Earlier quoted context omitted.
Even when I‘m logged out I expect at least information on my geographical location to seep into the archive via URLs addressing specific CDN endpoints or similar mechanisms.
Yup, this is why the ArchiveBox browser extension sends URLs to a separate server for archiving with an isolated burner profile. I should write a full article on the security implications at some point, there aren't many good top-down explanations of why this is a hard problem.
How does it save pages that are only available when you are logged in such as social networking pages?
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#147Earlier quoted context omitted.
You mean ArchiveBox still doesn’t deal with cookie popups? If so, it’s quasi not useful for EU based web sites.
It does, you just have to set up a chrome profile that has an extension to hide cookie popups, or use a profile where you've already accepted/closed them and have a session. You can archive with any chrome profile with arbitrary extensions enabled, so you can use uBlock, I still Don't care about cookies, Ghostery, etc.
How do you set this up? I found this relevant issue[1] but it doesn't explain how to get it working.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#148So, after reading through the comments and website, I just realized I used ArchiveBox a month or two ago for a very specific purpose. You see, I inherited a boat. This boat belonged to my father. He was not materialistic but he took very good care of the things he cared about, and he cared about this boat. It's an old 18' aluminum fishing/cruising boat built in the early 1960's. It's not particularly valuable as a co…
>because 10 or 20 years ago, there were quite a few active web forums containing informational/tutorial threads from the proud owners of these old boats. ... But the forums are gone, so a large chunk of knowledge on these boats is too, probably forever. These days, that kind of info would be locked up in a closed Discord chat somewhere, so you can forget about people 20 years from now ever seeing it.
Lots of private groups on facebook too
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#149Have you (and I wonder the same about archive.org) considered making a Merkle tree of the data that gets archived? Since data (including photos and videos) are getting easier to fake, it may be nice to have a provable record that at least a certain version of the data existed at a certain time. It would be most useful in case of some sort of oppressive regime down the line that wants to edit history. You'd want to pu…
Yup, already doing that in the betas. Thats what I'm referring to as the beginnings of a "content addressable store" in the article. In the closed source fork we currently store a merkle tree summary of each dir in a dotfile containing the sha256 and blake3 hash of all entries / subdirs. When a result is "sealed" the summary is generated, and the final salted hash can be submitted to Solana or ETH or some other netwo…
You might be interested in taking a look at SWHID (Software Hash IDentifiers), which defines a way (on its way to become an ISO standard) to reference files and directories with content-based identifiers, like swh:1:dir:d198bc9d7a6bcf6db04f476d29314f157507d505. Yes, it uses Merkle trees for filesystem hierarchy. https://www.swhid.org/specification/v1.1/5.Core_identifiers/
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#150I've tried to get started with ArchiveBox many times, but it was always quite buggy (Not working in Safari, a bit clunky to run,...) but I've noticed a lot of updates in the past months so I'm excited about it moving forward and giving it another shot.