Does anyone have recommendations for the hardware side of self-hosting something like that? How do I avoid bit rot?
Also I see django-ninja! Very cool.
ArchiveBox is evolving: the future of self-hosted internet archives
151–160 of 166 posts
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#152Earlier quoted context omitted.
Yup, this is why the ArchiveBox browser extension sends URLs to a separate server for archiving with an isolated burner profile. I should write a full article on the security implications at some point, there aren't many good top-down explanations of why this is a hard problem.
How does it save pages that are only available when you are logged in such as social networking pages?
You set up a chrome profile for archiving that's logged into all the sites you want to save. I recommend using burner accounts dedicated to archiving, so you'd have to add them to any private pages/groups you want to archive.
It is possible to use your main account for archiving but there are security risks (you cant share the snapshots without leaking session headers).
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#153Earlier quoted context omitted.
There's no reason to believe that the hashed and timestamped content was hosted at a particular domain, however (unless the content was signed by the author of course, then there's no Blockchain necessary). sure multiple peers could make some attestation that they saw it at that URL, but then you're back at square one of the reputation problem Internet archive as an institution with a reputation that holds up to a ju…
No, definitely not. I have no inherent reason to trust the people working at the Internet Archive over let's say close friend. For me trust is always a human to human concept, and no amount of tech or institutions will change that. The more people I hear making a claim, the more I'm likely to deem the claim(s) as true. This is even true regarding the claims that cryptographic algorithms have the properties that make…
Wow, thanks for sharing your perspective it's quite different from mine. For me reality is not democratic, number of people making a claim doesn't influence the truthiness of it.
I bring up judges because Internet archive captures have been used as evidence in court cases, the first one I pulled up [0] makes an interesting distinction on whether the archive's snapshots are merely hearsay:
The hearsay rule does not apply to the document (so far as it contains the representation) if the representation was made:
(a) by a person who had or might reasonably be supposed to have had personal knowledge of the asserted fact; or ...
[0] https://www.judgments.fedcourt.gov.au/judgments/Judgments/fc...
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#154Earlier quoted context omitted.
I can accept the other issues, but archivebox needs be private and secure by default. Sending everything to archive.org is bad default value and it erodes a certain level of trust in the project. Requiring "several important changes and security considerations" just makes a non-starter. The default settings should be "safe" for the default user, because as you mentioned in that post, 90% of users are never going to c…
> The default settings should be "safe" for the default user, I 100% agree, but because private archiving is doable but NOT 100% safe yet I cant make that mode the default. The difficult reality currently is that archiving anything non-public is not simple to make safe. Every capture will contain reflected session cookies, usernames, and PII, and other sensitive content. People don't understand that this means if the…
I don't think it's possible to remove information about yourself from a webpage before you share it. It's always possible to have crafted a website that sneaks reflected session information or the instance of archivebox's IP address into the main content. This can be a real response:
> And that was this week's newsletter! Congratulation for reading to the bottom, dear 198.51.100.1.
Even if the archivebox instance noted its own IP to do a search-and-replace like s|198\.51\.100\.1|XXX.XXX.XXX.XXX| on the snapshot it is about to create, it's possible to craft a response that obscures the presence of the information, such as by encoding the IP like this: MTk4LjUxLjEwMC4xCg==. I.e. steganography (https://en.wikipedia.org/wiki/Steganography).
Being able to anonymize archives before sharing them is something I would find interesting, but I don't think you can beat steganography, so I'm wondering what exactly you mean you plan to do.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#155Earlier quoted context omitted.
I can accept the other issues, but archivebox needs be private and secure by default. Sending everything to archive.org is bad default value and it erodes a certain level of trust in the project. Requiring "several important changes and security considerations" just makes a non-starter. The default settings should be "safe" for the default user, because as you mentioned in that post, 90% of users are never going to c…
As a custom tool built to archive stuff for archive.org, why would you expect that it can also do a completely opposite task, saving information privately? I can see why you would want such a tool, but it seems like a direct divergence from the core goal of the existing codebase.
> As a custom tool built to archive stuff for archive.org
Archivebox has no association with archive.org. Sending URLs to archive.org is just one of its features, which can also be turned off.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#156Earlier quoted context omitted.
Sorry which medium sized forum are you referring to? I love forums and want them to continue, I'm not sure where you got the idea that I dislike them as a medium. I was just pointing out that public sites in general have started to see some attrition a bit lately for a variety of reasons, and the tooling needs to keep with new mediums as they appear. I also make no apology for the content, in fact ArchiveBox is expli…
I guess he's talking about K_wi F_rms which was mentioned in one of the screenshots...
It's just a forum like any other and yet you're acting like it's, at least, the Devil 2.0.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#157Earlier quoted context omitted.
How does it save pages that are only available when you are logged in such as social networking pages?
You set up a chrome profile for archiving that's logged into all the sites you want to save. I recommend using burner accounts dedicated to archiving, so you'd have to add them to any private pages/groups you want to archive. It is possible to use your main account for archiving but there are security risks (you cant share the snapshots without leaking session headers).
That’s a very cool solution- gives the user explicit control
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#158Earlier quoted context omitted.
Ok that makes sense, I think archivebox works as the first step in a pipeline there, with some other tool doing the LLM analysis and query stuff.
Yep. That's what I've built for myself, I just can't really get at the data inside ArchiveBox until I upgrade.
How did you build it?
I can imagine an architecture where I throw everything into ArchiveBox, then run VectorDB as a plugin with Gradio or some such as the client.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#159Does this mean it's now possible to write plugins that dismiss cookie popups, solve captchas, scroll web pages etc.?
I have a private plugin with puppeteer support for stuff like this, currently charging clients money to use it to fund the open source development. The clients are people who are already legally allowed to evade CAPTCHAS (e.g. governments, NGOs doing research, lawyers collecting evidence, etc.) Unfortunatley I cant open source the CAPTCHA solving stuff myself, because it opens me up to liability, but if someone wants…
That’s a pretty savvy open-core strategy: paywall features that create risk of legal liability so they are only used when there’s a paying customer/operator to assume that risk.
Re: ArchiveBox is evolving: the future of self-hosted internet archives
#160Earlier quoted context omitted.
I guess he's talking about K_wi F_rms which was mentioned in one of the screenshots...
It's just a forum like any other and yet you're acting like it's, at least, the Devil 2.0.
what? I think you have posted in the wrong thread or something...
User A complained about a forum, user B asked what they were talking about, and I guessed as to the meaning of user A's complaint. So why am I acting like anything?