Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

31–40 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#31
post #23

Do y'all support archiving pages that are behind logins? Like using browser cookies?

Yes, but there's security concerns where you might accidentally leak your credentials / cookies if you publish your archive to the public.

https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv...

PS. I'm an archivebox user, not a dev or maintainer.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#32

Earlier quoted context omitted.

These are legitimate gripes that have plagued specific past releases, I hear your frustration. Please keep in mind this was a solo effort of a single developer, only worked on in my spare time over the last 7 years (up until very recently). The new v0.8 adds a BG queue specifically to deal with the issue of stalling when some sites fail. There was a system to do this in the past, but it was imperfect and mostly optim…

I can accept the other issues, but archivebox needs be private and secure by default. Sending everything to archive.org is bad default value and it erodes a certain level of trust in the project. Requiring "several important changes and security considerations" just makes a non-starter. The default settings should be "safe" for the default user, because as you mentioned in that post, 90% of users are never going to c…

Perhaps this data is "private" as in "personal property" and not "private" as in "confidential."

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#33

I've been using an instance of https://readeck.org/ for personal archives of web pages and I really like it, but I might try out ArchiveBox at some point too. I also run an instance of ArchiveTeam Warrior which is constantly uploading things to archive.org, and I like the direction ArchiveBox is heading with the distributed/federated archiving on the roadmap, so I may end up setting up an instance like that even if I…

I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution.

https://addons.mozilla.org/en-US/firefox/addon/single-file/

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#34

I've been using an instance of https://readeck.org/ for personal archives of web pages and I really like it, but I might try out ArchiveBox at some point too. I also run an instance of ArchiveTeam Warrior which is constantly uploading things to archive.org, and I like the direction ArchiveBox is heading with the distributed/federated archiving on the roadmap, so I may end up setting up an instance like that even if I…

I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution. https://addons.mozilla.org/en-US/firefox/addon/single-file/

I haven't looked at the on-disk format, I just use the browser interface. (It's fairly common for me to save something from my phone that I'll want to review on a computer later.)

Here's an example of an Amazon "review" I recently archived that has instructions for using a USB tester I have: https://readeck.home.nfriedly.com/@b/tCngVjkSFOrCbwb9DnY2yw

And, for comparison, here's the original: https://www.amazon.com/gp/customer-reviews/R3EF0QW6MAJ0VP

It'd be nice if I could edit out the extra junk near the top, but the important bits are all there.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#35
post #25

@nikisweeting ArchiveBox is awesome and we'd really love it to be more awesome. And sustainable! I've posted issues and PRs for showstopper issues that took months to get merged in: https://github.com/ArchiveBox/ArchiveBox/issues/991 https://github.com/ArchiveBox/ArchiveBox/pull/1026 You have the opportunity for the community to lean in on ArchiveBox. I understand it's hard to do everything as a solo dev, we've seen…

Let chat more. I'm almost ready to raise some seed money, hire a second staff dev or find a cofounder, and I'm looking for people that care deeply about the space.

It's only been during the last few months that I decided to go all in on the project, so this is still just the first few pages of a new chapter in the project's history.

(I should also mention that if you're a commercial entity relying on ArchiveBox, you can hire us for dedicated support and uptime guarantees. We have a closed source fork that has a much better test suite and lots of other goodies)

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#36

I've been using an instance of https://readeck.org/ for personal archives of web pages and I really like it, but I might try out ArchiveBox at some point too. I also run an instance of ArchiveTeam Warrior which is constantly uploading things to archive.org, and I like the direction ArchiveBox is heading with the distributed/federated archiving on the roadmap, so I may end up setting up an instance like that even if I…

I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution. https://addons.mozilla.org/en-US/firefox/addon/single-file/

Singlefile is excellent, Gildas is a great developer. ArchiveBox has had singlefile as one of its extractors built in for years :)

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#37
post #28

Somewhat similar topic, anyone has recommendations for a self-hosted internet website change monitoring system? I’ve been running Huginn for many years and it works well; however, I have a feeling the project is on its last leg. Also, it’s based on either text scraping (XPath/CSS/HTML and rss but it struggles with newer JS-based sites.

Changedetection.io

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#38
post #31
post #23

Do y'all support archiving pages that are behind logins? Like using browser cookies?

Yes, but there's security concerns where you might accidentally leak your credentials / cookies if you publish your archive to the public. https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv... PS. I'm an archivebox user, not a dev or maintainer.

Yes this is correct, with plans to make this easier in the near future via setup wizard that guides you through creating dedicated credentials for archiving.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#39

Earlier quoted context omitted.

I can accept the other issues, but archivebox needs be private and secure by default. Sending everything to archive.org is bad default value and it erodes a certain level of trust in the project. Requiring "several important changes and security considerations" just makes a non-starter. The default settings should be "safe" for the default user, because as you mentioned in that post, 90% of users are never going to c…

Perhaps this data is "private" as in "personal property" and not "private" as in "confidential."

It's intended for both but it currently requires extra setup to do "confidential" because there are security risks.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#40
The subline mentions "Auto-login", but the article never elaborates on this. Does this mean we will be able to more easily archive non-public websites?

Also, how do you plan to ensure data authenticity across a distributed archive? For example, if I archive someone's blog, what is stopping me from inserting inflammatory posts that they never wrote, and passing them off as the real deal? Slight update: I see you're using TLS Notary! That's exactly what I would have suggested!

Post reply on HN