Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

Post reply on HN
Print view View original

41–50 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#41

Post by nikisweeting »

pzmarzly wrote: post #22
toomuchtodo wrote: post #5

https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to ena…

Can you recommend some tools to manage mutable torrents? I.e. create them, edit them, download them and keep them downloaded up to date. BTW I recently tried using IPFS for a mutable public storage bucket and that didn't go well - downloads were very slow compared to torrents, and IPNS update propagation took ages. Perhaps torrents will do the job.

My plan is to use a separate control plane for the discovery/announcements of changes, and torrents just for the data transfer. The specifics are still being actively discussed, and it's a few releases away anyway.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#42

Post by toomuchtodo »

0cf8612b2e1e wrote: post #29
toomuchtodo wrote: post #5

https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to ena…

The Internet Archive serves a torrent file for every item they host I had no idea. I have found the IA serving speed to be pretty terrible. Are the torrents any better? Presumably the only ones seeding the files are IA themselves.

The benefit is not in seeding speed directly from IA, but the potential for distributed access and seeding of the item. Think of it as a filename of a zip file in a flat distributed filesystem, with the ability to cherrypick files that make up the item out via traditional bittorrent mechanisms. Anyone can consume each item via torrent, continue to seed, and then also access the underlying data. IA acts as the storage system of last resort (and the metadata index).

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#43

Post by 0x1ch »

nikisweeting wrote: post #21
wongarsu wrote: post #16

Does this mean it's now possible to write plugins that dismiss cookie popups, solve captchas, scroll web pages etc.?

I have a private plugin with puppeteer support for stuff like this, currently charging clients money to use it to fund the open source development. The clients are people who are already legally allowed to evade CAPTCHAS (e.g. governments, NGOs doing research, lawyers collecting evidence, etc.) Unfortunatley I cant open source the CAPTCHA solving stuff myself, because it opens me up to liability, but if someone wants…

Legally allowed to evade CAPTCHAs? LOL.

What world do we live in where evading a captcha is an illegal offense?

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#44

Post by ashildr »

nfriedly wrote: post #34
venusenvy47 wrote: post #33

Earlier quoted context omitted.

I've been using the Single File extension to save self-contained html files of pages I want to keep for posterity. I like it because any browser can open the files it creates. Is it easy to view the archive files from readeck? I haven't looked at fancier alternatives to my existing solution. https://addons.mozilla.org/en-US/firefox/addon/single-file/

I haven't looked at the on-disk format, I just use the browser interface. (It's fairly common for me to save something from my phone that I'll want to review on a computer later.) Here's an example of an Amazon "review" I recently archived that has instructions for using a USB tester I have: https://readeck.home.nfriedly.com/@b/tCngVjkSFOrCbwb9DnY2yw And, for comparison, here's the original: https://www.amazon.com/gp…

I was about to post a link to the same URL but archived using singleFile, which looks like the original at amazon. I didn‘t because I realized that I have absolutely no idea what additional information would be hidden in the file. In the worst case any component sent by Amazon and archived into the file may contain PII, even if I am “logged out“.

I‘m not saying that singleFile is bad in any way, I‘m using it a lot on multiple devices, but I‘m not sure whether sharing archives is a good idea™.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#45

Post by arminiusreturns »

404mm wrote: post #28

Somewhat similar topic, anyone has recommendations for a self-hosted internet website change monitoring system? I’ve been running Huginn for many years and it works well; however, I have a feeling the project is on its last leg. Also, it’s based on either text scraping (XPath/CSS/HTML and rss but it struggles with newer JS-based sites.

Why do you feel like Huginn is on its last leg? It's been in my list of things to play with for years now, but I never got around to it...

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#46

Post by nikisweeting »

ashildr wrote: post #44
nfriedly wrote: post #34

Earlier quoted context omitted.

I haven't looked at the on-disk format, I just use the browser interface. (It's fairly common for me to save something from my phone that I'll want to review on a computer later.) Here's an example of an Amazon "review" I recently archived that has instructions for using a USB tester I have: https://readeck.home.nfriedly.com/@b/tCngVjkSFOrCbwb9DnY2yw And, for comparison, here's the original: https://www.amazon.com/gp…

I was about to post a link to the same URL but archived using singleFile, which looks like the original at amazon. I didn‘t because I realized that I have absolutely no idea what additional information would be hidden in the file. In the worst case any component sent by Amazon and archived into the file may contain PII, even if I am “logged out“. I‘m not saying that singleFile is bad in any way, I‘m using it a lot on…

100%, this is the challenge of archiving logged in content.

It becomes un-shareable unless we use fake burner accounts for capture, or have really good sanitizing methods.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#47

Post by nikisweeting »

0x1ch wrote: post #43
nikisweeting wrote: post #21

Earlier quoted context omitted.

I have a private plugin with puppeteer support for stuff like this, currently charging clients money to use it to fund the open source development. The clients are people who are already legally allowed to evade CAPTCHAS (e.g. governments, NGOs doing research, lawyers collecting evidence, etc.) Unfortunatley I cant open source the CAPTCHA solving stuff myself, because it opens me up to liability, but if someone wants…

Legally allowed to evade CAPTCHAs? LOL. What world do we live in where evading a captcha is an illegal offense?

It doesn't matter whether or not it's actually legal, what matters is that the big platforms will sue you for trying, so you need a big bankroll to stand your ground.

At the very least they can bar you from accessing their sites as you're violating ToS that you accept upon signup.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#48

Post by nikisweeting »

Acrobatic_Road wrote: post #40

The subline mentions "Auto-login", but the article never elaborates on this. Does this mean we will be able to more easily archive non-public websites? Also, how do you plan to ensure data authenticity across a distributed archive? For example, if I archive someone's blog, what is stopping me from inserting inflammatory posts that they never wrote, and passing them off as the real deal? Slight update: I see you're us…

Auto log in is currently a service I provide for paying clients, and you can do it in the open source version manually with some extra config.

Working hard on making it more accessible in the future, and plugins should help!

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#49

Post by FiniteField »

Disappointing that a project that should ostensibly care about preserving the open, non-centralised internet takes the time to namedrop and talk about making "compromises" against preserving a well-known, medium-sized clearnet forum legally operated from a US-based LLC. Still-living independent forum sites in this day and age have unrivalled SNR of actual human-to-human communication, there should be no better candidate for archival. It's sad that a self-hosted archival tool has to apologise for any "evil" content it might be used for in the first place. Tape recorders do not require a disclaimer about people saying "hate speech" into them.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#50

Post by rodolphoarruda »

> "In an era where fear of public scrutiny is very tangible, people are afraid of archiving things for eternity. As a result, people choose not to archive at all, effectively erasing that history forever."

Really? I don't get that feeling at all. I use Evernote to archive anything I consider worth keeping. I wonder where such "fear of archiving" comes from.

Post reply on HN