Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

Post reply on HN
Print view View original

51–60 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#51

Post by rcarmo »

This is nice. I'm actually much more excited about the REST API (which will let me do searches and pull information out, I hope) than the plugin ecosystem, since the last thing I need is for another tool to have a half-baked LLM integration -- I prefer to do that myself and have full control.

Being able to do RAG on my ArchiveBox is something that I have very much wanted to do for over a year now, and it might finally be within reach without my going and hacking at the archived content tree...

Edit: Just looked at the API schema at https://demo.archivebox.io/api/v1/docs.

No dedicated search endpoint? This looks like a HUGE missed opportunity. I was hoping to be able to query an FTS index on the SQLlite database... Have I missed something?

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#52

Post by ashildr »

nikisweeting wrote: post #46
ashildr wrote: post #44

Earlier quoted context omitted.

I was about to post a link to the same URL but archived using singleFile, which looks like the original at amazon. I didn‘t because I realized that I have absolutely no idea what additional information would be hidden in the file. In the worst case any component sent by Amazon and archived into the file may contain PII, even if I am “logged out“. I‘m not saying that singleFile is bad in any way, I‘m using it a lot on…

100%, this is the challenge of archiving logged in content. It becomes un-shareable unless we use fake burner accounts for capture, or have really good sanitizing methods.

Even when I‘m logged out I expect at least information on my geographical location to seep into the archive via URLs addressing specific CDN endpoints or similar mechanisms.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#53

Post by 404mm »

arminiusreturns wrote: post #45
404mm wrote: post #28

Somewhat similar topic, anyone has recommendations for a self-hosted internet website change monitoring system? I’ve been running Huginn for many years and it works well; however, I have a feeling the project is on its last leg. Also, it’s based on either text scraping (XPath/CSS/HTML and rss but it struggles with newer JS-based sites.

Why do you feel like Huginn is on its last leg? It's been in my list of things to play with for years now, but I never got around to it...

It looks like it’s being maintained by a single remaining developer. No new features are being added, just some basic maintenance. The product as a whole still works well, so unless you find something better, I do recommend it. I run it in k3s and the image is probably the easiest way of maintaining it.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#54

Post by viraptor »

nikisweeting wrote: post #20
toomuchtodo wrote: post #14

Earlier quoted context omitted.

Keep in mind, what signing methodology you use is a function of who accepts it. If I can confirm "ArchiveTeam ripped this", that is is superior to whatever tlsnotary is doing with MPC, blockchain, distributed ledger, whatever (in my use case). Have to trust someone at the end of the day. ArchiveTeam's Warrior doesn't use tlsnotary, for example, and rips entire sites just fine.

The idea with TLSNotary is that you can have several universities or central agencies running signing servers but you dont have to share the cleartext content of your archives with them to get it signed. This dramatically changes what is possible with signing because previously to get ArchiveTeam's signature of approval, they would have to see the content themselves to archive it. With TLSNotary they can sign without…

Isn't that already possible with any kind of notary by giving them a sha256 of the content only? Or am I missing some distinction?

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#55

Post by nikisweeting »

rcarmo wrote: post #51

This is nice. I'm actually much more excited about the REST API (which will let me do searches and pull information out, I hope) than the plugin ecosystem, since the last thing I need is for another tool to have a half-baked LLM integration -- I prefer to do that myself and have full control. Being able to do RAG on my ArchiveBox is something that I have very much wanted to do for over a year now, and it might finall…

The /cli/list endpoint is the search endpoint you're looking for. It provides FTS but I can make it clearer in the docs, thanks for the tip.

As for the AI stuff don't worry, none of it is touching core, it's all in an optional community plugin only for those who want it.

I'm not personally a huge AI person but I have clients who are already using it and getting massive value from it, so it's worth mentioning. (They're doing some automated QA on thousands of collected captured and feeding results into spreadsheets)

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#56

Post by nikisweeting »

ashildr wrote: post #52
nikisweeting wrote: post #46

Earlier quoted context omitted.

100%, this is the challenge of archiving logged in content. It becomes un-shareable unless we use fake burner accounts for capture, or have really good sanitizing methods.

Even when I‘m logged out I expect at least information on my geographical location to seep into the archive via URLs addressing specific CDN endpoints or similar mechanisms.

Yup, this is why the ArchiveBox browser extension sends URLs to a separate server for archiving with an isolated burner profile.

I should write a full article on the security implications at some point, there aren't many good top-down explanations of why this is a hard problem.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#57

Post by nikisweeting »

rodolphoarruda wrote: post #50

> "In an era where fear of public scrutiny is very tangible, people are afraid of archiving things for eternity. As a result, people choose not to archive at all, effectively erasing that history forever." Really? I don't get that feeling at all. I use Evernote to archive anything I consider worth keeping. I wonder where such "fear of archiving" comes from.

A lot of people are retreating off public free-for-all platforms like Twitter to more siloed spaces like Discord, for many reasons, not just fear of archiving.

It all has the same effect of making it harder to archive though.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#58

Post by nikisweeting »

FiniteField wrote: post #49

Disappointing that a project that should ostensibly care about preserving the open, non-centralised internet takes the time to namedrop and talk about making "compromises" against preserving a well-known, medium-sized clearnet forum legally operated from a US-based LLC. Still-living independent forum sites in this day and age have unrivalled SNR of actual human-to-human communication, there should be no better candid…

Sorry which medium sized forum are you referring to?

I love forums and want them to continue, I'm not sure where you got the idea that I dislike them as a medium. I was just pointing out that public sites in general have started to see some attrition a bit lately for a variety of reasons, and the tooling needs to keep with new mediums as they appear.

I also make no apology for the content, in fact ArchiveBox is explicitly designed to archive the most vile stuff for lawyers and governments to use for long term storage or evidence collection. One of our first prospective clients was the UN wanting to use it to document Syrian war crimes. The point there was that we can save stuff without amplifying it, and that's sometimes useful in niche scenarios.

Lawyers/LE especially don't want to broadcast to the world (or tip off their suspect) that they are investigating or endorsing a particular person, so the ability to capture without publicly announcing/mirroring every capture is vital.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#59

Post by nikisweeting »

millvalleydev wrote: post #24

For devs like us, archivebox? or browsertrix-crawler? for scraping entire sites for our own uses, maybe to keep contents behind pay walls while we have subscriptions or maybe to feed them to local LLMs to ask?

For scraping entire sites browserteix is currently more suited until we add full depth recursive crawling in v0.9. For feeding to LLMs ArchiveBox MIGHT BE better (imho) because we extract the raw content and you likely don't need the whole WARC.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#60

Post by rcarmo »

nikisweeting wrote: post #55
rcarmo wrote: post #51

This is nice. I'm actually much more excited about the REST API (which will let me do searches and pull information out, I hope) than the plugin ecosystem, since the last thing I need is for another tool to have a half-baked LLM integration -- I prefer to do that myself and have full control. Being able to do RAG on my ArchiveBox is something that I have very much wanted to do for over a year now, and it might finall…

The /cli/list endpoint is the search endpoint you're looking for. It provides FTS but I can make it clearer in the docs, thanks for the tip. As for the AI stuff don't worry, none of it is touching core, it's all in an optional community plugin only for those who want it. I'm not personally a huge AI person but I have clients who are already using it and getting massive value from it, so it's worth mentioning. (They'r…

Thanks, I'll have a look.

My use for this is very different--I want to be able to use a specific subset of my archived pages (which is mostly reference documentation) to "chat" with, providing different LLM prompts depending on subset and fetching plaintext chunks as reference info for the LLM to summarize (and point me back to the archived pages if I need more info).

Post reply on HN