Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

Post reply on HN
Print view View original

71–80 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#71

Post by nikisweeting »

rcarmo wrote: post #60
nikisweeting wrote: post #55

Earlier quoted context omitted.

The /cli/list endpoint is the search endpoint you're looking for. It provides FTS but I can make it clearer in the docs, thanks for the tip. As for the AI stuff don't worry, none of it is touching core, it's all in an optional community plugin only for those who want it. I'm not personally a huge AI person but I have clients who are already using it and getting massive value from it, so it's worth mentioning. (They'r…

Thanks, I'll have a look. My use for this is very different--I want to be able to use a specific subset of my archived pages (which is mostly reference documentation) to "chat" with, providing different LLM prompts depending on subset and fetching plaintext chunks as reference info for the LLM to summarize (and point me back to the archived pages if I need more info).

Ok that makes sense, I think archivebox works as the first step in a pipeline there, with some other tool doing the LLM analysis and query stuff.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#72

Post by nikisweeting »

mikae1 wrote: post #66
nikisweeting wrote: post #64

Earlier quoted context omitted.

We're going to add TLSNotary support for real cryptographic signing, see my comments below :) Timestamping is also on my roadmap, definitely as a plugin (and likely paid) as it's more corporate users that really need it. We need to keep some of the really advanced attestation features paid to be able to support the rest of the business.

Thanks for the box! Any examples of other possible really advanced features that might go for-pay? Is there any chance you will make current free features for-pay? That'd be rather off-putting for me as a home user.

No, everything currently free will stay free.

The paid stuff currently is:

- per-user permissions & groups

- audit logging

- auto CAPTCHA solving

- burner credential management for FB/Insta/Twitter/etc. w/ auto phone based account verification ability

- custom JS scripts for expanding comments, hiding pop ups, etc.

- managed hosting + support

Some of this stuff ^ is going to become free in upcoming releases, some will stay paid. What I decide to make free is mostly based on abuse potential and legal ramifications, I'd rather have a say in how the risky stuff is used so that it doesn't become a tool weaponized for botting.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#73

Post by nikisweeting »

jasonfarnon wrote: post #69
petertodd wrote: post #63

You really should add timestamping to ArchiveBox. The easiest way to do that would be via my OpenTimestamps protocol, https://opentimestamps.org It's open source and free to use, and uses Bitcoin for the actual timestamps. Users of it do not need to make Bitcoin transactions themselves as a set of community calendar servers do that for you. You also don't need a Bitcoin node to create an OTS timestamp, and you can va…

I always wonder about this when someone gets in hot water based on something on the wayback machine and the person says the archive was tampered with. Can you elaborate on "prove that the archive was created in the past, prior to there being a reason to tamper it"? What exactly does opentimestamps certify?

OpenTimestamps alone can not currently prove anything because TLS session keys are symmetric. The client can forge anything and attest to it falsely. Unless you 100% trust the archiver (in which case you can trust their timestamps), you need TLSNotary or another reputable third party in the loop as a bare minimum.

But more critically: currently the legal standard for evidence is... screenshots. We have a lot of educating work to do before the public understands the value of attestation and signing.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#74

Post by dang »

the_gorilla wrote: post #19
hobs wrote: post #18

Earlier quoted context omitted.

As a custom tool built to archive stuff for archive.org, why would you expect that it can also do a completely opposite task, saving information privately? I can see why you would want such a tool, but it seems like a direct divergence from the core goal of the existing codebase.

[flagged]

We've banned this account for breaking the site guidelines. Please don't create accounts to break HN's rules with.

https://news.ycombinator.com/newsguidelines.html

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#75

Post by 404mm »

nikisweeting wrote: post #37
404mm wrote: post #28

Somewhat similar topic, anyone has recommendations for a self-hosted internet website change monitoring system? I’ve been running Huginn for many years and it works well; however, I have a feeling the project is on its last leg. Also, it’s based on either text scraping (XPath/CSS/HTML and rss but it struggles with newer JS-based sites.

Changedetection.io

Thank you! That looks great!

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#76

Post by giancarlostoro »

nikisweeting wrote: post #35
bravura wrote: post #25

@nikisweeting ArchiveBox is awesome and we'd really love it to be more awesome. And sustainable! I've posted issues and PRs for showstopper issues that took months to get merged in: https://github.com/ArchiveBox/ArchiveBox/issues/991 https://github.com/ArchiveBox/ArchiveBox/pull/1026 You have the opportunity for the community to lean in on ArchiveBox. I understand it's hard to do everything as a solo dev, we've seen…

Let chat more. I'm almost ready to raise some seed money, hire a second staff dev or find a cofounder, and I'm looking for people that care deeply about the space. It's only been during the last few months that I decided to go all in on the project, so this is still just the first few pages of a new chapter in the project's history. (I should also mention that if you're a commercial entity relying on ArchiveBox, you…

Do you guys have a Discord by chance? I have a close friend who is insanely passionate about archiving, he has a personal instance of archivebox, and is working on a Video Downloading project as well. He has used it almost everyday and archived thousands of news articles over years. He's aware of a lot of the nuances.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#77

Post by nyx »

nikisweeting wrote: post #35
bravura wrote: post #25

@nikisweeting ArchiveBox is awesome and we'd really love it to be more awesome. And sustainable! I've posted issues and PRs for showstopper issues that took months to get merged in: https://github.com/ArchiveBox/ArchiveBox/issues/991 https://github.com/ArchiveBox/ArchiveBox/pull/1026 You have the opportunity for the community to lean in on ArchiveBox. I understand it's hard to do everything as a solo dev, we've seen…

Let chat more. I'm almost ready to raise some seed money, hire a second staff dev or find a cofounder, and I'm looking for people that care deeply about the space. It's only been during the last few months that I decided to go all in on the project, so this is still just the first few pages of a new chapter in the project's history. (I should also mention that if you're a commercial entity relying on ArchiveBox, you…

It looks like you're doing great work here, thanks a bunch; looking forward to seeing this project develop.

Selling custom integrations, managed instances, white-glove support with an SLA, and so on seems like a reasonable funding model for a project based on an open-source, self-hostable platform. But I'm a little disheartened to read that you're maintaining a closed fork with "goodies" in it.

How do you decide which features (better test suite?) end up in the non-libre, payware fork of your software? If someone contributed a feature to the open-source version that already exists in the payware version, would you allow it to be merged or would you refuse the pull request?

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#78

Post by nikisweeting »

giancarlostoro wrote: post #76
nikisweeting wrote: post #35

Earlier quoted context omitted.

Let chat more. I'm almost ready to raise some seed money, hire a second staff dev or find a cofounder, and I'm looking for people that care deeply about the space. It's only been during the last few months that I decided to go all in on the project, so this is still just the first few pages of a new chapter in the project's history. (I should also mention that if you're a commercial entity relying on ArchiveBox, you…

Do you guys have a Discord by chance? I have a close friend who is insanely passionate about archiving, he has a personal instance of archivebox, and is working on a Video Downloading project as well. He has used it almost everyday and archived thousands of news articles over years. He's aware of a lot of the nuances.

We have a Zulip which is similar to discord (but self hosted and it has better threading): https://zulip.archivebox.io

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#79

Post by sunshine-o »

nikisweeting wrote: post #55
rcarmo wrote: post #51

This is nice. I'm actually much more excited about the REST API (which will let me do searches and pull information out, I hope) than the plugin ecosystem, since the last thing I need is for another tool to have a half-baked LLM integration -- I prefer to do that myself and have full control. Being able to do RAG on my ArchiveBox is something that I have very much wanted to do for over a year now, and it might finall…

The /cli/list endpoint is the search endpoint you're looking for. It provides FTS but I can make it clearer in the docs, thanks for the tip. As for the AI stuff don't worry, none of it is touching core, it's all in an optional community plugin only for those who want it. I'm not personally a huge AI person but I have clients who are already using it and getting massive value from it, so it's worth mentioning. (They'r…

I have been using ArchiveBox recently and love it.

About search, one thing I haven't yet figured out how to do easily is to plug it to my SearXNG instance as they only seem to support Elasticsearch, Meilisearch or Solr [0]

So this new plugin architecture will allow for a meilisearch plugin I guess (with relevancy ranking).

- [0] https://docs.searxng.org/dev/engines/offline/search-indexer-...

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#80

Post by beefnugs »

orblivion wrote: post #62

Have you (and I wonder the same about archive.org) considered making a Merkle tree of the data that gets archived? Since data (including photos and videos) are getting easier to fake, it may be nice to have a provable record that at least a certain version of the data existed at a certain time. It would be most useful in case of some sort of oppressive regime down the line that wants to edit history. You'd want to pu…

Not just all that nonsense, but also it makes a lot of sense to share just the parts from a website that matter like a single video etc without having to download an entire archive or the rest of the site
Post reply on HN