Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox is evolving: the future of self-hosted internet archives

docs.sweeting.me

Post reply on HN
Print view View original

91–100 of 166 posts

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#91

Post by dark-star »

nikisweeting wrote: post #58
FiniteField wrote: post #49

Disappointing that a project that should ostensibly care about preserving the open, non-centralised internet takes the time to namedrop and talk about making "compromises" against preserving a well-known, medium-sized clearnet forum legally operated from a US-based LLC. Still-living independent forum sites in this day and age have unrivalled SNR of actual human-to-human communication, there should be no better candid…

Sorry which medium sized forum are you referring to? I love forums and want them to continue, I'm not sure where you got the idea that I dislike them as a medium. I was just pointing out that public sites in general have started to see some attrition a bit lately for a variety of reasons, and the tooling needs to keep with new mediums as they appear. I also make no apology for the content, in fact ArchiveBox is expli…

I guess he's talking about K_wi F_rms which was mentioned in one of the screenshots...

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#92

Post by bityard »

So, after reading through the comments and website, I just realized I used ArchiveBox a month or two ago for a very specific purpose.

You see, I inherited a boat.

This boat belonged to my father. He was not materialistic but he took very good care of the things he cared about, and he cared about this boat. It's an old 18' aluminum fishing/cruising boat built in the early 1960's. It's not particularly valuable as a collectible but it is fairly rare and has some unique modifications. I spent a lot of time trying to dig up all of the info that I could on it, but this is one of those situations where most of the companies involved have been gone for decades and most everyone who was around when these were made are either dead or not really on the Internet.

It's a shame that I waited so long to start my research because 10 or 20 years ago, there were quite a few active web forums containing informational/tutorial threads from the proud owners of these old boats. I know because I have seen references to them. Some of the URLs are in archive.org, some are not. But the forums are gone, so a large chunk of knowledge on these boats is too, probably forever.

I did manage to dig up some interesting articles, pictures, and forum threads and needed a way to save them so that they didn't disappear from the web as well. There is probably an easier way to go about it, but in the end I ran ArchiveBox via Docker and set it to fetching what I could find and then downloaded the resulting pages as self-contained HTML pages.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#93

Post by nikisweeting »

nyx wrote: post #77
nikisweeting wrote: post #35

Earlier quoted context omitted.

Let chat more. I'm almost ready to raise some seed money, hire a second staff dev or find a cofounder, and I'm looking for people that care deeply about the space. It's only been during the last few months that I decided to go all in on the project, so this is still just the first few pages of a new chapter in the project's history. (I should also mention that if you're a commercial entity relying on ArchiveBox, you…

It looks like you're doing great work here, thanks a bunch; looking forward to seeing this project develop. Selling custom integrations, managed instances, white-glove support with an SLA, and so on seems like a reasonable funding model for a project based on an open-source, self-hostable platform. But I'm a little disheartened to read that you're maintaining a closed fork with "goodies" in it. How do you decide whic…

The idea with the plugin system is that plugins are just git repos containing /__init__.py, and you can add any set of git repo plugins you want to your instance.

The marketplace will work by showing all git repos tagged with the "archivebox" tag on github.

My approval is only needed for PRs to the archivebox core engine.

More info on free vs paid + reasoning why it's not all open source: https://news.ycombinator.com/item?id=41863539

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#94

Post by pabs3 »

0cf8612b2e1e wrote: post #29
toomuchtodo wrote: post #5

https://github.com/ArchiveTeam/grab-site might be helpful. I'm a fan of the ability to create WARC archives from a target, uploard the WARC files to object storage (whether that is IA, S3, Backblaze B2, etc), and then keep them in cold storage or serve them up via HTTPS or a torrent (mutable, preferred). The Internet Archive serves a torrent file for every item they host; one can do the same with WARC archives to ena…

The Internet Archive serves a torrent file for every item they host I had no idea. I have found the IA serving speed to be pretty terrible. Are the torrents any better? Presumably the only ones seeding the files are IA themselves.

The torrents have better speeds because they have WebSeeds for multiple IA servers, so you can download from multiple servers at once.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#96

Post by the_gorilla »

dang wrote: post #74
the_gorilla wrote: post #19

Earlier quoted context omitted.

[flagged]

We've banned this account for breaking the site guidelines. Please don't create accounts to break HN's rules with. https://news.ycombinator.com/newsguidelines.html

[dead]

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#97

Post by nikisweeting »

pabs3 wrote: post #95

Unfortunately ArchiveBox uses wget, so it produces non-standard WARC files. Sadly there are lots of things like this in the WARC ecosystem. https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem

Yes, this is true currently. If you need nice WARCs I recommend Browsertrix by our friends at Webrecorder instead.

Its on my roadmap to improve this eventually, but currently I'm focused on saving raw files to a filesystem, because it's more accessible to most users, and easier to pipe into other tools.

I encourage people to use ZFS to do deduping and compression at the filesystem layer.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#98

Post by nikisweeting »

dark-star wrote: post #91
nikisweeting wrote: post #58

Earlier quoted context omitted.

Sorry which medium sized forum are you referring to? I love forums and want them to continue, I'm not sure where you got the idea that I dislike them as a medium. I was just pointing out that public sites in general have started to see some attrition a bit lately for a variety of reasons, and the tooling needs to keep with new mediums as they appear. I also make no apology for the content, in fact ArchiveBox is expli…

I guess he's talking about K_wi F_rms which was mentioned in one of the screenshots...

Ahh that makes sense. Well all I can say to that is that it's not up to me what's evil. The point I was trying to make is: sometimes you want to archive something that you don't endorse / don't want to be publicly linked.

You might not want to amplify and broadcast the fact that you're archiving it to the world.

Re: ArchiveBox is evolving: the future of self-hosted internet archives

#100

Post by nikisweeting »

A4ET8a8uTh0 wrote: post #88

Those additions are welcome, but if I could request one -- I and one that it is very consistently requested -- feature: - backing up an entire page Yes, it is hard. Yes, for non-pure html pages is extra kind of painful, but that would honestly making archivebox go from nice to have to.. yes, I have an actual archive I can use when stuff goes down.

Do you mean backing up an entire domain? Like example.com/*

If so that's starting to roll out in v0.8.5rc50, check out the archivebox/crawls/ folder.

If you mean archiving a single page more thoroughly, what do you find is missing in Archivebox? Are you able to get singlefile/chrome/wget html when archiving?

Post reply on HN