Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Help preserve the internet with Archiveteam's warrior

selfhostedheaven.com

Post reply on HN
Print view View original

31–40 of 53 posts

Re: Help preserve the internet with Archiveteam's warrior

#31

Post by azeirah »

sandgiant wrote: post #5
Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

Can you provide some details on this? I'm curious how noise and signal are defined and measured in this case.

I disagree with the op. This is historical data and includes all kinds of interesting content. Even if severely uninteresting today it may still be really valuable 40 years from now as part of research into colloquial language, design, trends, influence of events etc.

Same reason why notes taken by random people 250 years ago are really valuable to historians today, even if it's just a todo list

Re: Help preserve the internet with Archiveteam's warrior

#32

Post by prox »

uniqueuid wrote: post #18
prox wrote: post #4

Earlier quoted context omitted.

I kind of wonder how we can make it searchable again. Is this included in this archiving effort? In any case wonderful work.

There is a standard set of tooling for indexing archives: CDX files. [1] They index WARC archives and can be used to quickly find records. You can build on top of this (and some systems do) to make a proper search front-end. But in general, these archives are NOT geared towards full-blown search because it would be pretty expensive to keep the indexes in hot cache. Plus you would need to deal with historical versions…

Thank you for that information!

Re: Help preserve the internet with Archiveteam's warrior

#33

Post by textfiles »

Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

I've been having fun with this post all day, but now I kind of need to know: Can you give examples of noise on the Archive?

Re: Help preserve the internet with Archiveteam's warrior

#34

Post by stjohnswarts »

Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

I would argue that the archive.org and saving the legacy of the internet is a far more important use of energy than making up imaginary digital currency pyramid schemes.

Re: Help preserve the internet with Archiveteam's warrior

#35

Post by myself248 »

camtarn wrote: post #27
uniqueuid wrote: post #18

Earlier quoted context omitted.

There is a standard set of tooling for indexing archives: CDX files. [1] They index WARC archives and can be used to quickly find records. You can build on top of this (and some systems do) to make a proper search front-end. But in general, these archives are NOT geared towards full-blown search because it would be pretty expensive to keep the indexes in hot cache. Plus you would need to deal with historical versions…

Ah, is the WARC format the reason it's called 'Warrior'? It seems like a very strange name for an archival program.

ArchiveTeam seems very guerrilla in their operations.

I always imagined the Warrior as a camo-faced archivist operating under cover of darkness, preserving data even in the most hostile Yahoo-occupied territory.

Re: Help preserve the internet with Archiveteam's warrior

#36

Post by myself248 »

causi wrote: post #28

Warrior is great for the community effort, but I wish someone would put some work into a modern local site archiver. HTTRACK just doesn't cut it anymore.

Oh jeez yeah. I've been going through https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-... the last few days and I've concluded that none of 'em are appropriate for someone with my level of software ineptitude.

Re: Help preserve the internet with Archiveteam's warrior

#37

Post by myself248 »

cxr wrote: post #25
RNAlfons wrote: post #14

Make it an easy installable/runable Windows application and it will spread like wildfire.

Why even require that? If the data in question is available over HTTP, it should be as easy as opening a page from the relevant origin in a browser tab, optionally opening a second tab for a "Warrior Dashboard", then invoking a bookmarklet on the former to slurp up data by XHR &tc. (If it's necessary to cross origins as the thing roves around, the dashboard can alert you to this while it continues doing what it can w…

That would be awesome, do you think you could write that?

Re: Help preserve the internet with Archiveteam's warrior

#38

Post by nix23 »

janandonly wrote: post #19
Thorentis wrote: post #3

The Internet Archive has a huge noise to signal ratio, very much in favour of noise. I admire the effort and regularly make use of the quality archives. However, I wonder if much like Bitcoin, tremendous energy and amounts of resources are being put towards very little of value.

Unlike the Archive, the "value" of Bitcoin can be measured: Today's market cap of BTC is $839.5B

>Today's market cap of BTC is $839.5B

Or Zero...depends who want to exchange it to real-stuff

Re: Help preserve the internet with Archiveteam's warrior

#39

Post by nix23 »

myself248 wrote: post #36
causi wrote: post #28

Warrior is great for the community effort, but I wish someone would put some work into a modern local site archiver. HTTRACK just doesn't cut it anymore.

Oh jeez yeah. I've been going through https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-... the last few days and I've concluded that none of 'em are appropriate for someone with my level of software ineptitude.

wget --recursive --page-requisites --adjust-extension --convert-links --no-parent https://YOURWEBPAGEHEREX.com

NO "--convert-links" if you want a "pure" non local browsable copy.

Re: Help preserve the internet with Archiveteam's warrior

#40

Post by lazyjeff »

myself248 wrote: post #36
causi wrote: post #28

Warrior is great for the community effort, but I wish someone would put some work into a modern local site archiver. HTTRACK just doesn't cut it anymore.

Oh jeez yeah. I've been going through https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-... the last few days and I've concluded that none of 'em are appropriate for someone with my level of software ineptitude.

For local archiving, I've been working on my own solution that's simply a background process running in your systray on Windows: https://irchiver.com/

There's some philosophical differences with ArchiveBox. 1) I'm more about automatic archiving of every web page, rather than the curation approach, 2) I prefer full-resolution screenshots over the actual source of the web page so you can save what you actually saw (so it works with dynamic pages, pages behind logins, etc.), 3) and I think full-text search is a key part of the archive so have implemented that.

Post reply on HN