Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Help preserve the internet with Archiveteam's warrior

selfhostedheaven.com

Post reply on HN
Print view View original

41–50 of 53 posts

Re: Help preserve the internet with Archiveteam's warrior

#41

Post by myself248 »

nix23 wrote: post #39
myself248 wrote: post #36

Earlier quoted context omitted.

Oh jeez yeah. I've been going through https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-... the last few days and I've concluded that none of 'em are appropriate for someone with my level of software ineptitude.

wget --recursive --page-requisites --adjust-extension --convert-links --no-parent https://YOURWEBPAGEHEREX.com NO "--convert-links" if you want a "pure" non local browsable copy.

Yes yes fine, and then I get throttled to 2 bytes/sec by the server. So I did some user-agent hijinks and set my delay to like 5000msec and that helped for a while, but my machine crashed and when I went to resume the task I was throttled again.

Re: Help preserve the internet with Archiveteam's warrior

#42

Post by jrwr »

cxr wrote: post #25
RNAlfons wrote: post #14

Make it an easy installable/runable Windows application and it will spread like wildfire.

Why even require that? If the data in question is available over HTTP, it should be as easy as opening a page from the relevant origin in a browser tab, optionally opening a second tab for a "Warrior Dashboard", then invoking a bookmarklet on the former to slurp up data by XHR &tc. (If it's necessary to cross origins as the thing roves around, the dashboard can alert you to this while it continues doing what it can w…

Full Archival with the standards required by the Internet Archive require that full unmodified headers are required, and unmodified content. This tends not to work well with modern browsers. Chrome and Firefox both fail at this currently. Someone is looking into a kind of modified Firefox to help with this. but its just not that how this system works. Now the Archive.org does have a API of sorts to say hay archive this URL, and a little working on the backend goes and does it..

What the Archive Team does is on a much more massive scale. Like SETI at home scale of scraping data across the internet. At almost every point we have had to make custom tools to ensure it meets our needs in our archival efforts.

Re: Help preserve the internet with Archiveteam's warrior

#43

Post by cxr »

myself248 wrote: post #37
cxr wrote: post #25

Earlier quoted context omitted.

Why even require that? If the data in question is available over HTTP, it should be as easy as opening a page from the relevant origin in a browser tab, optionally opening a second tab for a "Warrior Dashboard", then invoking a bookmarklet on the former to slurp up data by XHR &tc. (If it's necessary to cross origins as the thing roves around, the dashboard can alert you to this while it continues doing what it can w…

That would be awesome, do you think you could write that?

I'd definitely be interested in working on getting as close as possible if the grant money were to appear.

Re: Help preserve the internet with Archiveteam's warrior

#44

Post by cxr »

jrwr wrote: post #42
cxr wrote: post #25

Earlier quoted context omitted.

Why even require that? If the data in question is available over HTTP, it should be as easy as opening a page from the relevant origin in a browser tab, optionally opening a second tab for a "Warrior Dashboard", then invoking a bookmarklet on the former to slurp up data by XHR &tc. (If it's necessary to cross origins as the thing roves around, the dashboard can alert you to this while it continues doing what it can w…

Full Archival with the standards required by the Internet Archive require that full unmodified headers are required, and unmodified content. This tends not to work well with modern browsers. Chrome and Firefox both fail at this currently. Someone is looking into a kind of modified Firefox to help with this. but its just not that how this system works. Now the Archive.org does have a API of sorts to say hay archive th…

> standards required by the Internet Archive require that full unmodified headers are required

Sure, this would not be a solution for the Wayback Machine, but would be adequate[1][2] for lots of non-Wayback collections (of the sort that Archive Team is associated with).

1. https://twitter.com/textfiles/status/970912494284779520

2. http://ascii.textfiles.com/archives/4285

Re: Help preserve the internet with Archiveteam's warrior

#45

Post by TheTechRobo »

causi wrote: post #28

Warrior is great for the community effort, but I wish someone would put some work into a modern local site archiver. HTTRACK just doesn't cut it anymore.

There's github.com/ArchiveTeam/grab-site, but unfortunately it's not maintained very well.

Re: Help preserve the internet with Archiveteam's warrior

#46

Post by TheTechRobo »

cxr wrote: post #25
RNAlfons wrote: post #14

Make it an easy installable/runable Windows application and it will spread like wildfire.

Why even require that? If the data in question is available over HTTP, it should be as easy as opening a page from the relevant origin in a browser tab, optionally opening a second tab for a "Warrior Dashboard", then invoking a bookmarklet on the former to slurp up data by XHR &tc. (If it's necessary to cross origins as the thing roves around, the dashboard can alert you to this while it continues doing what it can w…

Similar: github.com/InternetArchive/warcprox

Re: Help preserve the internet with Archiveteam's warrior

#47

Post by janandonly »

nix23 wrote: post #38
janandonly wrote: post #19

Earlier quoted context omitted.

Unlike the Archive, the "value" of Bitcoin can be measured: Today's market cap of BTC is $839.5B

>Today's market cap of BTC is $839.5B Or Zero...depends who want to exchange it to real-stuff

Well, if you happen to have some bitcoins that you are willing to sell to me for less than their "market value" today, then please, get in touch with me...

The same goes for any other money/not-money's out there... If anyone has gold/silver/diamonds that he wants to get rid of for a price lower than the market value, then again, please get in touch with me....

Re: Help preserve the internet with Archiveteam's warrior

#48

Post by nix23 »

myself248 wrote: post #41
nix23 wrote: post #39

Earlier quoted context omitted.

wget --recursive --page-requisites --adjust-extension --convert-links --no-parent https://YOURWEBPAGEHEREX.com NO "--convert-links" if you want a "pure" non local browsable copy.

Yes yes fine, and then I get throttled to 2 bytes/sec by the server. So I did some user-agent hijinks and set my delay to like 5000msec and that helped for a while, but my machine crashed and when I went to resume the task I was throttled again.

>but my machine crashed

Maybe it's not the servers who throttle you then ;)

Re: Help preserve the internet with Archiveteam's warrior

#49

Post by nix23 »

janandonly wrote: post #47
nix23 wrote: post #38

Earlier quoted context omitted.

>Today's market cap of BTC is $839.5B Or Zero...depends who want to exchange it to real-stuff

Well, if you happen to have some bitcoins that you are willing to sell to me for less than their "market value" today, then please, get in touch with me... The same goes for any other money/not-money's out there... If anyone has gold/silver/diamonds that he wants to get rid of for a price lower than the market value, then again, please get in touch with me....

If you think you can compare the trust people have to gold compared to bitcoin your in a massive bubble.

Re: Help preserve the internet with Archiveteam's warrior

#50

Post by traverseda »

nix23 wrote: post #48
myself248 wrote: post #41

Earlier quoted context omitted.

Yes yes fine, and then I get throttled to 2 bytes/sec by the server. So I did some user-agent hijinks and set my delay to like 5000msec and that helped for a while, but my machine crashed and when I went to resume the task I was throttled again.

>but my machine crashed Maybe it's not the servers who throttle you then ;)

Wget will exhaust all available ram on a long enough crawl.
Post reply on HN