Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ArchiveBox: Open-source self-hosted web archive

github.com

Post reply on HN
Print view View original

11–20 of 63 posts

Re: ArchiveBox: Open-source self-hosted web archive

#11

Post by gwern »

amenod wrote: post #5

This sounds like a very good idea, but I'm having trouble making it work. For example, let's say I want to save a great website which will probably disappear soon ( https://launchaco.com ). I run `echo https://launchaco.com | ./archive` and then...? The generated index.html doesn't load css and js files. Or is this more for static content? Is there some tool that would allow one to make a copy of a modern SPA? Is tha…

> The generated index.html doesn't load css and js files. Or is this more for static content?

Why doesn't it do that? I thought that was the point of using Chrome as a headless browser to load all the dynamic elements into a final DOM so they could then be captured & serialized out: "ArchiveBox works by rendering the pages in a headless browser, then saving all the requests and fully loaded pages in multiple redundant common formats (HTML, PDF, PNG, WARC) that will last long after the original content dissapears off the internet."

Re: ArchiveBox: Open-source self-hosted web archive

#13

Post by [deleted] »

amenod wrote: post #5

This sounds like a very good idea, but I'm having trouble making it work. For example, let's say I want to save a great website which will probably disappear soon ( https://launchaco.com ). I run `echo https://launchaco.com | ./archive` and then...? The generated index.html doesn't load css and js files. Or is this more for static content? Is there some tool that would allow one to make a copy of a modern SPA? Is tha…

[deleted]

Re: ArchiveBox: Open-source self-hosted web archive

#14

Post by toomuchtodo »

platz wrote: post #4

This looks very nice As an aside, wget -r -k -np works surprisingly well for my offline needs. For permanent access I defer to archive.is

Consider https://github.com/ArchiveTeam/grab-site (ArchiveTeam/grab-site) in the future. There is a difference between fetching/mirroring content and recording the http header request and responses when retrieving all objects for a site.

I believe there is even a docker container for a quick pull and run.

Re: ArchiveBox: Open-source self-hosted web archive

#15

Post by jalopy »

This looks awesome. Is there an easy way to transfer my current chrome session / cookies to the archivebox chromium instance? Would love to my subscription websites (eg nytimes) to register my logged-in state and allow the capturing instance full logged-in access.

EDIT: Nevermind, should have RTFM - see CHROME_USER_DATA_DIR in https://github.com/pirate/ArchiveBox/wiki/Configuration.

This looks really, really awesome.

Re: ArchiveBox: Open-source self-hosted web archive

#16

Post by j88439h84 »

toomuchtodo wrote: post #14
platz wrote: post #4

This looks very nice As an aside, wget -r -k -np works surprisingly well for my offline needs. For permanent access I defer to archive.is

Consider https://github.com/ArchiveTeam/grab-site (ArchiveTeam/grab-site) in the future. There is a difference between fetching/mirroring content and recording the http header request and responses when retrieving all objects for a site. I believe there is even a docker container for a quick pull and run.

When I want to archive a site, don't I want the content, not just the headers? How do I save the content?

Re: ArchiveBox: Open-source self-hosted web archive

#18

Post by toomuchtodo »

j88439h84 wrote: post #16
toomuchtodo wrote: post #14

Earlier quoted context omitted.

Consider https://github.com/ArchiveTeam/grab-site (ArchiveTeam/grab-site) in the future. There is a difference between fetching/mirroring content and recording the http header request and responses when retrieving all objects for a site. I believe there is even a docker container for a quick pull and run.

When I want to archive a site, don't I want the content, not just the headers? How do I save the content?

When archiving, you want the request and response headers and the content. Grab-site does that (any tool really that'll write WARC files). Sorry if my comment was ambiguous in that regard.

https://en.wikipedia.org/wiki/Web_ARChive

https://www.loc.gov/preservation/digital/formats/fdd/fdd0002...

If you wanted to write WARC files out with wget, you'd use the options specified in https://www.archiveteam.org/index.php?title=Wget_with_WARC_o...

Post reply on HN