Ask HN: What is the best way to archive a webpage

#1

Post by badwolff » Wed, Mar 07, 2018, 5:20 PM UTC

I am working with some peers who have a website that links to and catalogs a number of resources (think blog posts).

It would be ideal for the administrators to be able to archive or have a copy of that web page on their server in case of the original post being deleted, links moving, servers being down etc.

Currently they are using http://archive.is to implement a half solution to this intent. It does not work for some websites and ideally they could host their own archived copy.

What are easy solutions to do this?

With Python I was thinking requests - but this would just grab the HTML and not images, or content generated by javascript.

Thinking Selenium, you could take a screenshot of the content - not the most user friendly to read.

What are some other solutions?

Re: Ask HN: What is the best way to archive a webpage

#2

Post by inceptionnames » Wed, Mar 07, 2018, 6:27 PM UTC

Save the page using the browser's Save feature and zip the created assets (a html file plus directory with graphics, js, css, etc) for ease of sharing.

Re: Ask HN: What is the best way to archive a webpage

#3

Post by cimmanom » Wed, Mar 07, 2018, 7:10 PM UTC

If it's not doing silly things like using Javascript to load static content, wget can do recursive crawls.

Re: Ask HN: What is the best way to archive a webpage

#4

Post by mdaniel » Wed, Mar 07, 2018, 8:54 PM UTC

I've enjoyed great success with various archiving proxies, including https://github.com/internetarchive/warcprox#readme and https://github.com/zaproxy/zaproxy#readme (which saves the content to an embedded database, and can be easier to work with than warc files). The benefit of those approaches over just save-as from the browser is that almost by definition the proxy will save all the components required to re-render the page, whereas save will only grab the parts it sees at that time.

If it's a public page, you can submit the URL to the Internet Archive, and benefit both you and them

Re: Ask HN: What is the best way to archive a webpage

#5

Post by anotheryou » Wed, Mar 07, 2018, 10:36 PM UTC

perma.cc looks sweet, but it's very limited for private people

Re: Ask HN: What is the best way to archive a webpage

#6

Post by tyingq » Thu, Mar 08, 2018, 3:12 AM UTC

If you’re okay with easy, but saves at a third party: https://www.npmjs.com/package/archive.is

Re: Ask HN: What is the best way to archive a webpage

#7

Post by ibn_ibid » Thu, Mar 08, 2018, 4:02 PM UTC

WebRecorder is your friend!

https://webrecorder.io

It can capture content such as images and JS and also do offline playback. You can use the web interface or run your own instance https://github.com/webrecorder/webrecorder

Re: Ask HN: What is the best way to archive a webpage

#8

Post by adultSwim » Sat, Mar 10, 2018, 6:45 AM UTC

Either curl or wget will get you pretty far. Learn one of them well. They are basically equivalent. I use curl.

For current web apps, there is an interactive archiver written in Python, Web Recorder. It captures the full bi-directional traffic of a session. https://webrecorder.io/ Web Recorder uses an internal Python library, pywb. That might be a good place to look. https://github.com/webrecorder/pywb

It looks like Selenium has done a lot of catching up on it's interface. I'd be curious how they compare now.

Talk to librarians about archiving the web. They made Internet Archive and have a lot of experience.

Re: Ask HN: What is the best way to archive a webpage

#9

Post by adultSwim » Sat, Mar 10, 2018, 6:51 AM UTC

anotheryou wrote: post #5

perma.cc looks sweet, but it's very limited for private people

Run it yourself https://github.com/harvard-lil/perma

Re: Ask HN: What is the best way to archive a webpage

#10

Post by anotheryou » Sun, Mar 11, 2018, 9:08 PM UTC

adultSwim wrote: post #9

anotheryou wrote: post #5

perma.cc looks sweet, but it's very limited for private people

Run it yourself https://github.com/harvard-lil/perma

I especially like them for their rescue plan and that it will stay online no matter what I do...