How to archive a website that's shutting down soon

#1

Post by marcua » Sun, Jan 09, 2022, 10:54 PM UTC

How to archive a website that's shutting down soon
blog.marcua.net

Re: How to archive a website that's shutting down soon

#2

Post by toomuchtodo » Mon, Jan 10, 2022, 2:13 AM UTC

Why not use grab-site?

https://github.com/ArchiveTeam/grab-site

Re: How to archive a website that's shutting down soon

#3

Post by weekay » Mon, Jan 10, 2022, 5:17 AM UTC

Easiest and simple self hosted option is archive box https://archivebox.io/

Re: How to archive a website that's shutting down soon

#4

Post by marcua » Mon, Jan 10, 2022, 1:51 PM UTC

toomuchtodo wrote: post #2

Why not use grab-site? https://github.com/ArchiveTeam/grab-site

This is great! It looks like you can even grab WARCs using `wget`: https://golangexample.com/put-a-web-archive-warc-on-an-s3-bu...

I'm curious: what sort of fidelity have you seen in `grab-site`'s rewritten static asset URLs? Having to fix URLs that weren't properly rewritten ended up taking me the most time.

Re: How to archive a website that's shutting down soon

#5

Post by marcua » Mon, Jan 10, 2022, 1:53 PM UTC

weekay wrote: post #3

Easiest and simple self hosted option is archive box https://archivebox.io/

Thank you! I wish I had found this sooner to try it out. Similar question to the other one I asked on this thread: what sort of fidelity have you seen in `archivebox`'s rewritten static asset URLs? Having to fix URLs that weren't properly rewritten ended up taking me the most time.

Re: How to archive a website that's shutting down soon

#6

Post by toomuchtodo » Mon, Jan 10, 2022, 4:51 PM UTC

marcua wrote: post #4

toomuchtodo wrote: post #2

Why not use grab-site? https://github.com/ArchiveTeam/grab-site

This is great! It looks like you can even grab WARCs using `wget`: https://golangexample.com/put-a-web-archive-warc-on-an-s3-bu... I'm curious: what sort of fidelity have you seen in `grab-site`'s rewritten static asset URLs? Having to fix URLs that weren't properly rewritten ended up taking me the most time.

grab-site doesn't rewrite URLs, it captures the entire request and response of each http request for an asset and stores as-is for archival purposes. The quality of any mutations or transformations performed will be governed by the tooling used to consume the generated WARCs.

More detail on the WARC format can be found below:

https://www.loc.gov/preservation/digital/formats/fdd/fdd0002...

https://en.wikipedia.org/wiki/Web_ARChive

http://fileformats.archiveteam.org/wiki/WARC

Re: How to archive a website that's shutting down soon

#7

Post by marcua » Mon, Jan 10, 2022, 4:57 PM UTC

toomuchtodo wrote: post #6

marcua wrote: post #4

Earlier quoted context omitted.

This is great! It looks like you can even grab WARCs using `wget`: https://golangexample.com/put-a-web-archive-warc-on-an-s3-bu... I'm curious: what sort of fidelity have you seen in `grab-site`'s rewritten static asset URLs? Having to fix URLs that weren't properly rewritten ended up taking me the most time.

grab-site doesn't rewrite URLs, it captures the entire request and response of each http request for an asset and stores as-is for archival purposes. The quality of any mutations or transformations performed will be governed by the tooling used to consume the generated WARCs. More detail on the WARC format can be found below: https://www.loc.gov/preservation/digital/formats/fdd/fdd0002... https://en.wikipedia.org/wik…

This is really helpful! Thank you so much!