Make Your Own Internet Archive with Archive Box • Hacker News Board

Re: Make Your Own Internet Archive with Archive Box

#21

Post by Triv888 » Wed, Jan 20, 2021, 2:20 AM UTC

ramraj07 wrote: post #20

bravura wrote: post #15

Earlier quoted context omitted.

This doesn't allow full text search easily, though.

ocr?

why go from text to image and back to text? seems wasteful and error prone...

Re: Make Your Own Internet Archive with Archive Box

#22

Post by ketamine__ » Wed, Jan 20, 2021, 2:41 AM UTC

How does archive.is trick news sites into showing content without the paywall? Is it pure user agent spoofing?

I'm wondering if this could be applied here.

Re: Make Your Own Internet Archive with Archive Box

#23

Post by LEARAX » Wed, Jan 20, 2021, 2:45 AM UTC

evc wrote: post #2

You will need a lot of disk storage right?

There are different extractors/services, and you can toggle them pretty easily. By default it screenshots everything, exports a PDF, saves like 4 different HTML copies and submits the link to the wayback machine. It also tries to extract important text, and stores that separately. You could easily configure it to only extract text, turn off some HTML extractors, or disable the PDF and screenshot captures if you want to prioritize disk space.

Re: Make Your Own Internet Archive with Archive Box

#24

Post by mikiem » Wed, Jan 20, 2021, 2:45 AM UTC

How can I use this to archive sites/pages that require logging in to see?

Re: Make Your Own Internet Archive with Archive Box

#25

Post by unnouinceput » Wed, Jan 20, 2021, 2:47 AM UTC

Quote: "..even if you instruct it to begin archiving a site then it can easily fail if that site’s robots.txt prevents crawling"

Huh? Does actually the big corporations care anymore about robots.txt? Nowadays is more of a "netiquette" than anything else. Google definitely ignores it. Dunno DuckDucGo what it does

Re: Make Your Own Internet Archive with Archive Box

#26

Post by tylorr » Wed, Jan 20, 2021, 2:57 AM UTC

blastro wrote: post #12

mikece wrote: post #7

This would be a nice thing to be able to run on a Synology NAS or other kind of device that typically has terabytes of storage.

that's what i do - there's a docker image, 1 line script + cron job. it archives an rss feed of links i gather

How do you generate that rss feed?

Re: Make Your Own Internet Archive with Archive Box

#27

Post by smnrchrds » Wed, Jan 20, 2021, 4:12 AM UTC

lazyjeff wrote: post #13

I feel like a simple automatic capture of timestamp + url + screenshot would already be very useful. This gives you a visual memory of the things you've seen on the web. I've wanted to develop this for a while, as a browser plugin. Being able to skim the past month or two click around the thumbnails would already be amazing. I've wanted to do that many times before to check if my memory was correct, or if a page chan…

> screenshot

Wouldn't it be more useful and take less space to use SignleFile?

Re: Make Your Own Internet Archive with Archive Box

#28

Post by blastro » Wed, Jan 20, 2021, 4:15 AM UTC

tylorr wrote: post #26

blastro wrote: post #12

Earlier quoted context omitted.

that's what i do - there's a docker image, 1 line script + cron job. it archives an rss feed of links i gather

How do you generate that rss feed?

getpocket.com

Re: Make Your Own Internet Archive with Archive Box

#29

Post by zeckalpha » Wed, Jan 20, 2021, 4:30 AM UTC

How long until this is a feature baked into a mainstream web browser? Archive, prefetch, cache, all variants on a theme. History, bookmarks, local search engine, all the same.

Re: Make Your Own Internet Archive with Archive Box

#30

Post by andai » Wed, Jan 20, 2021, 4:56 AM UTC

lazyjeff wrote: post #13

I feel like a simple automatic capture of timestamp + url + screenshot would already be very useful. This gives you a visual memory of the things you've seen on the web. I've wanted to develop this for a while, as a browser plugin. Being able to skim the past month or two click around the thumbnails would already be amazing. I've wanted to do that many times before to check if my memory was correct, or if a page chan…

A few years ago, in an attempt to increase productivity, I used a screen recorder that took a screenshot every 10 seconds and played it back at the end of every day. So I had a timelapse of how I was spending my time -- mostly online. It was very enlightening.

The most efficient format to store a sequence of screenshots in is video, because most of them will have heavily overlapping data.