Live data from Hacker News

Make Your Own Internet Archive with Archive Box

nixintel.info

31–40 of 81 posts

Re: Make Your Own Internet Archive with Archive Box

#31

How long until this is a feature baked into a mainstream web browser? Archive, prefetch, cache, all variants on a theme. History, bookmarks, local search engine, all the same.

I often wish that I could do a full text search of every page I've already visited.

Re: Make Your Own Internet Archive with Archive Box

#32
post #31

How long until this is a feature baked into a mainstream web browser? Archive, prefetch, cache, all variants on a theme. History, bookmarks, local search engine, all the same.

I often wish that I could do a full text search of every page I've already visited.

I'm working in that in my "self host the internet offline from your browsing history" project

https://github.com/c9fe/22120

It makes a web archive from everything you browse, and lately I've been working on the full text search

Re: Make Your Own Internet Archive with Archive Box

#34
post #7

This would be a nice thing to be able to run on a Synology NAS or other kind of device that typically has terabytes of storage.

It runs quite well in docker. I still feed my instance by hand but eventually need to write a firefox extension to push history semi-live.

Re: Make Your Own Internet Archive with Archive Box

#36

I feel like a simple automatic capture of timestamp + url + screenshot would already be very useful. This gives you a visual memory of the things you've seen on the web. I've wanted to develop this for a while, as a browser plugin. Being able to skim the past month or two click around the thumbnails would already be amazing. I've wanted to do that many times before to check if my memory was correct, or if a page chan…

> screenshot Wouldn't it be more useful and take less space to use SignleFile?

You'd think so, but code (even web code) needs to be executed and is brittle. Some of it doesn't even work as an archive even right after saving. All my images from 10-20 years ago work perfectly today. None of my code does without some major effort.

Re: Make Your Own Internet Archive with Archive Box

#37
post #21

Earlier quoted context omitted.

ocr?

why go from text to image and back to text? seems wasteful and error prone...

It's a hard problem to figure out what's readable text on a page, and what isn't. Even Google has a hard time figuring that out. OCR works very well with screenshots, and is purely computation time. But the real reason is generally just having timestamps, urls, and screenshots is good enough. I usually remember about when it was, and some words in the url, and don't need the heavyweight text search setup.

Re: Make Your Own Internet Archive with Archive Box

#38
post #21

Earlier quoted context omitted.

why go from text to image and back to text? seems wasteful and error prone...

It's a hard problem to figure out what's readable text on a page, and what isn't. Even Google has a hard time figuring that out. OCR works very well with screenshots, and is purely computation time. But the real reason is generally just having timestamps, urls, and screenshots is good enough. I usually remember about when it was, and some words in the url, and don't need the heavyweight text search setup.

Just hard with the "read more" buttons.

Re: Make Your Own Internet Archive with Archive Box

#39
post #31

How long until this is a feature baked into a mainstream web browser? Archive, prefetch, cache, all variants on a theme. History, bookmarks, local search engine, all the same.

I often wish that I could do a full text search of every page I've already visited.

Safari’s history used to crudely do that somewhat. You could open up the history and search for any word that appeared on a page you browsed and it would filter it to list the matching pages.

Re: Make Your Own Internet Archive with Archive Box

#40
post #30

I feel like a simple automatic capture of timestamp + url + screenshot would already be very useful. This gives you a visual memory of the things you've seen on the web. I've wanted to develop this for a while, as a browser plugin. Being able to skim the past month or two click around the thumbnails would already be amazing. I've wanted to do that many times before to check if my memory was correct, or if a page chan…

A few years ago, in an attempt to increase productivity, I used a screen recorder that took a screenshot every 10 seconds and played it back at the end of every day. So I had a timelapse of how I was spending my time -- mostly online. It was very enlightening. The most efficient format to store a sequence of screenshots in is video, because most of them will have heavily overlapping data.

Huh. That's a pretty nifty thing to do. Just wrote a python script to do that for me and it's running in the background right now. Shall be interesting to come back to it today in the evening. Do you recall how many seconds each screenshot would show itself in the final video (basically what was the framerate?). Currently considering about 4 frames per second but would love to get your take on it :)
Post reply on HN