Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Your Own Internet Archive with Archive Box

nixintel.info

Post reply on HN
Print view View original

61–70 of 81 posts

Re: Make Your Own Internet Archive with Archive Box

#61

Post by dgeiser13 »

hobo_mark wrote: post #53
matt_f wrote: post #33

Interesting side note: It seems like a lot of people in this thread have an interest in retaining a "replayable timeline" of their own browsing/reading history. There's probably enough support here to gather a few contributors for an open source project.

I seem to remember Google's Larry Page once proposed a similar thing in the early days, a product that would record all you read on your computer (to make it searchable later), but now I can't find it mentioned anywhere, am I imagining things?

If you use Google Chrome as your primary browser this exists at https://myactivity.google.com/item

Re: Make Your Own Internet Archive with Archive Box

#62

Post by nikisweeting »

dirtyid wrote: post #52

Tried this a while ago, disappointed at HD usage. My solution as heavy TTS user who has balabolka setup to read copied text which naturally leaves a log for future reference. There's extentions to auto copy highlighted text and append urls which makes entire flow straight forward. Log each day is around 1-5mbs of text saved in a big folder. Biggest limitation is trying to advance search unstructured text files by com…

I think you tried a very old version ;) all that has long since changed. As of v0.5 ArchiveBox has everything in a Sqlite3 DB and full-text search is implemented with Sonic.

Re: Make Your Own Internet Archive with Archive Box

#63

Post by nikisweeting »

throwawaysea wrote: post #44

Can you configure this tool to login to websites (for paid news subscriptions) and get past those paywalls?

Yeah, it supports it but there are security considerations if you're doing it for anything more serious than news content. See here: https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv...

Re: Make Your Own Internet Archive with Archive Box

#64

Post by nikisweeting »

ketamine__ wrote: post #22

How does archive.is trick news sites into showing content without the paywall? Is it pure user agent spoofing? I'm wondering if this could be applied here.

Yeup, just the reason why we expose the USER_AGENT options in ArchiveBox config ;)

https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overv...

I don't want to officially endorse using the Google bot user agent, but you're welcome to try it on your own and see if it improves the experience.

Re: Make Your Own Internet Archive with Archive Box

#65

Post by nikisweeting »

lazyjeff wrote: post #13

I feel like a simple automatic capture of timestamp + url + screenshot would already be very useful. This gives you a visual memory of the things you've seen on the web. I've wanted to develop this for a while, as a browser plugin. Being able to skim the past month or two click around the thumbnails would already be amazing. I've wanted to do that many times before to check if my memory was correct, or if a page chan…

Sounds like this might be what you want if you only want screenshots + timestamps and nothing else:

    archivebox oneshot --extract=screenshot 'https://example.com'
or

    archivebox add --extract=screenshot

Re: Make Your Own Internet Archive with Archive Box

#66

Post by nikisweeting »

jedimastert wrote: post #10

Is there a list of web page archive formats I could look at? There are a few things I'd love to do where it would be very handy to have one file per page

The main archive formats for web content are WARC, ZIM, Memento, and static HTML (e.g. from a tool like wget or Singlefile).

If you want 1 page per URL I recommend Singlefile.

Lots more info here if you want to compare different software options: https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-...

Re: Make Your Own Internet Archive with Archive Box

#67

Post by nikisweeting »

matt_f wrote: post #33

Interesting side note: It seems like a lot of people in this thread have an interest in retaining a "replayable timeline" of their own browsing/reading history. There's probably enough support here to gather a few contributors for an open source project.

There are a bunch of projects trying to do different flavors of this already, check out some of these:

https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-...

Re: Make Your Own Internet Archive with Archive Box

#68

Post by nikisweeting »

hobo_mark wrote: post #53
matt_f wrote: post #33

Interesting side note: It seems like a lot of people in this thread have an interest in retaining a "replayable timeline" of their own browsing/reading history. There's probably enough support here to gather a few contributors for an open source project.

I seem to remember Google's Larry Page once proposed a similar thing in the early days, a product that would record all you read on your computer (to make it searchable later), but now I can't find it mentioned anywhere, am I imagining things?

A "remember everything for me" tool is often called a "Memex" https://en.wikipedia.org/wiki/Memex

Re: Make Your Own Internet Archive with Archive Box

#69

Post by nikisweeting »

andai wrote: post #57
mail2merge wrote: post #55

Earlier quoted context omitted.

Wow, your reading comprehension is amazingly good. Yep, that's correct.

There's a link to more info on the Chrome thing but it 404s https://github.com/c9fe/22120/issues/57

http://web.archive.org/web/20201206153345/https://github.com...

Re: Make Your Own Internet Archive with Archive Box

#70

Post by nikisweeting »

Hey all, @pirate (ArchiveBox maintainer) here, thanks for posting this @adamhearn.

If you like ArchiveBox check out our new Twitter account for the project, https://twitter.com/ArchiveBoxApp we just opened it and we'll be posting announcements and prerelease sneak-peeks on there in the future.

Post reply on HN