Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Self-hosted offline Internet from your browsing history

github.com

Post reply on HN
Print view View original

91–100 of 166 posts

Re: Show HN: Self-hosted offline Internet from your browsing history

#91

Post by throwii »

abnry wrote: post #73

I am coding my own hacked together bookmarks manager. I can save any page with a click of the button using SingleFile (a fantastic Chrome extension, by the way!). Then a cronjob runs and puts it into a folder to be processed into a database, which generates a static html index and puts it in my Google Drive. Then it syncs offline on my chromebook. Which means that without internet, I can put my chromebook in tablet m…

I currently extract bookmarks from Firefox and Safari and store them inside a local database. Then a cronjob saves them to Wayback machine if a prior check revealed that they are currently not.. donating regularly for that. Mine just makes sure that the pages are not lost, but yours enables offline reading.

I'm uncertain what the best mechanism is, there are so many ways to solve it. From filtering to recrawling for new content to enabling more advanced features, there are so many possibilities.

Re: Show HN: Self-hosted offline Internet from your browsing history

#92

Post by jbc1 »

agumonkey wrote: post #75
abnry wrote: post #73

I am coding my own hacked together bookmarks manager. I can save any page with a click of the button using SingleFile (a fantastic Chrome extension, by the way!). Then a cronjob runs and puts it into a folder to be processed into a database, which generates a static html index and puts it in my Google Drive. Then it syncs offline on my chromebook. Which means that without internet, I can put my chromebook in tablet m…

one day this will be as famous as youtube-dl

https://github.com/pirate/ArchiveBox

Re: Show HN: Self-hosted offline Internet from your browsing history

#93

Post by noitpmeder »

fizixer wrote: post #88
caymanjim wrote: post #83

This is a neat idea, but I wish it would respect some basic Unix standards by default. Two big annoyances jump out at first glance: it assumes you want to use port 22120, and it puts its config in ~/22120-arc. Maybe both of these are configurable, but the directory is a terrible default. Use XDG (~/.config/22120) or at least use a hidden directory in the home dir. And the port it operates on should be completely conf…

I agree what you say while also pointing out that unix home directory has become a complete mess. Anyone (any installed software) can do whatever they like, there is no mechanism of enforcement, and advice in the form of constructive critque or comment is not even a drop in the bucket towards fixing the problem.

I find that I want to use a project _more_ if they follow best practices like using ~/.config/{app_name}. Attention to details like that usually indicate a higher quality piece of software overall.

Re: Show HN: Self-hosted offline Internet from your browsing history

#94

Post by neolog »

caymanjim wrote: post #83

This is a neat idea, but I wish it would respect some basic Unix standards by default. Two big annoyances jump out at first glance: it assumes you want to use port 22120, and it puts its config in ~/22120-arc. Maybe both of these are configurable, but the directory is a terrible default. Use XDG (~/.config/22120) or at least use a hidden directory in the home dir. And the port it operates on should be completely conf…

> Use XDG (~/.config/22120)

That's only for configuration files. The actual data should go in $XDG_DATA_HOME, by default ~/.local/share/22120/. Many languages have a tool (e.g. https://pypi.org/p/appdirs/ ) for finding the appropriate data/config/cache directory for each OS.

https://wiki.archlinux.org/index.php/XDG_Base_Directory

Re: Show HN: Self-hosted offline Internet from your browsing history

#95

Post by imhoguy »

yamrzou wrote: post #4

This uses Chrome DevTools Protocol in a pretty clever way. I used it to archive a highly interactive website and it worked like a charm. The README states: "It runs connected to a browser, and so is able to access the full-scope of resources (with, currently, the exception of video, audio and websockets, for now)" I wonder what kind of limitations makes it hard to intercept those resources like the rest of the conten…

Video stream is rarely accessed in one request. There may be prelight request to read format structure and of course any seek operation fragments it even more (Accept-Range header). That makes it hard to assemble reslurce for replay later.

Re: Show HN: Self-hosted offline Internet from your browsing history

#96

Post by niea_11 »

severine wrote: post #29
lxgr wrote: post #14

Very interesting project. I wish this was actually (optional) built-in behavior for browsers when bookmarking pages, or at least when adding to a "read later" list like Pocket/Instapaper etc. Pocket seems to offer something like this, but only in the premium version, so the "permanent archive" ironically seems to go away when unsubscribing. As a workaround, what if bookmarking a (public) page could actually ping it t…

I wish this was actually (optional) built-in behavior for browsers when bookmarking pages, or at least when adding to a "read later" list like Pocket/Instapaper etc. Sideshow Ask HN: Didn't Firefox mobile work like this? I could read the reader view items offline... Anyone knows what's happening with the whole bookmarks/collections situation?

I can confirm this. Pages you bookmarked in reader view were saved offline.

I use firefox nightly on android and the feature disappeared (I think) sometime this year when they re-did the whole ui. I can't find an article or bug ticket explaining the reason why.

Re: Show HN: Self-hosted offline Internet from your browsing history

#98

Post by dimtion »

abnry wrote: post #73

I am coding my own hacked together bookmarks manager. I can save any page with a click of the button using SingleFile (a fantastic Chrome extension, by the way!). Then a cronjob runs and puts it into a folder to be processed into a database, which generates a static html index and puts it in my Google Drive. Then it syncs offline on my chromebook. Which means that without internet, I can put my chromebook in tablet m…

You should checkout Shaarli[0] that has a thriving community.

[0] https://github.com/shaarli/Shaarli/

Re: Show HN: Self-hosted offline Internet from your browsing history

#99

Post by radlad »

rusk wrote: post #78
prima-facie wrote: post #33

Isn't this how the internet was supposed to work in the first place? I remember Netscape navigator having a 'go offline' icon in the corner.

Kind of. HTTP was designed with caching in mind, so the idea was that if you GET a page it should more or less not change and you could add headers and stuff to instruct proxy servers about whether to cache or not and for how long. I think you could use HEAD then to check if a page had changed ... The browser cache used to actually be quite dependable as an offline way to view pages but this seems to have fallen out…

> Kind of. HTTP was designed with caching in mind, so the idea was that if you GET a page it should more or less not change and you could add headers and stuff to instruct proxy servers about whether to cache or not and for how long. I think you could use HEAD then to check if a page had changed ...

Nothing in the HTTP spec has changed about this AFAIK. The internet still behaves this way.

> Anyway, you can no longer take this approach because bad people broke the Internet and now you have to have a great honking opaque TLS layer between you and the caching servers so there's no way for this optimisation to work any more.

You could certainly still do this, you just need to import the proxy's CA certificate into your browser. It's just not possible to do to others on your network without consent now. (This is a security feature, not a bug.)

Re: Show HN: Self-hosted offline Internet from your browsing history

#100

Post by ben509 »

segmondy wrote: post #76

I just wanna cache all my bookmarks, I rarely look at them, but when I do go look at them, a good chunk tend to have rotted. It will be awesome to cache all my bookmarks and then have option to recursively cache the path I'm in. I don't want to cache every page I visit, 90% is junk.

I'd recommend using Zotero as it's designed for researchers. It can save most pages for offline viewing, but also grabs metadata, and has specialized downloaders for papers and such.

Main downside is it's not trying to plug into existing bookmarks.

https://www.zotero.org/

Post reply on HN