Given the hard 'no' in the FAQ, does anyone know about a similar project for Firefox?
Could the "HAR" file I can save from Firefox' Network tab somehow be used for this? That looks to be a recording from the entire timeline, including payloads.
Show HN: Self-hosted offline Internet from your browsing history
71–80 of 166 posts
Re: Show HN: Self-hosted offline Internet from your browsing history
#72Earlier quoted context omitted.
You must be young :) These browsers were born in the era of dialup Internet that had per minute charges and/or long distance charges. At the very least you were tying up your family's phone line. Basically it's like paying for every minute your cable modem is plugged in. For the feature itself: Netscape had integration with the modem connectivity for the OS and would initiate a connection when you tried to visit a re…
I ran a BBS, my friend :) I'm quite familiar with modems. I just never used Windows (or the web!) until well past the Netscape era, so I'm not too familiar with the intersection of modems and early web browsers. > Netscape had integration with the modem connectivity for the OS and would initiate a connection when you tried to visit a remote page. That's not "integration with modem connectivity", that's just going thr…
I used to go into offline-mode so the browser would access pages from the cache when I went to their URLs. It wasn’t a ton, but it was enough that you could queue up a handful of sites, go offline and then, if you accidentally closed the tab, re-open it and see the caches version.
Re: Show HN: Self-hosted offline Internet from your browsing history
#73I am coding my own hacked together bookmarks manager. I can save any page with a click of the button using SingleFile (a fantastic Chrome extension, by the way!).
Then a cronjob runs and puts it into a folder to be processed into a database, which generates a static html index and puts it in my Google Drive.
Then it syncs offline on my chromebook. Which means that without internet, I can put my chromebook in tablet mode and do some nice reading. I've been very pleased so far.
Re: Show HN: Self-hosted offline Internet from your browsing history
#74This uses Chrome DevTools Protocol in a pretty clever way. I used it to archive a highly interactive website and it worked like a charm. The README states: "It runs connected to a browser, and so is able to access the full-scope of resources (with, currently, the exception of video, audio and websockets, for now)" I wonder what kind of limitations makes it hard to intercept those resources like the rest of the conten…
Video files are massive, so it may just be the case that archiving videos takes so long they didn't want to support it.
Re: Show HN: Self-hosted offline Internet from your browsing history
#75I am coding my own hacked together bookmarks manager. I can save any page with a click of the button using SingleFile (a fantastic Chrome extension, by the way!). Then a cronjob runs and puts it into a folder to be processed into a database, which generates a static html index and puts it in my Google Drive. Then it syncs offline on my chromebook. Which means that without internet, I can put my chromebook in tablet m…
one day this will be as famous as youtube-dl
Re: Show HN: Self-hosted offline Internet from your browsing history
#76I just wanna cache all my bookmarks, I rarely look at them, but when I do go look at them, a good chunk tend to have rotted. It will be awesome to cache all my bookmarks and then have option to recursively cache the path I'm in. I don't want to cache every page I visit, 90% is junk.
Re: Show HN: Self-hosted offline Internet from your browsing history
#77I just wanna cache all my bookmarks, I rarely look at them, but when I do go look at them, a good chunk tend to have rotted. It will be awesome to cache all my bookmarks and then have option to recursively cache the path I'm in. I don't want to cache every page I visit, 90% is junk.
>have option to recursively cache the path I'm in
It's interesting, what do you mean by that?
Re: Show HN: Self-hosted offline Internet from your browsing history
#78Isn't this how the internet was supposed to work in the first place? I remember Netscape navigator having a 'go offline' icon in the corner.
Kind of. HTTP was designed with caching in mind, so the idea was that if you GET a page it should more or less not change and you could add headers and stuff to instruct proxy servers about whether to cache or not and for how long. I think you could use HEAD then to check if a page had changed ...
The browser cache used to actually be quite dependable as an offline way to view pages but this seems to have fallen out of favour in the mid naughties. I remember how disgusted I was when I realised Safari was no longer me letting see a page unless it could contact the server and download the latest version.
I used to have a caching proxy server that would basically MITM my browsing and be more vigilant than even the cache and it really worked quite well. This was back in the 90s when every bit of your max 54kbs counted, or when you wanted to read something while your Dad or sister wanted to also use the phone.
Anyway, you can no longer take this approach because bad people broke the Internet and now you have to have a great honking opaque TLS layer between you and the caching servers so there's no way for this optimisation to work any more.
Of course it isn't really as important these days because we've got faster connections and interactions with the server are far less transactional and richer. But I still would like to have a way of tracking my own webusage and being able to go back in time without having to actually revisit each and every site.
These days you have to hack the browser because that's where your TLS endpoint emerges. Kaspersky tried this for their HTTP firewall application and there was ructions over that.
I'll defo take a look at this. Sounds just like what I've been looking for.
> Isn't this how the internet was supposed to work in the first place? I remember Netscape navigator having a 'go offline' icon in the corner.
Thinking back actually, if you forget about "the web"/HTTP - then yes actually - this is exactly how usenet worked and now I'm remembering that the "go offline" button used to download all your newsgroups along with your email and stuff so you could look at it all offline :-)
If you want something that's like Usenet these days check out Scuttlebut.
Re: Show HN: Self-hosted offline Internet from your browsing history
#79Isn't this how the internet was supposed to work in the first place? I remember Netscape navigator having a 'go offline' icon in the corner.
I remember that button, too, but I think it had more to do with connection charges than caching. In Netscape days, many people would have to pay by the minute to be connected to the internet. In those days, web pages generally contained far more information than they do now, and were less interactive. So you'd connect, load the content you wanted to see, disconnect, and then just sit there and read it for free, inste…
Back in the nineties the web was fairly new and people still used a thing called usenet quite a bit. You interacted with it kind of like email (Google groups is actually the final vestiges of it) - and the go offline button would just download all your emails and newsgroups and you got peruse them offline at your pleasure. It might seem strange also that back in those days you downloaded your emails from a server using POP3 rather than looking at them remotely (e.g. Web or IMAP), and you viewed them offline.
Re: Show HN: Self-hosted offline Internet from your browsing history
#80It looks like something I'd appreciate! I make a significant effort to archive things that I think I'll need.
Unfortunately it didn't work when I just tried installing it now (macOS 10.13.6, node v14.8.0).
MacBook-Pro:Desktop peter$ npx archivist1
npx: installed 79 in 8.282s
Preferences file does not exist. Creating one...
Args usage:
Updating base path from undefined to /Users/peter...
Archive directory (/Users/peter/22120-arc/public/library) does not exist, creating...
Created.
Cache file does not exist, creating...
Created!
Index file does not exist, creating...
Created!
Base path updated to: /Users/peter. Saving to preferences...
Saved!
Running in node...
Importing dependencies...
Attempting to shut running chrome...
There was no running chrome.
Removing 22120's existing temporary browser cache if it exists...
Launching library server...
Library server started.
Waiting 1 second...
{"server_up":{"upAt":"2020-11-11T21:48:25.324Z","port":22120}}
Launching chrome...
(node:33988) UnhandledPromiseRejectionWarning: Error: connect ECONNREFUSED 127.0.0.1:9222
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1144:16)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:33988) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:33988) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:33988) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'writeFileSync' of undefined
at ae (/Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:321:14209)
at Object.changeMode (/Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:321:8088)
at /Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:321:16174
at s.handle_request (/Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:128:783)
at s (/Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:121:879)
at p.dispatch (/Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:121:901)
at s.handle_request (/Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:128:783)
at /Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:114:2533
at Function.v.process_params (/Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:114:3436)
at b (/Users/peter/.npm/_npx/33988/lib/node_modules/archivist1/22120.js:114:2476)
(node:33988) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 3)
^CCleanup called on reason: SIGINT
MacBook-Pro:Desktop peter$