There’s a simple alternative to the current web • Hacker News Board

Re: There’s a simple alternative to the current web

#31

Post by idlewords » Sat, Aug 23, 2014, 6:27 PM UTC

purplerails wrote: post #21

idlewords wrote: post #14

Earlier quoted context omitted.

An early design idea I had for Pinboard was as a browser plugin that just saved everything it saw in passing to an upstream server. But the problem that stumped me was that there's much more downstream bandwidth than upstream on a typical residential connection, so it was hard to push things to a server in anything like real time. How did you end up dealing with this issue?

Pages are saved in the background. Nothing too fancy. I dedup when uploading: that helps a lot.

Okay, but how do you handle things like big PDFs or image gallery sites? Or pages that just pull in a lot of javascript includes? That stuff downloads in parallel, but then I would find myself trying to push it upstream through a little straw of bandwidth, sequentially.

Re: There’s a simple alternative to the current web

#32

Post by wyager » Sat, Aug 23, 2014, 6:27 PM UTC

This would not work with dynamic content.

We already have systems like this (bittorrent, freenet, etc.), and almost no one sees them as a viable replacement for the web because they can't do 99.9% of the things we want (social networks, forums, email, etc.)

Re: There’s a simple alternative to the current web

#33

Post by idlewords » Sat, Aug 23, 2014, 6:30 PM UTC

grmarcil wrote: post #28

I find Bret Victor's comparison between the internet and the LOC a little weird. I've always thought of the internet as a publishing/sharing medium, not an archive. There are plenty of books that go out of print within ten years, we just happen to have infrastructure beyond publishers (libraries) that preserve published copies.

I think it's significant that the Library of Congress has funding, an official mandate, employees, a clear legal status, and stores complete copies of the works in its catalog. A similar model would work great for the Internet (and archive.org is doing its best to fill the role).

Re: There’s a simple alternative to the current web

#34

Post by lingben » Sat, Aug 23, 2014, 6:31 PM UTC

purplerails wrote: post #25

lingben wrote: post #13

Earlier quoted context omitted.

sounds like evernote, if I'm mistaken, please enlighten me :)

1) saving is automatic 2) privacy-first architecture (e.g., plaintext is never uploaded, plaintext URLs are never uploaded etc)

ah, so it saves... everything? wow! that's nuts. I mean, where the heck are you going to store all of that?

and it is terribly inefficient to store say, 30,000 examples of the exact same article. or do you have a way to check and not store duplicates? if so, what if a blog post is saved today and has 10 comments and it is saved tomorrow by another person and it has 11 comments.

technically the page is different so it would be saved again

I think you need to explain exactly how it works a bit better or maybe I'm just not getting it

:)

Re: There’s a simple alternative to the current web

#35

Post by chimeracoder » Sat, Aug 23, 2014, 6:33 PM UTC

grmarcil wrote: post #28

I find Bret Victor's comparison between the internet and the LOC a little weird. I've always thought of the internet as a publishing/sharing medium, not an archive. There are plenty of books that go out of print within ten years, we just happen to have infrastructure beyond publishers (libraries) that preserve published copies.

> I find Bret Victor's comparison between the internet and the LOC a little weird. I've always thought of the internet as a publishing/sharing medium, not an archive.

The Library of Congress is archiving all tweets from the US[0], which I think is what he is referring to.

[0] http://www.businessinsider.com/library-of-congress-is-archiv...

Re: There’s a simple alternative to the current web

#36

Post by unicornporn » Sat, Aug 23, 2014, 6:42 PM UTC

The museums, libraries and cultural heritage institutions are working hard on digital preservation. I think there's a lot to learn from there. Check out this for an instance: http://www.lockss.org/about/how-it-works/

Re: There’s a simple alternative to the current web

#37

Post by [deleted] » Sat, Aug 23, 2014, 7:09 PM UTC

[deleted]

Re: There’s a simple alternative to the current web

#38

Post by azakai » Sat, Aug 23, 2014, 7:10 PM UTC

magila wrote: post #15

The fact that he thinks a federated wiki would be "simple" or "easy" leads me to believe he has not actually thought through the details of how it would work in practice.

Yes. 2 problems that immediately come to mind are

1. Copyright law.

2. Dynamic content.

Re: There’s a simple alternative to the current web

#39

Post by purplerails » Sat, Aug 23, 2014, 7:16 PM UTC

idlewords wrote: post #31

purplerails wrote: post #21

Earlier quoted context omitted.

Pages are saved in the background. Nothing too fancy. I dedup when uploading: that helps a lot.

Okay, but how do you handle things like big PDFs or image gallery sites? Or pages that just pull in a lot of javascript includes? That stuff downloads in parallel, but then I would find myself trying to push it upstream through a little straw of bandwidth, sequentially.

You're right, it takes longer to upload a page than to download it. And image gallery-like pages take long (I know because I save Imgur pages now than then :)). But in practice, this isn't a problem.

There are a couple of heuristics to avoid wastefully uploading pages: the full page is uploaded only if the reader expresses "sufficient interest" in the page. Currently the heuristic is 90 seconds of continuous reading of a page, or scrolling to the bottom. If a page is read for a minimum of 10 continuous seconds then only the text of the page is uploaded.

Static assets like JS files benefit from deduping: they take time to upload the first time, but subsequently processing them is much faster.

Typically, people read multiple pages in a browsing session: I rapidly open many tabs and then read each one for multiple seconds. There's a debug mode in Purple Rails in which a timer counts up when I switch to a tab. I find that typically spend 100+ (usually much more) seconds on a page that I read through to the end. This is usually enough time on a residential broadband connection (I have Sonic DSL) to finish uploading a page. I also use Purple Rails on a tethered 4G connection almost everyday: uploading is slower than DSL but it works.

Basically, by the time you finish reading a tab, the previous tab you read would have finished saving.

Like I said, nothing too fancy.

Re: There’s a simple alternative to the current web

#40

Post by hollerith » Sat, Aug 23, 2014, 7:17 PM UTC

purplerails wrote: post #12

I've been thinking about this for a while now. Please check out my web app to solve this problem: https://www.purplerails.com/ The main idea is to use a browser extension to automatically save pages that you read to the cloud (including the images, stylesheets etc) in the background. Saved pages are searchable and sharable.

I'd prefer for the pages to be saved to the hard drive of the machine running the web browser.

But maybe browser extensions cannot obtain permission to do that?