Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There’s a simple alternative to the current web

hapgood.us

Post reply on HN
Print view View original

101–110 of 148 posts

Re: There’s a simple alternative to the current web

#101

Post by thegeomaster »

hnha wrote: post #83
sroerick wrote: post #70

Earlier quoted context omitted.

Slightly related to this, the other day I tried to download every youtube video in my watch history. Turns out it's pretty much impossible. The Youtube API only delivers about 20 results, which is a bug that has existed for about 2 years. I tried manually loading the watch history page, and was only able to get about 1000 out of ~8000 results. When I selected those 1000 results and tried to add them to a playlist, th…

It baffles me how we got from "tracking is creepy" to "please track all I do on your site" in ~10 years. Your watch history should be stored by your browser on your machine. It is only your business and only locally it is fully in your control.

A week ago, I had my phone forcefully smashed into a floor. The screen ended up completely dead, but fortunately, the phone worked, and the remote debugging option was on, so I was able to pull all the data on it via ADB.

But if the motherboard or some other vital had component died, I would've beem locked out of my data. It's at times like this I'm glad that Google Keep syncs my notes to their servers, and that Viber stores my contacts' phone numbers remotely. If it hadn't been for them and my phone, say, fell in water and completely broke, I'd have lost all of that. Same argument for a hard drive death. In 2004 or so, we were expected to live with that risk. But I don't think we should have to now. Witnessing the explosion of the "cloud" buzzword everywhere, you'd think everyone in the world has convenient access to their own private space "in the cloud" (or, as those soooo-2000s people would say, "on the Internet"). And we do have access to such space, but for the most part, not quite so for the "private" thing.

Of course, ideally, I'd like to generate random keys, keep safe local backups of them and then sync encrypted data to remote servers. But I don't think the companies are eager to accept that as they probably owe a lot of their statistics and targeted advertising opportunities to big-scale mining of that plaintext data we provide them in exchange for convenience.

Re: There’s a simple alternative to the current web

#102

Post by ottonomy »

twoodfin wrote: post #53

I'd love it if browsers natively supported URI's derived from cryptographic hashes of content by looking them up in a distributed store a la BitTorrent. Imagine if Chrome supported such a thing, for example. Perfectly reliable cache-ability (or archive-ability), P2P hosting, ... All the good stuff for any web content that its creator wants to so expose, albeit at the price of immutability.

Especially under the current copyright regime, finding some solution that preserves the intent of the creator to publish in a fixed format, would be a great component of a distributed publishing system. I don't think this proposal has as good a fair use defense as the Internet Archive wayback machine does.

In the world today, we often think of publishing online as providing access to something under our control. I think a technology that aims to solve these problems should embody a different spirit, one closer to "making public". The word "mine" doesn't need to imply ownership in the sense of exclusive control. I mean, "My children" is at least as meaningful a relationship as "my property". Some kind of copyright license ability built into a distributed document publishing system would be nice.

Re: There’s a simple alternative to the current web

#103

Post by walterbell »

rando289 wrote: post #16
idlewords wrote: post #6

Clearly I'm a biased observer, but I really think people should take steps to archive stuff that is important to them. Of course it's terrible when large sites go offline and take vast swaths of the Internet with them, and we should continue to shame the ones that do it. At the same time, if something is really important to you, you shouldn't store it in the form of links to random third-party servers. One problem we…

It always seemed like such a regression to me when browsers disabled entire page caches and only ever show the freshest content. Before, if a page served 404, I could easily see my local cached version from my previous visit. Now, I'm shit out of luck.

What is amazing is that:

1) Never in the history of civilization has local knowledge storage (disk) and local compute been so cheap

2) Never have we had a larger free software ecosystem or more hackers to deploy free software locally

3) Never have we had more evidence about the differences in civilizational freedom between local and central storage/compute

What if backup/restore skills were taught alongside home economics? Is a home a place of both shelter & storage? Are we abandoning a set of possible futures because we want the "convenience" of "someone else" backing up our digital selves/souls?

"For want of a nail the shoe was lost.

For want of a shoe the horse was lost.

For want of a horse the rider was lost.

For want of a rider the battle was lost.

For want of a battle the kingdom was lost.

And all for the want of a horseshoe nail."

Re: There’s a simple alternative to the current web

#104

Post by idlewords »

sroerick wrote: post #86
lutusp wrote: post #78

Earlier quoted context omitted.

It clearly hasn't occurred to you that private collections evaporate over time, just like public ones. > I would like something that not only archives pages I visit, but also versions them and tracks changes. That's a huge storage requirement, you must realize this. If you're an avid Web browser, and if every archived page had to look as it originally looked (i.e. all the linked resources) you could accumulate severa…

I never said I was trying to save my entire web cache locally. Just bookmarks. I don't have Flash Player installed. Instead, I use youtube-dl for flash videos across many websites. I'm well aware of the storage implications of this kind of activity.

The storage implications are really not so bad. You would have trouble ever breaking 100 GB unless you are some kind of bookmarking mutant.

Re: There’s a simple alternative to the current web

#105

Post by idlewords »

Houshalter wrote: post #99

Link rot is a serious problem: http://www.gwern.net/Archiving%20URLs#link-rot >In a 2003 experiment, Fetterly et al. discovered that about one link out of every 200 disappeared each week from the Internet. McCown et al. (2005) discovered that half of the URLs cited in D-Lib Magazine articles were no longer accessible 10 years after publication [the irony!], and other studies have shown link rot in academic literature…

My own research (which I hope to publish soon) shows a slightly better link rot rate for bookmarked URLs (which are presumably ones people are most interested in keeping). The attrition rate I see so far is roughly linear and about 5% a year. Which is still shocking by any non-web standard, but a little better than the figures cited above.

Re: There’s a simple alternative to the current web

#106

Post by sillysaurus3 »

idlewords wrote: post #105
Houshalter wrote: post #99

Link rot is a serious problem: http://www.gwern.net/Archiving%20URLs#link-rot >In a 2003 experiment, Fetterly et al. discovered that about one link out of every 200 disappeared each week from the Internet. McCown et al. (2005) discovered that half of the URLs cited in D-Lib Magazine articles were no longer accessible 10 years after publication [the irony!], and other studies have shown link rot in academic literature…

My own research (which I hope to publish soon) shows a slightly better link rot rate for bookmarked URLs (which are presumably ones people are most interested in keeping). The attrition rate I see so far is roughly linear and about 5% a year. Which is still shocking by any non-web standard, but a little better than the figures cited above.

You do research using bookmarks on Pinboard as your dataset? May I ask how this data is used and disclosed to others?

Re: There’s a simple alternative to the current web

#107

Post by idlewords »

sillysaurus3 wrote: post #106
idlewords wrote: post #105

Earlier quoted context omitted.

My own research (which I hope to publish soon) shows a slightly better link rot rate for bookmarked URLs (which are presumably ones people are most interested in keeping). The attrition rate I see so far is roughly linear and about 5% a year. Which is still shocking by any non-web standard, but a little better than the figures cited above.

You do research using bookmarks on Pinboard as your dataset? May I ask how this data is used and disclosed to others?

https://blog.pinboard.in/2014/08/researching_link_rot/

Re: There’s a simple alternative to the current web

#108

Post by walterbell »

sroerick wrote: post #73
idlewords wrote: post #6

Clearly I'm a biased observer, but I really think people should take steps to archive stuff that is important to them. Of course it's terrible when large sites go offline and take vast swaths of the Internet with them, and we should continue to shame the ones that do it. At the same time, if something is really important to you, you shouldn't store it in the form of links to random third-party servers. One problem we…

My second reply, but, I think that this is really important. We only need to look at early film history to know how easy it is to lose massive parts of our history. Going back to old pages, I frequently get 404 results. For politically sensitive documents, the problem is much more widespread. I would like something that not only archives pages I visit, but also versions them and tracks changes. If there was a bookmar…

What do you think of this generalized architecture?

HARDWARE: differs depending on whether you want local search/analytics or just network storage.

For mobile use, either a VPN back to your personal home/cloud server, or a hackable wifi hard drive proxy, e.g. Seagate Wirless Plus + HackGFS.

For non-analytics home use, hackable router with USB3 storage and Linux software RAID, connected to a USB3 drive chassis with room for 2-4 disks.

For analytics home use, a microserver like HP N54L, Dell T20 or Lenovo TS140. Up to Xeon processor with ECC memory, plus 4-6 internal disks and up to 32GB RAM. Sold without a Windows tax, supports hardware virtualization and Linux. Possibly FreeBSD with ZFS.

SOFTWARE: generalized multi-tier cache AND compute. Camlistore and git-annex are tackling multi-device storage sync. For archives, we need a search interface that will query a series of caches, e.g. mobile > home > trusted friends private VPN (tinc overlay) > public paid cloud archive (pinboard et al) > public free cloud archive (archive.org).

It's important for usability to have a simple, local UX that will take a search string, propagate across all private/public federated tiers of storage and compute, then aggregate the metasearch results on the client.

With this approach, we can collectively pool resoures to improve on CommonCrawl.org, without locking up the 300TB index at AWS. This would turn web search engines into a secondary source, rather than a primary source. First search your archive + trusted friends, then trusted verticals (e.g. HN, StackOverflow), then a generic web search.

Let's be clear: the goal is not to archive "everything" in the world, only that which is personally important to the viewer. This attention metadata has long-term value. With this architecture, it is always optional to escalate a query to a public archive or search engine. Most importantly, there is technical autonomy and low-latency compute for local queries.

For web pages, wget of WARC formats (per HN advice on another thread) and wkhtmltopdf (available as Firefox plugin to print to PDF) will keep local archives. Recoll.org (xapian front-end with user-customizable python filters) on Linux will search full text and provide preview snippets, or lucene/solr can be adapted.

Re: There’s a simple alternative to the current web

#109

Post by sillysaurus3 »

idlewords wrote: post #107
sillysaurus3 wrote: post #106

Earlier quoted context omitted.

You do research using bookmarks on Pinboard as your dataset? May I ask how this data is used and disclosed to others?

https://blog.pinboard.in/2014/08/researching_link_rot/

To run the experiment, I am going to be drawing a few thousand links at random from the entire pool of Pinboard bookmarks. This will include private bookmarks, which make up about half the Pinboard collection.

You chose to include everyone's private bookmarks in your research without asking their consent? What?

I will publish some aggregate information about what I find, and use it to seek glory, and persuade people to sign up for archiving. But I won't release anything that could lead back to specific users or links.

There is roughly a boatload of evidence that anonymized datasets can be deanonymized in unexpected ways.

Even if you don't release any anonymized datasets, it's really not good that you decided to take such liberties with people's private links in the first place.

Re: There’s a simple alternative to the current web

#110

Post by idlewords »

sillysaurus3 wrote: post #109
idlewords wrote: post #107

Earlier quoted context omitted.

https://blog.pinboard.in/2014/08/researching_link_rot/

To run the experiment, I am going to be drawing a few thousand links at random from the entire pool of Pinboard bookmarks. This will include private bookmarks, which make up about half the Pinboard collection. You chose to include everyone's private bookmarks in your research without asking their consent? What? I will publish some aggregate information about what I find, and use it to seek glory, and persuade people…

Why would I need consent to study the global link rot rate? Publishing it reveals no information about users, either individually or in the aggregate.

I've made an effort to let anyone who wants to opt out of the research, because I know people can have strong feelings about privacy.

I agree with you that publishing an 'anonymized' dataset would be a violation of privacy guarantees. I wouldn't even do it for public bookmarks.

Post reply on HN