Live data from Hacker News

Internet Data Is Rotting

theconversation.com

101–110 of 116 posts

Re: Internet Data Is Rotting

#101
post #29

Earlier quoted context omitted.

Best use of time and money is probably http://pinboard.in/ ? Seems like someone I can trust not to sell out and shut down.

But if they die?

...or just run out of money?

Storage must be local, physicslly user-controlled. All online stuff is a backup only.

Re: Internet Data Is Rotting

#102
post #79

Back in the mists of time, I used to use wwwoffle proxy. It was great for low-latency links, but also had the benefit of keeping an offline archive of whatever you'd browsed. Project's still there, although not sure how well it does with the modern web. http://www.gedanken.org.uk/software/wwwoffle/ There are a bunch of more modern variations too: https://archivebox.io/ - "Your own personal internet archive" https://g…

Sadly, a lot of old-school proxies (squid, privoxy) are stymied by SSL/TLS connections. I think we're due for the idea that a proxy can be designated as a trusted intermediary, most especially if it's run on a personal basis. I'm sure this presents security issues, but it also avoids some.

> I think we're due for the idea that a proxy can be designated as a trusted intermediary, most especially if it's run on a personal basis.

We have that idea now; you designate the proxy as a trusted intermediary by accepting its certificate. The chain looks something like this:

    You: browser, take me to https://youtube.com
    Browser: proxy, get me https://youtube.com
    Proxy: YouTube, get me /
    YouTube: I'm youtube.com -- here's a certificate signed
             by the government of Egypt that proves it. And
             here are the contents of /
    Proxy (to browser): I'm youtube.com -- here's a
                        self-signed certificate attesting to
                        that. And here are the contents of /
    Browser (to user): SECURITY ALERT! SECURITY ALERT!
Configure your browser to accept that certificate, and your proxy can handle its own connection to youtube and just pretend, to your browser, that it is youtube.

Re: Internet Data Is Rotting

#103
post #55

Earlier quoted context omitted.

Good thing we are going to die and it won't matter. So let's think about what we can do for the generation that is about to be born.

Can you think of an even remotely feasible solution? As long as we have the technology, I don't see this stopping.

Europe seems to be going in the right way with the GDPR, no?

Re: Internet Data Is Rotting

#104

I think it's good that data is lost . Only items where someone gives enough of a duck to save it should be preserved. It's not as if physical paper content, which ends up recycled or in a landfill 99.999 of the time is any different. It's true that digital formats change but fighting that is the cost of preservation. A museum of software needs to also preserve the context on which software was run in order to save it…

I feel like you've never had to look for information or how to do something, and the only decent source is entirely gone or consists mostly of pictures, which are also mostly gone. Maybe somewhere at some point in time somebody saved it, but that copy gets lost and never makes it's way online again so you can find it. A lot of information is being lost in this way and I'm not sure why we should be fine with that.

Re: Internet Data Is Rotting

#105

I actually have a different problem -- not sure it is one that I can legally solve. I have 10 years of lovingly curated YouTube videos playlists, which, now when I look into the older ones, are a barren wasteland of "Video removed" or "Video not available". It is heartbreaking. Is there any way I can prevent this from happening?

I’d download and store the videos locally with youtube-dl.

I concur, youtube-dl is what I’ve been using during the last few years: whenever I find a YT video I might want to watch again, I now immediately download it. Learned the need for that the hard way.

Check out its options here: https://github.com/ytdl-org/youtube-dl/blob/master/README.md...

With --add-metadata you can embed the YT video description in the video file. The downloaded video file name will contain the YT identifier so you can still match them back if needed.

There is another option to save the metadata to a separate JSON file if you prefer that.

To download your playlists, give it each playlist’s URL instead of the video url:

    youtube-dl --add-metadata --ignore-errors 'https://www.youtube.com/watch?v=8GW6sLrK40k&list=RDQMc4l8l2aQrNo'
That example URL includes a specific video from the list, but will download all of them. It works just the same if you only give it the `list` parameter, but all links to playlists I’ve seen point to one of their videos.

The option --ignore-errors will jump over the unavailable videos instead of stopping.

Edit to add: If you want to download your playlists as separate directories, with each video file name including its original index in the playlist, see these examples in youtube-dl’s documentation: https://github.com/ytdl-org/youtube-dl/blob/master/README.md...

Re: Internet Data Is Rotting

#106
post #16

This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solved at all. The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of…

There used to be a Firefox extension called Scrapbook. I think it was one of the winners of one of the two first extension competitions.

It could easily download entire sites or just a subtree, including resources. It would the rewrite URLs as necessary to point to the downloaded documents instead of pointing to absolute URLs on tje Internet.

Still hoping that Mozilla will improve its extension API. I mean I can see the reason for not allowing everyone to pone around everywhere under the hood, but the current API is rather anemic it seems.

Re: Internet Data Is Rotting

#107
post #101

Earlier quoted context omitted.

But if they die?

...or just run out of money? Storage must be local, physicslly user-controlled. All online stuff is a backup only.

On top of this there is currently a push for a mix of censorship and 'compulsory removal' on the internet. For instance the Canadian Supreme Court ruling Canada can force Google to remove worldwide results for something, the UK censoring increasingly arbitrary things, or various 'right to be forgotten' type laws. And of course there are also intellectual property lobbyists and special interests pushing for ever more draconian rules on that front.

Any centralized online site that aggregates other data has an, at best, unclear future ahead of it. Really though, centralized is the key word. The internet so desperately needs to migrate to decentralization but we're collectively about as inertial as an 882 foot long, 46,328 ton tanker, built in 1909.

Re: Internet Data Is Rotting

#108

Earlier quoted context omitted.

I think if the last 10 years have taught us anything, it's that preserving the past does nothing to impede the changing moral zeitgeist. More records simply mean more people to attack for holding an opinion that has simply gone out of fashion. If the past decade wasn't characterized by tribalism and moral hysteria I'd be more inclined to worry about stringent preservation. At this point, I'm not really comfortable wi…

Though I bet I largely share your feelings, I feel like there is an interesting converse in terms of distant history looking back. Things like HP Lovecraft renouncing racism is only preserved because he wrote so many damn letters. It doesn't make the racist stuff he wrote less racist, but it allows a really clear view on how worldviews could change at the time. Perhaps the things we are storing will have historic or…

I think those are all fair points. I just wonder that now we have the same volume of content (if not quality) from many, many more people. I don't believe there's a lot of time and patience for those people, but the 'landmines' of their unpopular writing is accessible despite this.

Re: Internet Data Is Rotting

#109
post #83
post #16

This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solved at all. The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of…

Maybe I don't understand what you are saying, but I safe webpages all the time. Safari the creates a .webarchive file with everything in it. Isn't that what you want?

No. Because that's extra work. It should just be automatically done whenever I bookmark a page. In fact with disks so big now I could just dedicate a terabyte to just caching everything I visit. Make that indexed and searchable and a lot of what I use would be forever available offline. Pages could expire automatically if unvisited for some period of time, perhaps weighted by the number of times they were visited.

Perhaps I should try to resurrect my rusty and incomplete Firefox add-on skills.

Re: Internet Data Is Rotting

#110
post #101

Earlier quoted context omitted.

But if they die?

...or just run out of money? Storage must be local, physicslly user-controlled. All online stuff is a backup only.

why cant backup be local, user controlled?

revenue looks like hes doing just fine. https://blog.pinboard.in/2017/07/eight_years_of_victory/

https://news.ycombinator.com/user?id=idlewords

Post reply on HN