Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Internet Data Is Rotting

theconversation.com

Post reply on HN
Print view View original

81–90 of 116 posts

Re: Internet Data Is Rotting

#81

Post by dredmorbius »

mnl wrote: post #20

We should aim at browsing the Internet by date . We're moving everything there ignoring the fact that as it is there is no built-in permanence. We're accustomed to that after Gutenberg: it wasn't that easy losing every copy of an important document. Now it is, things disappear and we're getting into a drifting cultural bubble impossible to trace back. The Internet Archive is doing God's work, but it's not enough. If…

"Browsing by date" is pretty much Brewster Kahle's ideal, and is what the Internet Archive's Wayback Machine approximates, thanks in large part to WARC storage.

I'd also like to see a distinction between the idea of web servers, which are really publishers, and where archives are kept. Ideally not all in one single store, a/k/a the Internet Archive, but replicated fairly widely.

Re: Internet Data Is Rotting

#82

Post by dredmorbius »

Among the handy tools that can be used to save and access present, at-risk, and/or rotted data, are bookmarklets.

I've recently added two to my browser, "open in Wayback Machine" and "Save in Wayback Machine". Respectively:

    javascript:void(window.open('https://web.archive.org/web/*/'+location.href));

    javascript:void(window.open('https://web.archive.org/save/'+location.href));
This makes opportunistic archival and reference easy. There are also Wayback Machine / Internet Archive browser extensions.

(These are from the Internet Archive, not my work.)

For bulk archival, lists of URLs can be submitted to the IA's save address:

    https://web.archive.org/save/
(Used in the bookmarklet above as well.)

This can be automated with a simple shell script using any console or script-based HTTP agent, such as curl, wget, lynx, etc.

Re: Internet Data Is Rotting

#83

Post by Torwald »

gambler wrote: post #16

This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solved at all. The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of…

Maybe I don't understand what you are saying, but I safe webpages all the time. Safari the creates a .webarchive file with everything in it. Isn't that what you want?

Re: Internet Data Is Rotting

#84

Post by dredmorbius »

asdff wrote: post #70
fibonachos wrote: post #26

Earlier quoted context omitted.

Tangentially related, but the reader view of a page seems ideally suited for archival bookmarking. It's unfortunate that so many sites seem to be intentionally breaking the reader view functionality in order to maximize newsletter subscriptions and eyeball monetization. Recipe sites are the worst offenders IMO.

Pocket seems to have luck where other reader views and outlining services fail, and you can set up IFTTT to download pocket articles as html files to dropbox, but that's certainly not automatic and completely out of your way. I would have preferred it to archive as a plain text document but I couldn't figure it out through IFTTT.

Are the downloaded pages in original or Pocket/Reader view?

You can export (with some pain) your Pocket list. Fetching sources from there is ... somewhat painful. I've done it a couple of times, but it's not something I relish doing.

Re: Internet Data Is Rotting

#85

Post by dredmorbius »

Causality1 wrote: post #5

I read somewhere that the lifespan of the average hyperlink is only about two years. I count myself lucky I was introduced to the HTTRACK archiver program many years ago and thus have complete offline copies of many of my favorite websites of the early 00's.

According to the Internet Archive, it's closer to 100 days.

https://blogs.loc.gov/thesignal/2011/11/the-average-lifespan...

https://blog.archive.org/2013/10/25/fixing-broken-links/

Re: Internet Data Is Rotting

#86

Post by jancsika »

oceanplexian wrote: post #21
dev_dull wrote: post #13

I’m okay with internet rot and you should be too. I’m not sure where we got the idea that “our data must be preserved forever”. This can be especially harmful for teens and young adults whose indiscretions now follow them forever. Think of the privilege you had when you were younger. You could do something stupid and nobody could whip out a high def camera to record it and make it part of your history forever. Let it…

I'm OK with it because otherwise you are whitewashing history. For example I have recordings of the Colbert report going back to ~2005. Some of his skits released during that time would be classified as "hate speech" in 2019. Of course he, and mainstream broadcasting companies would love it if you didn't think about that. There are plenty of news clips and interviews where mainstream politicians (on Left AND Right) c…

> For example I have recordings of the Colbert report going back to ~2005. Some of his skits released during that time would be classified as "hate speech" in 2019.

Sanity check for 2019:

* "hate speech" should be a link

* when I click it, it should accordion out into links to recordings of Colbert Report that you classified as "hate speech" queued up to the relevant skits

* when I click one it should start streaming the data from all the peers who care about preserving old Colbert Report videos

* Copyright law should have changed a decade ago to protect people who rebroadcasting exact copies of data that the copyright owner/licensee initially publicly broadcast (where "exact" is verified by checking hashes)

Re: Internet Data Is Rotting

#87

Post by Hextinium »

patrick5415 wrote: post #74

Yeah it’s annoying that links get broken. But maybe it’s better this way. There’s something about modern tech that has turned all of us into digital horders. I (we?) have backups and backups of backups and redundant RAID servers with every version of every file so that no byte shall ever perish. I have essays still that I wrote in highschool nearly 20 years ago. To what end? I’m partial to material minimalism. Why no…

My common response is that there are real world costs to having too much stuff, data is pretty small to keep around. You can fit a RAID 5 box with 30 TB of space in a shoebox for $15/month, that is enough to keep pretty much any content that you ever consume so that if you want anything it still exists. My parents and grandparents hoarded files and documents to no end and I have been going through some of them and there is mostly garbage but there is some real gems in the rough. Willingly disposing of the internet through our own negligence is something I don't advocate for because there is some sort of value preserved in what we save for the next generation.

Re: Internet Data Is Rotting

#88

Post by thescriptkiddie »

gambler wrote: post #16

This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solved at all. The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of…

You're describing IPFS pinning.

https://docs.ipfs.io/guides/concepts/pinning/

Re: Internet Data Is Rotting

#89

Post by chessturk »

everdrive wrote: post #30
oceanplexian wrote: post #21

Earlier quoted context omitted.

I'm OK with it because otherwise you are whitewashing history. For example I have recordings of the Colbert report going back to ~2005. Some of his skits released during that time would be classified as "hate speech" in 2019. Of course he, and mainstream broadcasting companies would love it if you didn't think about that. There are plenty of news clips and interviews where mainstream politicians (on Left AND Right) c…

I think if the last 10 years have taught us anything, it's that preserving the past does nothing to impede the changing moral zeitgeist. More records simply mean more people to attack for holding an opinion that has simply gone out of fashion. If the past decade wasn't characterized by tribalism and moral hysteria I'd be more inclined to worry about stringent preservation. At this point, I'm not really comfortable wi…

Though I bet I largely share your feelings, I feel like there is an interesting converse in terms of distant history looking back.

Things like HP Lovecraft renouncing racism is only preserved because he wrote so many damn letters. It doesn't make the racist stuff he wrote less racist, but it allows a really clear view on how worldviews could change at the time. Perhaps the things we are storing will have historic or philosophical value.

I miss real privacy though.

Re: Internet Data Is Rotting

#90

Post by tobias2014 »

gambler wrote: post #16

This needs to be solved on the protocol level. Of course, the players who have control over our protocols are exactly the people who don't want this to be solved at all. The next best thing would be to redefine what "bookmarking" is. When I bookmark a page, I want it to be permanently stored on my local machine and full-text indexed. In fact, it's rather ridiculous that after 25 years browsers don't have anything of…

Use the wayback machine browser plugin allowing you to conveniently use the "save now" function which immediately saves the website to archive.org https://archive.org/web/
Post reply on HN