Live data from Hacker News

Link rot and content drift are endemic to the web

theatlantic.com

211–220 of 220 posts

Re: Link rot and content drift are endemic to the web

#212

Earlier quoted context omitted.

I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.

"But the wayback machine is pretty klunky and slow, honestly." archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes) usage 1: (simple html page of all results) echo https://www.theatlantic.com/article/619320/|1.sh >1.htm firefox ./1.htm usage 2: (retrieve last result) echo https://www…

Edit: Remove Wayback Machine's Javascript inserts (usage #2)

curl version

     #!/bin/sh
     read x0;
     x1=web.archive.org;
     curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original"|case $# in :)
     ;;0)( printf "     $x0
\n";
         sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\1/;s/ 

\n" ) ;;1)x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p)|exec tr -d '[\02\03\36]' \ |exec sed "s|/$y 1/; s//$z$x 2/;s//$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;"; esac
haproxy + nc version

     #!/bin/sh
     read x0;
     x1=web.archive.org;
     printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms 
     \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b 
     \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \
     |exec haproxy -D -f /dev/stdin;
     printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \
     \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80|case $# in :)
     ;;0)( printf "     $x0
\n";
         sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\1/;s/ 

\n" ) ;;1) x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036'); printf "GET %s HTTP/1.1\r\nHost: ${x1}\r\nConnection: close\r\n\r\n" \ $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \ |exec nc -vvn 127.21 80|exec tr -d '[\02\03\36]' \ |exec sed "s|/$y 1/; s//$z$x 2/;s//$y 2/;" \ |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;" esac; if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi

Re: Link rot and content drift are endemic to the web

#213

Earlier quoted context omitted.

Git chain hashes. Merkle trees are useful. They include hashes. They are not simply hashes. Content-addressable storage robust against variations is another approach.

I'm at a loss for your criticism. Didn't the original post say, "How they [hashes] get implemented in solving this problem is the question"? Merkle trees implement cryptographic hashes. >Git chain hashes. Git chains hashes? Yes, of course, along with diffs. How the permanent web will implement digests in various applications is the question. Whatever the answer is for link rot, the guaranteed solution will implement…

You may be trying to read too much into my statement.

It's simply that hashes, alone, do little, and that hash-free solutions might well exist as well.

Bald assertions of simple necessary and sufficient solutions are almost always mistaken.

Re: Link rot and content drift are endemic to the web

#214
post #34

Earlier quoted context omitted.

Maybe the reason it feels impossible to stem the tide of link rot is that it's takes tremendous energy to constantly increase the entropy of a system. And that energy has opportunity cost that no one really wants to talk about. The article has the unstated assumption that eternal preservation of all writing ever is a net benefit. I think it's worth having a discussion on that point.

> Maybe the reason it feels impossible to stem the tide of link rot is that it's takes tremendous energy to constantly increase the entropy of a system. I suppose you meant "decrease"

Yes, thanks. Somehow that's always been backwards in my brain. Maybe I'll remember it this time.

Re: Link rot and content drift are endemic to the web

#215
post #201

Earlier quoted context omitted.

You're not really providing YouTube an alternative to ad-supported model - you're only proving that lots of obnoxious ads can convert people into paying customers.

Yes, and how dare a business desire paying customers.

> we are already paying them with our data

Re: Link rot and content drift are endemic to the web

#216
post #52

Earlier quoted context omitted.

Or ICANN can create policies about domain squatting like GP described.

Or the FDA could put alternative-health scammers in jail. Back in the 1950s they put Wilhem Reich in jail, where he died. L. Ron Hubbard got the hint and left the country and when no country was safe he went to sea. Today people like Dr. Oz run alt-health scams continuously and nobody seems to go to jail or even get a fine.

FDA is American, ICANN operates worldwide

Re: Link rot and content drift are endemic to the web

#217

Visiting an old forum and all the pictures will be gone because, surprise, free image hosting doesn't make economic sense.

The web forum that I frequent most, https://forum.nasaspaceflight.com , has a policy of not allowing embedded images but requiring them to be attached on each post. The forum has been active for well over a decade now (site was founded in 2004) and has a thriving community that continues to grow. It is the community that keeps the forum alive instead of just one random company (although it technically is a company).…

Forums are not anachronistic - they are the future again. The only way to guard against the censorship of the current social media platforms is to not rely on them.

What always baffled me is why forums didn't embrace technologies we had in the BBS days. Offline readers were the greatest thing since sliced bread - you could use whatever interface to a message board you liked, whatever editor you liked, etc. It was a lot easier to quickly scan through literally thousands of messages with a native, local client that relying on the constant ping pong between your client and a remote server.

Decentralized aggregation is what we really need. A combination of RSS and DNS. Ways to foster creation and discovery of hand curated lists like the original Yahoo - but thousands of them. No reliance on Google, Facebook, Twitter, etc. It's a nice dream anyway...

Re: Link rot and content drift are endemic to the web

#218
I'd highlight two arguments which are mentioned in the article but shadowed by main topic of prevention-linkrot-by-archiving (no wonder, as it is written by Perma.cc co-founder):

1. Failure of Lumen/ChillingEffect initiative to prevent bogus takedown requests. Currently, anybody is able to takedown any page on the Internet by sending bogus requests and then takedown any mention about who did it.

2. Google's failure to “organize the world’s information and make it universally accessible and useful”. As the author said: “no such transparent, academic competitive search engine exists in 2021“.

I see some relation between those two.

Re: Link rot and content drift are endemic to the web

#219

yes. i am down to HN and Reddit. Google calendar and telegram. I don't even know how to find cool stuff online anymore. Google SERPs are all business driven now unless you're research news.

I wish there was something like Reddit that could be organized by topic, but had the simplicity of HN's design instead of the monstrocity Reddit has become. My guess is it would still succumb to the Reddit Hive Mind effect without a reasonably benevolent moderation team though. For all the times I've said that HN basically does the same thing, I have to admit that it is much better about keeping it in check.

Checkout my side project: https://upvotocracy.com -- its on github too

Re: Link rot and content drift are endemic to the web

#220
post #34

Earlier quoted context omitted.

Maybe the reason it feels impossible to stem the tide of link rot is that it's takes tremendous energy to constantly increase the entropy of a system. And that energy has opportunity cost that no one really wants to talk about. The article has the unstated assumption that eternal preservation of all writing ever is a net benefit. I think it's worth having a discussion on that point.

> Maybe the reason it feels impossible to stem the tide of link rot is that it's takes tremendous energy to constantly increase the entropy of a system. I suppose you meant "decrease"

No, they mean increase. Decreased entropy would be cheaper to store and maintain.
Post reply on HN