Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Link rot and content drift are endemic to the web

theatlantic.com

Post reply on HN
Print view View original

161–170 of 220 posts

Re: Link rot and content drift are endemic to the web

#161

Post by NoboruWataya »

majormajor wrote: post #56
wintermutestwin wrote: post #47

I am increasingly worried about the valuable content on YouTube. There are so many old live concerts, useful how-to videos and other cultural treasures amidst all the junk. I suspect that one day, they will make their ads unblockable by embedding them in the video files. I sure hope that some people are downloading the valuable stuff and stashing it away to load onto YouTube's successor.

How many how to videos, memes, concerts, etc, are really that important? In fifty years how many people will care? How many people should care because it would mean ignoring the huge volume of newer stuff? A hundred years? Two hundred? I haven't even read or seen many of the existing cultural artifacts we have from past decades and centuries, what would I do if orders of magnitudes more of them had been preserved? In…

Each individual meme or video might not be important, but then I don't think future historians are going to spend much time studying individual artifacts in detail in the same way that current historians do. We live in the age of big data and I think future historians will be focused on aggregating and automatically analysing that data. They probably won't be reading your comment or mine but they might analyse large sets of HN comments with a view to drawing conclusions about how particular demographics act, think and feel today. And if large swathes of those comments are lost the conclusions will be skewed, particularly if the loss is not random.

Our society is characterised by the constant generation and exchange of massive amounts of information. It's one of the things that sets us apart from previous generations. Preserving only a small subset of that data that we deem worthy or important will not allow future generations to fully understand today's society.

Re: Link rot and content drift are endemic to the web

#162

Post by azinman2 »

a1369209993 wrote: post #152
wintermutestwin wrote: post #47

I am increasingly worried about the valuable content on YouTube. There are so many old live concerts, useful how-to videos and other cultural treasures amidst all the junk. I suspect that one day, they will make their ads unblockable by embedding them in the video files. I sure hope that some people are downloading the valuable stuff and stashing it away to load onto YouTube's successor.

> that I should just pay a subscription fee to avoid their ad crap Fuck no. Do not do this . Use youtube-dl[0], and maintain local copies of anything useful you can find. 0: http://youtube-dl.org/

I’m happy to give YouTube my money to provide a real alternative to ad-supported models. Plus the vast majority of videos I watch id never want to see again.

Re: Link rot and content drift are endemic to the web

#163

Post by 8bitsrule »

Libraries (like Internet Archive for one) seem like the logical 'permalink' curators of information collections. I'm thinking, for example, of a large regional history site created by a a 501(c)(3) corporation. [https://www.historylink.org/]

Such efforts deserve a guaranteed permanent home with permanent funding. Another example: hundreds of 'Old-time-radio' programs and early TV shows have only emerged and survived because of enthusiasts. If they disappeared from Youtube ....

It'd be good to set up a universal 'permalink' library system (UPLS) like the DOI system and its 'persistent interoperable identifiers'. This could (and, at least in part, must) be publicly-funded. Anyone who wishes could apply for a unique ID. Then, subject to a set of specs from some participating library ("yes, we'll host that"), they could package that content with metadata. Backups must be ensured. If eventually the content needs to move (or can't find a home), the ID (and metadata) remains.

Re: Link rot and content drift are endemic to the web

#164

Post by 1vuio0pswjnm7 »

throwawayboise wrote: post #85
tda wrote: post #69

Earlier quoted context omitted.

On the bright side, using a tool like Internet archive it should be easy to filter out which articles were removed and/or edited by the BBC, in a way highlighting the most historically important articles.

I mean yes, if you are extremely motivated. But the wayback machine is pretty klunky and slow, honestly. And there's no good "diff" view that summarizes the changes to a URL over time AFAIK.

"But the wayback machine is pretty klunky and slow, honestly."

archive.org is much less annoying if one avoids using a bloated browser and Javascript to make HTTP requests

here is a more lightweight approach, not nearly as klunky/slow, IMO (393 bytes)

    usage 1: (simple html page of all results)
    echo https://www.theatlantic.com/article/619320/|1.sh >1.htm
    firefox ./1.htm

    usage 2: (retrieve last result)
    echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm
    firefox ./2.htm

    #!/bin/sh 
    read x0;
    x1=web.archive.org;
    curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \
    |case $# in :)
    ;;0)( printf "     $x0
\n";
        sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\1/;s/ 

\n" )
    ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p);
    esac
haproxy + nc version (965 bytes)

maybe it is faster than curl, maybe not; you be the judge

    #!/bin/sh
    read x0;
    x1=web.archive.org;
    printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms 
    \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b 
    \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \
    |exec haproxy -D -f /dev/stdin;
    printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \
    \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \
    |case $# in :)
    ;;0)( printf "     $x0
\n";
        sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\1/;s/ 

\n" )
    ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \
         $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \
         |exec nc -vvn 127.21 80;
    esac;
    if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi

Re: Link rot and content drift are endemic to the web

#165

Post by unethical_ban »

ergot_vacation wrote: post #59
kylestlb wrote: post #30

It's an Atlantic article so it's long, and several of the comments here show that people aren't actually reading the whole thing... but I did and it's worth the time. It's not only about links being dead, it's about the lack of transparency & audit when content is changed via takedown requests, it's about dead links showing up in decades-old supreme court decisions, it's about private industry's lack of incentive for…

Having read the whole thing, it seems partially good, martially misguided, and partially terrible. The overall bent is a hand-wringing about link rot, which I thought we mostly got over a decade ago. The Internet is fundamentally ephemeral. If you see something you like, save it so you can repost it later. If you rely on someone else to keep in up indefinitely, you're being foolish. Around the edges of that main disc…

>The overall bent is a hand-wringing about link rot, which I thought we mostly got over a decade ago. The Internet is fundamentally ephemeral. If you see something you like, save it so you can repost it later. If you rely on someone else to keep in up indefinitely, you're being foolish.

Did we? Should we?

Acknowledging the current state of affairs doesn't require accepting its flawed nature.

Imagine a world where gasp the BBC, NYTimes, etc. kept all versions of their articles available and online. Where the "pretty URL" shows the most recent version of a page, provides a permalink, and provides permalinks to all previous versions of a page.

I don't expect most sites to do this, but since someone else mentioned the BBC, I am targeting journalism as an example.

Re: Link rot and content drift are endemic to the web

#166

Post by crazygringo »

We need to accept that link rot and content drift are part of the web.

And realize that the best place to preserve history is the Internet Archive' Wayback Machine.

Kind of the same way newspapers were never responsible for maintaining their archives, but librarians did on microfiche (remember that?).

But I'd take it farther.

First, the Internet Archive ought to have an official partnership with the Library of Congress and other national libraries across the world, that help provide funding. It shouldn't have to rely on private donations.

And second, it's time browsers integrated with it -- if content no longer exists there should be a built-in option to easily check Wayback Machine with a single click, and use a heuristic to show the most recent "good" version.

In other words, let the Wayback Machine be not just a, but the place for the Internet's history. Let's make it official.

Re: Link rot and content drift are endemic to the web

#168

Post by floren »

robotnikman wrote: post #55
wintermutestwin wrote: post #47

I am increasingly worried about the valuable content on YouTube. There are so many old live concerts, useful how-to videos and other cultural treasures amidst all the junk. I suspect that one day, they will make their ads unblockable by embedding them in the video files. I sure hope that some people are downloading the valuable stuff and stashing it away to load onto YouTube's successor.

I set up a server specifically for downloading Youtube videos from all my playlists on a daily basis just for this reason. At some point I got fed up at seeing all the missing videos on my playlists (and not even knowing what was removed)

What do you use for automation? My first instinct is cron jobs running youtube-dl scripts against a set of playlist URLs, but I'd be interested to hear more.

Re: Link rot and content drift are endemic to the web

#169

Post by yesenadam »

jandrese wrote: post #145
CodeGlitch wrote: post #94

Earlier quoted context omitted.

Reading one of the BBC's technical articles, a cyber security news item, they had 3 errors in the first paragraph. I didn't bother reading to the end of the article. I'm glad I no longer pay for a TV license.

It has always been a constant of journalism that you read an article in your field and go "Wow, this is terrible, they got all of the details wrong". But then you turn around and trust the reporting on everything outside of your field of expertise.

https://en.wikipedia.org/wiki/Michael_Crichton#GellMannAmnes...

Re: Link rot and content drift are endemic to the web

#170

Post by brodouevencode »

makomk wrote: post #63

It's amazing how little some trusted institutions care about this. For example, the BBC has been bragging about how many people rely on their coverage of the pandemic, but have an obnoxious habit of repeatedly overwriting old articles with new ones on similar topics and not keeping the old versions available. The history of a once-in-a-century pandemic with huge local and global impacts is literally being overwritten…

How much of these overwrites are cover ups for failures? Certainly some percentage, just not sure what.
Post reply on HN