Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: t.co is adding a five-second delay to some domains

news.ycombinator.com

Post reply on HN
Print view View original

311–320 of 447 posts

Re: Tell HN: t.co is adding a five-second delay to some domains

#311

Post by dredmorbius »

gouggoug wrote: post #139
mutant_glofish wrote: post #78

Earlier quoted context omitted.

> We probably banned it for submissions because we want original sources at the top level. Then why web.archive.org isn't also banned? [1] And what about things which aren't available from the original source anymore? [1]: https://news.ycombinator.com/item?id=37130420

> Then why web.archive.org isn't also banned? Because web.archive.org is generally used for... ... things which aren't available from the original source anymore. While archive.is is generally used to bypass paywalls. These 2 websites have 2 very distinct missions and use-cases.

Whilst I agree with your characterisation as regards usage on HN, I will note that Archive Today actually is a quite useful archival tool, and often works on sites which the Internet Archive behaves poorly on.

I'd run across an instance of this when the Diaspora* pod I was on (the original public node, as it happens) ceased operations. I found myself wanting to archive my own posts, and was caught in something of a dilemma:

- The Internet Archive's Wayback Machine has a highly-scriptable method for submitting sites, in the form of a URL (see below). Once you have a list of pages you want to archive, you can chunk through those using your scripting tool of choice (for me, bash, and curl or wget typically). But it doesn't capture the comments on Diaspora* discussions.... E.g., https://web.archive.org/web/20220111031247/https://joindiasp...>

- Archive.Today does not have a mass-submission tool, and somewhat aggressively imposes CAPTCHAs at times. So the remaining option is manual submissions, though those can be run off a pre-generated list of URLs which somewhat streamlines the process. And it does capture the comments. E.g., https://archive.is/9t61g>

So, if you are looking to archive material, Archive Today is useful, if somewhat tedious at bulk.

(Which is probably why the Internet Archive is the far more comprehensive Web archive.)

Re: Tell HN: t.co is adding a five-second delay to some domains

#312

Post by mzs »

lapcat wrote: post #298
mzs wrote: post #297

Earlier quoted context omitted.

I don't see it with your URL either: % curl -gsSIw'foo %{time_total}\n' https://t.co/DzIiCFp7Ti | grep '^\(HTTP/\)\|\(location: \)\|\(foo \)' HTTP/2 301 location: https://www.threads.net/@chaco_mmm_room foo 0.123137 Doesn't matter if I do a HTTP/2 HEAD or GET: % curl -gsSw'%{time_total}\n' https://t.co/DzIiCFp7Ti 0.121503 HTTP/1.1 also shows no delay: % curl -gsSw'%{time_total}\n' --http1.1 https://t.co/DzIiCFp7Ti 0.…

> I don't see it with your URL either That's because you're not spoofing the User-Agent to be a browser rather than curl.

Oh that's it, thanks! In fact it it returns a 200 not 301 then:

  % curl -gsSw'%{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/DzIiCFp7Ti
  https://www.threads.net/@chaco_mmm_roomwindow.opener = null; location.replace("https:\/\/www.threads.net\/@chaco_mmm_room")4.690000
  % curl -gsSIw'%{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/DzIiCFp7Ti
  HTTP/2 200 
  ...
  content-length: 272
  ...
  x-response-time: 4524
  ...
  
  4.660211
The delay is not there for nyti.ms (anymore) but once you use the Safari UA it's handled as 200 response:

  % curl -gsSIw'foo %{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/4fs609qwWt https://t.co/iigzas6QBx | grep '^\(HTTP/\)\|\(location: \)\|\(foo \)'
  HTTP/2 200 
  foo 0.126043
  HTTP/2 200 
  foo 0.037255
It really does seem that twitter is adding a 4.5s delay to some sites from web browsers. Could be malicious, could be rot...

Re: Tell HN: t.co is adding a five-second delay to some domains

#313

Post by wakamex »

djvdq wrote: post #211
_a9 wrote: post #46

Earlier quoted context omitted.

Im not getting the same time delay with curl - `time wget https://t.co/4fs609qwWt ` -> `0m5.389s` - `time curl -L https://t.co/4fs609qwWt ` -> `0m1.158s`

And now add browser user-agent to the curl request and watch how slow it gets. - `time curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -L https://t.co/4fs609qwW ` -> 4.730 total - `time curl -L https://t.co/4fs609qwWt ` -> 1.313 total Same request, the only difference is user-agent.

your URLs are different.

Re: Tell HN: t.co is adding a five-second delay to some domains

#314

Post by mzs »

flutas wrote: post #310
mzs wrote: post #307

Earlier quoted context omitted.

No, in fact now t.co even returns an empty body with it's 301 response: % curl -vgsSw' &1 | grep '^

Firefox: https://www.threads.net/@chaco_mmm_room window.opener = null; location.replace("https:\/\/www.threads.net\/@chaco_mmm_room")

Turns-out it depends on the User-Agent: https://news.ycombinator.com/item?id=37139425

Re: Tell HN: t.co is adding a five-second delay to some domains

#315

Post by xg15 »

blowski wrote: post #37
88913527 wrote: post #5

The speed at which enshitification is being unleashed surprises me each and every day.

Enshitification is different. It’s when companies destroy a product with hundreds of changes that prioritise internal politics above what end users want. This is something else - just the ego of one rich guy petulantly satisfying his inner demons.

> just the ego of one rich guy petulantly satisfying his inner demons.

As that rich guy happens to be the CEO, how is this not the prime example of "prioritising internal politics above what end users want"?

Re: Tell HN: t.co is adding a five-second delay to some domains

#316

Post by lapcat »

PenguinCoder wrote: post #276
ender7 wrote: post #264

Earlier quoted context omitted.

I can replicate this behavior fairly easily in a browser. 1. Open incognito window in Chrome 2. Visit https://t.co/4fs609qwWt -> 5s delay 3. Open a second tab in the same window -> no delay 4. Close window, start a new incognito session 5. Visit https://t.co/4fs609qwWt -> 5s delay returns

What is that attempting to prove or replicate? Here's a simpler test I think replicates what I am indicating in GP comment, with regards to cookie handling: Not passing a cookie to the next stage; pure GET request: $ time curl -s -A "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" -e ";auto" -L https://t.co/4fs609qwWt > nocookie.html real 0m4.916s user 0m0.016s sys 0m0.018s Using `-b` to pass…

You're completely missing the point, which is that the 5 second delay doesn't exist at all for most t.co links, even without cookies. The delay only exists for a few Musk-hated domains.

Re: Tell HN: t.co is adding a five-second delay to some domains

#317

Post by dang »

rpruiz wrote: post #227
dang wrote: post #20

Earlier quoted context omitted.

Well, yes, many sites are banned on HN. Others are penalized (see e.g. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... ). None of this is secret, though we don't publish the lists themselves. Edit: about 67k sites are banned on HN. Here's a random selection of 10 of them: vodlockertv.com biggboss.org infoocode.com newyorkpersonalinjuryattorneyblog.com moringajuice.wordpress.com surrogacymumbai.com max…

So, is there an algorithm to be features in the front page? —other than upvotes. If a site can be banned, can another one be promoted?

Sorry, but I don't understand your question.

Re: Tell HN: t.co is adding a five-second delay to some domains

#318

Post by dang »

amadeuspagel wrote: post #218
dang wrote: post #20

Earlier quoted context omitted.

Well, yes, many sites are banned on HN. Others are penalized (see e.g. https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... ). None of this is secret, though we don't publish the lists themselves. Edit: about 67k sites are banned on HN. Here's a random selection of 10 of them: vodlockertv.com biggboss.org infoocode.com newyorkpersonalinjuryattorneyblog.com moringajuice.wordpress.com surrogacymumbai.com max…

Maybe "major media" should include tech media like The Register, Ars Technica, Tech Dirt, etc.. Unlike with media like the NYT, Bloomberg or Reuters, I've never seen a story for which these sites were the best source and much of what they publish is blogspam summarizing stories that have already been posted on HN, usually with a votebait title.

Yes, those sites are all downweighted. Whether they count as "major media" or not, they're classified the same way by HN's software, for more or less the same reason: they produce a lot of derivative and/or sensational and/or otherwise not-great-for-HN content, and they also produce substantive articles that are good for HN.

Re: Tell HN: t.co is adding a five-second delay to some domains

#319

Post by PenguinCoder »

mzs wrote: post #301
PenguinCoder wrote: post #236

Additional details I wrangled for this rabbit hole. I don't think it's t.co doing this intentionally, but rather poor handling of 'do you have our cookies or not'. Everyone in this thread _proving things_ without taking into account the complexity of the modern web. man curl -b, --cookie (HTTP) Pass the data to the HTTP server in the Cookie header. It is supposedly the data previously received from the server in a "S…

oh boy... -b takes an option which in your examples is -A and -e, then what follows is interpreted as a URL and you throw away the warnings: % curl -vgsSIw'> %{time_total}\n' -b -A "curl/8.2.1" https://t.co/DzIiCFp7Ti 2>&1 | grep '^\(* WARNING: \)\|\(Could not resolve host: \)\|>' * WARNING: failed to open cookie file "-A" * Could not resolve host: curl curl: (6) Could not resolve host: curl * WARNING: failed to open…

Alright thanks for explaining that . Here's what I see explicitly setting the cookiejar

    $ time curl -s -b cookies.txt -c cookies.txt -A "Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0" -e ";auto" -L https://t.co/DzIiCFp7Ti

    [t.co meta refresh page src]

    real     0m4.635s
    user   0m0.004s
    sys     0m0.008s

    $ time curl -b cookies.txt -c cookies.txt -A "wget/1.23" -e ";auto" -L https://t.co/DzIiCFp7Ti                        curl: (7)
    Failed to connect to www.threads.net port 443:  Connection refused
    real     0m4.635s
    user   0m0.011s
    sys     0m0.005s

    $ time curl -b cookies.txt -c cookies.txt -e ";auto" -L https://t.co/DzIiCFp7Ti                                       curl: (7)
    Failed to connect to www.threads.net port 443 Connection refused
    real     0m0.129s
    user   0m0.000s
    sys     0m0.013s
The failed to connects are threads.net likely blocking those user agents but the timing is there which is different than the first UA attempt.

Re: Tell HN: t.co is adding a five-second delay to some domains

#320

Post by DamonHD »

johngladtj wrote: post #247
DamonHD wrote: post #223

Earlier quoted context omitted.

You possibly haven't experienced how devious and determined and dishonest and unpleasant some bad actors are, including SPAMmers. (Even when doing the RightThing(TM) would probably be easier...) And, BTW, I occasionally get blocked by the mechanisms here, even though not doing anything bad, but understand that there is a trade-off.

I am fully aware of the issue. That's one of the costs with having a public website.

And that cost is so high that over the ~25Y+* that I have been running my own sites I have not had UGC on any of them, other then a very brief experiment, which showed me what utter relentless turds the bad actors can be.

Operators of public sites should NOT have to pay that tax. So you are best are not fully aware of the actual cost, IMHO.

Congrats to HN for striking a reasonable pragmatic balance.

*I had some of the first live (non-academic) Internet connectivity in the UK, and the very very first packets were hacking attempts...

Post reply on HN