Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter's Recommendation Algorithm

blog.twitter.com

Post reply on HN
Print view View original

931–940 of 1001 posts

Re: Twitter's Recommendation Algorithm

#931

Post by jsnell »

JoshCole wrote: post #811
jsnell wrote: post #597

Earlier quoted context omitted.

Why is lines of code the appropriate input here? Here's a different computation that is at least as plausible: There are hundreds of millions of users. Let's say 300M. Only a single one is special-cased in this code: the narcisistic CEO who reportedly went ballistic when his engagement metrics went down. The prior that it's a change done in response to his demands is 299999999/300000000. (But of course it was added b…

I think you are right, my estimate is much much too low. I'll explain my mistake and why I made it. I think I thought to use the estimate I did because Elon claimed he didn't know. The prior probability of not knowing something in a code base with millions of lines is very high, but contingent on his involvement in the change the prior that he is aware of it is much higher. So I started the estimate attempt with the…

> I don't agree with your 1/300M prior, because I'm aware that hot users get special treatment.

That's absolutely fair, and 1/300M was a reductio ad absurdum rather than a serious proposal. Not all users are equal, just like not all lines of code are equal :)

I have a few issues with the "hot user" theory, but they all boil down to the same point: no matter what the use case, you'd never want to do this with a single static user.

Does your infra require special-casing for accounts with more than 100M followers? That should be a flag in the account properties that gets flipped manually or automatically: if these users cause infra problems, you really don't want to be making code changes + full rollouts whenever a new user becomes hot.

Is this just a guard-rail metric, to make sure there's not some bug specifically affecting hot users that tanks their engagement? You'd want a much larger static set than a single account just to ensure there's a large enough number/variety of tweets to compute metrics from. A single user might take a break for a week, or might only be posting very specific kind of content for an extended period of time.

In any case, even if you chose to do this with a single user rather than a set of users, why would Musk be the obvious single choice? He wasn't the most followed Twitter account until two days ago. A year ago there must have been at least a couple of dozen accounts roughly as notable as Musk. The odds of him having been chosen as the special case still would not be very high.

> In my view there isn't a compelling reason for him to lie about this.

The reason to lie about this is that it makes him appear weak, needy, and a target of even more mockery. Given the purchase of Twitter seems to have been a vanity project, having this be exposed and leaving it in goes directly against his apparent goal.

Re: Twitter's Recommendation Algorithm

#932

Post by dmix »

dymk wrote: post #790
dmix wrote: post #576

Earlier quoted context omitted.

He didn’t say he knew what it did? It’s a good enough response to say that it shouldn’t be doing that period.

Generally, one should understand chesterton's fence before tearing it down

He was answering a question on the spot live on a video call. Careful consideration can come later by the devs who will investigate.

Must be nice to have people hold you to such high standards though. A sign of respect in a way. No matter how strenuous the takes.

Re: Twitter's Recommendation Algorithm

#933

Post by ngrilly »

TaylorAlexander wrote: post #204
ngrilly wrote: post #158

Earlier quoted context omitted.

What did you expect?

I don’t know if the parent’s expectations matter here. This is more about making sure others don’t misunderstand the meaning here.

Good point. I didn't see it like that. Thanks!

Re: Twitter's Recommendation Algorithm

#934

Post by whiddershins »

dmonitor wrote: post #285
phailhaus wrote: post #163

Great! But nothing is going to change until people realize that the problem is the feedback loop . It's not the recommendation engine itself, it's the fact that there's no way "out" of the feed that the engine produces. It recommends you stuff, you have little choice but to engage with it, and then it trains on that information . This is the problem with most of social media today. It is a very well known problem in…

reminds me of a story about a guy who was given a gift, a decorative plate with a rooster on it i think it was. didn’t care for it too much, but out of politeness put it on display on an empty cabinet he had. a while later someone noticed he had it and figured he liked it, so got him a similar decorative plate with a rooster on it. again, out of politeness, he put it next to the old one. now other people started to t…

I had a friend who this happened to with ducks. You would go to her house and there were little duck icons and duck figurines and duck themed things everywhere.

When questioned, it turns out she never cared about or liked ducks. Someone gave her one and then it became the go to present for her for every occasion, for decades.

Decades of ducks.

Re: Twitter's Recommendation Algorithm

#935

Post by lazyeye »

infogulch wrote: post #657
CathalMullan wrote: post #55

Earlier quoted context omitted.

Only used for metrics, apparently. [0] /** * These author ID lists are used purely for metrics collection. We track how often we are * serving Tweets from these authors and how often their tweets are being impressed by users. * This helps us validate in our A/B experimentation platform that we do not ship changes * that negatively impacts one group over others. */ [0]: https://github.com/twitter/the-algorithm/blob/7f…

So many unnecessarily cynical takes here. Let's say you were in charge of a large legacy system that some segment of customers complain about it not working for them as well as other segments. How would you know whether their complaints are valid unless you measured it? You have to know first. So measure it.

I've never seen so many "experts" speaking from a position of complete ignorance than I do on Hacker News.

Re: Twitter's Recommendation Algorithm

#936

Post by cunidev »

whiddershins wrote: post #934
dmonitor wrote: post #285

Earlier quoted context omitted.

reminds me of a story about a guy who was given a gift, a decorative plate with a rooster on it i think it was. didn’t care for it too much, but out of politeness put it on display on an empty cabinet he had. a while later someone noticed he had it and figured he liked it, so got him a similar decorative plate with a rooster on it. again, out of politeness, he put it next to the old one. now other people started to t…

I had a friend who this happened to with ducks. You would go to her house and there were little duck icons and duck figurines and duck themed things everywhere . When questioned, it turns out she never cared about or liked ducks. Someone gave her one and then it became the go to present for her for every occasion, for decades. Decades of ducks.

I am probably part of this problem - an enabler, so to say. I was hosted a nice lady with a house full of duck themed things (this was in southern England, probably not the same lol), and left her two (rather cute) duck-themed things when I moved assuming that she'd like those.

She was unequivocally happy when I did, but perhaps it was just because of the gesture.

(EDIT: also, the rooster/horse/duck example of feedback loop would make for a terrific blog post)

Re: Twitter's Recommendation Algorithm

#937

Post by bdw5204 »

tric wrote: post #11

From https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92... ( "author_is_elon", candidate => candidate .getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))), ( "author_is_power_user", candidate => candidate .getOrElse(AuthorIdFeature, None) .exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)), ( "author_is_democrat", candidate => candidate .getOrEls…

How hard would it be to replace this entire algorithm with the following pseudocode?

If !user.follows_author(author) then don't show tweet on timeline Else if tweet.timestamp is later than all other tweets show tweet first

This is vastly superior to any other possible recommendation algorithm because users can choose what tweets they see/don't see by whom they follow and everybody has an equal chance to have their tweets seen by their followers. When Twitter moved away from this, it rendered my timeline useless so I started just pulling up people's profiles to read their tweets in order and eventually deleted my (pseudonymous) account that had several thousand followers. Almost nobody was seeing my tweets anyway thanks to this algorithm and deleting the account did not prevent me from browsing accounts I'm interested in.

All Elon needed to do to fix Twitter was to reverse all of the bad changes they've made since 2015 or so and restore the platform to what it was in the late 00s/early 10s.

Re: Twitter's Recommendation Algorithm

#938

Post by amq »

Surprised no one mentioned this:

    s.SpaceSafetyLabelType.MedicalMisinfo -> MedicalMisinfo,
    s.SpaceSafetyLabelType.GenericMisinfo -> GenericMisinfo,
    s.SpaceSafetyLabelType.DmcaWithheld -> DmcaWithheld,
    s.SpaceSafetyLabelType.HatefulHighRecall -> HatefulHighRecall,
    ...
    s.SpaceSafetyLabelType.UkraineCrisisTopic -> UkraineCrisisTopic,
https://github.com/twitter/the-algorithm/blob/ec83d01dcaebf3...

Re: Twitter's Recommendation Algorithm

#939

Post by arcatech »

bdw5204 wrote: post #937
tric wrote: post #11

From https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92... ( "author_is_elon", candidate => candidate .getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))), ( "author_is_power_user", candidate => candidate .getOrElse(AuthorIdFeature, None) .exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)), ( "author_is_democrat", candidate => candidate .getOrEls…

How hard would it be to replace this entire algorithm with the following pseudocode? If !user.follows_author(author) then don't show tweet on timeline Else if tweet.timestamp is later than all other tweets show tweet first This is vastly superior to any other possible recommendation algorithm because users can choose what tweets they see/don't see by whom they follow and everybody has an equal chance to have their tw…

How is that a recommendation algorithm? The point of the "recommendation" part is to show you things you wouldn't normally see.

Re: Twitter's Recommendation Algorithm

#940

Post by cwkoss »

joshuamorton wrote: post #616
corbulo wrote: post #537

Earlier quoted context omitted.

This sentiment has high correlation to driving conclusions from a very time limited information set. This isn't the only part that is going to be posted to github. What is the net benefit from rushing to condemn something that can only be a net positive compared to the past alternatives? I don't understand the purpose of that approach. Help me.

> can only be a net positive compared to the past alternatives This seems to be unsubstantiated. Are you really claiming that selective disclosure is always superior to complete lack of transparency?

Yep "Limited Hangouts" are a common disinfo technique
Post reply on HN