Twitter's Recommendation Algorithm • Hacker News Board

Re: Twitter's Recommendation Algorithm

#961

Post by null4bl3 » Sun, Apr 02, 2023, 7:42 AM UTC

cbeach wrote: post #619

tric wrote: post #11

From https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92... ( "author_is_elon", candidate => candidate .getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))), ( "author_is_power_user", candidate => candidate .getOrElse(AuthorIdFeature, None) .exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)), ( "author_is_democrat", candidate => candidate .getOrEls…

The “author_is_elon” flag may have been assigned to him because Elon’s Twitter account has the most followers on the platform. So, for technical / performance reasons, changes to the algos might want to be benchmarked against this account in particular, because it’s the account most likely to be at the centre of capacity- / load-related issues.

Sure. Every serious application has a special flag for their boss man right?

Re: Twitter's Recommendation Algorithm

#962

Post by DuckFeathers » Sun, Apr 02, 2023, 7:49 AM UTC

corbulo wrote: post #201

It's disappointing the comments are so obsessed with the political angle to this that there's a total lack of appreciation (or discussion) of opening up the most influential social media platform in the world.

Attacks on Elon has been growing since he started calling out corruption.

Re: Twitter's Recommendation Algorithm

#963

Post by rblion » Sun, Apr 02, 2023, 7:55 AM UTC

First thing I would like to see gone is business bros sharing 'guides' after you follow them, threatening to start charging real soon. Go fuck yourself, get a real job.

Re: Twitter's Recommendation Algorithm

#964

Post by dymk » Sun, Apr 02, 2023, 8:26 AM UTC

dmix wrote: post #932

dymk wrote: post #790

Earlier quoted context omitted.

Generally, one should understand chesterton's fence before tearing it down

He was answering a question on the spot live on a video call. Careful consideration can come later by the devs who will investigate. Must be nice to have people hold you to such high standards though. A sign of respect in a way. No matter how strenuous the takes.

Making commitments to things without understanding the basics isn’t something to be respected, it’s just dumb

Re: Twitter's Recommendation Algorithm

#965

Post by WinstonSmith84 » Sun, Apr 02, 2023, 9:38 AM UTC

amq wrote: post #938

Surprised no one mentioned this: s.SpaceSafetyLabelType.MedicalMisinfo -> MedicalMisinfo, s.SpaceSafetyLabelType.GenericMisinfo -> GenericMisinfo, s.SpaceSafetyLabelType.DmcaWithheld -> DmcaWithheld, s.SpaceSafetyLabelType.HatefulHighRecall -> HatefulHighRecall, ... s.SpaceSafetyLabelType.UkraineCrisisTopic -> UkraineCrisisTopic, https://github.com/twitter/the-algorithm/blob/ec83d01dcaebf3...

Yes, this thread is particularly interesting

https://twitter.com/aakashg0/status/1641976869460275201

Speaking about Ukraine, it seems to be literally a Twitter policy violation ... https://github.com/twitter/the-algorithm/blob/main/visibilit...

Re: Twitter's Recommendation Algorithm

#966

Post by jalapenos » Sun, Apr 02, 2023, 10:07 AM UTC

CathalMullan wrote: post #55

tric wrote: post #11

From https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92... ( "author_is_elon", candidate => candidate .getOrElse(AuthorIdFeature, None).contains(candidate.getOrElse(DDGStatsElonFeature, 0L))), ( "author_is_power_user", candidate => candidate .getOrElse(AuthorIdFeature, None) .exists(candidate.getOrElse(DDGStatsVitsFeature, Set.empty[Long]).contains)), ( "author_is_democrat", candidate => candidate .getOrEls…

Only used for metrics, apparently. [0] /** * These author ID lists are used purely for metrics collection. We track how often we are * serving Tweets from these authors and how often their tweets are being impressed by users. * This helps us validate in our A/B experimentation platform that we do not ship changes * that negatively impacts one group over others. */ [0]: https://github.com/twitter/the-algorithm/blob/7f…

Correct. It's a binary metric. Did the number go up, yes/no (kept job / not).

Re: Twitter's Recommendation Algorithm

#967

Post by dmak » Sun, Apr 02, 2023, 10:29 AM UTC

d_sc wrote: post #947

I think they have a bug here here: https://github.com/twitter/the-algorithm/blob/7f90d0ca342b92... Code: ( "has_gte_10k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 1000))), Should be: ( "has_gte_10k_favs", _.getOrElse(EarlybirdFeature, None).exists(_.favCountV2.exists(_ >= 10000))),

They might be trying to preserve the previous tag label.

Re: Twitter's Recommendation Algorithm

#968

Post by dmak » Sun, Apr 02, 2023, 10:33 AM UTC

roddylindsay wrote: post #226

For ranking the candidates these predictions are combined into a score by weighting them: "recap.engagement.is_favorited": 0.5 "recap.engagement.is_good_clicked_convo_desc_favorited_or_replied": 11* (the maximum prediction from these two "good click" features is used and weighted by 11, the other prediction is ignored). "recap.engagement.is_good_clicked_convo_desc_v2": 11* "recap.engagement.is_negative_feedback_v2":…

Could you link to the code on github?

Re: Twitter's Recommendation Algorithm

#969

Post by jimkleiber » Sun, Apr 02, 2023, 10:59 AM UTC

makeitdouble wrote: post #625

eecc wrote: post #79

Earlier quoted context omitted.

Wouldn’t that make them easy prey of “spam SEO”. However, given the framework isn’t it still possible to guess the models?

The spam SEO issue should be dealt/thought about _before_ engaging in the whole adventure, and having to guess how it could work if decently implemented properly defeats the "open source" spirit of it. More credits would be given if the very idea of open sourcing the algorithm hasn't already been discussed to death with predictions of the difficult points and how it probably won't happen in any sane way.

Makes me wonder if a way to override people SEO hacking the algorithm is to create a market of open-source algorithms that each individual can choose and then it's not trying to hack THE algorithm but having to hack many and not knowing which algorithm an individual is using.

Re: Twitter's Recommendation Algorithm

#970

Post by hijodelsol » Sun, Apr 02, 2023, 11:09 AM UTC

mkj wrote: post #847

junto wrote: post #826

Did anyone else notice this below? I can’t even begin to imagine how many CPU’s that would require and what the cost must be… just for a recommendation engine. > The pipeline above runs approximately 5 billion times per day and completes in under 1.5 seconds on average. A single pipeline execution requires 220 seconds of CPU time, nearly 150x the latency you perceive on the app.

5e9 * 220 / 3600 / 24 implies they are using 12 million cpu cores continuously? That seems nearly implausible, but perhaps it's true?

I immediately did the same calculation, the climate impact per user also seems non-negligible. Doing some back of the envelope maths, 20W per core server power consumption equals 240.000kWh per hour. At 500g CO2eq/kWh this gives us roughly a billion kilograms of CO2eqs per year. At approx 300M MAUs this is roughly 3.5kg/user/year. Not completely off the charts but still important, reducing the time to 120 CPU seconds per execution would have the similar impact as 300M people not traveling 10-15km by car.

Energy costs per user is also interesting, if at all close, at 0.25c per kWh the power consumption cost per user would be greater than 5$ per year.