Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

twitter/the-algorithm

github.com

Post reply on HN
Print view View original

161–170 of 403 posts

Re: twitter/the-algorithm

#161

Post by layer8 »

AviationAtom wrote: post #112
Irishsteve wrote: post #7

Many people have commented that it is empty. However what they do not realize is that there has never actually been an algorithm and that is why it is empty.

You don't think recent news of the EU compelling social media companies to disclose their algorithms has anything to do with this?

No, the repository is a joke (IME). Regarding the EU, the details haven’t even been nailed down yet, and the timeline is that it will only become a requirement in 2024 I believe. The goal also isn’t to publish “the algorithm”, but to give researchers and civil society representatives access to the training data and information about the most important factors controlling the algorithm, so they can make assessments on the basis of that. It is not about open-sourcing code.

Re: twitter/the-algorithm

#162

Post by ironmagma »

jonwinstanley wrote: post #27
influx wrote: post #9

Earlier quoted context omitted.

There's not an algorithm to order timelines?

If it’s a list of tweets ordered based on a kajillion ML data points that varies per user is it still an algorithm? And does every user have their own algorithm? And could it be made readable to a human?

In that case, the definition of “the algorithm” is the training set.

Re: twitter/the-algorithm

#163

Post by suadeo »

axg11 wrote: post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

As someone who’s also worked in this area, I disagree with this take.

There is typically clear objective function of a recommendation system.

What Twitter is optimizing for is what’s of interest here. And some of the hidden business rules. It’s likely these are specified in the code in an obvious way.

How exactly they achieve that is the part that is complex and relatively indecipherable.

It’s possible that it’s designed in such a way the optimization objectives are also unclear, but that would indicate a bad design and be to the detriment of the company and users.

Re: twitter/the-algorithm

#164

Post by bigfudge »

Barrin92 wrote: post #79

whatever will show up in this repo, I hope people realize that depending on what data you put into some algorithm you can get whatever output you want, and twitter is never going to (and neither can or should they) publish everyone's personal information and interaction on the site. So I'm not sure what the ultimate point of this exercise is other than producing faux-transparency.

I still think there could be lots of interest, even without the data. In fact some of the most interesting parts would be the algorithm in the broadest sense — how tech interacts with company policy and SOP. For one small example, what aspects of moderation/banning happen automatically without any further human intervention?

Re: twitter/the-algorithm

#165

Post by lamontcg »

axg11 wrote: post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

Did you just tell me to go write a map/reduce function in Erlang?

Re: twitter/the-algorithm

#166

Post by actuator »

axg11 wrote: post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

Yeah, this makes no sense. There is no golden algorithm that Facebook, Tiktok or Twitter has figured out.

All these feed rankings are complex combinations of features, models coupled with weights and filters. On top of this abuse detection layers are added.

Unless Musk is planning to open source user data to show what all the "scores" and "features" for all the entities are and how they were reached to, this will make no sense. The whole argument against some people being downranked has been, why me? Just writing a whitepaper to tell the general methodology, is not going to make that go away.

On top of that, exposing every vector through which you measure and stop abuse, will just allow for more sophisticated abuse.

Re: twitter/the-algorithm

#167

Post by whimsicalism »

threeseed wrote: post #88
whimsicalism wrote: post #75

Earlier quoted context omitted.

What are you basing this on? I wouldn't assume response prediction is using a "fairly basic ML model," it can be a lot more than that.

If it's a simple, constrained problem where the number of available features is low then inherently the complexity can never be high. I've built hundreds of models and run a ML company and I don't believe it's technically possible for this rule not to be the case.

I work in basically this field. Tweets have text and often images and users with relationships with other users posting and liking them, the potential feature space was never going to be low dimensional.

Re: twitter/the-algorithm

#168

Post by bilekas »

Irishsteve wrote: post #7

Many people have commented that it is empty. However what they do not realize is that there has never actually been an algorithm and that is why it is empty.

This, personally I was thinking in light of the new upcomming sale, the current Twitter staff might decide to 'open source' some algorithm. Which would be an effective method of protest against a sale like that.

But there definitely is a relationship algo that could be considered theirs, like all social medias inflating the bubbles users all feel.

Re: twitter/the-algorithm

#169

Post by KrishnaShripad »

mhh__ wrote: post #116
axg11 wrote: post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

Was it the algorithm or was it a "guy in a room" which/who decided to block New York Post's article on Hunter Biden's laptop scandal?

Re: twitter/the-algorithm

#170

Post by ironmagma »

hotpotamus wrote: post #155
jdrc wrote: post #33

I know the algorithm i use, it ends with ORDER BY date DESC.

Is that actually possible on Twitter? (I have an account but don't use it.) I do believe them that when a user follows several 100+ other users, it would probably be impossible to read every tweet chronologically.

Impossible to read, maybe. Impossible to display, definitely not.
Post reply on HN