Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

twitter/the-algorithm

github.com

Post reply on HN
Print view View original

121–130 of 403 posts

Re: twitter/the-algorithm

#121

Post by cscurmudgeon »

axg11 wrote: post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

Just because it has evolved to be a complex monster that is unnaccountable doesn't mean it has to be that way in the future.

There are ways to translating trained ML models and associated systems into understandable hierarchical rules.

Twitter's timeline is NOT AGI.

Re: twitter/the-algorithm

#122

Post by kaladin-jasnah »

hunterb123 wrote: post #30
mrintellectual wrote: post #15

This seems to be a practical joke by a Twitter engineer as opposed to an actual release.

Or a commit is being prepared? It's only 20 minutes old.

https://twitter.com/elonmusk/status/1518677066325053441 states that Elon Musk is to looking "make the algorithms open source."

Re: twitter/the-algorithm

#123

Post by mhh__ »

andrewmcwatters wrote: post #61
bastardoperator wrote: post #48

Earlier quoted context omitted.

This is most likely the correct answer. I doubt there is a single piece of code or algorithm that controls all of twitter. From my limited perspective it's just a catch phrase to simplify what is likely tens of thousands of lines of code.

Well, that's an algorithm. If you have to draw a diagram showing how edge cache nodes affect a user's results based on whether or not they signed in from New York or Phoenix, or if the downtime of a cluster affects the relevance rankings of tweets for a photography enthusiast, then guess what... That's an algorithm.

Yes it's an algorithm, but as software engineers we shouldn't pander to people who actually think there is a small flow chart dictating what happens in your search results. It's idiotic or dishonest.

Re: twitter/the-algorithm

#124

Post by [deleted] »

andrewmcwatters wrote: post #47
Irishsteve wrote: post #7

Many people have commented that it is empty. However what they do not realize is that there has never actually been an algorithm and that is why it is empty.

If there is a total set of data, and a subset of it is produced for users, there is an algorithm. `SELECT TOP` is an algorithm. A ML model with a trillion parameters is an algorithm. There is no "no algorithm."

[deleted]

Re: twitter/the-algorithm

#125

Post by cellis »

captainmuon wrote: post #54
krapp wrote: post #31

Earlier quoted context omitted.

There probably is not "an algorithm" on a site as large and complex as Twitter, no. There are probably dozens if not hundreds of algorithms spread throughout the codebase which affect the timeline for individual users, possibly even code entirely self-generated by ML systems.

Well sure there is "ORDER BY" something , and there are batch jobs to calculate this something from different variables. Maybe it is partially a ML black box, but I'm pretty sure they are not flying completely blind. There's got to be an internal Wiki or some PPTs where they list what goes into the mix. Number of likes, retweets. Attached media, hashtags, trending or not. Do they do sentiment analysis and use positiv…

The trouble is, there could be literally millions of ORDER BY queries. Hell, each user could be given a fine-tuned cnn…

Re: twitter/the-algorithm

#127

Post by renewiltord »

memish wrote: post #57
mostlysimilar wrote: post #22

Earlier quoted context omitted.

Twitter Locks Down Product Changes After Agreeing to Musk Bid - https://finance.yahoo.com/news/twitter-locks-down-product-ch...

> Twitter imposed the temporary ban to keep employees who may be miffed about the deal from “going rogue,” according to one of the people. What could a rogue employee do?

Once upon a time they could perform all sorts of admin-level actions. One of my friends was there back in the day and said you could literally delete someone's account or use the built-in impersonation to see someone else's DMs. Another thing they could do back then was leak PII of specific accounts. I think it was the Saudis who paid someone to get someone's name. A friend of mine was part of the FBI team that caught one of those spies.

Re: twitter/the-algorithm

#128

Post by zeckalpha »

Railsify wrote: post #108
axg11 wrote: post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

I get what you're saying. I'd imagine this repo will be closer to pseudo code than "complete" code for all systems that a tweet flows through. For example, the system that flags/remove picture of hotdogs would likely be represented as "if image contains hotdog: weight: -500"

That could be a starting point for a DSL that realizes the algorithm.

Re: twitter/the-algorithm

#129

Post by temp8964 »

axg11 wrote: post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

I don’t understand people like you. Just because Elon Musk used the term “the algorithm” in conversation, it must be interpreted literally? And for someone like Elon Musk, he doesn’t know the complexity and need you to point out?

Re: twitter/the-algorithm

#130

Post by kaladin-jasnah »

influx wrote: post #42
EMIRELADERO wrote: post #26

Kind of unrealistic but I hope Twitter now open-sources not only the algorithm but also the Rails monolith itself. Would be kind of interesting to see how everything is done

The rails monolith is long gone.

That gives even more reason to open-source it, right? If Twitter isn't using that codebase anymore.
Post reply on HN