Kind of unrealistic but I hope Twitter now open-sources not only the algorithm but also the Rails monolith itself. Would be kind of interesting to see how everything is done
The rails monolith is long gone.
twitter/the-algorithm
81–90 of 403 posts
Re: twitter/the-algorithm
#82First of all, "the algorithm" is probably hundreds of thousands of lines of code, including all the tedious boilerplate like cache policies and multi-AZ logic.
And second of all, doesn't the algorithm include machine learning components, which are trained on terabytes of data? That data will likely be impossible to open source. And open sourcing the neural nets without the training data is mostly meaningless from a transparency perspective?
Re: twitter/the-algorithm
#83Earlier quoted context omitted.
Twitter Locks Down Product Changes After Agreeing to Musk Bid - https://finance.yahoo.com/news/twitter-locks-down-product-ch...
Looks like a wise move to keep potential destructive protests at bay.
Re: twitter/the-algorithm
#84Makes me wonder how Twitter employees internally are handling the news. If they are celebrating or commiserating?
Re: twitter/the-algorithm
#85Earlier quoted context omitted.
Twitter Locks Down Product Changes After Agreeing to Musk Bid - https://finance.yahoo.com/news/twitter-locks-down-product-ch...
> Twitter imposed the temporary ban to keep employees who may be miffed about the deal from “going rogue,” according to one of the people. What could a rogue employee do?
- Rogue Twitter Employee Briefly Shuts Down Trump's Account [0]
[0] https://www.nytimes.com/2017/11/02/us/politics/trump-twitter...
Re: twitter/the-algorithm
#86Earlier quoted context omitted.
Or a commit is being prepared? It's only 20 minutes old.
It just seems unlikely that the algorithm would be open-sourced right after a deal for Twitter is agreed upon (but before it actually goes through). I've never seen a buyout of this scale done by an individual, but I imagine the SEC and several other parties will need to be involved. At the minimum, I would make a private Github repo first, add all relevant commits, and then make it public once there's actually conte…
The engineer from the OSS team at Twitter linked to it and said "watch this space"
https://twitter.com/willnorris/status/1518694675909013504
Sure I guess it could be that engineer gearing up for a joke, but I think it's more likely that it's a real release.
Re: twitter/the-algorithm
#87Re: twitter/the-algorithm
#88Anyone who actually uses Twitter already knows the algorithm: * Chronological - reverse sort by date * Home - for all of the followed topics, recommended topics, retweets and tweets in the past day determine the estimated level of engagement, include the highest and reverse sort by date. This is likely to be a fairly basic ML model. It will be uncontroversial, technically unsophisticated and of no practical use to an…
What are you basing this on? I wouldn't assume response prediction is using a "fairly basic ML model," it can be a lot more than that.
I've built hundreds of models and run a ML company and I don't believe it's technically possible for this rule not to be the case.
Re: twitter/the-algorithm
#89Earlier quoted context omitted.
This is most likely the correct answer. I doubt there is a single piece of code or algorithm that controls all of twitter. From my limited perspective it's just a catch phrase to simplify what is likely tens of thousands of lines of code.
Well, that's an algorithm. If you have to draw a diagram showing how edge cache nodes affect a user's results based on whether or not they signed in from New York or Phoenix, or if the downtime of a cluster affects the relevance rankings of tweets for a photography enthusiast, then guess what... That's an algorithm.
Re: twitter/the-algorithm
#90Earlier quoted context omitted.
There probably is not "an algorithm" on a site as large and complex as Twitter, no. There are probably dozens if not hundreds of algorithms spread throughout the codebase which affect the timeline for individual users, possibly even code entirely self-generated by ML systems.
Well sure there is "ORDER BY" something , and there are batch jobs to calculate this something from different variables. Maybe it is partially a ML black box, but I'm pretty sure they are not flying completely blind. There's got to be an internal Wiki or some PPTs where they list what goes into the mix. Number of likes, retweets. Attached media, hashtags, trending or not. Do they do sentiment analysis and use positiv…