Live data from Hacker News

twitter/the-algorithm

github.com

81–90 of 403 posts

Re: twitter/the-algorithm

#81
post #42

Kind of unrealistic but I hope Twitter now open-sources not only the algorithm but also the Rails monolith itself. Would be kind of interesting to see how everything is done

The rails monolith is long gone.

In that case the same applies for the microservices

Re: twitter/the-algorithm

#82
I don't understand the concept of open-sourcing "the algorithm".

First of all, "the algorithm" is probably hundreds of thousands of lines of code, including all the tedious boilerplate like cache policies and multi-AZ logic.

And second of all, doesn't the algorithm include machine learning components, which are trained on terabytes of data? That data will likely be impossible to open source. And open sourcing the neural nets without the training data is mostly meaningless from a transparency perspective?

Re: twitter/the-algorithm

#83
post #40

Earlier quoted context omitted.

Twitter Locks Down Product Changes After Agreeing to Musk Bid - https://finance.yahoo.com/news/twitter-locks-down-product-ch...

Looks like a wise move to keep potential destructive protests at bay.

Yep. I imagine Twitter has a employee base with a very strong activist streak. I'm sure some of them wouldn't mind martyring themselves to make a virtuous statement.

Re: twitter/the-algorithm

#84

Makes me wonder how Twitter employees internally are handling the news. If they are celebrating or commiserating?

I am sure there is a small set of Vocal employee's blasting internal communications with the end of the world messages, a majority of employee's just wanting to keep working and get their salary, and a different small set of employees hoping Elon purges the first set of employee's making the work place better.

Re: twitter/the-algorithm

#85
post #57

Earlier quoted context omitted.

Twitter Locks Down Product Changes After Agreeing to Musk Bid - https://finance.yahoo.com/news/twitter-locks-down-product-ch...

> Twitter imposed the temporary ban to keep employees who may be miffed about the deal from “going rogue,” according to one of the people. What could a rogue employee do?

Twitter has a history with this.

- Rogue Twitter Employee Briefly Shuts Down Trump's Account [0]

[0] https://www.nytimes.com/2017/11/02/us/politics/trump-twitter...

Re: twitter/the-algorithm

#86

Earlier quoted context omitted.

Or a commit is being prepared? It's only 20 minutes old.

It just seems unlikely that the algorithm would be open-sourced right after a deal for Twitter is agreed upon (but before it actually goes through). I've never seen a buyout of this scale done by an individual, but I imagine the SEC and several other parties will need to be involved. At the minimum, I would make a private Github repo first, add all relevant commits, and then make it public once there's actually conte…

Sure it's fast but it seems the deal is done. A practical joke by an employee seems more unlikely than an official release.

The engineer from the OSS team at Twitter linked to it and said "watch this space"

https://twitter.com/willnorris/status/1518694675909013504

Sure I guess it could be that engineer gearing up for a joke, but I think it's more likely that it's a real release.

Re: twitter/the-algorithm

#88

Anyone who actually uses Twitter already knows the algorithm: * Chronological - reverse sort by date * Home - for all of the followed topics, recommended topics, retweets and tweets in the past day determine the estimated level of engagement, include the highest and reverse sort by date. This is likely to be a fairly basic ML model. It will be uncontroversial, technically unsophisticated and of no practical use to an…

What are you basing this on? I wouldn't assume response prediction is using a "fairly basic ML model," it can be a lot more than that.

If it's a simple, constrained problem where the number of available features is low then inherently the complexity can never be high.

I've built hundreds of models and run a ML company and I don't believe it's technically possible for this rule not to be the case.

Re: twitter/the-algorithm

#89

Earlier quoted context omitted.

This is most likely the correct answer. I doubt there is a single piece of code or algorithm that controls all of twitter. From my limited perspective it's just a catch phrase to simplify what is likely tens of thousands of lines of code.

Well, that's an algorithm. If you have to draw a diagram showing how edge cache nodes affect a user's results based on whether or not they signed in from New York or Phoenix, or if the downtime of a cluster affects the relevance rankings of tweets for a photography enthusiast, then guess what... That's an algorithm.

This shouldn't be downvoted. Any black box with inputs and outputs is a function over discrete data, otherwise known as an algorithm.

Re: twitter/the-algorithm

#90
post #31

Earlier quoted context omitted.

There probably is not "an algorithm" on a site as large and complex as Twitter, no. There are probably dozens if not hundreds of algorithms spread throughout the codebase which affect the timeline for individual users, possibly even code entirely self-generated by ML systems.

Well sure there is "ORDER BY" something , and there are batch jobs to calculate this something from different variables. Maybe it is partially a ML black box, but I'm pretty sure they are not flying completely blind. There's got to be an internal Wiki or some PPTs where they list what goes into the mix. Number of likes, retweets. Attached media, hashtags, trending or not. Do they do sentiment analysis and use positiv…

Maybe not. It could all be event-driven, distributed, and just pulling off various prioritized queues.
Post reply on HN