Live data from Hacker News

twitter/the-algorithm

github.com

251–260 of 403 posts

Re: twitter/the-algorithm

#251
post #116
post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

And that hasn’t happened? Google got fined by the EU for manipulating search results. It definitely happens.

https://amp.usatoday.com/amp/1248099002

https://www.vox.com/2017/6/27/15878980/europe-fine-google-an...

Re: twitter/the-algorithm

#252
post #188
post #116

Earlier quoted context omitted.

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

"Among the elite, and within Twitter specifically, there is much more inclination to ban the right. I say this as someone whose political views, if you force them onto the left-right spectrum, probably end up about 80% toward the left. E.g. I've spent millions over the past several elections supporting the Democrats. It used to be that censorship was something the right did, and free speech was something the left wer…

I’m pro censorship. We have and need speech norms.

The left and right are not equal. The left does not rely as much on lies to advocate its positions, and the left is not as oriented around destruction and regression as the right.

We need a more robust understanding of speech than “allowed or not”. Emphasis & volume matters.

Re: twitter/the-algorithm

#253
I'm very technical and I think it would still be valuable to have a list of all the things that weight into the timeline view, even without the models or underlying data.

Like, there's no public admission right now of whether "shadow banning" or "ghost banning" is even officially a thing!

Some transparency seems unquestionably more powerful than none, and we can work from there.

Re: twitter/the-algorithm

#254

At the time of posting, Will Norris (the open source lead at twitter, admin of their github account presumably) posted this. It has 44 retweets, 193 likes, 17 quote tweets, on github it has 1.6k stars. That seems... bizarre to me?

Nope. People are just excited that the Twitter cesspool might finally improve.

Re: twitter/the-algorithm

#255
The government, at federal, state and local levels, all rely on Twitter to conduct official taxpayer funded work. Taxpayer funded work should not happen on proprietary systems that operate with zero oversight or public transparency.

Elon polled Twitter users about this and the response was overwhelmingly in favor of open source and transparency. Everyone on Twitter got a vote.

If you oppose transparency, as many now are, you lose your credibility. So it’s another one of Elon’s people hacks, and look at all the morons falling for it.

Re: twitter/the-algorithm

#256
post #116
post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

> I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

Even if that's precisely true, is it not good to be creating a more trusted space for everyone? The grievances, regardless of merit, are mostly coming from the right. If you want to create a service that caters to all you're going to have to address their concerns. If he can do that in a way that is fair to all, it sounds like a win to me.

Re: twitter/the-algorithm

#257
post #241

Earlier quoted context omitted.

> That's why people on the far right keep starting lame Twitter alternatives. You don't see people on the far left doing that. They don't need to because they have cool Twitter alternatives. Like Mastodon.

Which is also the Trump alternative, iirc. Just the far left seems to respect the license.

No, it's not THE trump alternative.

It existed before.

Open source, such that anyone can use it

Re: twitter/the-algorithm

#258
post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

I guess I kind of think about these things not as algorithms, but as a collection of frontends and backends. Collectively, any human request (typing [ ramen shops near me ] as an example) will be handled by a bunch of different code, typically that code is structured as RPCs.

We can think of the main interaction as being a query which is an RPC payload. The contents contain the user request and a wide amount of other context (either referenced by a collection of keys like cookies, or materialized like fields that specify the user's age) and the response is a web page which contains sections (the web search response to the query, as well as the ads; either these could be rendered to two different frames, or interspersed, by the result presentation engine).

That query -> frontend translates into a tree or a graph of requests which collect up various bits of contextual data required to satisfy the query. For example, the query terms might be rewritten slightly and then sent to a web search backend which searches/ranks documents and returns the top matching documents on the organic web, or sent to an ads backend that returns the top matching bidders for those query terms. Again, just RPC/responce, although the actual context that the frontend and backend systems are dealing with, and use to modify the result, are truly enormous.

Each of those backend systems itself was produced with an enormous amount of data processing and contextual data that is available at serving time. All of this is implemented using various algorithms; everything from the TCP algorithms that manage bandwidth to the neural networks doing inference on the joint product of the user context and the query context and the ad context, and the logging system that writes the queries and their clicks to centralized storage for more ML training.

In theory though you could set up a system that compiled the full web stack, and ran the end to end of a user query, dumping all the intermediate RPCs, etc, from a modestly sized instantiation of the production system. and people could sit down and inspection what terms affected query result order, or which pages were omitted at which part of the filtering, or what data was logged.

It would be hell for a team to maintain and keep up to date wrt the production system, but many folks do this any way to have a simple version of the system around so they can make quick changes and see if it breaks part of the complex system without doing a full deployment.

Re: twitter/the-algorithm

#260

At the time of posting, Will Norris (the open source lead at twitter, admin of their github account presumably) posted this. It has 44 retweets, 193 likes, 17 quote tweets, on github it has 1.6k stars. That seems... bizarre to me?

That's more stars than I'll ever have on any of my repos. Maybe I made a mistake not joining a faang
Post reply on HN