Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

twitter/the-algorithm

github.com

Post reply on HN
Print view View original

211–220 of 403 posts

Re: twitter/the-algorithm

#211

Post by thorgutierrez »

shrimpx wrote: post #82

I don't understand the concept of open-sourcing "the algorithm". First of all, "the algorithm" is probably hundreds of thousands of lines of code, including all the tedious boilerplate like cache policies and multi-AZ logic. And second of all, doesn't the algorithm include machine learning components, which are trained on terabytes of data? That data will likely be impossible to open source. And open sourcing the neu…

If it's too complicated, there is a good chance Elon will ask to simplify it until it can be open sourced.

Re: twitter/the-algorithm

#212

Post by almog »

hotpotamus wrote: post #155
jdrc wrote: post #33

I know the algorithm i use, it ends with ORDER BY date DESC.

Is that actually possible on Twitter? (I have an account but don't use it.) I do believe them that when a user follows several 100+ other users, it would probably be impossible to read every tweet chronologically.

Assuming we're talking about the timeline "latest" mode (rather than "top"), IIRC in the past the used to do the home timeline processing on the read paths, but have long since opted to optimize for read latency, thus the heavy lifting is performed during the insertion, so in broad strokes, when someone post a tweet, the three main things that happen are:

1. Insertion of tweet to tweets table.

2. Insertion of that tweet-id to the home timelines of all that user's followers.

3. Insertion of that tweet-id to the user-timeline of that user.

On the read-path, if I'm not mistaken, the only join that happen is between the requested timeline and the tweets table (which is replicated across cluster of machines but not partitioned, or at least I remember reading that was the case not many years ago)

Re: twitter/the-algorithm

#213

Post by lamontcg »

ironmagma wrote: post #162
jonwinstanley wrote: post #27

Earlier quoted context omitted.

If it’s a list of tweets ordered based on a kajillion ML data points that varies per user is it still an algorithm? And does every user have their own algorithm? And could it be made readable to a human?

In that case, the definition of “the algorithm” is the training set.

Twitter itself is probably the training set for twitter.

Re: twitter/the-algorithm

#214

Post by newman314 »

asabjorn wrote: post #179
mhh__ wrote: post #116

Earlier quoted context omitted.

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

Quoted post unavailable.

The Epoch Times is not a credible source.

“The Epoch Times is a far-right[16] international multi-language newspaper and media company affiliated with the Falun Gong new religious movement”

https://en.m.wikipedia.org/wiki/The_Epoch_Times

Re: twitter/the-algorithm

#215

Post by MangoCoffee »

axg11 wrote: post #95

I've worked on very large scale recommendation systems at a FAANG. If Twitter's system resembles anything like ours, the concept of publishing or open sourcing "the algorithm" doesn't make sense. Even if we were to open source all associated code and publish all related documents it would be very difficult to make sense of the entire system. That is precisely why companies such as Twitter A/B test the hell out of eve…

i'm looking forward to see what's going to be on it.

Re: twitter/the-algorithm

#216

Post by triyambakam »

KrishnaShripad wrote: post #169
mhh__ wrote: post #116

Earlier quoted context omitted.

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

Was it the algorithm or was it a "guy in a room" which/who decided to block New York Post's article on Hunter Biden's laptop scandal?

To those that are probably triggered by this and likely even uninformed, the "left" view has been that the laptop story was fake, but it turns out it was real.

Re: twitter/the-algorithm

#217

Post by mistermann »

mhh__ wrote: post #150
tyingq wrote: post #140

Earlier quoted context omitted.

Not saying Google is turning down conservative search results, but they absolutely can. Using the same cycle of human raters and tweaking weights they used to push down comparison shopping sites. See http://graphics.wsj.com/google-ftc-report/

It's the man in a room part I was emphasising. It's simultaneously massively incorrect about how software actually works, misunderstands social dynamics inside companies, and massively stokes the fire when it comes to conspiracy theories along these lines (in this case at least)

Are you claiming that you have comprehensive objective knowledge of what did and did not happen in privileged Twitter operations across time, or is this more like your opinion on the matter? And if the latter, how is it that some people's opinions "don't make sense" or "are dog whistles", but yours are epistemically flawless (assuming you actually believe them to be true, of course)?

Re: twitter/the-algorithm

#218

Post by tomrod »

asabjorn wrote: post #179
mhh__ wrote: post #116

Earlier quoted context omitted.

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

Quoted post unavailable.

Interesting. I reckon Google wasn't causal here, but rather access to unfiltered information was. A goodly portion of conservatives/conservative-leaners I know and love are fairly gullible to Fox News and other bastions of ten-truths-with-a-lie approaches. Unfiltered information and data literacy sometimes helps to overcome that problem -- 2% - 5% of the population sounds about right for that shift to my prior.

Your mileage may vary.

Also, The Epoch Times is a far right source -- not trustworthy generally.

Re: twitter/the-algorithm

#219

Post by akhmatova »

commandlinefan wrote: post #152
mhh__ wrote: post #116

Earlier quoted context omitted.

Of course it doesn't make sense. I think it's just a dog whistle to the people who believe google have a guy in a room somewhere turning the "conservative search results" lever down a notch during elections.

> turning the "conservative search results" lever It may not have been algorithmic, but it definitely happened.

You know this how? Because you were that guy?

Re: twitter/the-algorithm

#220

Post by EricE »

madeofpalk wrote: post #156
34679 wrote: post #146

Earlier quoted context omitted.

I don't believe the point was ever to convey an complete understanding of the system, but to allow for the identification of "red flags": Specific functions that would be deemed controversial or counter to equal opportunity speech.

Is the idea somewhere in the-algorithm there's a function called `derankGOPMembers()`? Twitter's been pretty transparent in how it "deranks" certain accounts [1]. What more would come from opening the code that certainly not include the actual database of "no no terms" (if you were to believe that exists)? [1] https://blog.twitter.com/en_us/topics/company/2022/our-ongoi...

Twitter is transparent in how they do shadow bans? They still deny they are a thing but it's clear to see the network effects of them. Twitter is transparent - that's the most hilarious thing I have read today!
Post reply on HN