Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Real World Recommendation System

blog.fennel.ai

Post reply on HN
Print view View original

141–150 of 156 posts

Re: Real World Recommendation System

#141

Post by charcircuit »

q-base wrote: post #139
charcircuit wrote: post #62

Earlier quoted context omitted.

YouTube, TikTok, and Twitter all work well for me.

Does YouTube actually work for someone? I bit the bullet after not really giving any other input than what I watched and started marking "Not interested" or "Don't recommend channel" but I cannot see that it has had any effect at all it keeps recommending rubbish "sensationalist" videos. I know that they probably optimize for ads etc. but if they actually showed me videos I would like to see, then I would spend more…

Yes, it works great for me. I don't really use the negative feedback. Most recommendation I guess are new uploads from creators I like or videos I haven't watched from creators I like. It shows me videos from unknown creators to me that match my interests as they go viral. It shows some random old videos that just randomly go viral again. It's a good experience.

Re: Real World Recommendation System

#142

Post by dr_kiszonka »

q-base wrote: post #139
charcircuit wrote: post #62

Earlier quoted context omitted.

YouTube, TikTok, and Twitter all work well for me.

Does YouTube actually work for someone? I bit the bullet after not really giving any other input than what I watched and started marking "Not interested" or "Don't recommend channel" but I cannot see that it has had any effect at all it keeps recommending rubbish "sensationalist" videos. I know that they probably optimize for ads etc. but if they actually showed me videos I would like to see, then I would spend more…

These sensationalist videos are a plague for me. It only takes watching two or three regular videos on a specific topic for YouTube to hone in on it and start populating my feed with them. I find it very disappointing that they exist in every area that I am interested in, including relatively niche ones like functional programming and mechanical keyboards. Although, I have yet to come across a sensationalist video about prolog ;)

Re: Real World Recommendation System

#143

Post by faizshah »

priansh wrote: post #4

The main issue with deploying these systems right now is the technical overhead to develop them out. Existing solutions are either paid and require you to share your valuable data, or open source but either abandoned (rip Crab) or inextensible (most rely on their own DB or postgres). I’d love to see a lightweight, flexible recommendation system at a low level, specifically the scoring portion. There are a few flexibl…

Yahoo released Vespa as open source: https://vespa.ai/

It has everything you need at a platform level to build a production recommendation system given that it’s the engine that powered a lot of yahoo product’s search and recommendation capabilities. I have been experimenting with it, the number of capabilities are immense. It’s really an untapped resource.

Take a look at the features: https://vespa.ai/features and the ranking syntax: https://docs.vespa.ai/en/ranking.html

Really cool stuff, I haven’t even scratched the surface of what it can do.

Re: Real World Recommendation System

#144

Post by faizshah »

colesantiago wrote: post #45
mrfox321 wrote: post #29

Earlier quoted context omitted.

TikTok

Why TikTok in particular? What is the engineering story behind TikTok's recommendation system? How did they get it right?

Not a direct answer to your question, but at least in English there isn’t that much stuff on how TikTok’s recommendation system works internally. This is the best breakdown I have found on TikTok’s recommendation system internals: https://leehanchung.github.io/2020-02-18-Tik-Tok-Algorithm/

There’s a more technical recent paper from bytedance as well: https://arxiv.org/pdf/2007.07203.pdf

And another recent one on bytedance user profile system, this paper gives the deepest understanding of their recommendation system: https://www.cs.princeton.edu/courses/archive/spring21/cos598...

There’s a good breakdown of the recent nytimes article on TikTok’s internals here: https://read.deeplearning.ai/the-batch/issue-122/

If anyone has found anything better let me know.

If you try searching yourself you will want to try switching between the keywords “douyin” “toutiao” “bytedance” and “tiktok.”

Re: Real World Recommendation System

#145

Post by Shish2k »

samhw wrote: post #98
efsavage wrote: post #90

Earlier quoted context omitted.

> mostly backed by hundreds of thousands of Mysql instances Kind of. It's part of the recipe but one you find at these large tech companies (I've worked at FB and GOOG) is they have the resources to bend even large/standard projects like MySQL to their will, while ideally preserving the good ideas that made them popular in the first place. There are wrappers/layers/modifications/etc that eventually evolve to subsume…

I mean, this post from a year ago makes it sound not that non-standard : https://engineering.fb.com/2021/07/22/data-infrastructure/my... What you're describing sounds like you mean something on the level of Cockroach, talking the Postgres wire protocol but implemented entirely independently underneath (which came indirectly out of Google). Facebook's MySQL deployment sounds more like a heavily-patched-but-basically-M…

FB uses mysql in two very different ways - for the giant social-network database, mysql is basically a key-value store used as the storage layer for the graph database built on top. Then for the thousands of small utility databases (small enough to fit on a single machine) it’s used in a very vanilla way.

Re: Real World Recommendation System

#146

Post by q-base »

charcircuit wrote: post #141
q-base wrote: post #139

Earlier quoted context omitted.

Does YouTube actually work for someone? I bit the bullet after not really giving any other input than what I watched and started marking "Not interested" or "Don't recommend channel" but I cannot see that it has had any effect at all it keeps recommending rubbish "sensationalist" videos. I know that they probably optimize for ads etc. but if they actually showed me videos I would like to see, then I would spend more…

Yes, it works great for me. I don't really use the negative feedback. Most recommendation I guess are new uploads from creators I like or videos I haven't watched from creators I like. It shows me videos from unknown creators to me that match my interests as they go viral. It shows some random old videos that just randomly go viral again. It's a good experience.

That's interesting and perhaps I should not be that surprised. If it did not work at all, then they would probably have changed it. Their KPI is probably ad revenue, but still if it worked for no one, then I would expect their revenue to decline as people spend their time elsewhere. But then again, where will you go and watch medium to long videos created by "normal" people? Vimeo? Perhaps youtube's interactions are despite their algorithm and not because.

Re: Real World Recommendation System

#147

Post by q-base »

dr_kiszonka wrote: post #142
q-base wrote: post #139

Earlier quoted context omitted.

Does YouTube actually work for someone? I bit the bullet after not really giving any other input than what I watched and started marking "Not interested" or "Don't recommend channel" but I cannot see that it has had any effect at all it keeps recommending rubbish "sensationalist" videos. I know that they probably optimize for ads etc. but if they actually showed me videos I would like to see, then I would spend more…

These sensationalist videos are a plague for me. It only takes watching two or three regular videos on a specific topic for YouTube to hone in on it and start populating my feed with them. I find it very disappointing that they exist in every area that I am interested in, including relatively niche ones like functional programming and mechanical keyboards. Although, I have yet to come across a sensationalist video ab…

I have the exact same experience. I want to spend time on the platform and I can find videos that I like and would have loved for the algorithm to find for me, but I really have to search very thoroughly for them myself.

Re: Real World Recommendation System

#148

Post by charcircuit »

q-base wrote: post #146
charcircuit wrote: post #141

Earlier quoted context omitted.

Yes, it works great for me. I don't really use the negative feedback. Most recommendation I guess are new uploads from creators I like or videos I haven't watched from creators I like. It shows me videos from unknown creators to me that match my interests as they go viral. It shows some random old videos that just randomly go viral again. It's a good experience.

That's interesting and perhaps I should not be that surprised. If it did not work at all, then they would probably have changed it. Their KPI is probably ad revenue, but still if it worked for no one, then I would expect their revenue to decline as people spend their time elsewhere. But then again, where will you go and watch medium to long videos created by "normal" people? Vimeo? Perhaps youtube's interactions are…

>ad revenue

Not directly, most people believe they optimize for session time. It tries to serve you a result that will keep you on the platform watching videos as opposed to needing to keep scrolling or leaving the platform. They truely do want to serve the best videos that they can to you that they think you will be interested in watching. Thankfully for YouTube you watching videos and YouTube getting ad rev is correlated.

>where will you go and watch medium to long videos created by "normal" people?

No one forces you to watch that format of video. You can go on TikTok, Twitch, Netflix, etc and be entertained.

Re: Real World Recommendation System

#149

Post by samhw »

notriddle wrote: post #97
chudi wrote: post #92

Earlier quoted context omitted.

If You have 10000 posts that You have to sort it in some way and the user just going to see 20 of those, the sorting is the recommendation system, people are just used to think of products, movies and songs, but in those platforms the users are the products

This hardly seems like a reasonable way to characterize Netflix, which has a personal recommendation system, especially compared to HN, which is ad supported yet gives the same recommendations to everyone.

My reading of their comments is that they are trying to say that social media and news media can be characterised as having recommendation systems too, not just song and movie platforms (I don't know who exactly they're arguing against – I've never heard anyone say that recommendation systems can only be for songs and movies).

I don't think they're really paying much attention to the dimension you're splitting it along, i.e. whether the recommendations are personalised for each user. The huge important idea they have in their head is that recommendations can apply to user-generated social media content too.

Re: Real World Recommendation System

#150

Post by hbn »

oofbey wrote: post #85
hbn wrote: post #79

Earlier quoted context omitted.

> Facebook is now Meta, Google is now Alphabet Eh, the new parent company names aren't really what people know them as still. I don't think most people are even aware that Google has a parent company. I have a friend that works at Google, and that's what we say. I don't think him or anyone would ever say he works at Alphabet.

Yeah, Meta will likely stick because Zuckerberg and crew are actively trying to run away from the dumpster fire they lit with Facebook. But Google is still Google, and probably always will be. Just like Youtube is still Google, and Waymo is still Google.

I'm going out of my way to not say Meta for that exact reason. They haven't proved they deserve to shed their horrible reputation yet. They'll always be the Facebook we know and hate until they're less dystopian.
Post reply on HN