Live data from Hacker News

Building an Internet Scale Meme Search Engine

findthatmeme.com

11–20 of 154 posts

Re: Building an Internet Scale Meme Search Engine

#11
post #6
post #3

Would love to see that load balancer implementation, as I'm a scrub and this project fascinates me.

nginx makes it really simple to setup a load balancer, it defaults to distributing requests equally between all upstream hosts but you can always assign weights to each of your server. https://docs.nginx.com/nginx/admin-guide/load-balancer/http-...

Yep, this is exactly what I'm running on the raspberry pi LB. Nginx makes it super easy!

Re: Building an Internet Scale Meme Search Engine

#12
Love the inventiveness.

My question is about the image distribution costs. All the memes on the site seem to be coming straight off an object storage, all that bandwidth consumption has got to add up(?). Some sort of a CDN might help depending on the search patterns.

Re: Building an Internet Scale Meme Search Engine

#13
Last time I looked into OCR stuff I came to a similar conclusion (though I didn't implement anything back thne). It would be really nice to have "open source" models that had similar performance, without having to deal with the iphone cluster hackery.

Re: Building an Internet Scale Meme Search Engine

#14

This is remarkable. I'd love to see that combined with some kind of sentiment analysis like Microsoft offers, just to see if something useful comes out of it. Sometimes, I don’t know the exact words when looking for a meme, but once I see it, I know that’s the one.

Semantic search using CLIP embeddings / other LLM embeddings could be an amazing addition too.

Re: Building an Internet Scale Meme Search Engine

#16

My #1 recommendation for anyone thinking about the convoluted OCR solution: use a cheap OCR API and save yourself months of time / hassle / upkeep. Google's OCR API is a good place to start, but AWS has one too and dozens of others out there.

If you have to process tens of millions of photos though the cost gradually becomes forbidding.

Re: Building an Internet Scale Meme Search Engine

#17

Earlier quoted context omitted.

(Not the author) Maybe they leveraged https://knowyourmeme.com/ ? But that can't possibly have all the random memes, could it?

Author here: KnowYourMeme is one of many sites that memes are continually ingested from (any site that has memes I try to ingest regularly) :)

Amazing work! Also, thank you for making that feed on the main page, been laughing for a while here :D

Re: Building an Internet Scale Meme Search Engine

#18
Very inventive. Admittedly when I read the first few paragraphs, I was thinking “he’s got to have $40K of iPhones doing image processing” but you made a good point about being able to use iPhones with screen and other damage.

What was your average price per iPhone, if you don’t mind disclosing?

Post reply on HN