Show HN: Subreddit Finder - Trained on 4M Reddit Posts from 4K Subreddits

Title: Build recommendations. Message: "I'd like to upgrade some components. My current rig has an old i7 and an RTX 2060. Looking for something midrange that can handle modern games at high settings (but maybe not ultra)."

Matches: Nvidia (19%), IndianGaming (8%), GamingLaptops (8%), pcgamingtechsupport (6%)

Title: Travel advice. Message: "I'm returning to Ireland in July from the USA. My visa is up. I know I will have to self-quarantine for two weeks. I cannot move back to my family home due to elderly parents. Are there any recommendations for people in this sort of situation? I'm happy to pay for a hotel, but don't want to put a hotel worker at risk. We have an old house down in Wexford I could stay in, but would involve taking a train when I arrive, and the HSE guidance says not to take public transport. Any advice?"

Recommendations: LegalAdviceUK (8%), IWantOut (7%), AskUK (5%)

Overall I think this was pretty good, even if it wasn't perfect. I thought it would struggle more with the second one (maybe getting confused and suggesting vacation planning subreddits). A little controversial that it kept suggesting "UK" reddits for a question about Ireland though :)

Indeed, I got some ML metrics on a test split that gives me an idea of its accuracy :) But it's just an estimation, so indeed I am looking out for feedback to know its real performance so I can debug bad cases and fix those with more data or a better model.

The test performance on subreddit r/hearthstone is 0.21 f1-score, which is not great. And looking at the confusion matrix for r/hearthstone it gets often confused with:

r/BobsTavern r/CompetitiveHS r/customhearthstone r/Blizzard

If you are curious, I uploaded the metrics (precision, recall, f1-score) and confusion matrix on the test dataset on a Google Spreadsheet.

https://docs.google.com/spreadsheets/d/1NBY1o85ZiNpcm4tcYhKk...

The sheet 'confusion_matrix_gt2' can be used to find similar subreddits.

You're most likely using simple average pooling, which is why many users are getting results that don't look right to them. Try a chunking approach, where you get a vector for each chunk of the document and horizontally concatenate those together (if your vectors are 50d, and do 5 chunks per doc, than you get a 250d fixed vector for each document regardless of length). This partially solves the issue of highly diluted vectors which is responsible for the poor results that some users are reporting. You can also do "attentive pooling" where you pool the way a transformer head would pool - though that's an O(N^2) operation so YMMV

If you have the GPU compute, try something like BERT, or GPT-2 which is fine-tuned on all of reddit. Better yet, try vertically concatenating all of the word-embeddings models you can together (just stack the embeddings from each model) if you have the compute

To respond to your comment (since HN isn't letting me post cus I'm 'posting too fast')

You can use cheaper and more effective approaches for getting the subword functionality you want.

Look up "Byte Pair Embeddings". That will also handle the OOV problem but for far less CPU/RAM overhead. BERT also does this for you with its unique form of tokenization.

A home CPU can fine-tune FastText in a day on 4 million documents if you're able to walk away from your computer for awhile. Shouldn't cost you anything except electricity. If you set the number of epochs higher, you'll get better performance but correspondingly longer times to train.

For BERT/GPT-2, you'll maybe want to fine-tune a small version of the model (say, the 117m parameter version of GPT-2) and then vertically concatenate that with the regular un-fine-tuned GPT-2 model. That should be very fast and hopefully not expensive (and also possible on your home GPU)