Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Quid uses deep learning with small data

quid.com

Post reply on HN
Print view View original

11–20 of 49 posts

Re: How Quid uses deep learning with small data

#11

Post by bowlesbe »

ryanschneider wrote: post #6
SubiculumCode wrote: post #5

I can't seem to find where the sample size is mentioned. It mentions that Quid has 50,000 company descriptions, but is n=50,000 tiny in thr ML/DeepLearning world? I do neuroscience research and where I am coming from I have maybe n=150to200 per class. And that is not generally regarded as a tiny sample.

The issue is those 50,000 descriptions aren't labeled good/bad. Someone had to pick a subset of them and label them, so my guess is they did this for maybe 100 or 200 descriptions.

This is correct. We had 300-400 examples of each

Re: How Quid uses deep learning with small data

#12

Post by jackschultz »

bowlesbe wrote: post #9
jackschultz wrote: post #8

Curious how you guys got training data for this. Did someone have to go through and rate whether or not a sentence was quality or not? And how many training examples did you use? You say it was "difficult to develop a large set" but I'm curious how large that set actually was. Edit: Also, do you think more data or a "better" or "more sophisticated" model would make the results better? I would guess more data would tr…

Thanks for the comment! There were a few hundred sentences of each, collected internally from from a wide number of descriptions. Yes, I'd definitely agree- more data is what we need here for further model improvements.

I actually had this issue recently when trying to get training data for a project of mine as well [0], so I built an app [1] as a way to more easily classify documents.

Basically I have simpler interfaces and the ability for multiple people to quickly answer questions like this on a set of data. Easily exportable in the end as well. If you're interested in using that to get some more data on sentences, let me know. I'm really curious how much better the results get with more data, and this could help.

[0] https://bigishdata.com/2016/11/01/classifying-country-music-...

[1] https://fierce-mountain-21498.herokuapp.com/

Re: How Quid uses deep learning with small data

#14

Post by rspeer »

bowlesbe wrote: post #10
rspeer wrote: post #2

The baseline I'd like to see this compared to is the not-very-deep-learning "bag of tricks" that's conveniently implemented in fastText [1]. [1] https://github.com/facebookresearch/fastText

Great point! I considering using fasttext as a baseline, however in practice fasttext really didn't work well at all with the small data set, much worse than the tfidf baseline. I think Fasttext's classification approach might not work well with such a small dataset. I'm not sure but I suspect its because it tries to learn embeddings - but there just isn't anywhere near enough data for that. I'd love an outside persp…

Fair enough. That's a useful comparison to know about.

But I'm wondering how you get around that with the neural net. In the post, you said there are only a few hundred labeled examples, right? How can a neural net with hundreds of parameters set those parameters to anything reasonable, and not overfit, when there are about as many parameters as examples?

Re: How Quid uses deep learning with small data

#15

Post by IanCal »

cocktailpeanuts wrote: post #13

Wait, isn't "deep learning with small data" just machine learning, after all the buzzwords cancel themselves out? I thought the whole point of "deep learning" is its approach to using data.

The point of deep learning is using a deep graph, like a neural network with a lot of layers, not the amount of data.

However, picking millions of parameters with small amounts of data is unlikely to work well.

Re: How Quid uses deep learning with small data

#16

Post by bowlesbe »

rspeer wrote: post #14
bowlesbe wrote: post #10

Earlier quoted context omitted.

Great point! I considering using fasttext as a baseline, however in practice fasttext really didn't work well at all with the small data set, much worse than the tfidf baseline. I think Fasttext's classification approach might not work well with such a small dataset. I'm not sure but I suspect its because it tries to learn embeddings - but there just isn't anywhere near enough data for that. I'd love an outside persp…

Fair enough. That's a useful comparison to know about. But I'm wondering how you get around that with the neural net. In the post, you said there are only a few hundred labeled examples, right? How can a neural net with hundreds of parameters set those parameters to anything reasonable, and not overfit, when there are about as many parameters as examples?

Great question and I share your intuition but I think its all properly regularizing your model. I guess for neural networks, Dropout works really darn well as a regularization strategy. I could have tried to see whether performance dropped significantly without dropout.

Re: How Quid uses deep learning with small data

#17

Post by bunderbunder »

IanCal wrote: post #15
cocktailpeanuts wrote: post #13

Wait, isn't "deep learning with small data" just machine learning, after all the buzzwords cancel themselves out? I thought the whole point of "deep learning" is its approach to using data.

The point of deep learning is using a deep graph, like a neural network with a lot of layers, not the amount of data. However, picking millions of parameters with small amounts of data is unlikely to work well.

It seems, at first blush, like using a very complex model to fit a very small amount of data is a recipe for some serious overfitting.

Re: How Quid uses deep learning with small data

#18

Post by cocktailpeanuts »

IanCal wrote: post #15
cocktailpeanuts wrote: post #13

Wait, isn't "deep learning with small data" just machine learning, after all the buzzwords cancel themselves out? I thought the whole point of "deep learning" is its approach to using data.

The point of deep learning is using a deep graph, like a neural network with a lot of layers, not the amount of data. However, picking millions of parameters with small amounts of data is unlikely to work well.

yeah that's what i was trying to say. Saying you can construct a deep graph using small data feels like "I'm gonna become a millionaire in ten years, only by operating this single lemonade stand"
Post reply on HN