Learning from Imbalanced Classes
svds.com
Learning from Imbalanced Classes
1–10 of 14 posts
Re: Learning from Imbalanced Classes
#2Solid article. Especially important are the suggestions to consider a probabilistic evaluation metric.
If you want to work with really imbalanced data, try working with data from the LHC. There were on order of 1000 Higgs in a year, while there are around 600 million proton-proton collisions a second.
Re: Learning from Imbalanced Classes
#3In addition to unbalanced classes, where let's say 2% of data is from a minority class and 98% is from the majority, how do people handle when 2% of data is from the minority and 98% is of an unknown class, but pulled from a known distribution?
Re: Learning from Imbalanced Classes
#4In addition to unbalanced classes, where let's say 2% of data is from a minority class and 98% is from the majority, how do people handle when 2% of data is from the minority and 98% is of an unknown class, but pulled from a known distribution?
Re: Learning from Imbalanced Classes
#5Solid article. Especially important are the suggestions to consider a probabilistic evaluation metric. If you want to work with really imbalanced data, try working with data from the LHC. There were on order of 1000 Higgs in a year, while there are around 600 million proton-proton collisions a second.
LHC data = EPIC
Re: Learning from Imbalanced Classes
#6There is a current Kaggle competition that is a good learning opportunity for imbalanced data. The goal is to predict parts that will be rejected by quality control.
Re: Learning from Imbalanced Classes
#7I usually find that SVM's do really well with very unbalanced data due to its hinge loss
Re: Learning from Imbalanced Classes
#8Solid article. Especially important are the suggestions to consider a probabilistic evaluation metric. If you want to work with really imbalanced data, try working with data from the LHC. There were on order of 1000 Higgs in a year, while there are around 600 million proton-proton collisions a second.
LHC data = EPIC
This is one of those few moments when saying Big Data is actually a well-used term.
Re: Learning from Imbalanced Classes
#9In addition to unbalanced classes, where let's say 2% of data is from a minority class and 98% is from the majority, how do people handle when 2% of data is from the minority and 98% is of an unknown class, but pulled from a known distribution?
I am not completely following what you mean by "98% is of an unknown class, but pulled from a known distribution".
Are you suggesting that: 2 percent of samples are positive, drawn from p(x|y=1)
98 percent of samples are drawn from a distribution p(x), but may be either positive or negative?
The setting that you described above is called "positive and unlabeled (PU)" learning. This paper: http://cseweb.ucsd.edu/~elkan/posonly.pdf is one of the seminal articles on the topic (although the equation on the bottom of page 214 contains a statement that may not necessarily hold true). There are quite a lot more recent papers on this topic.
Re: Learning from Imbalanced Classes
#10In addition to unbalanced classes, where let's say 2% of data is from a minority class and 98% is from the majority, how do people handle when 2% of data is from the minority and 98% is of an unknown class, but pulled from a known distribution?
I am not completely following what you mean by "98% is of an unknown class, but pulled from a known distribution". Are you suggesting that: 2 percent of samples are positive, drawn from p(x|y=1) 98 percent of samples are drawn from a distribution p(x), but may be either positive or negative? The setting that you described above is called "positive and unlabeled (PU)" learning. This paper: http://cseweb.ucsd.edu/~elka…
[deleted]