Live data from Hacker News

Learning from Imbalanced Classes

svds.com

11–14 of 14 posts

Re: Learning from Imbalanced Classes

#11
post #8
post #5

Earlier quoted context omitted.

LHC data = EPIC

This is one of those few moments when saying Big Data is actually a well-used term.

There's a really great episode of Linear Digressions[0] (a Data Science podcast) that goes into the different scales of data that exist in the world. Everyone thinks of Google and Facebook as Big Data, but the Australian Square Kilometer Array Pathfinder collects 7.5B TB of data per second!

[0]: https://soundcloud.com/linear-digressions/whats-the-biggest-...

Re: Learning from Imbalanced Classes

#12
post #8

Earlier quoted context omitted.

This is one of those few moments when saying Big Data is actually a well-used term.

There's a really great episode of Linear Digressions[0] (a Data Science podcast) that goes into the different scales of data that exist in the world. Everyone thinks of Google and Facebook as Big Data, but the Australian Square Kilometer Array Pathfinder collects 7.5B TB of data per second! [0]: https://soundcloud.com/linear-digressions/whats-the-biggest-...

And it's going to end up quite a bit more than that.

Check out slides 21 and 22 from [1]. There are parts that will process 4 PB/s (!)

[1] http://www.slideshare.net/SparkSummit/distributed-data-proce...

Re: Learning from Imbalanced Classes

#13
Quite an interesting article!

I had two research projects on finding low quality - deleted and closed - questions on Stack Overflow [0, 1]. Since, these question classes always suffered from imbalanced class problems (2% closed and 8% deleted) - I decoded to under sampling of the majority class. Later, I used an ensemble - Gradient Decision Boosting Tree, Random Forrest - for classification. However, rather than taking bootstrap samples - I took various random samples from the majority class (equivalent to the minority class) and built many ensemble classifiers for each random sample. In the end, I checked the variation on the final classification results. It just seemed intuitive to me. I had no idea about Wallace et al.!

[0] Denzil Correa and Ashish Sureka. 2014. Chaff from the wheat: characterization and modeling of deleted questions on stack overflow. In Proceedings of the 23rd international conference on World wide web (WWW '14). ACM, New York, NY, USA, 631-642. DOI: http://dx.doi.org/10.1145/2566486.2568036

[1] Denzil Correa and Ashish Sureka. 2013. Fit or unfit: analysis and prediction of 'closed questions' on stack overflow. In Proceedings of the first ACM conference on Online social networks (COSN '13). ACM, New York, NY, USA, 201-212. DOI=http://dx.doi.org/10.1145/2512938.2512954

Re: Learning from Imbalanced Classes

#14
post #12

Earlier quoted context omitted.

There's a really great episode of Linear Digressions[0] (a Data Science podcast) that goes into the different scales of data that exist in the world. Everyone thinks of Google and Facebook as Big Data, but the Australian Square Kilometer Array Pathfinder collects 7.5B TB of data per second! [0]: https://soundcloud.com/linear-digressions/whats-the-biggest-...

And it's going to end up quite a bit more than that. Check out slides 21 and 22 from [1]. There are parts that will process 4 PB/s (!) [1] http://www.slideshare.net/SparkSummit/distributed-data-proce...

It's not a secret that in the process of automatic manufacturing there are terabytes of data produced in a matter of minutes. OF course how many of such data is of any use is another story, but anyway it still requires storage and processing power capable of handling that volumes.
Post reply on HN