Working on my cli frontend to speedtest. Trying to figure out why I'm getting inconsistent speed readings
Ask HN: What is your weekend project?
41–50 of 50 posts
Re: Ask HN: What is your weekend project?
#42I just completed this one, it was to be done till sunday but I was really excited for this, so finished it today. A machine learning driven web application firewall http://fsecurify.com/fwaf-machine-learning-driven-web-applic...
That's really cool! Quick, unsolicited 2¢:
1. Your data is severely imbalanced, so accuracy is a very misleading metric to use here. From what I see, you have a 1:20 imbalance (malicious vs non-malicious distribution). This affects both the metrics and induces bias in classification.
2. I'd like to add to the other comment asking you for calibration curves and see what your minority class performance looks like in terms of precision, recall, f-beta, average precision (area under precision-recall curve).
3. Then, try and see if resampling helps or hurts the predictive performance- it typically speaks to the level of noise and small disjuncts in the data.
4. I see you've done a 0.2 split for test-train, but try and eliminate split bias by using stratified cross validation. This would ensure that you didn't just get lucky with random seed = 42 and get a really great test set.
All of these can be implemented using sklearn and imbalanced-learn [0]. Not included- deeper dive into cost sensitive and adversarial techniques. Let me know if you have any more questions and keep up the good work!
[0]: https://github.com/scikit-learn-contrib/imbalanced-learn
Source: PhD in imbalanced machine learning.
Re: Ask HN: What is your weekend project?
#43https://www.findlectures.com Right now it has an index of ~70k conference talks / lectures / speeches. I'm working on improving it to get slide text and audio quality (for ranking), and getting more historical content.
How do you pay for the bandwidth costs for scraping?
I'm doing everything over my home network
Re: Ask HN: What is your weekend project?
#44Re: Ask HN: What is your weekend project?
#45https://www.findlectures.com Right now it has an index of ~70k conference talks / lectures / speeches. I'm working on improving it to get slide text and audio quality (for ranking), and getting more historical content.
what is the stack you used?
Re: Ask HN: What is your weekend project?
#46I just completed this one, it was to be done till sunday but I was really excited for this, so finished it today. A machine learning driven web application firewall http://fsecurify.com/fwaf-machine-learning-driven-web-applic...
That's really cool! Quick, unsolicited 2¢: 1. Your data is severely imbalanced, so accuracy is a very misleading metric to use here. From what I see, you have a 1:20 imbalance (malicious vs non-malicious distribution). This affects both the metrics and induces bias in classification. 2. I'd like to add to the other comment asking you for calibration curves and see what your minority class performance looks like in te…
Thank you so much for these suggestions, I'll surely try these and will let you know.
One thing to add, the data is not that much imbalanced. I only used 100000 non malicious and 50,000 malicious so its 2:1 actually. I didn't use all the non malicious queries.
Thanks again.
Re: Ask HN: What is your weekend project?
#47https://www.findlectures.com Right now it has an index of ~70k conference talks / lectures / speeches. I'm working on improving it to get slide text and audio quality (for ranking), and getting more historical content.
what is the stack you used?
It's mostly Node right now. I started documenting it in more detail (just notes right now) - https://www.findlectures.com/articles/2017/01/22/Software-Ar...
Re: Ask HN: What is your weekend project?
#48https://www.findlectures.com Right now it has an index of ~70k conference talks / lectures / speeches. I'm working on improving it to get slide text and audio quality (for ranking), and getting more historical content.
I didn't know i wanted this. How do you scrap all that data?
I wrote up some more info here on data acquisition here (just notes right now) - https://www.findlectures.com/articles/2017/01/22/Software-Ar...
Re: Ask HN: What is your weekend project?
#49Earlier quoted context omitted.
That's really cool! Quick, unsolicited 2¢: 1. Your data is severely imbalanced, so accuracy is a very misleading metric to use here. From what I see, you have a 1:20 imbalance (malicious vs non-malicious distribution). This affects both the metrics and induces bias in classification. 2. I'd like to add to the other comment asking you for calibration curves and see what your minority class performance looks like in te…
Thank you so much for these suggestions, I'll surely try these and will let you know. One thing to add, the data is not that much imbalanced. I only used 100000 non malicious and 50,000 malicious so its 2:1 actually. I didn't use all the non malicious queries. Thanks again.
Sorry, just saw this after all these days. Do check out Adversarial Machine Learning as a really cool next step.