Lessons Learned from Benchmarking Fast Machine Learning Algorithms
blogs.technet.microsoft.com
Lessons Learned from Benchmarking Fast Machine Learning Algorithms
1–10 of 21 posts
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#2The GPU versions are performing surprisingly bad. To even match CPU performance, you need a training set in the tens of millions, and even far beyond that, a doubling of speed seems to be the best you can hope for.
Compare to, for example, tensorflow, where it isn't uncommon to see a 10x speedup even for moderately-sized training sets.
(I say "surprising" in the sense that I'm surprised; I don't know the algorithms used for decision trees, and it may well be that they are less amendable to GPU-parallelization than the NN- and matrix algorithms I've worked with)
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#3It's interesting that LightGBM was initially promoted as being more accurate than XGB, but that claim always seemed marginal at best and was hard to reproduce.
Other investigations show the same thing about training speed though, eg https://medium.com/implodinggradients/benchmarking-lightgbm-...
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#4The GPU versions are performing surprisingly bad. To even match CPU performance, you need a training set in the tens of millions, and even far beyond that, a doubling of speed seems to be the best you can hope for. Compare to, for example, tensorflow, where it isn't uncommon to see a 10x speedup even for moderately-sized training sets. (I say "surprising" in the sense that I'm surprised; I don't know the algorithms u…
In decision trees more than half of the optimization time is spent doing sorting (sorting the set at each node to find the optimal splitting for the key at that node), in neural nets it's almost all matrix multiplies. That's where the speedup difference comes in the CPU v GPU comparison.
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#5It's interesting that LightGBM was initially promoted as being more accurate than XGB, but that claim always seemed marginal at best and was hard to reproduce. Other investigations show the same thing about training speed though, eg https://medium.com/implodinggradients/benchmarking-lightgbm-...
I'd recommend rewording the last sentence as it seems to imply that training speed has same "hard to reproduce" numbers, when in fact I believe you meant to say that LightGBM is indeed faster.
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#6It would be interesting if they compared training speed with CatBoost [0].
I remember seeing a paper where they managed to avoid getting stuck in local optimum in terms of number of learners, and the more trees you add better the result.
Logloss results seem to confirm there's a superior tree algorithm going on there in CatBoost.
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#7It's interesting that LightGBM was initially promoted as being more accurate than XGB, but that claim always seemed marginal at best and was hard to reproduce. Other investigations show the same thing about training speed though, eg https://medium.com/implodinggradients/benchmarking-lightgbm-...
I'd recommend rewording the last sentence as it seems to imply that training speed has same "hard to reproduce" numbers, when in fact I believe you meant to say that LightGBM is indeed faster.
Yes, fair point. Edited, thank.
Actually, that's very, very strange. The edit I made doesn't seem to be what is above. I said something like " same (edit: that LightGBM is faster to train than XGB) thing". I mean it's harmless, but very odd.
Did someone else edit my post? @dang ?
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#8Seems to be a project under Microsoft's Distributed Machine Learning Toolkit (http://www.dmtk.io/).
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#9The GPU versions are performing surprisingly bad. To even match CPU performance, you need a training set in the tens of millions, and even far beyond that, a doubling of speed seems to be the best you can hope for. Compare to, for example, tensorflow, where it isn't uncommon to see a 10x speedup even for moderately-sized training sets. (I say "surprising" in the sense that I'm surprised; I don't know the algorithms u…
In decision trees more than half of the optimization time is spent doing sorting (sorting the set at each node to find the optimal splitting for the key at that node), in neural nets it's almost all matrix multiplies. That's where the speedup difference comes in the CPU v GPU comparison.
There are fast parallel sorting algorithms that should be able to take advantage of GPUs. Maybe they didn't implement them?
Re: Lessons Learned from Benchmarking Fast Machine Learning Algorithms
#10Earlier quoted context omitted.
I'd recommend rewording the last sentence as it seems to imply that training speed has same "hard to reproduce" numbers, when in fact I believe you meant to say that LightGBM is indeed faster.
Yes, fair point. Edited, thank. Actually, that's very, very strange. The edit I made doesn't seem to be what is above. I said something like " same (edit: that LightGBM is faster to train than XGB) thing". I mean it's harmless, but very odd. Did someone else edit my post? @dang ?
This is off topic but I have been told if you have questions to the site you email it to them. I think the contact email is at the bottom.