Show HN: Fast Rolling Quantiles for Python
1–10 of 12 posts
Re: Show HN: Fast Rolling Quantiles for Python
#2Awesome! Rolling quantiles were a major bottleneck in my work a couple years ago, so much so that I ended up removing them and using approximations instead. It'll be good to try this if I ever go back to that code.
Re: Show HN: Fast Rolling Quantiles for Python
#3Wouldn’t it make more sense to speed up the current APIs used by lots of programs over introducing a new API?
Re: Show HN: Fast Rolling Quantiles for Python
#4Wouldn’t it make more sense to speed up the current APIs used by lots of programs over introducing a new API?
Perhaps, but as far as I'm aware, none of them are as flexible as my interface (to support e.g. streaming, pipelining.) I have not looked at all of pandas' time-series functionality that closely.
Re: Show HN: Fast Rolling Quantiles for Python
#5Wouldn’t it make more sense to speed up the current APIs used by lots of programs over introducing a new API?
Perhaps, but as far as I'm aware, none of them are as flexible as my interface (to support e.g. streaming, pipelining.) I have not looked at all of pandas' time-series functionality that closely.
I see, it's interesting that ypu wrote it all in C, pandas in Python
Re: Show HN: Fast Rolling Quantiles for Python
#6What's the advantage of this over Pandas' standard rolling quantiles?
import numpy as np
import pandas as pd
s_input = pd.Series(np.random.randn(1000))
s_p10 = s_input.rolling(10).quantile(0.1)
Re: Show HN: Fast Rolling Quantiles for Python
#7What's the advantage of this over Pandas' standard rolling quantiles? import numpy as np import pandas as pd s_input = pd.Series(np.random.randn(1000)) s_p10 = s_input.rolling(10).quantile(0.1)
See the "benchmarks" section:
https://github.com/marmarelis/rolling-quantiles#benchmarking...
Re: Show HN: Fast Rolling Quantiles for Python
#8Might be worth mentioning: `bottleneck` package, implemented in C, with various moving window functions - https://bottleneck.readthedocs.io/en/latest/reference.html#m...
Re: Show HN: Fast Rolling Quantiles for Python
#9This is pretty cool. The title would be a bit more descriptive if it were “Fast Rolling Quantile Filters for Python”, since the high-pass/low-pass filter functionality seems to be the focus.
The README mentions it uses binary heaps - if you’re willing to accept some (bounded) approximation, then it should be possible to reduce memory usage and somewhat reduce runtime by using a sketching data structure like Dunning’s t-digest: https://github.com/tdunning/t-digest/blob/main/docs/t-digest....
There is an open source Python implementation, although I haven’t used it and can’t vouch for its quality: https://github.com/CamDavidsonPilon/tdigest
Re: Show HN: Fast Rolling Quantiles for Python
#10Awesome! Rolling quantiles were a major bottleneck in my work a couple years ago, so much so that I ended up removing them and using approximations instead. It'll be good to try this if I ever go back to that code.
[deleted]