61–70 of 91 posts
Earlier quoted context omitted.
A Software Engineer. I'm just specialised a bit in DevOps, Data Engineering, and (beware buzzword) MLOps
What's MLops? Is it what I imagine? Maintaining repos of training/data, APIfying the pipeline, deploying an ML processing pipeline with CI/CD, etc?
Stuff I've used: - Prophet - seems to be the current 'standard' choice - ARIMA - Classical choice - Exponential Moving Average - dead simple to implement, works well for stuff that's a time series but not very seasonal - Kalman/Statespace model - used by Splunk's predict[1] command (pretty sure I always used LLP5) I did some anomaly detection work, in business transactions, and found the best way was to create a sort…
> - 2-3 anomalies in a row > - high deviation from expected > - multiple models all detected anomaly This is basically what statistical process control charts do for you. If you haven't learned about it already, I can recommend looking it up!
Easiest is to use cvxpy with your own objective function. You can easily add seasonality regularization etc. other things are too much black box. Also pivot tables. They are free now in online version of google sheets and online excel. Set the time as a row field and it will automatically aggregate. Or if you want irregular spacing you can group by 100 samples.
Edit: I do question why you think this is "easiest". Do you have some really unusual/specific modeling needs?
Former Reuters Research Director here. When modeling time series, you will want a model that is sensitive both to short term and longer term movements. In other words, a Long Term Short Term Memory (LSTM). Sepp Hochreiter invented this concept in his Master's thesis supervised by Jürgen Schmidhuber in Munich in the 1990s; today, it's the most-cited type of neural network. Here are papers describing it: https://people…
LSTMs have been going the way of the dinosaurs since 2018. If you really need a complex neural network (over 1D convolution approaches), transformers are the current SOTA. Example implementation in "temporal fusion": https://pytorch-forecasting.readthedocs.io/en/stable/tutoria... Mind you, in practice I've found these DL approaches overkill for simple problems of the "trends + cyclics + noise" kind.
Lots of people have already made good library recommendations, so I will make a non-recommendation for all the data science students out there: stop thinking about libraries, and start thinking about models. "What library do I use?" is the wrong question. "What model do I use?" is the right question. Libraries are just part of the process of answering that question. That said, high quality implementations of interest…
Earlier quoted context omitted.
A Software Engineer. I'm just specialised a bit in DevOps, Data Engineering, and (beware buzzword) MLOps
What's MLops? Is it what I imagine? Maintaining repos of training/data, APIfying the pipeline, deploying an ML processing pipeline with CI/CD, etc?
1. Time series based forecast based on revenue (the one OP is referring to). All the statistical time-series models come here. I primarily used H2O.ai for this.
2. Conversion based revenue forecast (input -> pipeline, output -> revenue). This proved to be quite tricky as there was a time lag between pipeline creation and revenue conversion
3. Delphi-method: Got the sales/pre-sales folks on-ground to predict a bottom-up number and used that as a forecast.
Finally, I combined them by applying weightages to the above approaches - based on how accurate they were on the test dataset.
IMHO, Like many of them have pointed out - the model/assumptions are more important than the library. The job of a data scientist is to make the prediction as reliable and explainable as possible.