Live data from Hacker News

Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

microprediction.com

51–60 of 82 posts

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#51
post #49
post #42

As someone who spent a good part of my professional career in forecasting and time-series analysis I would like to point out that "point-forecasts" are mostly useless in many import practical applications such as FinTech, e-commerce, sports betting, etc. Point-forecast models such as Prophet fail to give you a meaningful measure of uncertainty of the predicted value. A much better approach are probabilistic forecasti…

Can you elaborate a bit? Prophet can use MCMC sampling and includes uncertainty in its forecasts.

Prophet with MCMC can produce probabilistic forecasts. But how to choose MCMC priors and measure accuracy of the probabilistic forecast are open questions.

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#52
post #43
post #42

As someone who spent a good part of my professional career in forecasting and time-series analysis I would like to point out that "point-forecasts" are mostly useless in many import practical applications such as FinTech, e-commerce, sports betting, etc. Point-forecast models such as Prophet fail to give you a meaningful measure of uncertainty of the predicted value. A much better approach are probabilistic forecasti…

Do you have any favourite libraries for producing such?

> Do you have any favourite libraries for producing such?

For modern Deep Learning based probabilistic forecasting you can try DeepAR with parametric likelihood function [0] or Multi-Horizon Quantile RNN (non-parametric) [1]. The implementations of these models in Pytorch and MXnet are scattered all over the place.

[0] https://arxiv.org/abs/1704.04110

[1] https://arxiv.org/abs/1711.11053

EDIT: formatting

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#53
To close, let me say that this post ended up being more negative than I expected and, like Nessie, my opinion may rise in the future when I understand the implications of the Prophet generative model better, and either modify it or find better ways to identify its strengths. The unanswered question here is why Prophet is so popular, and this surely merits a better explanation than I have given. I think there are probably statistical angles I am not seeing - something reflecting the fact that people are voting with their eyeballs when they use Prophet.

(from the article)

- The author

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#54
post #14

My experience in general is that most time series model are inadequate in predicting time series except for very trivial cases of seasonalities or simple linear/nonlinear trends. I think that you can throw any model you like at the problem but all you will do is overfit most of the time.

This is to be expected, because pure time-series models (Holt-Winters, ARIMA, etc.) only capture behavior of historical data (autoregressive, i.e. yₖ = f(yₖ₋₁, yₖ₋₂, ...)). If the patterns of interest aren't primarily time-based patterns, then time series models wouldn't be predictive.

In my experience, the time-series models that are reliably predictive typically aren't purely autoregressive but contain exogeneous variables as well (i.e. yₖ = f(yₖ₋₁, yₖ₋₂, ..., xₖ, xₖ₋₁, xₖ₋₂...), like ARX models). These models don't only capture relationships to historical patterns but to other driving/causal variables.

Price forecasts are often modeled as time-series models, but this assumes that prices only have time-based patterns which is often not true. In my domains of interest for instance, time has tangible yet limited effect on prices -- prices are driven more by variables like weather and certain types of market activity.

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#55
post #51
post #49

Earlier quoted context omitted.

Can you elaborate a bit? Prophet can use MCMC sampling and includes uncertainty in its forecasts.

Prophet with MCMC can produce probabilistic forecasts. But how to choose MCMC priors and measure accuracy of the probabilistic forecast are open questions.

[deleted]

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#56
I'll try to give my perspective, though it's mostly expanding on em500's [1] 3rd paragraph. I think you're asking the wrong question about prophet - most users don't care "how good is it?" but instead "is it good enough?", and then "how easy is it to use?"

Prophet solves a broad class of easy problems that a lot of ordinary businesses have: you have several years of basically regular data (sales or page views or store foot traffic) that you know has yearly/weekly/daily (if you have sub-daily data) cycles, and you want to give a reasonable prediction to the business so they can plan for the upcoming week/month/year. And you want to remove the periodic effects so you can see the underlying trends.

Imagine someone, lets call them Bill, who might be called a data scientist, or business analyst or just assistant operations manager, for a medium-large business. Bill has the last 5 years of sales/views/traffic data in the database (anything before that is in a bunch of excel spreadsheets on the share drive), and knows just enough python to be dangerous. Bill can probably explain an R-squared value but is not an expert at statistics by any measure. He wants to fit the data, but has several problems:

1) the weekly trend does not line up with the yearly data, as the year starts on a different weekday.

2) Those damn public holidays, some of them occur on a specific date, some of them on the "first Monday of the month", and some of them seem to change almost randomly year-to-year.

3) The reporting system was down for a couple of weeks in June and Feb last year, and the numbers for the first few years were copied from excel, so sometimes are missing the first or last day of the month.

Prophet comes by default with yearly/weekly seasonality. Prophet comes out-of-the-box with a simple way to import holidays, and even a way to specify your own. Prophet doesn't require any cleaning, or special procedures to deal with missing data. And it is quick and easy to use, and get nice-looking, broadly reasonable graphs out (with the above mentioned, consistent data). And that solves the business problem.

And Bill's probably heard of it because it is (a) popular already, and (b) has Facebook's name attached.

That's my take as to why, even if it is not even close to the most accurate method, Prophet is so broadly popular.

[1] https://news.ycombinator.com/item?id=27697274

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#57
post #17
post #14

My experience in general is that most time series model are inadequate in predicting time series except for very trivial cases of seasonalities or simple linear/nonlinear trends. I think that you can throw any model you like at the problem but all you will do is overfit most of the time.

Personally I find there is one important factor, and one factor alone. Context. Seasonality is the context that there's a seasonal driver at play. The context of a public holiday can explain a decrease in sales on that day. The context of a football match can explain a spike in transport demand near a stadium. The context of the presence of a heat dome predicted by pressure data can explain record temperature figures…

> The context of a public holiday can explain a decrease in sales on that day.

As an aside, the way this is handled is with a calendar like the https://nrf.com/resources/4-5-4-calendar

I wrote a time library once that handled calendars like this automagically, so you can roll the context in if you're clever enough.

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#58
post #42

As someone who spent a good part of my professional career in forecasting and time-series analysis I would like to point out that "point-forecasts" are mostly useless in many import practical applications such as FinTech, e-commerce, sports betting, etc. Point-forecast models such as Prophet fail to give you a meaningful measure of uncertainty of the predicted value. A much better approach are probabilistic forecasti…

It's worth noting that one of the original authors of Prophet (Sean Taylor) basically agrees with this and is now on a team at Lyft doing probabilistic forecasting for their marketplace [0].

[0] https://twimlai.com/causal-models-in-practice-at-lyft-with-s...

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#59
Is there a way to adapt time-series products like this to analysis of categorical data, in my case audit logs. I'd love to be able to suss out patterns/clusters/sequences but most of the stuff I've looked at requires you to do unnatural things like flip the categorical content into distinct dimensions with a calculated rate metric.

Re: Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?

#60

To close, let me say that this post ended up being more negative than I expected and, like Nessie, my opinion may rise in the future when I understand the implications of the Prophet generative model better, and either modify it or find better ways to identify its strengths. The unanswered question here is why Prophet is so popular, and this surely merits a better explanation than I have given. I think there are prob…

The title is a bit strong too, but maybe that’s what you were going for.
Post reply on HN