101–110 of 114 posts
Earlier quoted context omitted.
Stan is nice but its GPL license is taboo in my corporate environment :( . I am puzzled how they managed to release Prophet under BSD with such a dependency.
Stan has a BSD core. Prophet must avoid the GPLv3 interfaces.
Earlier quoted context omitted.
> Probably just help verify that the stock market is a random walk with a meager trend upwards that doesn't beat inflation + trading costs. That doesn't sound right. Let me clear that up for you. Since 1950: S&P 500 Annual Price Change: 7.2% S&P 500 Annual Div Dist: 3.6% S&P 500 Annual Total Return: 11.0% Annual Inflation: 3.8% Annual Real Price Change: 3.3% Annual Real Total Return: 7.0 % Buying the straight S&P 500…
Buying the S&P 500 in 1950 and holding 67 years does. One sample tells you nothing about randomness. What if you buy in August 1929? What if you hold for a more realistic 20 or 30 years from peak earning years to retirement?
Annual Total Return: 9.1%
Annual Real Total Return: 5.9%
Annual Total Return: 9.8%
Annual Real Total Return: 7.0%
If you're interested, here's a simulator that looks at historic market data. You'll note that even the lowest possible percentile of 30-year holding periods will still yield a 3.43% inflation-adjusted total return: https://dqydj.com/sp-500-historical-return-calculator-popout...
So...How much will this do at forecasting stock prices? =) Very cool though --- I would be interested to dive into the methods they've implemented sometime in the near future!
It would probably perform pretty poorly as other has suggested. This is mainly due to the fact that stock prices by itself is a pretty non-stationary dataset/measurement. Most of these probabilistic models are poorly equipped to make accurate predictions for non-stationary data since it's trends are seemingly similar to noise.
-Mandelbrot, in the foreward to Multifractals and 1/f Noise.
it's worth saying that Mandelbrot was apparently a large influence to E Fama, who proposed the efficient market hypothesis in the first place.
Earlier quoted context omitted.
> the stock market is a random walk with a meager trend upwards that doesn't beat inflation + trading costs. That assumes that the efficient-market hypothesis holds true, but it has yet to be thoroughly proven or disproven... (and funds like Medallion would strongly suggest otherwise for the medium term: https://www.bloomberg.com/news/articles/2016-11-21/how-renai... )
It doesn't assume the Effiecient Market hypothesis - empirical studies of returns support random returns without the imposing a model (non-parametric tests). That's not to say returns are actually random, but in any given time range, it appears to be.
Or are you saying that movements aren't actually random, and only appear to be?
Earlier quoted context omitted.
... and if you like Clojure, you might try Bayadera, which has its own engine running the analysis on the GPU. http://github.com/uncomplicate/bayadera
I'm pretty interested in this as I do most of my work on the JVM and I love trying this out on our stream processor at work. Cloned and tried to build it but I'm getting an error regarding uncomplicate:commons:0.3.0-SNAPSHOT being unavailable on clojars. Is that something you currently have installed to your local maven repo? I don't see it here: https://clojars.org/repo/uncomplicate/commons/ I can get it to build wi…
Earlier quoted context omitted.
... and if you like Clojure, you might try Bayadera, which has its own engine running the analysis on the GPU. http://github.com/uncomplicate/bayadera
I'm pretty interested in this as I do most of my work on the JVM and I love trying this out on our stream processor at work. Cloned and tried to build it but I'm getting an error regarding uncomplicate:commons:0.3.0-SNAPSHOT being unavailable on clojars. Is that something you currently have installed to your local maven repo? I don't see it here: https://clojars.org/repo/uncomplicate/commons/ I can get it to build wi…
This is an interesting project, and in one of the areas where almost all businesses could do better. Anecdotally, there is a ton of money left on the table by established businesses that do it poorly, which also leaves lots of room for resume-padding technical experience. So anything that claims to improve the state of the art of automated forecasting is definitely worth watching. That being said this claim in point…
True. fwiw, I worked on the same project at Twitter 4 years back - the Facebook folks call it capacity planning at scale, we called it capacity utilization modeling. The goal was the same - there are all these "jobs" - 10s of 1000s of programs running on distributed clusters, hogging CPU, memory and disk. Can we look at a snapshot in time of the jobs usage, and then predict/forecast what the next quarter jobs usage would be ? If you get these forecasts right ( within reasonable error bounds ), the folks making purchasing decisions ( how many machines to lease for the next quarter for the datacenters) can save a bundle.
From an engineering pov, every job would need to log it's p95 and p99 CPU usage, memory stats, disk stats...Since Twitter was running some 50k programs back then (2013ish) on these Mesos clusters, the underlying C++ API had hooks to obtain CPU and memory stats, even though the actual programs running were all coded up in Scala (mostly), or python/Ruby (bigger minority), or C/Java/R/perl ( smaller minority ). There's an interesting Quora discussion on why Mesos was in C++ while rest of Twitter is Scalaland...mostly because you can't do these sort of CPU/memory/disk profiling in the jvmland as well as you can in C++.
OK, so you now have all these CPU stats. What do you do with them ? Before you get to that, you have the usual engineering hassles - how often should you obtain the CPU stats ? Where would you store them ?
So at Twitter we got these stats every minute ( serious overkill :) and stored them in a monstrous JSON ( horrible idea given 50000 programs * number of minutes in day * all the different stats you were storing :))
So every day I'd get a gigantic 20gb JSON from infra, then I'd have to do the modeling.
In those days, you couldn't find a single Scala JSON parser that would load up that gigantic JSON without choking. We tried them all. Finally we settled on GSON - Google's JSON parser written in Java, that handled these gigantic jsons with no hiccups.
Before you get to the math, you would have to parse the JSON and build a data structure that would store these (x,t) tuples in memory. You had 50k programs, so each program would get a model, each model originated from a shitton of (x,t) tuples, the t being minutely and the fact that some of these programs had been running for years, meant you had very large datasets.
The math was relatively straightforward...I used so called "LAD" - least absolute deviation from mean, as opposed to simple OLS, because least squares wasn't quite predictive for that use case. Building the LAD modeling thing in Scala was somewhat interesting...Most of the work was done by the commons math Apache libraries, I mostly had to ensure the edge cases wouldn't throw you off, because LAD admits multiple solutions to the same dataset - it's not like OLS where you give it a dataset and it finds a unique best fit line. Here you'd have many lines sitting in an array, depending on how long you let the Simplex solver run. Then came the problem of visualizing these 50,000 piecewise line models using javascript heh heh. The front end guys had a ball with the models I spit out.
If someone's doing this from scratch these days, NNs would be your best bet. Regime changes are a big part of that.
Earlier quoted context omitted.
I'm pretty interested in this as I do most of my work on the JVM and I love trying this out on our stream processor at work. Cloned and tried to build it but I'm getting an error regarding uncomplicate:commons:0.3.0-SNAPSHOT being unavailable on clojars. Is that something you currently have installed to your local maven repo? I don't see it here: https://clojars.org/repo/uncomplicate/commons/ I can get it to build wi…
Please also note that it currently requires OpenCL 2.0 compatibility, and is optimized for AMD GPUs. I plan to add CUDA support later this year.
Earlier quoted context omitted.
Buying the S&P 500 in 1950 and holding 67 years does. One sample tells you nothing about randomness. What if you buy in August 1929? What if you hold for a more realistic 20 or 30 years from peak earning years to retirement?
Bought way back in August 1929: Annual Total Return: 9.1% Annual Real Total Return: 5.9% Bought in January 1987, held for a realistic 30 years: Annual Total Return: 9.8% Annual Real Total Return: 7.0% There's always going to be some deviation, but over any given multi-decade holding period, you will generally end up with a predictable 5-9% annualized (inflation-adjusted) return. That is more than zero. My point stand…
If not, look at http://www.macrotrends.net/1319/dow-jones-100-year-historica...
Let's buy in August 1929 at 5338.69, and sell 20 years later, in August 1949, at 1822.87 (inflation-adjusted). Congratulations, you lost two thirds of your money.
Sell 30 years later instead? August 1959, at 5525.23. Wow, after 30 years you're up almost 3.5%!