Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Looking at the data behind prediction markets

asteriskmag.com

Post reply on HN
Print view View original

21–30 of 55 posts

Re: Looking at the data behind prediction markets

#21

Post by idiotsecant »

torben-friis wrote: post #16

Given the request about engaging with this specific article: >Ive thought hard about how to sell prediction markets to consumers. In 2020, I created Google’s current internal prediction market. Since then, I’ve served as the CTO of Metaculus, a non-market-based crowd-forecasting website, and now run FutureSearch, a startup that provides AI forecasters and researchers. I feel like openly saying you professionally try…

Its based on high quantity decision making, and quantity is a sort of quality if you squint and turn your head

Re: Looking at the data behind prediction markets

#22

Post by dimbletimbers »

idiotsecant wrote: post #21
torben-friis wrote: post #16

Given the request about engaging with this specific article: >Ive thought hard about how to sell prediction markets to consumers. In 2020, I created Google’s current internal prediction market. Since then, I’ve served as the CTO of Metaculus, a non-market-based crowd-forecasting website, and now run FutureSearch, a startup that provides AI forecasters and researchers. I feel like openly saying you professionally try…

Its based on high quantity decision making, and quantity is a sort of quality if you squint and turn your head

"Quantity has a quality all its own" —known economic genius, Joseph Stalin

Re: Looking at the data behind prediction markets

#23

Post by [deleted] »

rpjt wrote: post #6

Random aside: I distinctly remember getting on a phone call with people from the SEC (US Gov't) with the goal of understanding if I could legally start a prediction market. This was during 2020 or 2021. I recall them saying basically "no way" and that it wouldn't be legal, and would be rife with abuse. Fun times.

[deleted]

Re: Looking at the data behind prediction markets

#24

Post by ddp26 »

bo1024 wrote: post #19

Nice article. One small comment, it's very hard to conclude anything about accuracy over time because the comparisons may not be apples to apples. For example if there used to be lots of questions about if it will rain in Boston and now there are lots of questions about if it will rain in Phoenix, it will look like predictions are getting more accurate, but the questions are just getting easier.

Author here. Agree, and I wrote in that section "Absolute accuracy is hard to compare across markets on one platform, and across platforms, because different forecasting questions have different difficulties. I addressed this by tracking similar markets on a single platform over time."

Even doing this, it's not apples-to-apples. One thing is, in this article, I filter only to "interesting" markets, so that controls for the % that are "easy" as you describe.

Re: Looking at the data behind prediction markets

#25

Post by ddp26 »

infinitewars wrote: post #3

It sounds like they should be called "indicator markets" rather than "prediction markets", as the data shows they largely just summarize the current knowledge, with little predictive ability.

It's true they are "just" summarizing current knowledge. But there are better and worse summaries of current knowledge!

Some summaries, like on some prediction markets, have objective accuracy that is much better than chance.

Re: Looking at the data behind prediction markets

#26

Post by ddp26 »

jonahx wrote: post #20
Beijinger wrote: post #18

Most people don't know, that "prediction markets" are acutally based on an idea by DARPA in 2002, after 9/11/2001.

Prediction markets, by any reasonable definition, existed long before 2002.

Yeah. People have put together a Prediction Market Database [1] (in a Google sheet), I think it's pretty well sourced and shows a good number of both real money and play money prediction markets from before 2002.

DARPA did have a big role though, too.

[1] https://docs.google.com/spreadsheets/d/1vGjnJPxdnBKwag3Q9Uy_...

Re: Looking at the data behind prediction markets

#27

Post by NDlurker »

rpjt wrote: post #6

Random aside: I distinctly remember getting on a phone call with people from the SEC (US Gov't) with the goal of understanding if I could legally start a prediction market. This was during 2020 or 2021. I recall them saying basically "no way" and that it wouldn't be legal, and would be rife with abuse. Fun times.

Should have just went for it

Re: Looking at the data behind prediction markets

#28

Post by ddp26 »

ghaff wrote: post #7

I dove into the prediction markets rabbit hole a number of years back. And I’ve personally seen witnessed scenarios where the wisdom of crowds seems to really work. What I have not really—including in this piece—read is rigorous theory of what makes it effective or not. There are hints here and in the Wisdom of Crowds book but I’ve never read a really comprehensive theory.

Author here. Hal Varian pointed me to this 1992 paper, which I think is still considered the canonical empirical piece on what is actually going on in trading behavior that leads to accuracy (or not): https://www.jstor.org/stable/2117471

Re: Looking at the data behind prediction markets

#29

Post by ddp26 »

tech_ken wrote: post #14

One thing that really jumps out to me is the lack of a performance gap between the 90-day and 30-day resolution times. If 2-months of new information doesn't lead to materially improved forecast, then to me this seems to strongly reinforce the takeaway that these markets aren't really forecasting, so much as "the oracle is largely saying what other oracles already say, just updated faster." Am I misunderstanding the…

Author here. Great point, and I think this is due to what another commenter points out, that the questions are different.

The right test of this is to take the _same_ markets that run for 90+ days, and check accuracy 90 days out vs 30 days out. I've done this on other prediction market datasets, though not on Kalshi and Polymarket, and found that forecasts are in fact more accurate 30 days out.

I agree that if they weren't, that would be incredibly suspicious!

Re: Looking at the data behind prediction markets

#30

Post by cwmoore »

dimbletimbers wrote: post #22
idiotsecant wrote: post #21

Earlier quoted context omitted.

Its based on high quantity decision making, and quantity is a sort of quality if you squint and turn your head

"Quantity has a quality all its own" —known economic genius, Joseph Stalin

“More is different.” - Nobel laureate Phillip W. Anderson

https://www.tkm.kit.edu/downloads/TKM1_2011_more_is_differen...

Post reply on HN