Smart Folk Often Full of Crap, Study Finds

The concept of calibration - actually being right at the rate that matches your estimated percent confidence - is important for machine learning. It's important to know how well you can trust the prediction of a ML model (ironically to that you can do something like send low confidence predictions for manual review, which apparently might not be that calibrated either)... Deep learning models in particular are not good at accurately estimating confidence, so there is a whole body of work on getting better confidence estimates, e.g. [0]. It's nice to see that people make the same kind of mistakes.

[0]Guo et al., https://arxiv.org/abs/1706.04599

I am not completely sure I believe that Black Magin Woman was first performed by Feetwood Mac [sic].

And I confess I have no idea what a stop side is. But apparently I was correct to guess that it does not have six sides.

And I remain dubious that the Mona Lisa was painted by someone other than Leonardo.

Yeah, I had the thought that the math might work out in such a way that this is the expected result.

That is, if you say you are "50%" confident, it really means you have no confidence - simply guessing would get you 50%, so you can't go less than that. But on the other end, for people who say they're 100% or 95% confident, it's pretty much all downside since you can't go over 100%.

Job applicants for junior trading (market making) roles are often given similar questions. An example question might be:

"Make me a market on the height of the empire state building."

By this the interviewer means: what price would you buy at and what price would you sell at?

So for example I might respond 1000/2000. which means I'm buying at 1000ft and selling at 2000ft. Presumably, I believe the true height is somewhere between these two numbers (ideally 1500ft), at a certain confidence level. The interviewer might ask you to quote a spread given a confidence interval.

The interviewer might then lift your bid or ask and then ask you to make a new market. There's a couple factors at play here:

1. The interviewer has more information than you (presumably), so whether he bought from or sold to you is valuable information. You might want to raise your quotes if he buys in order to capitalize on the new information.

2. You want to decrease your risk. So for example if he just lifted your bid, you might want to lower your quotes: offer a lower price to buy and a lower price to sell. A good market maker capture the difference between the bid-ask spread: the price for which you buy at and the price that you sell at (like any other middle man). By lowering your quotes, you are increasing the probability that you will flatten out your risk (you are not trying to speculate on the height of the empire state building).

These two things are in contention, and the interviewer wants to see how you think about it/deal with it.

Also the interviewer wants to ascertain whether your confidence levels are correct over multiple questions. ie if the confidence level is 80%, then you should be wrong (true value lies outside your range) about 20% of time. It is not good to be either under or over confident. Accurately dealing with uncertainty is what they are looking for.

Anyways I thought this was a very similar process that you were referring to, except you are dealing with money. Which always helps to clarify things!

Google will easily convince you of the first two. The last one is speculation.