Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Data Can’t Do

newyorker.com

Post reply on HN
Print view View original

91–100 of 106 posts

Re: What Data Can’t Do

#91

Post by erichahn »

contravariant wrote: post #77
erichahn wrote: post #22

Earlier quoted context omitted.

Isn't the point of ML exactly that you don't know the underlying distribution? How is this ever assumed in any way? ML is not parametric statistics.

Well, all optimization problems are equivalent to a maximum likelihood estimate for a corresponding probability distribution so you may make more implicit assumptions than you think. Typical ML methods just have a huge distribution space that can fit almost anything from which they pick just 1 option. This has two downsides: Since your distribution space is several times too large by design you lose the ability to sa…

I mean yes, there is parametric ML (maximum likelihood, MAP, GMMs, ...) and there is non-parametric ML (everything neural network, SVM, GBM, random forrests, ...).

I'd argue that the latter had bigger success in the past since the prior on the data distribution is usually wrong in real life. Think about a prior for image data distributions or the same in nlp. Forget about it.

Re: What Data Can’t Do

#92

Post by carlosf »

sjburt wrote: post #81
_dps wrote: post #72

Earlier quoted context omitted.

It's not characteristic of all forecasting, only purely empirical forecasting. Definitionally, the only way to reason about risk that doesn't appear in training data is non-empirical (e.g. a priori assumptions about distributions, or worst cases, or out-of-paradigm tools like refusing to provide predictions for highly non-central inputs). DL is not any better (or worse) than any other purely empirical method at answe…

I think it's wrong to assume that non-empirical methods can be reliably trusted to give better results. Humans are terrible at avoiding bias or evaluating risks, especially for uncommon events.

Food for thought: if every method for predicting event x is terrible, then you might as well not try to predict x and build your life in such way that you never expose yourself to the risk of x happening.

Re: What Data Can’t Do

#93

Post by lmm »

mjburgess wrote: post #21
lmm wrote: post #17

Does the use of statistics actually amplify misunderstanding, or merely reveal misunderstandings that were already there? In any of these examples given - predicting rearrests, infant mortality, or so on - it's hard to imagine that someone not using numbers would have reached a conclusion that was any closer to the truth. Data has its limits, but the solution is usually - maybe even always - more data, not less.

It's pretty trivial to predict things without "data". Data just means using some measurement system to obtain measurements of some target phenomenon. Many targets cannot be measured, or have not occurred to be measured. Reasoning counter-factually is trivial: What would happen if I dropped this object in this place in which an object, of this kind, has never been dropped before? Well apply relevant models, etc. and "…

But where does that model come from if not from data? We might use some logical principles to inform our model - but don't those principles themselves also ultimately have to be inferred from data about the world?

Re: What Data Can’t Do

#95

Post by sidpatil »

sfink wrote: post #50
carlosf wrote: post #4

I am increasingly worried with people applying ML in everything without any rigour. Statical inference generally only works well in very specific conditions: 1 - You know the distribution of the phenomenon under study (or make an explicit assumption and assume the risk of being wrong) 2 - Using (1), you calculate how much data you need so you get an estimation error below x% Even though most ML models are essentially…

Personally, I think the main problem with ML is simpler: it works well for interpolation, and is crap for extrapolation. If the outputs you want are well within the bounds of your training data set, ML can do wonders. If they aren't, it'll tell you that in 20 years everyone will be having -0.2 children and all the other species on the planet will start having to birth human babies just so they can be thrown into the…

> If they aren't, it'll tell you that in 20 years everyone will be having -0.2 children and all the other species on the planet will start having to birth human babies just so they can be thrown into the smoking pit of bad statistical analysis.

https://xkcd.com/605/

Re: What Data Can’t Do

#96

Post by Nasrudith »

lupire wrote: post #54
dsaavy wrote: post #38

Earlier quoted context omitted.

> *Numbers don’t lie, except when they do. Harford is right to say that statistics can be used to illuminate the world with clarity and precision. They can help remedy our human fallibilities. What’s easy to forget is that statistics can amplify these fallibilities, too. As Stone reminds us, “To count well, we need humility to know what can’t or shouldn’t be counted.”* I do have a problem with her conclusion here. Ar…

> Are numbers really lying if it's actually an incorrect data collection method or conflicting definitions of criteria for generation of certain numbers Obviously it's a figurative metaphor, but it's pretty clearly a case of "this supposedly objective factual calculation is presenting an untruth."

You can still be very misleading with objectively true calculations. "There is very low stress on the patient's arteries and only a very small tear." - said patient bled to death and has only atmospheric stress now that their veins are bloodless. Less than 0.1% of their total vein area has a rip in it.

Re: What Data Can’t Do

#97

Post by clircle »

sfink wrote: post #50
carlosf wrote: post #4

I am increasingly worried with people applying ML in everything without any rigour. Statical inference generally only works well in very specific conditions: 1 - You know the distribution of the phenomenon under study (or make an explicit assumption and assume the risk of being wrong) 2 - Using (1), you calculate how much data you need so you get an estimation error below x% Even though most ML models are essentially…

Personally, I think the main problem with ML is simpler: it works well for interpolation, and is crap for extrapolation. If the outputs you want are well within the bounds of your training data set, ML can do wonders. If they aren't, it'll tell you that in 20 years everyone will be having -0.2 children and all the other species on the planet will start having to birth human babies just so they can be thrown into the…

Outside of simple time series, I'm not aware of any good way to extrapolate.

Re: What Data Can’t Do

#98

Post by jrochkind1 »

mdip wrote: post #13

Kind of love the initial story in the article about 48-hour wait times. I had a stint writing conferencing software for quite some time, and every once in a while we'd come across a customer requirement that had capabilities which were obvious to us developers "would be misused". As a result, we did the "Thinking, Fast and Slow" pre-mortem to help surface other ways that the system could be attacked (along with what…

What is 'the "Thinking, Fast and Slow" pre-mortem'?

Re: What Data Can’t Do

#99

Post by jrochkind1 »

strathmeyer wrote: post #37
mdip wrote: post #13

Kind of love the initial story in the article about 48-hour wait times. I had a stint writing conferencing software for quite some time, and every once in a while we'd come across a customer requirement that had capabilities which were obvious to us developers "would be misused". As a result, we did the "Thinking, Fast and Slow" pre-mortem to help surface other ways that the system could be attacked (along with what…

How do we read more than the initial story? Do we have to pay to read it? There is no indication on the webpage there is more than two paragraphs other than the advertisement for the author's book.

Not what I see, including in a Chrome incognito window (which shows me the whole article). Could you have an ad blocker or other extension that is going wrong? Do you use an unusual browser?

Re: What Data Can’t Do

#100

Post by tootie »

ppod wrote: post #34

This author has published a couple of articles like this at the New Yorker They all have this in common: the author works through some interesting and in some ways unusual cases where data or statistics have been improperly or naively applied, with some social costs. I really enjoy the articles themselves. Then the New Yorker packages it up with a cartoon and a headline and subheadline like "Big Data: When will it ea…

It's ironic that you accuse them of sensationalizing headlines by making up a sensationalized headline. The headline for this article is pretty neutral and sets up an interesting article that teases a very deep topic with references to books that explain it further. And the cartoon was pretty funny. This is exactly what I expect from the New Yorker and I'm rarely disappointed.
Post reply on HN