Live data from Hacker News

What Data Can’t Do

newyorker.com

31–40 of 106 posts

Re: What Data Can’t Do

#31
post #28
post #4

I am increasingly worried with people applying ML in everything without any rigour. Statical inference generally only works well in very specific conditions: 1 - You know the distribution of the phenomenon under study (or make an explicit assumption and assume the risk of being wrong) 2 - Using (1), you calculate how much data you need so you get an estimation error below x% Even though most ML models are essentially…

i think this actually gets at what makes applied ML distinct from statistics as a practice, even though there is a ton of overlap. statisticians make assumptions 1 and 2, and think of themselves as trying to find the "correct" parameters of their model. people doing applied ML typically assume they don't know 1 (although they might implicitly make some weak assumptions like sub-gaussian to avoid fat tails, etc.) and…

Yes - this is pretty much exactly how I explain the difference between machine learning and statistics.

Despite using similar models, the expertise required for 'doing statistics' (statistical inference) is actually very different from machine learning. Machine learning fits into the 'hacker mentality' well - try stuff out see what works. To do statistical inference effectively, you really do need to spend time learning the theory. They both require deep skills - but the skills are surprisingly different considering it's often the same underlying model.

Re: What Data Can’t Do

#32
post #4

I am increasingly worried with people applying ML in everything without any rigour. Statical inference generally only works well in very specific conditions: 1 - You know the distribution of the phenomenon under study (or make an explicit assumption and assume the risk of being wrong) 2 - Using (1), you calculate how much data you need so you get an estimation error below x% Even though most ML models are essentially…

As currymj commented, this isn't accurate for ML, only for classical statistics.

In ML (or more specifically deep learning), we make no distribution-based assumptions, other than the fundamental assumption that our training data is "distributed like" our test data. Thus, there aren't issues with fat-tailed distributions since we make no such normality assumptions. Indeed, with the use of autoencoders, we don't assume a single distribution, but rather a stochastic process.

I suppose you could say statistics is less "empirical" than ML in the sense that it is axiom-based, whether that is a normality assumption of predictions about a regression line or stock prices following a Wiener process. By contrast, ML is less rationalist by simply reflecting data.

Re: What Data Can’t Do

#33
post #28
post #4

I am increasingly worried with people applying ML in everything without any rigour. Statical inference generally only works well in very specific conditions: 1 - You know the distribution of the phenomenon under study (or make an explicit assumption and assume the risk of being wrong) 2 - Using (1), you calculate how much data you need so you get an estimation error below x% Even though most ML models are essentially…

i think this actually gets at what makes applied ML distinct from statistics as a practice, even though there is a ton of overlap. statisticians make assumptions 1 and 2, and think of themselves as trying to find the "correct" parameters of their model. people doing applied ML typically assume they don't know 1 (although they might implicitly make some weak assumptions like sub-gaussian to avoid fat tails, etc.) and…

"Every parameter is a nuisance parameter" is a great way to put it.

Re: What Data Can’t Do

#34
This author has published a couple of articles like this at the New Yorker They all have this in common: the author works through some interesting and in some ways unusual cases where data or statistics have been improperly or naively applied, with some social costs. I really enjoy the articles themselves.

Then the New Yorker packages it up with a cartoon and a headline and subheadline like "Big Data: When will it eat our children?" or "Numbers: Do they even have souls?", and serves it up to their technophobic audience in a palatable way.

https://www.newyorker.com/contributors/hannah-fry

Re: What Data Can’t Do

#35
"once a useful number becomes a measure of success, it ceases to be a useful number"

Two other unintended consequences of incentives I learned in economics:

1. Increasing fuel efficiency does not reduce gas consumption. People just use their car more often.

2. Asking people to pay-per-bag for garbage pickup resulted in people dumping trash on the outskirts of town.

Edit: Did more research after downvote. Definitely double check things you learn in college

1. The jury is still out: https://en.wikipedia.org/wiki/Jevons_paradox

2. Seems false https://en.wikipedia.org/wiki/Pay_as_you_throw#Diversion_eff...

Re: What Data Can’t Do

#36

The article seems to stop pretty early, as if something is missing. It’s an anecdote about a government incentive to have doctors see patients within 48 hours causing doctors to refuse scheduling patients later than 48 hours in order to get the incentive bonus. This is not an example of limits of data, but an example of perverse incentives.

Did you use reader mode? because I noticed that when I use firefox's reader mode it will cut off part of the article on the New Yorker site.

We clicked on the link and were presented with two paragraphs. New Yorker articles usually don't show up correctly, I don't know why they allowed to be posted here. Acting like we don't know how to read a webpage is gaslighting.

Re: What Data Can’t Do

#37
post #13

Kind of love the initial story in the article about 48-hour wait times. I had a stint writing conferencing software for quite some time, and every once in a while we'd come across a customer requirement that had capabilities which were obvious to us developers "would be misused". As a result, we did the "Thinking, Fast and Slow" pre-mortem to help surface other ways that the system could be attacked (along with what…

How do we read more than the initial story? Do we have to pay to read it? There is no indication on the webpage there is more than two paragraphs other than the advertisement for the author's book.

Re: What Data Can’t Do

#38
post #34

This author has published a couple of articles like this at the New Yorker They all have this in common: the author works through some interesting and in some ways unusual cases where data or statistics have been improperly or naively applied, with some social costs. I really enjoy the articles themselves. Then the New Yorker packages it up with a cartoon and a headline and subheadline like "Big Data: When will it ea…

> *Numbers don’t lie, except when they do. Harford is right to say that statistics can be used to illuminate the world with clarity and precision. They can help remedy our human fallibilities. What’s easy to forget is that statistics can amplify these fallibilities, too. As Stone reminds us, “To count well, we need humility to know what can’t or shouldn’t be counted.”*

I do have a problem with her conclusion here. Are numbers really lying if it's actually an incorrect data collection method or conflicting definitions of criteria for generation of certain numbers (like the example used in the second to last paragraph)? She seems to be pointing out a more important fact, which is that people don't question underlying data, how it was collected, and the choices those data collectors made when making a data set. People tend to take data and conclusions drawn from it as objective realities, when in reality data is way more subjective.

Re: What Data Can’t Do

#39

> doctors would be given a financial incentive to see patients within forty-eight hours. Not measuring that from the first contact that the patient made is simply dishonest. "Call back in three days to make the appointment, so I can claim you were seen within 48 hours, and therefore collect a bonus" amounts to fraud because the transaction for obtaining that appointment has already been initiated. I mean, they could…

It's really hard to design a not game-able metric. The problem here seems to be that doctors are under-provisioned for some reason, and so long wait times are a form of load shedding for the system. Without addressing this core issue, which individual clinics have little control over because they are generally boxed in by regulations over who can administer medical care, except to rush appointments (which they're probably already doing), there's not much they can do to solve the problem, so all they can do is try to game the rules or not get the bonuses.

Re: What Data Can’t Do

#40

> doctors would be given a financial incentive to see patients within forty-eight hours. Not measuring that from the first contact that the patient made is simply dishonest. "Call back in three days to make the appointment, so I can claim you were seen within 48 hours, and therefore collect a bonus" amounts to fraud because the transaction for obtaining that appointment has already been initiated. I mean, they could…

It may amount to fraud but in the context of that service, no record was kept of calls not resulting in an appointment. My wife was a receptionist at a GPs around the time the article mentions and in some cases it was worse than that - if you phoned and asked for an appointment, if they couldn't give you one within 48 hours they wouldn't offer one at all - telling you to call back later / the next day.

Although the New Yorker piece has leaned on the bonus angle the way it was discussed publicly was that doctors weren't allowed to offer you appointments outside the 48 hour window [0].

It was a very silly interpretation of the rules, but I think GPs felt it was too rigid and therefore stuck to the letter rather than the spirit.

0 - http://news.bbc.co.uk/1/hi/health/3682920.stm

Post reply on HN