Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Fall of Big Data

argmin.net

Post reply on HN
Print view View original

11–20 of 35 posts

Re: The Fall of Big Data

#11

Post by loorinm »

sudoscript wrote: post #5

> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think. I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the cou…

Yes. This is exactly what happened and what so many liberals (myself included) don't want to hear but have to come to terms with. It's in our best interest to share the prosperity we enjoy with those less fortunate, or we create a breeding ground for anger and ignorance that will come back to truly make our lives hell.

Same thing applies in the middle east where we've bred terrorism from poverty.

This is our collective fault. We can't ignore them anymore. The only way forward is to empathize with our fellow human beings whose suffering is real and whose anger is dangerous.

Re: The Fall of Big Data

#12

Post by dajohnson89 »

sudoscript wrote: post #5

> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think. I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the cou…

It's not just poors who voted for trump. So I think calling a vote for trump a vote for survival is a stretch.

Re: The Fall of Big Data

#13

Post by ramblenode »

Clickbait.

The TLDR:

A) Polls were biased and we don't have many elections on which to build good models.

B) Social media created echo chambers.

C) Other people's politics are sci-fi and more people should agree with me.

Re: The Fall of Big Data

#14

Post by matt4077 »

huffpopo wrote: post #6

A) This is NOT Big Data B) This election was 101 in ML. The biases were obvious and easy to undo. C) I called it and won a lot of money https://news.ycombinator.com/item?id=12863445 D) Best not to take advice from people who are constantly wrong E) Big Data and ML are still awsome :) See below from my linked post a week before the election: The problem with Bayesian is that it's often over confident in the model. Err…

Congratulations on the money, but it doesn't really proof anything, especially not (B). There were hundreds of people working on this, considering all sorts of possible sources of errors. Any bias that is "obvious and easy to undo" was priced in.

And the polls were actually not as bad – the difference was below the 3% from 2012. It just happened to swing the result.

Re: The Fall of Big Data

#15

Post by matt4077 »

huffpopo wrote: post #9
huffpopo wrote: post #6

A) This is NOT Big Data B) This election was 101 in ML. The biases were obvious and easy to undo. C) I called it and won a lot of money https://news.ycombinator.com/item?id=12863445 D) Best not to take advice from people who are constantly wrong E) Big Data and ML are still awsome :) See below from my linked post a week before the election: The problem with Bayesian is that it's often over confident in the model. Err…

NOTE: The link only appears to work for my old account?? Was I shadow banned? Perhaps banning decenting opinions are a big reason why this election was such a supprise to everyone. https://news.ycombinator.com/threads?id=ttyltrful

Not banned / save the conspiracies for another day / "dissenting" / "surprise"

Re: The Fall of Big Data

#16

Post by lsc »

sudoscript wrote: post #5

> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think. I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the cou…

>There is real desperation in this country. That's why Trump is President. People didn't vote for him because they are racist. They voted for him because they worry about putting food on their tables. They worry about being left behind in a world that has fewer and fewer ways for the average Joe to dream about a future. They're angry, rightfully so, and in Trump, they found someone willing to listen

How do you square the poverty story above with the fact that trump ran on a fiscally conservative platform, within a fiscally conservative party that promises to reduce the safety net?

Re: The Fall of Big Data

#17

Post by matt4077 »

sudoscript wrote: post #5

> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think. I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the cou…

This is the narrative currently being pushed, but let's not forget its weaknesses:

- A blue-collar senior in Appalachia is really unlikely to have enough contact with those "liberal college students" to be bothered with their constant references to Judith Butler etc.

- If the issue is with racism or homophobia, I don't see a compromise happening: minority rights aren't subject to popular approval. Democracy isn't two wolves and a sheep deciding on dinner.

- Trump voters are by and large not the poorest. The white guy with missing teeth living in a trailer is the wrong cliche, because he's unlikely to vote at all. These are conservatives, many retired.

- Every time some politician talks about "real America", he's trying to delegitimize liberals. I fail to see how that is less insulting.

- People who decide to punish the weakest of the weak, just because it's a convenient way to annoy me, are indeed worse people.

- The left should get back to talking about economics, yes. But I doubt that these voters would appreciate it, seeing as the last time the left made the case against neoliberalism, most of them probably complained about "the smelly hippies who should get a job instead of taking about some 99% or 1% BS"

- there are quite a few improvements for a factory worker who lost his job that the left has implemented, or would like to if not for a complete boycott of government by the right: universal healthcare, universal income, consumer financial protection (payday-loan and similar specifically targeting poor people). Strong anti-trust regulation. Redirecting money from the military to investments in infrastructure/education/health...

Re: The Fall of Big Data

#18

Post by quicklyfrozen »

lsc wrote: post #16
sudoscript wrote: post #5

> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think. I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the cou…

>There is real desperation in this country. That's why Trump is President. People didn't vote for him because they are racist. They voted for him because they worry about putting food on their tables. They worry about being left behind in a world that has fewer and fewer ways for the average Joe to dream about a future. They're angry, rightfully so, and in Trump, they found someone willing to listen How do you square…

I have to assume that many don't want a safety net; they want to be able to be able to support themselves.

Re: The Fall of Big Data

#19

Post by sudoscript »

lsc wrote: post #16
sudoscript wrote: post #5

> The third major failure has been a general apathy about politics amongst my colleagues here in the Bay Area. When many of the best minds in machine learning have decided that the most existential threat to civilization is the rise of Skynet, we have had a major failure of group think. I cannot agree more. I am also in the Bay Area, and I feel surrounded by people who are deeply out of touch with the rest of the cou…

>There is real desperation in this country. That's why Trump is President. People didn't vote for him because they are racist. They voted for him because they worry about putting food on their tables. They worry about being left behind in a world that has fewer and fewer ways for the average Joe to dream about a future. They're angry, rightfully so, and in Trump, they found someone willing to listen How do you square…

Not everyone believes that the solution to poverty is welfare. I imagine many of his voters believe the solution looks more like protectionism. Which is basically what his plan is.

I'm not claiming they're right. I'm just claiming that you can be a fiscal conservative and still think you have the solution to poverty.

Re: The Fall of Big Data

#20

Post by huffpopo »

matt4077 wrote: post #14
huffpopo wrote: post #6

A) This is NOT Big Data B) This election was 101 in ML. The biases were obvious and easy to undo. C) I called it and won a lot of money https://news.ycombinator.com/item?id=12863445 D) Best not to take advice from people who are constantly wrong E) Big Data and ML are still awsome :) See below from my linked post a week before the election: The problem with Bayesian is that it's often over confident in the model. Err…

Congratulations on the money, but it doesn't really proof anything, especially not (B). There were hundreds of people working on this, considering all sorts of possible sources of errors. Any bias that is "obvious and easy to undo" was priced in. And the polls were actually not as bad – the difference was below the 3% from 2012. It just happened to swing the result.

By "obvious" I meant to ML practitioners (esp frequentists which is more typical for big data). Not everyone. So not priced in. It is not an efficient market. There are huge information asymmetries. I will give you that calling it ML 101 was over the top.

If you dig into the polling breakdown and compare the individual polls (states as well as national) with the actuals it is clear that the polls were bad. Just because the aggregate error was less off than it otherwise could have been is more luck than anything. It could be argued that you can assume a gaussian error distribution so a more precise aggregate is to be expected but I would argue that that is not a safe assumption.

TLDR; Just because it was close doesn't make it a good model.

As an aside. All of my Bayesian friends lost money and all of my Frequents friends made money :)

Post reply on HN