Data Science: Reality Doesn't Meet Expectations • Hacker News Board

Re: Data Science: Reality Doesn't Meet Expectations

#121

Post by roystonvassey » Wed, Apr 08, 2020, 7:21 AM UTC

proverbialbunny wrote: post #64

hectormalot wrote: post #5

> Moreover, you may quickly realize much of this work is repetitive and while time-consuming, is “easy”. In fact, most analyses involve a great deal of time to understand the data, clean it and organize it. You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. This. Universities and online challenges…

>You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. I don't get why building a model people consider to be the "fun" part. That's mostly spitting data in, watching a loading screen, and then observing the output. That's not fun, that's boring. The fun part is looking at the data and gleaming all th…

> That's not fun, that's boring. The fun part is looking at the data and gleaming all these potential patterns from it, seeing what potential is there and what could be

Exactly! This is the reason why I love my job. It gets even better when you uncover a non-intuitive insight.

Re: Data Science: Reality Doesn't Meet Expectations

#122

Post by Rainymood » Wed, Apr 08, 2020, 7:23 AM UTC

Can definitely relate to this. Work for big consulting firm (F500) as a data scientist, end up in this weird software engineer/ml engineer hybrid role.

I personally love it but am doing more pure software engineering now as the infrastructure is not there and I need to build it myself.

Re: Data Science: Reality Doesn't Meet Expectations

#123

Post by mcrad » Wed, Apr 08, 2020, 7:49 AM UTC

starpilot wrote: post #90

kqr wrote: post #33

Earlier quoted context omitted.

I took a data visualisation class in uni that handled this really cleverly. The second assignment sounded very easy. The teacher provided links to the sources where we could find data. Most people figured that with such a simple assignment (not significantly harder than the first one, which was also easy-ish) they could put off doing it until the last moment. Most people failed. This real world data needed hours upon…

This is universal to STEM degrees I think. In mechanical engineering classes you analyze a beam, in real life you analyze an assembly with 50 components that have undergone 100 revisions with 20 different materials and loading from 4 directions that vary with time. Oh, and you have 4 sensors to give you information to analyze critical stresses. But one of them is broken, and Bob who can fix it is on PTO until next Mo…

> This is universal to STEM degrees I think. In mechanical engineering classes you analyze a beam, in real life you ...

Hard to believe this. Don't these degrees require rigorous laboratory assignments where the student learns to differentiate best case scenario with real world uncertainties? STEM is not just some IT certification

Re: Data Science: Reality Doesn't Meet Expectations

#124

Post by itsmefaz » Wed, Apr 08, 2020, 7:50 AM UTC

proverbialbunny wrote: post #64

hectormalot wrote: post #5

> Moreover, you may quickly realize much of this work is repetitive and while time-consuming, is “easy”. In fact, most analyses involve a great deal of time to understand the data, clean it and organize it. You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. This. Universities and online challenges…

>You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. I don't get why building a model people consider to be the "fun" part. That's mostly spitting data in, watching a loading screen, and then observing the output. That's not fun, that's boring. The fun part is looking at the data and gleaming all th…

Can you please elaborate on the feature engineering part a little bit?

Re: Data Science: Reality Doesn't Meet Expectations

#125

Post by chosenbreed37 » Wed, Apr 08, 2020, 9:13 AM UTC

scottlocklin wrote: post #30

> I attended a 12-week data science bootcamp in mid-2016. ... Yeah, well there's your problem, my dude. I've been doing what might be described as "data science" since I quit physics in 2004. Aka before the term existed. It's a great area to work in for intelligent people who want to use their brains to impact the real world; vastly better than what people get paid to do in physics. If customers don't know what the t…

I don't think the op would care much for your delivery but you make some great points.

> If your work product isn't in front of the decision makers, you've also failed: they can tell the bottom line impact and will reward you accordingly.

This one in particular stood out. There is an aspect of salesmanship (or navigating corporate hierarchies) to the role. Things will not be obvious to the decision makers. Perhaps the data scientist has to take some responsibility in bringing their work to the fore.

Re: Data Science: Reality Doesn't Meet Expectations

#126

Post by michaelscott » Wed, Apr 08, 2020, 9:22 AM UTC

Barrin92 wrote: post #110

I'm generally confused by the hype around ML and 'data science'. it seems like CS has somehow regressed to the behavourism era of psychology or economics before the Lucas critique. The problem with all this data talk isn't just about implementation or bad structure, the limitations of putting all your bets on inductive reasoning are systemic. The insights that economists had in the 70s and 80s was that reasoning from…

I only see ML and data science as having real value when considered as a single component of a larger system, most of which will not consist of anything close to ML. Many real world environments are too entropic to see much accuracy from ML models except in very, very limited bands (facial recognition, for example).

As other commenters here have posted, without the integration of data science into both the business needs and the rest of the existing tech stack it will remain a fun school course activity.

Re: Data Science: Reality Doesn't Meet Expectations

#127

Post by sgt101 » Wed, Apr 08, 2020, 9:37 AM UTC

vasili111 wrote: post #102

sgt101 wrote: post #27

Disclaimer: I use the term Data Scientist throughout this post; however, popular titles such as Data Analyst, Data Engineers and BI analyst are randomly applied by people who know nothing, and these people share none of the responsibilities of a Data Scientist. I have never had hopes about the potential impact of being a Data Scientist. I felt every company should be a “data company”, but everything I knew told me th…

How important is Ph.D for data science?

I think it's quite important - or an equivalent.

From about 2012 to 2018 I went round a lot of universities, conferences and companies doing presentations and I used to often ask the audience for a definition of data science (in the hope of getting a good one). The best one I heard came at the University of Bath where someone (I know who, but he didn't say it to back it with his reputation so it's not fair to name him - it wasn't me though) said "Just drop the data, it's science".

I totally think that - Data Science is about doing Science with found and evolving data sources, we aren't often able to construct our experiments from scratch, but we often get to augment them, but we always start from the data we are given - which is why it's a sub-field.

In any case - the Ph.D's I have employed have almost all known how to do Science, and it has really helped. Some people without a Ph.D. learn to do it. Experimental Ph.D's are best.

Maths and theoretical Physics Ph.D's are generally not able to do this!

Re: Data Science: Reality Doesn't Meet Expectations

#128

Post by Underqualified » Wed, Apr 08, 2020, 9:43 AM UTC

mcrad wrote: post #123

starpilot wrote: post #90

Earlier quoted context omitted.

This is universal to STEM degrees I think. In mechanical engineering classes you analyze a beam, in real life you analyze an assembly with 50 components that have undergone 100 revisions with 20 different materials and loading from 4 directions that vary with time. Oh, and you have 4 sensors to give you information to analyze critical stresses. But one of them is broken, and Bob who can fix it is on PTO until next Mo…

> This is universal to STEM degrees I think. In mechanical engineering classes you analyze a beam, in real life you ... Hard to believe this. Don't these degrees require rigorous laboratory assignments where the student learns to differentiate best case scenario with real world uncertainties? STEM is not just some IT certification

As a mechanical engineer : No, my education didn't.

The problem is that most real world problems take too much time to really solve to fit in any modern ciriculum.

Re: Data Science: Reality Doesn't Meet Expectations

#129

Post by itsmefaz » Wed, Apr 08, 2020, 9:49 AM UTC

Barrin92 wrote: post #110

I'm generally confused by the hype around ML and 'data science'. it seems like CS has somehow regressed to the behavourism era of psychology or economics before the Lucas critique. The problem with all this data talk isn't just about implementation or bad structure, the limitations of putting all your bets on inductive reasoning are systemic. The insights that economists had in the 70s and 80s was that reasoning from…

> CS has somehow regressed to the behavourism era of psychology or economics before the Lucas critique?

Can you please elaborate on this please?

Re: Data Science: Reality Doesn't Meet Expectations

#130

Post by mikorym » Wed, Apr 08, 2020, 10:02 AM UTC

I am sorry to sound like I am being obstinate, but my opinion about this is that as a society, since the early 90s, we have put way to much focus on "tech" than we have put on plain old mathematics or foundational science.

I don't mean manufacturing (which is doing really well), but companies like Microsoft, Google, Facebook (and even Apple) and others do encourage you to try to compete against their founders (or maybe society does that) rather than focusing on being solid mathematically. Yes, Google pays people well with those skills, but movies portray mostly their founders, emphasising how rich they are, while mathematicians are generally portrayed as weird. Society as a whole puts more emphasis on Bill Gates than on fundamental researchers.

In fact, if you really want to have a rich representative, you can pick the Simons guy. (See, I don't even know his name.) His Medallion hedge fund was built on mathematics. Ironically, Bill Gates is these days one of the biggest financial supporters of people with science skills that he doesn't have.

It is a fad to be a techie. Mathematics is not a fad, although it does have internal fads.