Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Science: Reality Doesn't Meet Expectations

dfrieds.com

Post reply on HN
Print view View original

81–90 of 168 posts

Re: Data Science: Reality Doesn't Meet Expectations

#81

Post by Swizec »

throwaway713 wrote: post #54

As a research-oriented data scientist at one of the larger tech companies, I can confirm that even here, a lot of people are unsure about what exactly data scientists are supposed to do. My most frequent request is "tell us why metric X dropped", to which the answer is often a subtle combination of many different factors (often random fluctuation) that doesn't lead to a pleasing actionable result in the sense of "her…

> The work that is most valuable to a business is not exciting all of the time

This probably describes just about every job in a for-profit business.

Re: Data Science: Reality Doesn't Meet Expectations

#82

Post by VHRanger »

rafiki6 wrote: post #40
smeeth wrote: post #37

Earlier quoted context omitted.

I always thought the non-specificity of the term Data Science was a strange criticism for those in the tech industry to make. How many types of SWE are there? Front-end, back-end, full-stack, devops, security, QA... I agree wholeheartedly with your recommendation. Like any other job, each company has different needs and expectations and if you want something else out of the role you'd best avoid that company.

Frankly I have the same criticism of those who use the term software engineer. Engineering is a pretty established profession with a set of standards, ethics and practices. Most of us who work in software are not engineers. We are developers. Similarly, a scientist is one who follows the scientific method to do research. So by that logic a data scientist should be a person who uses the scientific method to do researc…

A decent amount of data scientists work on AB tests which is science on production data.

Actually I'd say that's as common as the failed ML projects

Re: Data Science: Reality Doesn't Meet Expectations

#83

Post by FridgeSeal »

mr_toad wrote: post #50
rafiki6 wrote: post #40

Earlier quoted context omitted.

Frankly I have the same criticism of those who use the term software engineer. Engineering is a pretty established profession with a set of standards, ethics and practices. Most of us who work in software are not engineers. We are developers. Similarly, a scientist is one who follows the scientific method to do research. So by that logic a data scientist should be a person who uses the scientific method to do researc…

> the scientific method to do research on data Exploratory data analysis is often overlooked and underrated.

Ppphhh we don’t need to do exploratory data analysis or prepare the days, don’t you know that neural networks will do all that themselves!

Doesn’t yield the right results? Clearly not enough data.

Still doesn’t work? Change to whatever the latest model google or fb is using and try again.

/s

Re: Data Science: Reality Doesn't Meet Expectations

#84

Post by Vaslo »

I’ve been doing an MS in Data Science very slowly due to work and 2 new kids. Finishing the degree this year in year 4. I was very excited about the prospect of doing something different. A few things have changed for me.

1). I am hearing about Data Science Teams being furloughed during these times. That isn’t happening in my function (Corporate Finance). I am glad to be secure even though I enjoy much of the data sci work.

2) I’m able to apply Data Science concepts in my current role, and it’s adding a lot of job security and providing me with exposure. I am much less interested now in moving to straight Data Science and instead am applying my learnings in my current role as a sort of in-house Data Science guy. But I have a lot to learn to be honest.

3). There seem to be a lot of “thought leaders” acting like they are big experts in the area and really don’t know anything many of us amateur scientists don’t know. They pull perfect clean datasets and show these magic transformations they just copy from others to get YouTube hits or Twitter followers. That just never happens in real life, and many leaders are seeing this and losing interest in this function in the returns they are getting from sole data science folks.

Re: Data Science: Reality Doesn't Meet Expectations

#85

Post by RayVR »

Vaslo wrote: post #84

I’ve been doing an MS in Data Science very slowly due to work and 2 new kids. Finishing the degree this year in year 4. I was very excited about the prospect of doing something different. A few things have changed for me. 1). I am hearing about Data Science Teams being furloughed during these times. That isn’t happening in my function (Corporate Finance). I am glad to be secure even though I enjoy much of the data sc…

This isn’t unique to data science. I personally know people in finance that are poor coders and even worse quants, yet they go around lecturing at universities.

Re: Data Science: Reality Doesn't Meet Expectations

#86

Post by mattmanser »

apohn wrote: post #79
throwaway713 wrote: post #54

As a research-oriented data scientist at one of the larger tech companies, I can confirm that even here, a lot of people are unsure about what exactly data scientists are supposed to do. My most frequent request is "tell us why metric X dropped", to which the answer is often a subtle combination of many different factors (often random fluctuation) that doesn't lead to a pleasing actionable result in the sense of "her…

>But in terms of career progression and job safety, the risk is just way too high, at least for me personally. I save the highly mathematical stuff for a hobby. I think the sad truth is that this is the reality of work no matter if you are a Data Scientist or not. What you thought you would be doing to show your worth and climb the ladder gets blurred in with KPIs you didn't set, politics you didn't create, goals and…

Sounds more like it simply doesn't work very well, rather than any of the reasons you listed.

It's often the case, I remember when that stupid Amazon infographic was going around about decreased load times meaning big upswings in conversions.

A client paid for a significant project to reduce load times, which we succeeded in to a huge degree with most of the pages going from 1.5-3 seconds secs down to 250-500 ms. Absolutely no meaningful swing in conversions at all. I've done this a few times since, but never seen conversion move at all when I've done performance improvements.

Nada, zilch. I honestly think it's absolute bullshit. I've always suspected since that it was someone massaging figures in Amazon to justify their job.

Re: Data Science: Reality Doesn't Meet Expectations

#87

Post by antipaul »

rafiki6 wrote: post #36
antipaul wrote: post #29

Earlier quoted context omitted.

100% agree with the article. The top misconceptions are spot on. I’m at a big place where data science hype among leadership couldn’t be bigger.

If I may ask, what were you told when you interviewed that convinced you to join?

Reasons for joining: 70% just needed a job, 20% location (silicon valley - wanted to be in tech ecosystem), 10% was combo of: I was told it's a small, entrepreneurial team with undefined remit (so opportunities to forge own path - turned out to be true) and it was impressive in many non-technical ways (company mission, campus, resources, etc)

And no reasons relating to technical or data science know-how on the part of the team/company ;) I already knew coming in that the industry is technologically backwards (big healthcare co)

Re: Data Science: Reality Doesn't Meet Expectations

#88

Post by starpilot »

Swizec wrote: post #81
throwaway713 wrote: post #54

As a research-oriented data scientist at one of the larger tech companies, I can confirm that even here, a lot of people are unsure about what exactly data scientists are supposed to do. My most frequent request is "tell us why metric X dropped", to which the answer is often a subtle combination of many different factors (often random fluctuation) that doesn't lead to a pleasing actionable result in the sense of "her…

> The work that is most valuable to a business is not exciting all of the time This probably describes just about every job in a for-profit business.

If jobs were exciting, they wouldn't have to pay you to do it.

Re: Data Science: Reality Doesn't Meet Expectations

#89

Post by antipaul »

AndrewKemendo wrote: post #68

A really easy way that I try to explain things to people is like this: You can't compress information until you have it in a format that is appropriate for compression. That is: You can't compress (apply/create algorithms) information (data) until you have it (instrumented data collection) in a format (schema) that is appropriate for efficient compression (structured logging/cleaning). 99% of that is Data Engineering…

Totally agree. Non-tech companies that think they need "data science" should instead put same effort into (data) engineering.

It's not quite 99% of the effort but close enough ;)

Search "data science hierarchy of needs"

Re: Data Science: Reality Doesn't Meet Expectations

#90

Post by starpilot »

kqr wrote: post #33
hectormalot wrote: post #5

> Moreover, you may quickly realize much of this work is repetitive and while time-consuming, is “easy”. In fact, most analyses involve a great deal of time to understand the data, clean it and organize it. You may spend a minimal amount of time doing the “fun” parts that data scientists think of: complex statistics, machine learning and experimentation with tangible results. This. Universities and online challenges…

I took a data visualisation class in uni that handled this really cleverly. The second assignment sounded very easy. The teacher provided links to the sources where we could find data. Most people figured that with such a simple assignment (not significantly harder than the first one, which was also easy-ish) they could put off doing it until the last moment. Most people failed. This real world data needed hours upon…

This is universal to STEM degrees I think. In mechanical engineering classes you analyze a beam, in real life you analyze an assembly with 50 components that have undergone 100 revisions with 20 different materials and loading from 4 directions that vary with time. Oh, and you have 4 sensors to give you information to analyze critical stresses. But one of them is broken, and Bob who can fix it is on PTO until next Monday, so...

Internships are supposed to fill this gap but it'd be nice if all students could get a taste of real world systems and data. For tech, maybe if they could partner with the IT department at the school to get them exposed to real, messy data. Maybe there are some teaching datasets with over a billion rows that people could play around with.

Post reply on HN