Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, data science

ryxcommar.com

Post reply on HN
Print view View original

121–130 of 415 posts

Re: Goodbye, data science

#121

Post by numbsafari »

adamsmith143 wrote: post #67
drgiggles wrote: post #58

Unfortunately it seemed pretty clear from the start that this is what data science would turn into. Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. For research and analysis data scientists must have a strong understanding of underlying statistical theo…

On the flip side you used to have statisticians writing code that is frankly unusable in a Production environment. You would weep at the R code I've seen and had to turn into something to actually produce business value.

My first job basically consisted of taking code in FORTRAN and translating it into C++ with robust testing and engineering, and then frontending that code into a ton of spreadsheet packages. So you had quanta doing quant work, software engineers doing software engineering, and analysts and traders being analysts and traders, instead of having quants fail at all three, which is more or less what data science is.

Re: Goodbye, data science

#122

Post by throwaway2037 »

mschuster91 wrote: post #47
jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

Not everyone is wired that way. Personally, I have taken apart and reassembled most of the tech stuff I have at home simply because it interests me how things work (and broke and repaired a non-negligible amount of them in the process, to add), I've dabbled in repairing cars, gas boilers, do my own electricity work... but in my social circle, I'm pretty much the only one. And as I grew older, managed to land myself a…

You wrote: >

Is this true in Germany?

Re: Goodbye, data science

#123

Post by _Wintermute »

TrackerFF wrote: post #68
itissid wrote: post #59

I have never understood the what a good ML engineer couldn't do and a Data scientist could in _majority_ situations. When you need a decision to be made based on data its just common sense risk analysis added together with basic statistics. I feel some good field training in statistics(Look up Andrew Gelman) a couple of good courses on Linear, Bayesian Regression is all you need, rest is just engineering skill. The d…

IMO a data scientist should also be a domain expert, in the same way analysts are. But of course, too many view DS as some abstract skill where domain knowledge is not needed, and where the methodology will solve all problems / provide insight.

I agree completely, but if a data scientist should be a domain expert, surely we should just focus more on programming and quantitative skills in these fields?

Re: Goodbye, data science

#124

Post by jghn »

drgiggles wrote: post #58

Unfortunately it seemed pretty clear from the start that this is what data science would turn into. Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. For research and analysis data scientists must have a strong understanding of underlying statistical theo…

I worked adjacent to the data science field when it was in its infancy. As in I remember people who are now household names in the field debating what it should be called.

At the time I considered going down that path, but decided I did not have anywhere near the statistics & math knowledge to get very far. So I stuck with the path I had been on. Over time I saw a lot of acquaintances jumping into the data science game. I couldn't figure out how they were learning this stuff so fast. At some point I realized that most of them knew less than I did when I decided I didn't know enough to even begin that journey.

Of course, I was comparing myself against the giants of the field and not the long tail of foot soldiers. But it made for a great example to me of how with just about everything there's a small handful of people who are the primary movers, and then everybody else.

Re: Goodbye, data science

#125

Post by rafaelero »

codeulike wrote: post #119
drgiggles wrote: post #58

Unfortunately it seemed pretty clear from the start that this is what data science would turn into. Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. For research and analysis data scientists must have a strong understanding of underlying statistical theo…

Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. I dont know anything about Data Science but as a bystander with a mathematical background thats what I assumed was going on so its kindof interesting to see it spelt out like that. Like you've put words to…

That's because businesses don't require a deep level of math knowledge.

Re: Goodbye, data science

#126

Post by dqpb »

samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

> Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results.

This is to be expected from an information theory point of view. It's why "fake news" will always be a thing.

Re: Goodbye, data science

#127

Post by itissid »

1. Fact: The reality of most ML modeling on data is a lot of models don't need to be giant ass interaction machines with 10^80 features. Many non linear models with a Naive Bayes or other simple non parametric models and adding a few Boolean Random Indicator Variables to your regression will go a long way to make your models non linear and get you to 80% F1 score. These are all practical problems and none of them require just a pure DS but also ML skills

2. Fact: You need to deal with bigger problems in ML like data set class imbalance, calibrating responses to the right scalar range(figuring out what that range even is in terms of domain). This is not taught in schools, just like writing Software is not taught in schools. One needs to be in the field to learn these skills and an ML Engineer can pick these up as much as a DS.

Re: Goodbye, data science

#128

Post by bootsmann »

adamsmith143 wrote: post #65
CapmCrackaWaka wrote: post #30

Earlier quoted context omitted.

I've seen this a LOT in my professional group. Many people (who often have PhDs!!) I interview for data science positions seem to know absolutely nothing about the algorithms they use professionally, or how to optimize them, or why they are a good fit for their use case, etc etc etc. I usually see through LinkedIn that these same people are now in impressive-sounding positions at other companies. I had one candidate…

Where are these jobs where you can interview this badly and still get hired because in my experience DS interviews are extremely hard and often expect people to have very high Stats skills as well as Data Structures/Algo skills at FAANG level.

I think the issue here is that "data science" encompasses two very distinct branches of work. One answers to business needs and the other produces data based solutions for the product itself i.e you might have a data scientist who A/B tests your website design so you minimize your churn rate and the other is the team at uber eats who maintains the recommendation engine. While the distinction might not always be as sharp, the former makes up the bulk of data scientists in the market (and I suspect the OP is in that boat) with comparably simple interviews while the rest is the 5 step interview process with hackerrank test you are more familiar with.

Re: Goodbye, data science

#129

Post by samvher »

2devnull wrote: post #108
samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

This gave me a chuckle. If you read the feature article you understand that this is also because management wants “decision driven data.” They have an idea and use ds to provide charts and tables to support their idea. The harder the idea is to support, the greater value data science is able to provide. I guess data science is inferior to research in this way. People care about research methods, rigor, etc… Maybe dat…

I did read the article - some of the problems with judgements of work quality also come up with (hypothetical) well-intentioned truth-seeking non-political long-term-optimizing managers who just don't happen to be stats experts.

Re: Goodbye, data science

#130

Post by fn-mote »

lotsofpulp wrote: post #117
etempleton wrote: post #84

Earlier quoted context omitted.

I become wary any time someone utters the phrase, "show me the data" or any variation there of. There is a specific type of leader who thinks that within the data lurks a magical solution just waiting to be discovered. There is also the leader who uses data as a trump card to win arguments and these folks are perhaps even worse. This is not new. The origination of the phrase, "lies, damned lies, and statistics," can…

I have only heard “show me the data” when someone wants someone else to support a claim. I do not see why this would necessarily be a bad thing.

It's BS because the people asking for the data do not have the sophistication to actually do a reasonable _analysis_ of the data. Or criticize an existing analysis.

Unfortunately, as many posters here are pointing out, there's plenty of ways to do a correct-looking analysis of the data to get evidence to support your agenda.

Maybe your agenda is right and maybe it's not, but I'd love to hear a story of someone standing up and saying "your consultant submitted a report with glaring flaws, they should not be paid and you should reconsider X." It's more likely the little company just goes out of business or the big company buries the failure.

Post reply on HN