Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, data science

ryxcommar.com

Post reply on HN
Print view View original

21–30 of 415 posts

Re: Goodbye, data science

#21

Post by oneoff786 »

frgtpsswrdlame wrote: post #7

Read this yesterday and absolutely loved it. Especially feel the pain regarding working with management. I think there's an accountability that comes with evaluating management decisions with data that nobody really wants. I still have a lot of half-formed thoughts/opinions about this but it really feels like data-driven requires strict discipline but the data people who would be accountable for that discipline both…

The thing is that anymore anything short of a deep net is equally difficult to implement as a linear regression, if not easier due to nulls and categoricals.

Also, modern tooling makes a lot of these models more than explainable enough for a lot of cases… 10% is a lot

Re: Goodbye, data science

#22

Post by spaniard89277 »

Correct me if I'm wrong because I'm on the receiving end of such models, but I feel that many times a couple of linear regressions, surveys and qualitative work with customers could land much better results.

I say so because I've had time to read some of the reports that DS teams produce to drive decisions in my BIGCORP and it makes very little sense most of the times.

And we suffer from it because we have direct contact with clients, but nobody cares about my department opinion, they will rather believe in some model where I can see insane dispersion in datapoints when they plot em in reports, conclussions by people who clearly has zero understanding of our business.

I'm forced to make decisions on how to treat certain customers, by entering data into some software and being given an output I can't challenge, which produces lots of insane and unfair situations.

Also, IDK how they clean and treat their data, but if they're relying on our ERP's data, good luck. Our CRM if full of BS because most employees rush to put whatever it allows to continue as they need to keep up with KPIs, so they aren't trying to make nice comments and check everything is ok.

Re: Goodbye, data science

#23

Post by mark_l_watson »

oneoff786 wrote: post #19
jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

Seems pretentious to me. I’ve never bothered to look through many things I use. I look extensively at how to use them and what the API offers. I have a good intuition for how most models work. I don’t really care about the specifics of the implementations. I have more important things to do. The hacker mentality, imo, is about identifying what’s useful for you to explore to accomplish whatever you need. Often that’s…

I agree with you. Super powers are seeing value in doing something, and then finding the easiest and most efficient path to get there.

That said, sometimes I do like to read the code in libraries I use but often this is more for enjoyment with occasionally learning something interesting.

Re: Goodbye, data science

#24

Post by Icathian »

This hit all the same high notes I was feeling when I quit Data Science to become a software engineer. It's an infinitely better gig and I encourage all my colleagues with enough chops to make the same switch.

Re: Goodbye, data science

#25

Post by jstx1 »

adammarples wrote: post #20
jstx1 wrote: post #9

Earlier quoted context omitted.

Data engineers don't work with machine learning at all. In fact one of the reasons why it developed as a job title over time waas specifically to differentiate the people who work with data but don't do any statistics or ML. If a DE who is doing "datasets for ML" decides to call themselves an ML engineer, they're just getting a bit too creative with the job titles (maybe they want a career change, more money, they th…

Depends. If their favourite data engineer says "Oh hey, I can write tensorflow too", then guess who get the job of to "productionizing" their crappy data science notebooks?

You have two much more likely options:

1. The person who developed the notebook is responsible for productionizing it. (No, it's not all crappy notebooks and some data scientists can indeed write high quality code).

2. You have someone like an ML engineer whose job it is to do this.

What you're describing seems like the least likely option; at least on the teams I've worked on "I can write tensorflow" would get you nowhere if that's not already a part of your job description.

Re: Goodbye, data science

#26

Post by Dragony »

Many things in this article, especially about the problems with the Data Science role, resonate with me (low value work with low expectations for quality). Funny thing is I have never worked in Data Science. Rather I've worked in Software Development. The summary at the bottom about Data engineering seems like the dream job to me. But I don't think it's because I'm interested in doing Data engineering specifically. I think it's because doing things that actually have an impact day to day is fullfilling. The last job I had totally lost me after ignoring security problems in favor of surface level things like updating CTA labels or similar. Have other Software Devs had this experience?

Re: Goodbye, data science

#27

Post by r-zip »

oneoff786 wrote: post #19
jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

Seems pretentious to me. I’ve never bothered to look through many things I use. I look extensively at how to use them and what the API offers. I have a good intuition for how most models work. I don’t really care about the specifics of the implementations. I have more important things to do. The hacker mentality, imo, is about identifying what’s useful for you to explore to accomplish whatever you need. Often that’s…

That's all fine, as long as you still understand the underlying assumptions and pitfalls. Many people who skim documentation and throw things together haphazardly do not.

Re: Goodbye, data science

#28

Post by boringg »

jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished.

I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre-determine if I think the time spent pursuing that rabbit hole has any value to it. Granted you never know the outcome - it is alway a gamble.

Re: Goodbye, data science

#29

Post by boringg »

aulin wrote: post #5
jstx1 wrote: post #3

As someone who also wants to move away from data science, data engineering is the last thing I would want to do. I think DE comes with many of the same problems and it's also a very ill-defined career track; I wouldn't recommend it to anyone. ML engineer or backend developer seem like much more appealing job profiles.

would you highlight some of the biggest differences between ML engineering and data engineering? I believe they're sometimes used interchangebly especially if "data" is "datasets" for ML.

Data Engineer = build the data pipelines; ML Engineer = implement the models

Re: Goodbye, data science

#30

Post by CapmCrackaWaka »

samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

I've seen this a LOT in my professional group. Many people (who often have PhDs!!) I interview for data science positions seem to know absolutely nothing about the algorithms they use professionally, or how to optimize them, or why they are a good fit for their use case, etc etc etc. I usually see through LinkedIn that these same people are now in impressive-sounding positions at other companies.

I had one candidate who was in charge of a multi-armed-bandit project at their current company. I asked them how it worked, and how they settled on that. Their response was "you know, I'm not really sure, the code was set up when I got there". He had been there for over a year, and could tell me nothing!

> A common one I've seen quite many times is people using a flawed validation strategy (e.g. one which rewards the model for using data "leaked" from the future), or to rely on in-sample results too much in other ways.

It's funny you mention this, we have a direct competitor who does this and advertises flawed metrics to clients. Often times our clients will come back to us saying "XYZ says they can get better performance", the performance in this case being something which is simply impossible without data leakage or some flawed validation strategy.

Post reply on HN