Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, data science

ryxcommar.com

Post reply on HN
Print view View original

31–40 of 415 posts

Re: Goodbye, data science

#31

Post by Reimersholme »

agomez314 wrote: post #10

"Managers will say they want to make data-driven decisions, but they really want decision-driven data" Ooofff. This is too true. How often is the case that data is collected to test hypotheses vs confirming priors?

Rather than wanting to confirm priors, I believe this usually is a problem with neither the PM nor the data scientist ensuring that the problem formulation is good enough before diving in. I.e., what data would be needed to actually test the hypothesis? Do we have that data or not? Is the hypothesis even formulated in a way to be falsified in theory?

I've seen so many analysis tasks where data scientists without questioning went away for a few weeks to crunch data and come back with some random graphs and statistics that are completely useless as decision support.

Re: Goodbye, data science

#32

Post by boringg »

Coles notes: Data Engineer - more money, more clout, less analysis / interesting projects, more job security, more infra style work Data Science - less money, a lot of random projects (Sometimes totally overqualified for), more analytical, don't have as much clout / confusion & lots of people don't actually understand capabilities.

Re: Goodbye, data science

#33

Post by 6gvONxR4sf7o »

oneoff786 wrote: post #19
jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

Seems pretentious to me. I’ve never bothered to look through many things I use. I look extensively at how to use them and what the API offers. I have a good intuition for how most models work. I don’t really care about the specifics of the implementations. I have more important things to do. The hacker mentality, imo, is about identifying what’s useful for you to explore to accomplish whatever you need. Often that’s…

If you think it’s about implementation details, you’re misunderstanding. It’s about understanding the principles behind it.

As an example, it’s more about understanding the statistics and linear algebra around estimating uncertainty in GLM regression estimates, than about reading the code for how the statsmodels library implements that.

Re: Goodbye, data science

#34

Post by cjf4 »

One of the things that always sort of annoys me about complaints that "management doesn't listen to data (science)" is the lack of awareness they consist of.

It turns out that data work is limited by all the same things every other part of the business is limited by: the need to make quick decisions, institutional imperative, the beliefs of decision makers, the ability to communicate well/influence, and so on.

Having better access or skill with data doesn't give you a pass on these things, despite the suggestions otherwise from laments such as this.

Re: Goodbye, data science

#35

Post by cmmeur01 »

xyzelement wrote: post #11

// it was often personally unfulfilling (e.g. tuning a parameter to make the business extra money). He lost me here. Something I've always loved about being an engineer (and now in product) is that something small we do/tweak can have big impact. If you tuned a parameter and that actually had tangible impact on the business, that's like the best case scenario and should be celebrated (vs doing some cool rocket scienc…

And all that extra profit is hovered up by the people above you that had nothing to do with it. Validated engineering cost savings should be treated like sales, the engineer gets a percentage.

Re: Goodbye, data science

#36

Post by Davertron »

boringg wrote: post #28
jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished. I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre…

Well good luck then, in my experience the most free time I've ever had in my life was during college. I squandered massive amounts of that time doing things completely unrelated to education, and I definitely don't regret doing that. College isn't just about book learning after all. But still, BY FAR, college is the time of my life when I had the most free time to do whatever I wanted.

Re: Goodbye, data science

#37

Post by 6gvONxR4sf7o »

samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

This is the defining pain point for data science, in my experience. There’s no simple ground truth to test competence against.

If someone tells you that the data says their work is good, the only real way to know if they’re right or wrong is to look at what the data says yourself. If 99% of the work is building and 1% is checking something like latency, then you’re likely to have more than one set of eyeballs on that 1%. But if 99% of the work is putting the data together and doing the analysis, then you’re unlikely to have more than one person ever look at that part.

So incompetence goes unchecked (or worse, it is rewarded).

Re: Goodbye, data science

#38

Post by webnrrd2k »

spaniard89277 wrote: post #22

Correct me if I'm wrong because I'm on the receiving end of such models, but I feel that many times a couple of linear regressions, surveys and qualitative work with customers could land much better results. I say so because I've had time to read some of the reports that DS teams produce to drive decisions in my BIGCORP and it makes very little sense most of the times. And we suffer from it because we have direct con…

The word "conclussion" is especially appropriate in this context -- it could easily be pronounced like a combination of "conclusion" and "concussion".

Re: Goodbye, data science

#39

Post by nisegami »

boringg wrote: post #28
jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished. I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre…

I think this inclination to be curious can still be apparent even when someone doesn't have the time to pursue that inclination. It will be more subtle, but I think it's something rather fundamental that applies in broad ways across our lives.

Re: Goodbye, data science

#40

Post by zhdc1 »

jstx1 wrote: post #9
aulin wrote: post #5

Earlier quoted context omitted.

would you highlight some of the biggest differences between ML engineering and data engineering? I believe they're sometimes used interchangebly especially if "data" is "datasets" for ML.

Data engineers don't work with machine learning at all. In fact one of the reasons why it developed as a job title over time waas specifically to differentiate the people who work with data but don't do any statistics or ML. If a DE who is doing "datasets for ML" decides to call themselves an ML engineer, they're just getting a bit too creative with the job titles (maybe they want a career change, more money, they th…

Somewhat surprised that there's a separate job category for what sounds like large-scale data cleaning and aggregation work (which IMO is 90%+ of the effort involved with data science).

Anyway, I'm going to go back to my 5K+ lines of code for an upcoming conference submission - almost all of which involve data cleaning and aggregation - and think about how I could be making a 2x more than I am now.

Thanks Hacker News.

Post reply on HN