Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, data science

ryxcommar.com

Post reply on HN
Print view View original

151–160 of 415 posts

Re: Goodbye, data science

#151

Post by _dain_ »

Icathian wrote: post #24

This hit all the same high notes I was feeling when I quit Data Science to become a software engineer. It's an infinitely better gig and I encourage all my colleagues with enough chops to make the same switch.

yep, exact same feeling here. I had several years as a "data scientist" and it was a an almost totally bullshit job. the org bought into the hype and hired a cohort of us straight out of university, but then couldn't find anything data-science-y for us to actually do. what I actually ended up doing 95% of the time was taping together dodgy excel-based workflows using python scripts. it gave me a visceral appreciation for Conway's Law. the other 5% was when I got to do some genuinely interesting mathematical work, but that wasn't "data science" either, it was more like operations research. I lived for that stuff, but there wasn't enough of it.

so I jumped ship and became a software engineer. better pay and more interesting problems.

Re: Goodbye, data science

#152

Post by jacobyoder »

Foobar8568 wrote: post #145
CapmCrackaWaka wrote: post #120

Earlier quoted context omitted.

> Are your customers not sophisticated? Are they unable (or unwilling) to follow up on defects and outright lies? You would probably be depressed if you knew who our customers were, and how technologically unsophisticated they are.

I manage at a client an application which is the actual leader (most top right and by far) in Gartner magic quadrant for its category, and for years, I have never seen a product this bad, where the implementors and supports are clueless of their own product. And obviously it's buggy as hell. Lies and deceptions.

The people who make the decisions don't use the product. That's almost always the root cause of this stuff. I worked on a system for my state - another vendor came in and 'took over' all the functionality my system handled. Supposedly. 7 years later, my system powers the exception to the mandate to 'use system X', because... they refuse to provide the functionality that they sold the state. Contractually, "we provide feature ABC", but the reality is.. they don't. I even provided them our code to use - it was paid for with public money, they should just integrate it and then sell it to other people to make their product better. They can't even be bothered to take the code and integrate it... they prefer to continually lie and say "we provide feature ABC" when... they don't. It's beyond insane. A large majority of the people on the ground know it's bad/lacking/broken, but ... they have 0 voice in the matter.

Re: Goodbye, data science

#153

Post by usgroup »

There are good research jobs in industry which are serious and mathematical. However they also require you to be serious and mathematical. I’d venture to say at this stage that most “data scientists” are either self taught segues from adjacent fields or have a shallow relevant background.

The serious places don’t want you… so you end up at the place that can’t tell the difference, and the self fulfilling prophecy begins.

Re: Goodbye, data science

#154

Post by tomrod »

fnands wrote: post #110

Tangentially, I also think the term data scientist has been so abused as to almost be meaningless at this point. When I was applying for jobs it could range from anything from "knows how to use MS Excel" to "Can train large language models at scale". Personally I went for ML Engineering. My company at some point hired people as data scientists (some of my more senior colleagues still have the title, despite doing the…

Agreed.

I've run a "data science consultancy" in some form or fashion for three years now.

When people say "data science" they mean one of three things:

(1) MLE

(2) Data Management

(3) Data Analysis or Business Intelligence (applications of the same skillsets).

(1) has a lot of ongoing innovation, be it in MLOps, autoML, mapping frontier ML to business cases, etc. Innovation is expensive if the investment strategy is unprincipled. (2) is a critical and essential part of making data a usable asset. Management is expensive if it exists solely as a control process and gatekeeps access and use. (3) is core and will never get away from the adhocs and the standard flows, but the inferences are often dubious or not logically justifiable and requires depth of statistical knowledge (rare) to do well -- and courage to call out BS.

Very few people have the depth to do all three. What I have found is that many businesses hope for capacity in all three, plus some basic SWE, in the hope that they can decrease labor expenses. Not an irrational hope, to be frank, but ultimate the iron law of business holds: you can have it good, fast, or cheap -- pick two and be happy with one.

My core observation (and one I see validated based on client interest and experience) is that this is not new and has happened before -- it is the hype cycle in action. The digitization process (including moving to digital and then moving to Web) had a similar cycle. When you treat "data science" like its a silver bullet it will generally fail to do anything but suck budget. When you embed it with your technology teams and treat it as an iterative add, as useful as devops, etc., you have a better chance for value add.

Re: Goodbye, data science

#155

Post by ChrisMarshallNY »

I have no axe to grind, irt data science/engineering, as I have no experience in either.

However, it seems this person's biggest gripe is with good old crap management; the bane of business for hundreds of years.

This line stood out:

> Companies all over were consistently pursuing things that could be reasoned about a priori as being insane ideas– ideas any decently smart person should know wouldn’t work before they’re tried.

That pretty much summarizes why I have been told that today's companies want only young people. Us "olds," are "negative naysayers," who say things like "You know that the laws of physics forbid this, right?" or "I tried that, a couple of years ago. It didn't work out, and here's why...".

Apparently, young people are able to do the impossible, because they haven't been told it's impossible, and mixing "olds" with them, spoils the soup, by telling them it's impossible (or maybe a lot more difficult that they imagine).

Re: Goodbye, data science

#156

Post by scottLobster »

_jx7j wrote: post #95
samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

>Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build…

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response in the media/government, to petty crime being outright tolerated, to in my own career I've at times burned through enough money badly enough (albeit with good intentions) that I thought I was going to be fired, only to be told in a performance review I was doing a good job (grateful to stay employed but WTF, I would have fired or at least demoted me). Importantly, the motivation for this lack of consequence doesn't seem to stem from a desire for forgiveness or positive reinforcement or any mechanism that might make things better.

It seems like there's a general apathy/nihilism that's growing in society, whereas by contrast my entire education from childhood up I was held to strict standards and reliably punished when I failed to meet them, and this was in US public schools (albeit a highly ranked school district) and a public university. That or I was just raised in a bubble, and the historical examples I referenced growing up and reference to this day are just a case of survivorship bias, and all the bullshit that was alongside them back in the day has simply been forgotten. I'm not sure, but it is disappointing how little people at large seem to give a shit. Maybe it's a side-effect of the obesity epidemic and people just have less energy or something

Re: Goodbye, data science

#157

Post by lagt_t »

whatever1 wrote: post #55
lagt_t wrote: post #54

Earlier quoted context omitted.

You don't look at single outcomes with statistics.

See? “Better luck next time”. Not being mean to you, just showing how typically the goal posts are moved. To give you an example from physics, if you find just one experiment that goes against your model, you immediately invalidate the model. You don’t just make grand claims that the model in general works.

Quantum mechanics says otherwise

Re: Goodbye, data science

#158

Post by kjkjadksj »

scottLobster wrote: post #156
_jx7j wrote: post #95

Earlier quoted context omitted.

>Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build…

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

Part of it too is the stressful expectations school puts on you that as you’ve found, don’t actually exist in the real world.

Re: Goodbye, data science

#159

Post by jmount »

samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

Super point. I can't resist repeating it back. When incorrect work outperforms correct work in superficial evaluation, it is then selected for.

Re: Goodbye, data science

#160

Post by scoobitydoobap »

_dain_ wrote: post #151
Icathian wrote: post #24

This hit all the same high notes I was feeling when I quit Data Science to become a software engineer. It's an infinitely better gig and I encourage all my colleagues with enough chops to make the same switch.

yep, exact same feeling here. I had several years as a "data scientist" and it was a an almost totally bullshit job. the org bought into the hype and hired a cohort of us straight out of university, but then couldn't find anything data-science-y for us to actually do. what I actually ended up doing 95% of the time was taping together dodgy excel-based workflows using python scripts. it gave me a visceral appreciation…

Could you elaborate on what kind of "software engineering" you now do? For someone who also would like to get out of data science, mentions of "I became a software engineer" don't really help to clarify what kind of SWE is feasible for a data scientist with decent programming chops to get into.
Post reply on HN