Live data from Hacker News

Goodbye, data science

ryxcommar.com

271–280 of 415 posts

Re: Goodbye, data science

#271
post #59

I have never understood the what a good ML engineer couldn't do and a Data scientist could in _majority_ situations. When you need a decision to be made based on data its just common sense risk analysis added together with basic statistics. I feel some good field training in statistics(Look up Andrew Gelman) a couple of good courses on Linear, Bayesian Regression is all you need, rest is just engineering skill. The d…

I think the qualifying term here is "good". I've worked with a surprising number of MLEs that don't really understand gradient descent or how most models really work under the hood. They certainly couldn't implement most things from scratch if they needed to (neither could most data scientists). I used to think an MLE was a solid engineer who also had a strong quantitative and numerical computing background. The kind…

>They certainly couldn't implement most things from scratch if they needed to (neither could most data scientists).

Could most CS folks actually implement Linux or Chromium from scratch?

Re: Goodbye, data science

#272
> The only way to win is to become a stooge.

I sometimes feel like that as a software dev, though, when management pushes some changes or "fixes" that aren't useful and won't fix anything.

Re: Goodbye, data science

#273
post #95

Earlier quoted context omitted.

>Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build…

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

> as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect.

I wouldn't say fewer uniformly, but certainly very noisy. Some have their lives destroyed for minor or non-existent misdeeds, others get away with egregious crimes.

Re: Goodbye, data science

#274
post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

I become wary any time someone utters the phrase, "show me the data" or any variation there of. There is a specific type of leader who thinks that within the data lurks a magical solution just waiting to be discovered. There is also the leader who uses data as a trump card to win arguments and these folks are perhaps even worse. This is not new. The origination of the phrase, "lies, damned lies, and statistics," can…

if you torture data long enough, it will confess to anything

Re: Goodbye, data science

#275

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

Seems pretentious to me. I’ve never bothered to look through many things I use. I look extensively at how to use them and what the API offers. I have a good intuition for how most models work. I don’t really care about the specifics of the implementations. I have more important things to do. The hacker mentality, imo, is about identifying what’s useful for you to explore to accomplish whatever you need. Often that’s…

I see what you're saying, but the post above seems to indicate that understanding those models SHOULD be important to you if your job is to run those models and explain results and make corporate decisions based off your forecasts. You shouldn't be just passing data through and thinking that it's not your job to actually understand things. The subtlety matters a lot as the software hitting some edge case could completely skew the results. There is a vague line drawn somewhere that tells you what is necessary to learn and what is superfluous. Finding the line isn't easy, but those that label too much as superfluous will likely get more erroneous results and that is a problem.

With regards to your API statement, I'm just as guilty regarding reading the code, but I do run some manual tests to ensure that my script calling the database actually does what I think it should. Is that good enough? Who knows :)

Re: Goodbye, data science

#276
post #154
post #110

Tangentially, I also think the term data scientist has been so abused as to almost be meaningless at this point. When I was applying for jobs it could range from anything from "knows how to use MS Excel" to "Can train large language models at scale". Personally I went for ML Engineering. My company at some point hired people as data scientists (some of my more senior colleagues still have the title, despite doing the…

Agreed. I've run a "data science consultancy" in some form or fashion for three years now. When people say "data science" they mean one of three things: (1) MLE (2) Data Management (3) Data Analysis or Business Intelligence (applications of the same skillsets). (1) has a lot of ongoing innovation, be it in MLOps, autoML, mapping frontier ML to business cases, etc. Innovation is expensive if the investment strategy is…

Curious, what kinds of clients pay money for data science consulting? And does it feel like a sustainable business moving forward?

Re: Goodbye, data science

#277
post #113

Earlier quoted context omitted.

If I work my ass off and my model recommends a few extra units, I don't see another dime, so where's the motivation?

Un-ironically: the pride of a job well done? Most people in software on this site a very well paid and well treated, the least we can do is do our job right.

Pride doesn't pay the bills my dude. Meanwhile my boss will probably get a nice bonus for boosting those KPIs. So I don't buy that one.

Re: Goodbye, data science

#278
Great post. A few random comments...

> The median data scientist is horrible at coding and engineering in general. The few who are remotely decent at coding are often not good at engineering in the sense that they tend to over-engineer solutions, have a sense of self-grandeur, and want to waste time building their own platform stuff (folks, do not do this).

> It was obvious that there is a general industry-wide need for people who are good at both data science and coding to oversee firms’ data science practices in a technical capacity.

The job of overseeing a crowd of stubborn self-important over-engineerers sounds pretty thankless.

> 23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college. So many careers are being ruined before they’ve even started because data science kids went straight from undergrad to being the third data science hire at a series C company where the first two hires either provide no mentorship, or provide shitty mentorship because they too started their careers in the same way.

Startups are a low-paid job with a lottery ticket for a little dash of excitement. You get what you pay for.

> ...I live in constant anxiety that someone will pop quiz me with questions like “what is the formula for an F-statistic,” and that by failing to get it right I will vanish in a puff of smoke. So my brain tells me that I must always refresh myself on the basics.

Focusing on the basics is better than pretending to understand fancy things, but even this level of "continuous professional training" or whatever you want to call it is, to me, a bit off the mark. We can look up formulas whenever we want these days. We need more meaningful ways to test our understanding of things.

Re: Goodbye, data science

#279
post #162

Earlier quoted context omitted.

The alternative is to trust in most circumstances (you did hire me after all), and do thorough analysis once in a while as a gut check.

And how do you do that thorough analysis without asking to see the data?

The person above wasn’t complaining about data science as a whole, they were complaining about data science theater. The scenario where as long as you put some numbers in your bosses face, they could care less what the real implications are. In cases where you’re doing thorough analysis, you should look for the data yourself, rather than ask someone to market it to you

Re: Goodbye, data science

#280

I have no axe to grind, irt data science/engineering, as I have no experience in either. However, it seems this person's biggest gripe is with good old crap management; the bane of business for hundreds of years. This line stood out: > Companies all over were consistently pursuing things that could be reasoned about a priori as being insane ideas– ideas any decently smart person should know wouldn’t work before they’…

[deleted]
Post reply on HN