Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, data science

ryxcommar.com

Post reply on HN
Print view View original

251–260 of 415 posts

Re: Goodbye, data science

#251

Post by tomrod »

omega3 wrote: post #14
jstx1 wrote: post #9

Earlier quoted context omitted.

Data engineers don't work with machine learning at all. In fact one of the reasons why it developed as a job title over time waas specifically to differentiate the people who work with data but don't do any statistics or ML. If a DE who is doing "datasets for ML" decides to call themselves an ML engineer, they're just getting a bit too creative with the job titles (maybe they want a career change, more money, they th…

> Data engineers don't work with machine learning at all. This very much depend on the company. From experience DE is used as a catch-all title.

I haven't come across that. Unless the job title is being polluted like DS was (to include all aspects of data management), DE is specifically about data pipelines and not the models generating those clusters, predictions, or classifications.

Re: Goodbye, data science

#252

Post by grepLeigh »

I'm reading a lot of sour grapes here, but I'd like to offer a more optimistic future snapshot (if you're a data scientist and happen to be reading).

I agree that many companies hire data scientists with only a vague idea about how to utilize them, but the same is true of software people in general. "Software is eating the world" and so is the practice of extracting value from data.

The margin on software is high - often more than 95% - so there's a lot of room to screw up and "figure it out" as a business. I think that's why there's a low bar for software and data management compared to, say, an automotive manufacturing line manager.

But that's where the opportunity is, if you're a budding data scientist:

- The business might not know how to effectively use/manage/train/mentor you.

- Upper management might have 20+ years of line of business experience, but will need your help to understand how your team can impact the business.

- You're going to need to seek out ways to impact the bottom line of the business.

All of the above is a recipe for leaping forward in your career. Since data science is a relatively new field, the demand for senior leadership FAR outstrips the available supply.

If you can learn how to effectively manage yourself, your team mates, and your function within the business - you have a ton of negotiating leverage and can name your price.

Source: I'm a data person who "retired" in their early 30s. Now I do all the research and hard science I want. ;)

Re: Goodbye, data science

#253

Post by jrochkind1 »

asdff wrote: post #245
jrochkind1 wrote: post #191

Earlier quoted context omitted.

Not just tech companies either. Go bigger. It's possibly 60% of activity in US white collar economy. (I can't speak for those who work with their bodies, could be or not).

Former body worker. 100% of us worked, because if you had nothing to do the boss will invent something for you to do or send you home.

Are you saying that meant you did a lot of useless work?

Re: Goodbye, data science

#254

Post by antipaul »

alexpetralia wrote: post #205
chasely wrote: post #42

Earlier quoted context omitted.

I've been pitched by many "data-driven" vendors offering predictions. They often have very impressive accuracy metrics (RMSE, R2, etc). When I dive into the details these metrics are often reported using in-sample predictions. I see this pointing to any of the following: a) DS teams overpromising the accuracy of their approaches b) marketing driving the narrative and DS getting pulled along c) incompetence from the D…

Are these inferential statistics not designed to be in-sample? I would imagine predictive statistics use more out-of-sample metrics like precision and recall.

If you do it right…

That’s the problem: these metrics often come from overfitted or in-sample data, and are completely unrealistic when it comes to expected generalization performance.

I’m at the point where I never trust performance metrics anymore. Or rather, the worse they are, the more I trust them!

Re: Goodbye, data science

#255

Post by mrtksn »

angry_moose wrote: post #72

In a recent past life, I was a HPC (high performance computing) administrator for a mid size company (just barely S&P400) who was in the transportation industry, so I had a lot of interactions with the "data science" team and it was just a fascinating delusion to watch. Our CTO did the "Quick, this is the future! I'll be fired if I don't hop on this trend" panic thing and picked up a handful of recent grads and gave…

In retrospect, maybe they made the right call for themselves when the money was pouring. Probably everyone involved was paid very well for the charade. The ethics can be questionable but maybe its some kind of wealth redistribution, after all the people with money are trusted to make the calls and them falling for this maybe simply means the money is beter off somewhere else.

Re: Goodbye, data science

#256

Post by random314 »

adamsmith143 wrote: post #65
CapmCrackaWaka wrote: post #30

Earlier quoted context omitted.

I've seen this a LOT in my professional group. Many people (who often have PhDs!!) I interview for data science positions seem to know absolutely nothing about the algorithms they use professionally, or how to optimize them, or why they are a good fit for their use case, etc etc etc. I usually see through LinkedIn that these same people are now in impressive-sounding positions at other companies. I had one candidate…

Where are these jobs where you can interview this badly and still get hired because in my experience DS interviews are extremely hard and often expect people to have very high Stats skills as well as Data Structures/Algo skills at FAANG level.

The ML interviews at FAANG are absurdly simple. Design YouTube recommendations for which canned answers are readily available.

A simple stats question. If I double the number of samples, how much will the confidence interval change? Most FAANG ML engineers can't answer this question.

Re: Goodbye, data science

#257

Post by f1shy »

CapmCrackaWaka wrote: post #30
samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

I've seen this a LOT in my professional group. Many people (who often have PhDs!!) I interview for data science positions seem to know absolutely nothing about the algorithms they use professionally, or how to optimize them, or why they are a good fit for their use case, etc etc etc. I usually see through LinkedIn that these same people are now in impressive-sounding positions at other companies. I had one candidate…

Is this the US? I'm concern about the extremely low level or the bar to get a PhD in Europe... and I'm wondering if that is a global problem, or only Europe.

Re: Goodbye, data science

#258

Post by wheelinsupial »

bo1024 wrote: post #225
boringg wrote: post #186

Earlier quoted context omitted.

No personal attack taken but your experience and points fail to win me over. The difference probably belies in the rigor of the program. It sounds like you are working in a non-engineering based program. In our engineering programs we had 40 hours of class time + lab time per week. I had a concurrent arts degree at the same time which is was, in comparison, incredibly light workload - though concurrently it took time…

You posted this elsewhere in the thread, where I replied that this is not normal in the U.S. Can I ask what university and degree program it is where students have 40 hours of class and lab time per week?

They mentioned engineering programs in their comment.

40 hours isn't normal even for engineering programs. Every engineering program I've looked at has higher course hour and credits required. Obviously I haven't looked at every single engineering program at every engineering school, so there probably exists some counter example showing it's no different than arts or science..

Where I studied, we had one semester with 40.5 hours of lecture, lab, and tutorials. One other semester was around 38 or 39 hours. The rest were in the mid-twenties for lecture, lab, tutorial. My program wasn't the typical engineering program, but all of the other engineering schools where I went (Western Canada) did require more credits and more class hours than science and arts and business programs. There may have been some exceptions with honors programs (meaning they have to take 132 credits vs 120 credits and write a thesis) in arts and science that put them closer to engineering programs, but these have limited enrollment.

Re: Goodbye, data science

#259

Post by 3pt14159 »

> Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case.

Story time.

There was once a junior data scientist at Shopify that had learned Python and SQL and was tasked to figuring out how to fix their "broken app store recommendation engine" but since they didn't know Ruby, they asked for my help in figuring out what was going on.

Well somewhere in the soup of math was a fuzz factor at the very top. Think of it like

    factor = 0.something  # Not 100% sure what the decimal portion was.
    some_complicated_math_that_maxed_out_at_one_pt_zero() + rand(factor)
Now the thing about ruby is that rand is basically broken for floats.

    Negative or floating point values for max are allowed, but may give surprising results.
https://ruby-doc.org/core-2.4.0/Kernel.html#method-i-rand

So basically what they thought they were doing was introducing a bit of randomness that would hinder others from reverse engineering their algorithm. What they actually did was make the recommendation algorithm fifty percent total noise. Yes, it's true. On every load half the recommended app scores were noise.

They fixed the bug and I'm sure a ton of balance sheets for businesses around the world are markedly different know because of it, but I never heard of it again.

This is one of the core problems with data science.

The lack of feedback.

Re: Goodbye, data science

#260

Post by clircle »

r-zip wrote: post #235
usgroup wrote: post #153

There are good research jobs in industry which are serious and mathematical. However they also require you to be serious and mathematical. I’d venture to say at this stage that most “data scientists” are either self taught segues from adjacent fields or have a shallow relevant background. The serious places don’t want you… so you end up at the place that can’t tell the difference, and the self fulfilling prophecy beg…

Can you recommend a few such places? :)

Look up the technical presenters at your favorite math or stats conference and look at the orgs they work for. That's how you can tell which orgs are serious about solving research problems and spreading solutions.
Post reply on HN