Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, data science

ryxcommar.com

Post reply on HN
Print view View original

371–380 of 415 posts

Re: Goodbye, data science

#371

Post by wodenokoto »

starwind wrote: post #71

I made this same transition from data science to data engineering about 18 months ago and I've never looked back. I hated working with bad code and dealing with arrogant phds who don't value good code. I've seen so many terrible Jupyter Notebooks just copied and pasted into VS Code and the data scientist just washed their hands of it calling it "production ready." Here's a conversation I've had multiple times: Me: ha…

I don’t get it. As a data engineer aren’t you the one who has to deal with the DS code? Whereas a DS someone else has to deal with your code?

Re: Goodbye, data science

#372

Post by logicchains »

EdwardDiego wrote: post #348
nostrademons wrote: post #182

Earlier quoted context omitted.

Parenting & the public education system is a very artificially constructed bubble designed to reinforce and reward "good" behavior, where "good" is usually defined as "that which makes life easier for my caregivers". That gives kids a falsely inflated sense of how much everything matters: your caregivers want you to mind your behavior, because then they don't have to, even if you would've been perfectly fine playing…

It so happens that "good" behaviour that we seek to embed in our children is generally the same as behaviour that is good for society. Broadly the values I've tried to instil in my children break down as - Take responsibilities seriously - Apply your best efforts - Be considerate - Cultivate empathy - Value yourself - Don't be a dick - When you screw up, admit it, and make amends These, when applied, lead to the beha…

> It so happens that "good" behaviour that we seek to embed in our children is generally the same as behaviour that is good for society.

The behaviour that most of the school system seeks to embed in children is primarily "obey and do as you're told, don't question", which is far from good.

Re: Goodbye, data science

#373

Post by gammarator »

lrc wrote: post #216
scottLobster wrote: post #156

Earlier quoted context omitted.

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

My own thought (I know there is a great deal of room for disagreement) is that the J6 crowd saw no consequences for the attempts to obstruct the Brett Kavanaugh confirmation and thought the rules had changed. One of them was shot in the neck, many others are still incarcerated two years later despite a clear constitutional right to a speedy trial. I'd rather the Capitol Police had just cracked heads at this point. Yo…

I must have missed the part where the Kavanaugh protesters showed up in body armor with zip ties and plans to hold members of the Senate Judiciary Committee hostage.

Re: Goodbye, data science

#374

Post by mayankkaizen »

boringg wrote: post #28
jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished. I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre…

You must have been a very sincere and disciplined student. :)

In my case, when I was studying, I had all the time in the world. During that time, I did try to learn many things but I didn't go deep or weren't consistent. I mostly wasted times in goofing around. Looking back, the amount of time I wasted during college years has become a biggest pain of my current life when I don't have any skill or time to learn those skills.

Re: Goodbye, data science

#375

Post by ShredKazoo »

drgiggles wrote: post #58

Unfortunately it seemed pretty clear from the start that this is what data science would turn into. Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. For research and analysis data scientists must have a strong understanding of underlying statistical theo…

>Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge

An important thing people miss is that shallow statistical knowledge can cause subtle failures, but shallow software engineering knowledge can cause subtle failures too.

A junior frontend developer will write buggy code, notice that the UI is glitched, and fix the bug. A junior data analyst will write buggy code, fix any bugs which cause the results to be obviously way off, but bugs which cause subtler problems will go unfixed.

Writing correct code without the benefit of knowing when there is a bug is challenging enough for senior developers. I don't trust newbie devs to do it at all.

Context here is I used to work in email marketing and at one point I was reading some SQL that one of the data scientists wrote and observed that it was triple-counting our conversions from marketing email. Triple-counting conversions means the numbers were way off, but not so far off as to be utterly absurd. If I hadn't happened to do a careful read of that code, we would've just kept believing that our email marketing was 3x as effective as it actually was.

So, it's impossible to know how much of a problem this is. But there is every reason to believe it is a significant problem, and lots of code written by data scientists is plagued by bugs which undermine the analysis. (When's the last time you wrote a program which ran correctly on the first try?) Any serious data science effort would enforce stern practices around code review, assertions, TDD, etc. to make the analysis as correct as possible -- but my impression is it is much more common for data analysis to be low-quality throwaway code.

Re: Goodbye, data science

#376

Post by throwingit0 »

mmcnl wrote: post #170
samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

This is somewhat captured in the article as well. "Managers will say they want to make data-driven decisions, but they really want decision-driven data. If you strayed from this role– e.g. by warning people not to pursue stupid ideas– your reward was their disdain, then they’d do it anyway, then it wouldn’t work (what a shocker). The only way to win is to become a stooge." In science, a good scientific result can be…

>There is often little appreciation for the "science" in data science.

It feels like even Google falls prey to this at times: they keep redoing the same A/B test until it comes up in favor of the change (or the designer whose pet project it is runs out of political capital, presumably).

Re: Goodbye, data science

#377

Post by random314 »

adamsmith143 wrote: post #283
random314 wrote: post #256

Earlier quoted context omitted.

The ML interviews at FAANG are absurdly simple. Design YouTube recommendations for which canned answers are readily available. A simple stats question. If I double the number of samples, how much will the confidence interval change? Most FAANG ML engineers can't answer this question.

And then reverse a binary tree?

What's your point? The question about sample sizes is arcane trivia?

Re: Goodbye, data science

#378

Post by manicennui »

drgiggles wrote: post #58

Unfortunately it seemed pretty clear from the start that this is what data science would turn into. Data science effectively rebranded statistics but removed the requirement of deep statistical knowledge to allow people to get by with a cursory understanding of how to get some python library to spit out a result. For research and analysis data scientists must have a strong understanding of underlying statistical theo…

I don't know many software engineers who have the ability to design and implement robust production systems.

Re: Goodbye, data science

#379

Post by koide »

nostrademons wrote: post #182
scottLobster wrote: post #156

Earlier quoted context omitted.

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

Parenting & the public education system is a very artificially constructed bubble designed to reinforce and reward "good" behavior, where "good" is usually defined as "that which makes life easier for my caregivers". That gives kids a falsely inflated sense of how much everything matters: your caregivers want you to mind your behavior, because then they don't have to, even if you would've been perfectly fine playing…

I would say that there is indeed a problem with modern tech companies where things matter even less than they should. It's not a problem of being raised in a context where everything matters, but that many companies, especially tech are totally care free with their vc money, and we can see that changing in the last months of downturn.

Raising children to care is good and takes lots of effort.

Raising children to leave parents alone usually means the children end up not caring or worse.

Re: Goodbye, data science

#380

Post by roland_nilsson »

'Like bro, you want to do stuff with “diffusion models”? You don’t even know how to add two normal distributions together! You ain’t diffusing shit!'

This made me laugh :) I'm in academia and this post does remind me of certain students...

Post reply on HN