Live data from Hacker News

Goodbye, data science

ryxcommar.com

161–170 of 415 posts

Re: Goodbye, data science

#161
post #95

Earlier quoted context omitted.

>Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build…

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

Enforcing consequences is difficult as laws and bureaucracies become ever more complex.

This gives plenty of space for opportunists and tricksters to hide.

You don’t ever have to fear being beheaded by the people whose life savings you stole and you don’t have to face consequences if you have a good lawyer.

To do well in todays world learn all the rules and where the loop holes lie. Violating the spirit of the law is fine as long as you can lawyer around the letter of it.

Re: Goodbye, data science

#162

Earlier quoted context omitted.

It's BS because the people asking for the data do not have the sophistication to actually do a reasonable _analysis_ of the data. Or criticize an existing analysis. Unfortunately, as many posters here are pointing out, there's plenty of ways to do a correct-looking analysis of the data to get evidence to support your agenda. Maybe your agenda is right and maybe it's not, but I'd love to hear a story of someone standi…

So whats the alternative? "Just trust me"?

The alternative is to trust in most circumstances (you did hire me after all), and do thorough analysis once in a while as a gut check.

Re: Goodbye, data science

#164
post #95

Earlier quoted context omitted.

>Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build…

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

I think you hit the nail on the head there with the survivorship bias and the raised in a bubble comments. Most people are raised in a bubble because children generally can't cope with how messy and complicated the world is. And systems and companies that last a long time can point to how successful they were because of their good decisions while ignoring their equally bad decisions that really should have undone them had they not been lucky.

The older I get, the more I realize how fragile a lot of human systems really are, but I suspect it has always been this way and it won't change significantly any time in my lifetime.

Your comment itself sound somewhat nihilistic, so I hope you're doing well mentally!

Re: Goodbye, data science

#165
post #131

Earlier quoted context omitted.

On the flip side you used to have statisticians writing code that is frankly unusable in a Production environment. You would weep at the R code I've seen and had to turn into something to actually produce business value.

There is a bit of a joke that a data scientist is someone who can do better stats then the average SWE and can write better code than the average statistician. Both of those are relatively low bars to clear though

The way I heard the joke was "a data scientist is someone who's not good enough at math to be a statistician, and not good enough at programming to be a software engineer."

Maybe a little harsh...

Re: Goodbye, data science

#167
post #28

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished. I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre…

Nope, nope and nope again. I refute this utterly, as a teaching academic.

Contact hours at most universities are around 2-4 hours per week per 15-credit module. To gain a degree, you have to take 120 credits a year, typically two terms of 4 x 15 credit modules, or 8-16 hours of contact per week maximum with the entire summer off.

You therefore have at least 24 hours a week to study on your own to bring your working week up to 40 hours. Maybe you're working, fair enough. But if you don't have time to study subjects in depth then you need to reduce your working hours. If you can't, then by definition you are not a full-time student.

This is not a personal attack on you. Perhaps you were genuinely studious and spent all your time poring over the coursework. It is a commentary on the whole academic sector where we repeatedly see students do nothing for most of the time and spend the last 2 weeks cramming and putting in substandard assessments, then blame the course material/their lecturers/their anxiety etc. for their poor results. And of course the leadership teams lap it up and tell us to make our courses easier.

Re: Goodbye, data science

#168

Earlier quoted context omitted.

> Are your customers not sophisticated? Are they unable (or unwilling) to follow up on defects and outright lies? You would probably be depressed if you knew who our customers were, and how technologically unsophisticated they are.

I manage at a client an application which is the actual leader (most top right and by far) in Gartner magic quadrant for its category, and for years, I have never seen a product this bad, where the implementors and supports are clueless of their own product. And obviously it's buggy as hell. Lies and deceptions.

Gartner is it’s own confidence trick. They don’t rate you unless you pay them to rate you. It’s manufactured reputation extortion by another name.

Re: Goodbye, data science

#169
As someone who recently switched from DS to platform engineering, this post really captures why I also jumped ship.

> Shitty code & shitty data science

In my opinion the bar should be higher for code quality but also general engineering know-how in data science. You'd be surprised how many are uncomfortable with git, using the command line, interacting with APIs, managing environments, etc. Being able to only work within a jupyter notebook is not good enough, at all. Otherwise, you end up with people who's entire job it is to productionize and deploy the code which is a waste of time and effort.

> Poor mentorship

There is either a lack of quality leadership and mentorship or an inability for upper management to see the value in hiring for it. What ends up happening is you have a team who doesn't know how to grow, scale, or work together. They instead focus on building models and learning statistics when they should be focusing on building systems and process for helping the business scale analytics and building models when appropriate.

I enjoyed data science but found it to also not matter in the implementation that everyone thinks it should be. Data science isn't building nothing but ML models. In most companies, in my opinion, it is actually about scaling analytics. Being able to reach further up into data engineering, get raw data, explore it, shape it, give it back to DE to automate, and then automate the delivery of data to upper management and guide them through using it. If the team thinks their job is to just build models, everyone is going to have a miserable, miserable time.

Re: Goodbye, data science

#170
post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

This is somewhat captured in the article as well.

"Managers will say they want to make data-driven decisions, but they really want decision-driven data. If you strayed from this role– e.g. by warning people not to pursue stupid ideas– your reward was their disdain, then they’d do it anyway, then it wouldn’t work (what a shocker). The only way to win is to become a stooge."

In science, a good scientific result can be bad for business. There is often little appreciation for the "science" in data science.

Post reply on HN