Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, data science

ryxcommar.com

Post reply on HN
Print view View original

201–210 of 415 posts

Re: Goodbye, data science

#201

Post by teraflop »

gpresot wrote: post #194

I realise the article was written for a specific audience for which this may be obvious, but what is the difference between data scientist and data engineer (in terms of what their job is)?

Generally speaking:

"Data engineering" means building systems that can manipulate data (e.g. storing, retrieving, and delivering it). There are usually fairly well-defined functional requirements about what the system is supposed to do, plus goals about performance and reliability that might be slightly more nebulous.

"Data science" means building systems that can draw conclusions from data. The functional requirement is usually some form of "accuracy", as measured somehow against some kind of human evaluation of the same conclusion.

Concretely: a data engineer might be asked to build a system that can ingest every tweet posted to Twitter, and return the 10 most widely-used hashtags in the last hour. A data scientist might be asked to build a system that looks at a tweet and figures out what language it's written in, or whether it's spam, or whether an attached image is pornographic.

Re: Goodbye, data science

#202

Post by daydream »

PragmaticPulp wrote: post #196
scottLobster wrote: post #156

Earlier quoted context omitted.

Not to get too off topic, but as a 35 year old engineer it seems the world in general has far fewer consequences than I was raised to expect. Everything from businesses with bullshit ideas flourishing at a loss, to January 6 even being possible (politics aside I expected the Capitol Police to crack a lot more skulls than they did once people started smashing windows), to the whole FTX situation and the tepid response…

Consequences often catch up slowly. It took years for Elizabeth Holmes to be sentenced because it takes time to collect evidence, build an airtight case, and give people their due process. As I get older, I'm actually noticing more and more consequences catching up with people, albeit slowly. The people I knew who drank heavily through their 20s and 30s are in much worse shape than basically anyone who made an effort…

> Consequences often catch up slowly.

Agreed. Few people defy gravity; in the end most hit the ground.

The phrase “slowly, then suddenly” comes to mind.

Re: Goodbye, data science

#203

Post by quickthrower2 »

pessimizer wrote: post #174
vsareto wrote: post #141

Earlier quoted context omitted.

For establishing competence, you still have to dig in to see what caused the slowness. A regular user can't tell you that.

> For establishing competence, you still have to dig in to see what caused the slowness. Not as management. You just have to see that other people's similar sites are not slow with the same resources, therefore it is possible for your site not to be slow. You don't have to know why you're failing to know that the totality of the people you hired were not as good as the people those others hired. This is of course bar…

Software Engineering is one of the few knowledge working areas where you can actually test the result in various ways as a layman. You can flush the toilet before paying the plumber to a large extent and hire another counter-team called QA. QA themselves are tested by future production bugs.

In other disciplines it is way more fuzzy. If you are in the conclusion business and there isn’t a clear path to test your conclusion in the short term you can bullshit away!

Re: Goodbye, data science

#204

Post by stonemetal12 »

mr_gibbins wrote: post #167
boringg wrote: post #28

Earlier quoted context omitted.

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished. I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre…

Nope, nope and nope again. I refute this utterly, as a teaching academic. Contact hours at most universities are around 2-4 hours per week per 15-credit module. To gain a degree, you have to take 120 credits a year, typically two terms of 4 x 15 credit modules, or 8-16 hours of contact per week maximum with the entire summer off. You therefore have at least 24 hours a week to study on your own to bring your working w…

Where do you teach? Where I have gone, 1 credit meant 1 hr of lecture and an expected 2 hr of study outside of lecture. Therefore 15 credits means 45 hours a week of study before you get curious about your field.

For example here is Purdue's handbook on credit guidelines:

https://www.purdue.edu/registrar/forms/Semester_Credit_Hours...

Re: Goodbye, data science

#205

Post by alexpetralia »

chasely wrote: post #42
samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

I've been pitched by many "data-driven" vendors offering predictions. They often have very impressive accuracy metrics (RMSE, R2, etc). When I dive into the details these metrics are often reported using in-sample predictions. I see this pointing to any of the following: a) DS teams overpromising the accuracy of their approaches b) marketing driving the narrative and DS getting pulled along c) incompetence from the D…

Are these inferential statistics not designed to be in-sample?

I would imagine predictive statistics use more out-of-sample metrics like precision and recall.

Re: Goodbye, data science

#206

Post by Hasz »

boringg wrote: post #28
jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished. I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre…

I see lots of concurring and dissenting opinions here, and will add one more:

For context, I double majored in two adjacent subjects, physics and math. I went to a state school that has a very strong physics program. I also worked in physics lab for the last ~2 years, and graduated a semester early. While I did OK academically, I had no desire to run the gauntlet again in grad school, and left to work in tech.

I have never, ever, been as a busy as I was in college, nor do I ever want to be. I think that's a good thing! I have much more time to explore things that don't pan out, to do things I know are not "productive" (i.e, play video games), and am generally happier.

Apart from quality of life improvements, I think there are additional financial and intellectual benefits to not being overly burdened -- the time to explore topics that were not immediately adjacent to my field of study results in extremely useful skill development and better cross-pollination of ideas.

Re: Goodbye, data science

#207

Post by disgruntledphd2 »

whiplash451 wrote: post #173
samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

You should pay the price for data leakage very quickly in production. Does management look at slides or AB test dashboards?

People don't pay attention to production metrics, and a noisy problem (like marketing or whatnot) can often be pretty bad for a looooonnnnnggg time before anyone notices.

Re: Goodbye, data science

#208

Post by etempleton »

thinkharderdev wrote: post #193
lotsofpulp wrote: post #117

Earlier quoted context omitted.

I have only heard “show me the data” when someone wants someone else to support a claim. I do not see why this would necessarily be a bad thing.

> I have only heard “show me the data” when someone wants someone else to support a claim. I've heard it a lot in situations where somebody is demanding a level of rigor that they themselves do not live up to. This is usually soon after they have framed the conversation around a solution that they want to pursue that also lacks any supporting data. That is to say, being data driven is on net good but it can also just…

This is exactly what I am referring to.

I am not talking about in the instance of claim verification. I have seen a number of instances where a leader just wants to see data. Not any specific data, just all of the data. There is a belief that data can solve problems if only they had enough of it.

Re: Goodbye, data science

#209

Post by CapmCrackaWaka »

bootsmann wrote: post #128
adamsmith143 wrote: post #65

Earlier quoted context omitted.

Where are these jobs where you can interview this badly and still get hired because in my experience DS interviews are extremely hard and often expect people to have very high Stats skills as well as Data Structures/Algo skills at FAANG level.

I think the issue here is that "data science" encompasses two very distinct branches of work. One answers to business needs and the other produces data based solutions for the product itself i.e you might have a data scientist who A/B tests your website design so you minimize your churn rate and the other is the team at uber eats who maintains the recommendation engine. While the distinction might not always be as sh…

Yes we definitely fall into more traditional "predictive modeling" data science than deep learning / recommendation algo roles.

Re: Goodbye, data science

#210

Post by ifyoubuildit »

yunwal wrote: post #162
ifyoubuildit wrote: post #147

Earlier quoted context omitted.

So whats the alternative? "Just trust me"?

The alternative is to trust in most circumstances (you did hire me after all), and do thorough analysis once in a while as a gut check.

And how do you do that thorough analysis without asking to see the data?
Post reply on HN