Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Goodbye, data science

ryxcommar.com

Post reply on HN
Print view View original

91–100 of 415 posts

Re: Goodbye, data science

#91

Post by BeFlatXIII »

dpkirchner wrote: post #41
Davertron wrote: post #36

Earlier quoted context omitted.

Well good luck then, in my experience the most free time I've ever had in my life was during college. I squandered massive amounts of that time doing things completely unrelated to education, and I definitely don't regret doing that. College isn't just about book learning after all. But still, BY FAR, college is the time of my life when I had the most free time to do whatever I wanted.

Middle and high school is where a lot of students learn to stop being curious due to a lack of time. College demands far fewer hours per day, but it can be hard to forget what was taught previously.

I have to agree with you. So many of my professors have been vocally disappointed with their students for their lack of intellectual curiosity after it had been beaten out of them through the overstuffed schedules and pointless busy work of K–12.

Re: Goodbye, data science

#92

Post by rjtavares »

tgtweak wrote: post #90

I feel like in the near future there will be a more formal hybrid role between data engineering and data science - like devops or full stack developers. The best data scientists I have worked with (ML mostly) have been incredible data engineers as well - some of them former sysadmins, backend developers or DBAs themselves. They know where to get the data, how to set up pipelines and jobs, how to make sure they run pr…

I agree, up to a point. I feel that companies up to a few billion dollars in stable revenues in a non data intensive business don't need teams of Data Scientists, but would benefit from some people having Data Science skills (both Data Engineers and Business Analysts).

Re: Goodbye, data science

#93

Post by warinukraine »

jmhammond wrote: post #13

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I d…

> But there’s also a part of me that’s just like, how can you not be curious? How can you write Python for 5 years of your life and never look at a bit of source code and try to understand how it works, why it was designed a certain way, and why a particular file in the repo is there? How can you fit a dozen regressions and not try to understand where those coefficients come from and the linear algebra behind it? I dunno, man.

Because there's a lot of things out there which are also interesting, and you don't have time to do all of them, so you choose. And different people choose differently.

Re: Goodbye, data science

#94

Post by vsareto »

6gvONxR4sf7o wrote: post #37
samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

This is the defining pain point for data science, in my experience. There’s no simple ground truth to test competence against. If someone tells you that the data says their work is good, the only real way to know if they’re right or wrong is to look at what the data says yourself. If 99% of the work is building and 1% is checking something like latency, then you’re likely to have more than one set of eyeballs on that…

That's the same for many tech jobs. Competence is often only a local thing, subject to politics, reputation, and appearances. There's also no ground truth because the ground changes so fast. No one knows if the technologies mentioned in the OP will be popular 5-10 years from now.

Re: Goodbye, data science

#95

Post by _jx7j »

samvher wrote: post #6

> Nobody knew or even cared what the difference was between good and bad data science work. Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case. In my experience it's even a little bit worse than that. Approaches that are wrong from a statistics point of view are more likely to generate impressive seeming results. But the flaws are often subtle. A…

>Meaning you could absolutely suck at your job or be incredible at it and you’d get nearly the same regards in either case.

One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build something amazing, or build a house of cards, nobody cares as long as the money people are happy, even if the business use case turns out to be wrong in the long run.

>With a short-term focus they also won't really care, because they can still put these results in marketing materials and impress most outsiders as well.

Forget Data Science, you see this in KPIs as well. Say a crappy metric has to be moved by Q2 next year and people will destroy the company to move it.

I feel like Data Science is just one of those areas where you are exposed to a wider range of people and get to feel the full crapola of the insanity of working in a corporation. For lots of roles (e.g. Engineering) you get to hide in a hole behind layers of people and not see some of this insanity.

Re: Goodbye, data science

#96

Post by nicoburns »

Davertron wrote: post #36
boringg wrote: post #28

Earlier quoted context omitted.

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished. I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre…

Well good luck then, in my experience the most free time I've ever had in my life was during college. I squandered massive amounts of that time doing things completely unrelated to education, and I definitely don't regret doing that. College isn't just about book learning after all. But still, BY FAR, college is the time of my life when I had the most free time to do whatever I wanted.

I had quite a lot of free time when in college, but I still feel like I had less time to pursue my interests. Reason being that the course itself was intellectually demanding while also being quite prescribed about what you had to learn. Meaning I ended up using all my mental capacity grinding through a bunch of stuff that my professors wanted me to learn, leaving me with much less time to go off and learn what interested me.

Both before and since I've had more free capacity to pursue learning for it's own sake.

Re: Goodbye, data science

#97

Post by rcarmo »

My $DAYJOB has a very high match to this. In fact, I used to do data and analytics full-time for a long while (a lot of Spark, basically, mostly cutting through the usual lambda and K-architecture hype, bulding data lakes and "lakehouses") in several F500 companies.

I could not agree more with the overall sentiment that data science is overblown in terms of reproducible results because the people doing it just don't have an actual process or good leadership focus (which is not just a startup problem...).

So much so that I stayed stauncihily on the "data engineering" track because it was much more concrete in terms of technology, performance drivers, and business outcomes than the folk who sold pipedreams of magical AI models that would provide amazing analytics overnight.

Turns out that if you can't get at, scrub and actually _use_ the data, figuring out trends or training models doesn't happen, so I focused on making at least that 50% of the project happen and leave nice, tidy infrastructure, workflows and schemas for the data science folk to go through.

I also had the good fortune to work with some very organized, knowledgeable ML folk who actually understood how things worked, but some partners and customers had... incredibly disorganized "data scientists" that would leave stuff scattered all over the place (including private copies of datasets on their laptops when we had nice, secure remote sandboxes for them that even did data masking to avoid leaking sensitive data).

Personally, I blame a lot of this on lack of certifications or professional training that emphasises _process_. Otherwise it's exactly the same problem we've had for the past 20 years in BI departments: People doing their own Excel sheets because "SQL is hard" and nobody can do ETL properly.

(Full disclosure: I am an MS FTE, spent something like 10 years doing analytics almost full time, and have presented on how to do Data Science at scale a few times: https://carmo.io/talks)

Re: Goodbye, data science

#98

Post by noelsusman »

adamsmith143 wrote: post #65
CapmCrackaWaka wrote: post #30

Earlier quoted context omitted.

I've seen this a LOT in my professional group. Many people (who often have PhDs!!) I interview for data science positions seem to know absolutely nothing about the algorithms they use professionally, or how to optimize them, or why they are a good fit for their use case, etc etc etc. I usually see through LinkedIn that these same people are now in impressive-sounding positions at other companies. I had one candidate…

Where are these jobs where you can interview this badly and still get hired because in my experience DS interviews are extremely hard and often expect people to have very high Stats skills as well as Data Structures/Algo skills at FAANG level.

It's different at a lot of non-tech companies. I'm in the nonprofit world and my interview barely had any technical component at all.

Re: Goodbye, data science

#99

Post by throwawaaarrgh »

I've never met a data scientist who could do anything more than basic statistics combined with the Python skills of a fifth grader (that's probably insulting to today's fifth graders tho). I honestly have no idea what they're supposed to be doing or why they're paid so much money. Pay a high school junior for the same and get better work. And where's the scientific method? Where's the experiments and rigor?

Re: Goodbye, data science

#100

Post by nvrspyx »

Davertron wrote: post #36
boringg wrote: post #28

Earlier quoted context omitted.

To counter your professor opinion. The amount of extra time available as a student that I had to pursue things of interest was in the negative. All academic time was spent getting course content accomplished. I am a naturally curious individual but time limitations prevent further exploration in most circumstances. Additionally there is a relevancy factor weighed on top of it. If something looks curious I have to pre…

Well good luck then, in my experience the most free time I've ever had in my life was during college. I squandered massive amounts of that time doing things completely unrelated to education, and I definitely don't regret doing that. College isn't just about book learning after all. But still, BY FAR, college is the time of my life when I had the most free time to do whatever I wanted.

This was true for my undergrad, but my graduate program demands almost all of my free time, including weekends. Although, this may mostly be due to a drastic change in field of study from the two (social science to computer science) where I probably have to dedicate more time than those that already have knowledge/experience in this field.
Post reply on HN