Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The revenge of the data scientist

hamel.dev

Post reply on HN
Print view View original

21–30 of 44 posts

Re: The revenge of the data scientist

#21

Post by kokken »

I don't understand the framing of the assumption.

Was the data scientist role only about building NLP models? Are the LLms gonna build Churn prediction models? Tell the PM why stopping the A/B test halfway through is a bad idea? Push back on loony ideas of applying ML to predicting sales from user horoscopes?

Maybe the role is a bit tinier in scope than 10 years ago, but I see it as a good thing. If you looked at DS positions on job search sites the role descriptions would be all over the place, maybe now at least we'll see it consolidate.

Re: The revenge of the data scientist

#22

Post by HeytalePazguato »

This matches what I've seen working with automated systems. The watching part is genuinely underrated. Evals give you a score. Watching gives you intuition about failure modes you didn't know to test for.

Sitting with a running system teaches you things you would never think to measure.

Re: The revenge of the data scientist

#23

Post by LaserPineapple »

__mharrison__ wrote: post #10

I just spent yesterday applying Kaparthy's autoresearch on an ML problem. I teach ML for a living and was amazed with what the tokens gave back to me after many rounds of experiments. If Kaggle was still a thing, AI would generally beat it. The challenge I've seen is that most data science/ml modeling work is quite weak. Folks don't even know the basic tools well. Not sure if giving AI to them will really open up man…

Is Kaggle no longer a thing?

Re: The revenge of the data scientist

#24

Post by kj4211cash »

Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

You recognize that you haven't really needed strong mathematical (or coding) skills to create models for some time. Data Scientists add value by knowing how to translate business speak into XGBoost type model and interesting XGBoost model results into business speak. And, frankly, often by being some of the smartest people in the room. The math is occasionally helpful for speaking the language of the XGBoost model. And picking only people who are decent at math (and coding) helps ensure the smart factor. How much of that will really change with AI? I've also seen Business stakeholders try to use the chatbot to bypass the Data Scientist. Typically it's not long before there is a design decision or an interesting result the Business stakeholders don't understand. That's why I think there will be demand for Data Scientists. Not exactly evaluation and monitoring. And definitely not gatekeeping building of LLM solutions. Often the opposite, called in to explain and debug the Business stakeholders' slop.

Re: The revenge of the data scientist

#25

Post by laichzeit0 »

Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

I don’t actually even know what people are hinting at when they say that LLMs replace the need for building custom models. Regression models? People are using LLMs instead of say building a Bayesian hierarchical model? That’s not possible. Time series modeling using an LLM? Also ridiculous. Recommender systems? Ok maybe, still utterly ridiculous and abysmally slow.

For anything NLP sure, it definitely wins. However, I’ve just recently used some big fancy OpenAI model to actually just label thousands of text data for me, just so I could build a classifier with CatBoost. Guess what, inference speed is at a guaranteed sub 100ms and it costs $0 in tokens. The”AI Engineer” solution here would be just run every classification request through an LLM.

AI Engineering is going to have the same problem we had when Data Science as a term arrived and you had every Statistician saying they’re just re-inventing everything that exists in statistics, poorly.

Re: The revenge of the data scientist

#26

Post by Caum »

laichzeit0 wrote: post #25
Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

I don’t actually even know what people are hinting at when they say that LLMs replace the need for building custom models. Regression models? People are using LLMs instead of say building a Bayesian hierarchical model? That’s not possible. Time series modeling using an LLM? Also ridiculous. Recommender systems? Ok maybe, still utterly ridiculous and abysmally slow. For anything NLP sure, it definitely wins. However,…

[flagged]

Re: The revenge of the data scientist

#27

Post by jimbokun »

twelfthnight wrote: post #12
__mharrison__ wrote: post #10

I just spent yesterday applying Kaparthy's autoresearch on an ML problem. I teach ML for a living and was amazed with what the tokens gave back to me after many rounds of experiments. If Kaggle was still a thing, AI would generally beat it. The challenge I've seen is that most data science/ml modeling work is quite weak. Folks don't even know the basic tools well. Not sure if giving AI to them will really open up man…

I agree AI could probably do a decent job on Kaggle problems. Of course, almost no DS job is building models with well-defined objectives and perfect data. The DS and MLE folks I work with mostly spend their time reframing ill-posed product requests into ML systems that can be maintained and improved with feedback loops. A _huge_ part of a DS is saying "No" to bad ideas posed by non-experts. The issue with LLMs is al…

The data scientist is like in house lawyers in that respect.

Re: The revenge of the data scientist

#28

Post by efavdb »

I'm a data-scientist now, and a fan of claude code for implementing things. But I have to say, I'm constantly surprised by how "dumb" chatgpt is as a math research partner. I will ask it a math question I'm thinking about, get a confident answer back, only to realize hours to days later that it was 180 degrees backwards. I'm so frustrated right now with this that I'm almost ready to stop asking it such questions at all. I'm aware this seems to contrast strongly with other math-people's enthusiasm e.g., Terrance Tao. Unclear why my mileage varies.

Much of my work takes the form above -- in other words figuring out what to do. once i've decided, it can of course spit out the boilerplate code much faster than I could, and I appreciate that. But for the moment I think I still have some job security thanks to the first issue.

Re: The revenge of the data scientist

#29

Post by vicchenai »

The monitoring and evaluation piece is underrated. In my experience the hardest part isn't building the initial LLM pipeline, it's knowing when the thing quietly broke. Domain expertise matters a lot there because you need to design evals that actually catch the failure modes that matter for your specific data distribution.

Re: The revenge of the data scientist

#30

Post by codebolt »

Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

As a AI-aware software engineer currently creating systems that integrate with LLM provider APIs for my company- who also has no idea what an eval is or how a data scientist thinks about RAG. I honestly don't see what value a data scientist would bring to the table for my team. Maybe someone would care to enlighten me?
Post reply on HN