The revenge of the data scientist • Hacker News Board

Re: The revenge of the data scientist

#41

Post by valzam » Thu, Apr 02, 2026, 4:13 PM UTC

fn-mote wrote: post #39

valzam wrote: post #37

Earlier quoted context omitted.

Do those use cases need LLMs? Probably not. but if good results can be had with a day of prompting (in addition to the stuff mentioned in the article, which you have to do anyway) and a smaller model like Haiku gives good results why would you build a classifer before you have literally millions of customers? The LLM solution will be much more flexible because prompts can change more easily than training data and inp…

> Do those use cases need LLMs? Probably not. One of the points of the article is the importance of gathering data to support your conclusions. > prompts can change more easily than training data Training data is real, and prompts are not. I don’t think this is an apples to apples comparison.

I don't disagree that very numerical tasks like revenue forecasting are not a good fit for LLMs. But neither did a lot of data scientist concerns themselves with such things (compared to business analysts and the like). Software to achieve this has been commoditized.

Re: The revenge of the data scientist

#42

Post by thesz » Thu, Apr 02, 2026, 7:12 PM UTC

kj4211cash wrote: post #24

Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

You recognize that you haven't really needed strong mathematical (or coding) skills to create models for some time. Data Scientists add value by knowing how to translate business speak into XGBoost type model and interesting XGBoost model results into business speak. And, frankly, often by being some of the smartest people in the room. The math is occasionally helpful for speaking the language of the XGBoost model. A…

  > You recognize that you haven't really needed strong mathematical (or coding) skills to create models for some time.

And then there goes something like this [1], where researchers failed to control for p-value: "In this particular setting, emergent abilities claims are possibly infected by a failure to control for multiple comparisons. In BIG-Bench alone, there are ≥220 tasks, ∼40 metrics per task, ∼10 model families, for a total of ∼10^6 task-metric-model family triplets, meaning probability that no task-metric-model family triplet exhibits an emergent ability by random chance might be small."

[1] https://arxiv.org/abs/2304.15004

Re: The revenge of the data scientist

#43

Post by whattheheckheck » Fri, Apr 03, 2026, 2:07 AM UTC

__mharrison__ wrote: post #10

I just spent yesterday applying Kaparthy's autoresearch on an ML problem. I teach ML for a living and was amazed with what the tokens gave back to me after many rounds of experiments. If Kaggle was still a thing, AI would generally beat it. The challenge I've seen is that most data science/ml modeling work is quite weak. Folks don't even know the basic tools well. Not sure if giving AI to them will really open up man…

Karparthys autoresearch is just automated overfitting no?

Re: The revenge of the data scientist

#44

Post by mscbuck » Fri, Apr 03, 2026, 11:12 AM UTC

twelfthnight wrote: post #12

__mharrison__ wrote: post #10

I just spent yesterday applying Kaparthy's autoresearch on an ML problem. I teach ML for a living and was amazed with what the tokens gave back to me after many rounds of experiments. If Kaggle was still a thing, AI would generally beat it. The challenge I've seen is that most data science/ml modeling work is quite weak. Folks don't even know the basic tools well. Not sure if giving AI to them will really open up man…

I agree AI could probably do a decent job on Kaggle problems. Of course, almost no DS job is building models with well-defined objectives and perfect data. The DS and MLE folks I work with mostly spend their time reframing ill-posed product requests into ML systems that can be maintained and improved with feedback loops. A _huge_ part of a DS is saying "No" to bad ideas posed by non-experts. The issue with LLMs is al…

Yeah, once you move onto legitimate business evaluation metrics (where Precision@k or Recall@k don't actually fit your business model without modification), GPTs just seem to suffer without context, and hey, knowing the context is part of what gives a data scientist his value.