Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The revenge of the data scientist

hamel.dev

Post reply on HN
Print view View original

31–40 of 44 posts

Re: The revenge of the data scientist

#31

Post by DeathArrow »

Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

So you consider data scientist akin to QA? They are just validating LLM based solutions?

Re: The revenge of the data scientist

#32

Post by DeathArrow »

Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

I believe data scientists and ML engineers should not be conflated.

Re: The revenge of the data scientist

#33

Post by disgruntledphd2 »

DeathArrow wrote: post #32
Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

I believe data scientists and ML engineers should not be conflated.

I believe that the massive splitting of data roles over the past decade is both a product of ZIRP and premature optimisation.

Re: The revenge of the data scientist

#34

Post by alexhans »

redhale wrote: post #11
Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

I agree with your take. I don't really see why evals are assumed to be exclusively in the domain of data scientists. In my experience SWEs-turned-AI Engineers are much better suited to building agents. Some struggle more than others, but "evals as automated tests" is, imo, so obvious a mental model, and can be so well adapted to by good SWEs, that data scientists have no real role on many "agent" projects. I'm not sa…

[dead]

Re: The revenge of the data scientist

#35

Post by jappgar »

laichzeit0 wrote: post #25
Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

I don’t actually even know what people are hinting at when they say that LLMs replace the need for building custom models. Regression models? People are using LLMs instead of say building a Bayesian hierarchical model? That’s not possible. Time series modeling using an LLM? Also ridiculous. Recommender systems? Ok maybe, still utterly ridiculous and abysmally slow. For anything NLP sure, it definitely wins. However,…

You're right. For years the real impediment to "AI" products at many companies was the sheer crappiness of ML frameworks which were built by and for grad students, not professional engineers.

When LLMs appeared it was just so much easier to use then as an uber model and leave behind the training and inference infrastructure (if you can even call it that).

Now that LLMs can code I expect we'll be coding up custom model pipelines more and more... but only when we stop subsidizing LLMs.

Re: The revenge of the data scientist

#36

Post by schnitzelstoat »

Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

I think most use-cases will still use simpler models like XGBoost etc. rather than LLM's. Customer segmentation is a really common use-case with no need for an LLM. Same for revenue/LTV forecasting.

Perhaps they can use the LLM to write and deploy these models without needing a Data Scientist but that seems risky to say the least.

In my company, the most Data Scientist-adjacent people are the Data Analysts but they tend not to have programming experience beyond SQL and basic Python and they aren't used to using the terminal etc.

Re: The revenge of the data scientist

#37

Post by valzam »

schnitzelstoat wrote: post #36
Flashtoo wrote: post #8

These are good practices to keep in mind when setting up GenAI solutions, but I'm not convinced that this part of the job will allow "data scientist" as a profession to thrive. Here's my pessimistic take. Data scientists were appreciated largely because of their ability to create models that unlock business value. Model creation was a dark magic that you needed strong mathematical skills to perform - or at least that…

I think most use-cases will still use simpler models like XGBoost etc. rather than LLM's. Customer segmentation is a really common use-case with no need for an LLM. Same for revenue/LTV forecasting. Perhaps they can use the LLM to write and deploy these models without needing a Data Scientist but that seems risky to say the least. In my company, the most Data Scientist-adjacent people are the Data Analysts but they t…

Do those use cases need LLMs? Probably not. but if good results can be had with a day of prompting (in addition to the stuff mentioned in the article, which you have to do anyway) and a smaller model like Haiku gives good results why would you build a classifer before you have literally millions of customers?

The LLM solution will be much more flexible because prompts can change more easily than training data and input tokens are cheap.

Re: The revenge of the data scientist

#38

Post by schnitzelstoat »

kokken wrote: post #21

I don't understand the framing of the assumption. Was the data scientist role only about building NLP models? Are the LLms gonna build Churn prediction models? Tell the PM why stopping the A/B test halfway through is a bad idea? Push back on loony ideas of applying ML to predicting sales from user horoscopes? Maybe the role is a bit tinier in scope than 10 years ago, but I see it as a good thing. If you looked at DS…

Exactly - in my company we had some NLP models in Customer Service (bag-of-words for classifying tickets) but everywhere else it was just classification or regression problems.

So yeah, the bag-of-words model got replaced with a chatbot several years ago (when chatbots were all the rage back in like 2017) and will probably get replaced again with an LLM-enhanced chatbot soon. But the meat and potatoes are those classification and regression models and they aren't going anywhere.

Re: The revenge of the data scientist

#39

Post by fn-mote »

valzam wrote: post #37
schnitzelstoat wrote: post #36

Earlier quoted context omitted.

I think most use-cases will still use simpler models like XGBoost etc. rather than LLM's. Customer segmentation is a really common use-case with no need for an LLM. Same for revenue/LTV forecasting. Perhaps they can use the LLM to write and deploy these models without needing a Data Scientist but that seems risky to say the least. In my company, the most Data Scientist-adjacent people are the Data Analysts but they t…

Do those use cases need LLMs? Probably not. but if good results can be had with a day of prompting (in addition to the stuff mentioned in the article, which you have to do anyway) and a smaller model like Haiku gives good results why would you build a classifer before you have literally millions of customers? The LLM solution will be much more flexible because prompts can change more easily than training data and inp…

> Do those use cases need LLMs? Probably not.

One of the points of the article is the importance of gathering data to support your conclusions.

> prompts can change more easily than training data

Training data is real, and prompts are not. I don’t think this is an apples to apples comparison.

Post reply on HN