Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

31–40 of 367 posts

Re: We don't need data scientists, we need data engineers

#31

Post by serjester »

jariel wrote: post #9
avs733 wrote: post #2

I teach engineers for a living. I struggle to see how this is not just a straw man argument based on colloquial usage of terms. It is just inferences drawn based on job ads that are rarely written by people doing the job and instead are effectively human-as-seo-optimized so the best candidates can find the job they hopefully fit for and not be too confused to apply for it.

So I think that the delineation between the scientist working with the content, and the Engineers who actually provide the mechanics for it is very fair. If there is a question mark here - it's really how much value are we deriving from all of these data people? Where is all the ML that's changing our lives? Search, Alexa and TikTok, I can see it. In the future obviously vision systems for autonomous cars etc.. But I…

I used to work at a legacy automaker and you’d be shocked at how much ML has changed certain areas of the business. It used to take an entire department to sort warranty claims and it’s now mostly automated. Aluminum part defects are now spotted automatically on the plant floor. Don’t even get me started with telematics data.

Most software isn’t consumer facing but just because you don’t see it doesn’t mean it’s not changing things around you. ML tends to be overhyped but your assessment is too pessimistic.

Re: We don't need data scientists, we need data engineers

#32

Post by ziml77 »

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up.

I also end up having to be the one to talk to data vendors to understand their data feeds and essentially translate that for the data scientists. Having to sit in the middle is annoying for me and suboptimal for the business.

Re: We don't need data scientists, we need data engineers

#33

Post by dpayonk »

AndrewKemendo wrote: post #25
alex_anglin wrote: post #10

Earlier quoted context omitted.

The aspiration that GP was getting to was that less cleaning is required as a result of better data engineering, I believe.

Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning." Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.

If we only had to use 1st party data, that might be easier. But then again, if you’re building your product incrementally, you’re still going to have instrumentation holes that you may or may not be able to partially backfill.

Re: We don't need data scientists, we need data engineers

#34

Post by superbcarrot »

ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

I think that this is more of a problem with the specific people that you have worked with and it isn't inherent to the role of a data scientist.

Re: We don't need data scientists, we need data engineers

#36

Post by secondcoming »

C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

Indeed, adtech is a great place to work for anyone interesting in working with data. And yes, people working in adtech hate, and block, ads too.

Re: We don't need data scientists, we need data engineers

#38

Post by alexpetralia »

slt2021 wrote: post #35

Full Stack Data Scientist (data janitor + data engineer + ML engineer + ML Ops + Business Analyst) is the future

These are incredibly disparate skill sets. Of course anyone would want to hire someone like this, and far more would claim to possess such a broad skill set, but in practice it is extremely rare.

You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy), technical skills (application programming, advanced databases, advanced machine learning, Excel modeling) and finally some business domain knowledge.

This is an uncommon intersection of skills.

Re: We don't need data scientists, we need data engineers

#39

Post by nerdponx »

ziml77 wrote: post #32

Having to deal with data scientists, I absolutely agree. The thing that I've seen that lands in the "lab" vs production distinction is that these people expect their data to be pristine. They flip out when the world isn't as perfect as their models want. Leads to me as just a normal software developer having to do the data analysis and figure out how to clean it up. I also end up having to be the one to talk to data…

Instead of sneering at "having to deal with" data scientists, consider that the data scientists themselves would often much rather have data engineers and dev ops people involved in the process.

Data scientists like to quip that 80% of the job is data cleaning, with the remaining 20% divided up arbitrarily among other tasks as suited the joke. In some shops nowadays, it's more like 45% data cleaning, 45% data engineering/ops/programming just trying to make your results available to the rest of your organization, and 10% research.

If I can spend less time learning/doing software engineering and devops and more time doing actual data science, that's great. At a previous job, my team was clamoring for more data engineer hiring, and part of the reason our projects were slipping and starting to fail was lack of data engineering support. Our tooling was shit, our processes were shit, our code was shit, and access to (and trust of) our data sources was especially wet and stinky shit.

It made the daily work of doing data science a miserable slog of ad-hoc duct-tape solutions, and it contributed to us being generally ineffective as a team.

All of this would have been fixed if we had one competent data engineer with some actual real-world data/ML engineering experience and good communication/advocacy skills. Let alone two or three!

Post reply on HN