Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

71–80 of 367 posts

Re: We don't need data scientists, we need data engineers

#71

Post by superbcarrot »

sidlls wrote: post #66
superbcarrot wrote: post #34

Earlier quoted context omitted.

I think that this is more of a problem with the specific people that you have worked with and it isn't inherent to the role of a data scientist.

It’s becoming more inherent, especially as the field is populated with people who have no experience with the “science” part. That is, with the very real and ubiquitous problem of collecting and cleaning data to make it fit for scientific study. Even theoretical physicists, for example, participate in and rely on empirical data collection, and understand deeply how messy and fraught with error it is. I don’t see the…

> with people who have no experience with the “science” part

It's interesting that you put it that way because a lot of the other complaints in this thread are that the people who expect their data to be ready for use are exactly the people with science experience but without the relevant technical background.

Re: We don't need data scientists, we need data engineers

#72

Post by jesseryoung »

4 years ago I moved from a role where I primarily wrote C# as an architect on a web application, to an architect helping to build a data warehouse. The contrast in tooling, discipline and information available to build anything in the data world is so stark it had me questioning my career decisions. Sure, you can read Kimball and Inmon and I'm sure there are a handful of others out there - but there are drastically fewer than what you can find in the application development space.

Things are getting better, Visual ETL tools are falling out of favor to proper coded ETL (spark, dbt, etc) and data teams are starting to see the value of actually engineering a solution instead of just throwing it over the wall to a DBA to deal with. But tooling, and general information on the web is still lacking. Pushing data engineers over "etl developers" or "bi developers" (or "data scientists") will drastically improve any organizations ability to actually deliver real analytics and hopefully an industry wide push will raise all ships.

Re: We don't need data scientists, we need data engineers

#73

Post by mywittyname »

superbcarrot wrote: post #48
mywittyname wrote: post #12

I always felt that tech-focused data scientists should also be required to know how process data end-to-end; at minimum, from a SQL database to deployed model, but knowing how to collect & clean data is important too. It seems like the industry is trying fill the gap that was created by a glut of people without math/cs backgrounds going into 5-week data science courses who then need hand-holding when they get real jo…

> Data science & engineering should be treated as a single collection of skill-sets. This can be tough because there could be a lot in that skill set. You can't realistically expect someone to have solid knowledge of statistics including specialising in the sub-field and type of algorithms that your product needs, and also be able to write good code and act as a developer, and also have solid knowledge of all the too…

These are all skills taught in standard computer science programs. Granted, some are electives, like high-level stats. But even back in 2010, data science electives were available to fill the gaps. I took three DS&E classes in college with projects that were end-to-end platforms, where you'd have to collect, clean, and analyze the data, then build, test, and deploy models from it.

I would certainly hope that college courses are even more comprehensive after 10 years and an explosion in interest for the field.

Also, much like being a full stack developer, a full stack data engineer doesn't need to know everything at a master level. But that you can at least handle tasks at most points in the chain.

Re: We don't need data scientists, we need data engineers

#74

Post by program-iscuous »

That's interesting post, however there's large bias in my opinion in how the analysis is done.

You have a stratified sample of companies in their early stages, I think it's quite normal for most companies in their early stages to prioritize data engineers rather than data scientists.

Data scientist comes after the data engineer, and if you have a data scientist and not a data engineer then probably the data scientist does both jobs. On the other hand, data engineer is not dependent on a data scientist.

To conclude, I think that indeed there are more data engineer positions because there are too many "data scientists", however the true difference is not as large as in your analysis.

Re: We don't need data scientists, we need data engineers

#75

Post by thecolorgreen »

nojito wrote: post #67
thecolorgreen wrote: post #61

I'm a SWE and data engineering actually sounds super interesting to me. Unfortunately, my day-to-day doesn't provide opportunities to work with the massive amounts of data we generate. I've looked into learning this stuff online but courses like DataCamp seem too basic (I have experience with Python and data cleaning in a research setting along with some academic ML experience) or downright a bit scammy. Many of the…

We just hired a SWE turned Data Engineer. You don't need to handle massive data to make the transition. I believe all the person did was build a small but robust pipeline that took some API response data, cleaned it, populated some sqlite dbs, replicated it for a small team to use and kept it updated every few days automatically.

That's good news for me. For a minute, I genuinely thought I'd have to waste time going through a bunch of stuff I already knew to learn a tiny bit and earn a certificate to prove my skills.

Re: We don't need data scientists, we need data engineers

#77

Post by eeZah7Ux »

mongol wrote: post #76

Why is data scientist a profession in IT but not for example computer scientist? Many IT professionals studied computer science but they don't call themselves scientists in their line of work.

Don't give developer ideas. People with a javascript bootcamp and 2 years experience are already called "senior engineer".

Re: We don't need data scientists, we need data engineers

#78

Post by spacemanmatt »

Data engineer, here. Or at least, that has been my title a couple of times.

Some data is inherently trash but a huge part of the data quality problem is sources who are allowed to produce trash that everyone else has to clean up, when it would be way more efficient for them to quit producing trash.

Not to pick on any one institution, but SOAP seems to be a read flag that the service will also deliver some screwy data.

Re: We don't need data scientists, we need data engineers

#79

Post by spacemanmatt »

bitcharmer wrote: post #49
AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.

Is this sentiment perhaps due to someone "practicing CS" on your engineering schedule? What's the real harm you're describing?

Re: We don't need data scientists, we need data engineers

#80

Post by spacemanmatt »

digitalsushi wrote: post #8
amelius wrote: post #4

In other words: we need plumbers.

Or, if the data is food, we have chefs making incredible plates, but we need wait staff to get it to the people who want to eat it. I would love to be on that wait staff; as an infradev I feel like the process is very close to me but I am struggling to break into it.

I would put data engineering on the supply side of the chef: This would be ingredients, delivery scheduling, and pre-prep functions. That sort of thing.
Post reply on HN