We don't need data scientists, we need data engineers • Hacker News Board

Re: We don't need data scientists, we need data engineers

#251

Post by dijksterhuis » Thu, Jan 14, 2021, 9:53 PM UTC

mgh2 wrote: post #242

AndrewKemendo wrote: post #25

Earlier quoted context omitted.

Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning." Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.

Do data engineers have good analysis skills? Do business analysts have good engineering skills? I don't think either of them can fill the data scientist role. The scientific training and mindset (scientific method, hypothesis, experiment setup, etc.) to even create an accurate model is an undervalued skill here no? Even if data cleaning is automated, these skills cannot be easily learned. There is a reason why so man…

> Do data engineers have good analysis skills?

Yes.

> Do business analysts have good engineering skills?

Depends on the analyst.

> I don't think either of them can fill the data scientist role.

> The scientific training and mindset (scientific method, hypothesis, experiment setup, etc.) to even create an accurate model is an undervalued skill here no? Even if data cleaning is automated, these skills cannot be easily learned.

It's not about replacing data scientists with data engineers, it's about both roles working together to make everything more efficient.

The hiring rate for data scientists has plateaued. The industry doesn't need any more of them. Why? Because data scientists often can't solve problems fast enough. It's a commonly quoted statistic that 70% of any data science task is data cleansing and/or etl. A data engineer's job is to take that 70% and turn it into 10%. The data engineer saves the data scientist time, meaning they can focus on what they're supposed to do -- build models.

Re: We don't need data scientists, we need data engineers

#252

Post by pmiller2 » Thu, Jan 14, 2021, 9:59 PM UTC

mywittyname wrote: post #241

mushbino wrote: post #228

Earlier quoted context omitted.

Engineers get paid well in SV because they are in demand, have lots of employment opportunities, and therefore are more difficult to retain.

And because their contributions can be tied back to revenue. You need both, demand for talent, as well as the ability & justification to pay for it. Engineers are in high demand all over the world. But most companies do not profit enough from technology to justify similar paying SV salaries.

Not always. Frequently, the connection between code that’s written today and revenue tomorrow is tenuous and difficult to package in a way that says “look at me! I’m valuable!”

And, then there are those somewhat rare occasions where a project is not intended to increase revenue, and may even decrease it. At my last employer, we guesstimated that a project I worked on for months could possibly have ended up costing us $2M per year in revenue. That was both accepted and expected, because we were doing it to gain goodwill with users, but in such a way that it might end up pissing off a small minority of our customers.

I really wish, just once, I could work on a project and put underneath it on my resume “Increased revenue by X%,” because I’ve never worked on anything that was so easy to directly trace back to the top line.

Cost savings are another story, because engineers can fairly easily quantify how much less money is being spent by doing $THING a bit more efficiently....

Re: We don't need data scientists, we need data engineers

#253

Post by _RPL5_ » Thu, Jan 14, 2021, 10:01 PM UTC

proverbialbunny wrote: post #189

a_zaydak wrote: post #185

Earlier quoted context omitted.

Thanks for the feedback! Seems like you and I both have had a bit of experience being first engineering hires at startups but have had very different experiences when it comes to rolls or a data scientist. I appreciate that.

Np. There is a common trend in the industry where a company hires on a data scientist, doesn't know the data prerequisites (specifically labeled data), the data scientist struggles, after a while the company fires the data scientist. This leaves the company with a bad taste in their mouth. In recent years I tend to get hired on as a specialist to help fix this. (And yes, I've been the first engineer hired on too.) Wh…

Thank you for your comments! They are very insightful. To piggyback a bit:

Assuming you are a competent data "analyst" who wants to become a data engineer, how would you go about it? Is "go back to school and get a CS degree" the answer? I suppose this question is very broad, but I am curious if a practitioner like you has an opinion.

---

To give some context:

I recently graduated with a STEM PhD, and looking to move into data science. Reading the comments, I feel like I fall into the "pointless data scientist" cohort derided in this thread. Eg: I am very comfortable doing typical analytical work & occasionally training models inside a notebook, but I am neither a cutting-edge theoretical statistician nor a data engineer.

I've been trying to improve on the engineering side. For example, I did a project recently where I set up a rudimentary pipeline that continuously pings an API, uploads the data to a cloud database, then serves up the analysis via a Flask app. For me this was a big step up from just doing notebooks on a csv file :)

But moving beyond the basics, I am not sure what to study next. Hence my question. If you have any suggestions, I would greatly appreciate it!

Re: We don't need data scientists, we need data engineers

#254

Post by mywittyname » Thu, Jan 14, 2021, 10:13 PM UTC

Ansil849 wrote: post #95

Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.

I think a lot of people feel like data scientists get all the credit and the fun work, while we have to do all the heavy lifting and boring stuff that they need. There's this idea that data science is a special and unique skillset that couldn't possibly be possessed by a simple software engineer. Despite being largely a subdomain of CS.

I remember an era before "data scientist" was a job title. When we (programmers) would analyze data to see if we had enough information available to identify the problem, if not, fix that, then come up with a strategy to solve it, test, and finally deploy the model. The fun part was trying different solutions and analyzing the data. It also felt awesome to deploy a product that worked like "magic." Product owners didn't know or care what a neural net was, they were just happy it worked.

Now there are tons of data scientist out there who take the easy, fun, rewarding work and try to skip over the nitty gritty implementation details. Then management thinks engineering is incapable of doing such work, and the only time we get the opportunity to do something fun is to do so behind the scenes.

Re: We don't need data scientists, we need data engineers

#255

Post by notretarded » Thu, Jan 14, 2021, 10:13 PM UTC

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

Please don't change the status quo. I love my cushy job.

Re: We don't need data scientists, we need data engineers

#256

Post by jsinai » Thu, Jan 14, 2021, 10:17 PM UTC

alexpetralia wrote: post #94

C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

The other thing I'd emphasize here is dealing with "state". Data is effectively state. As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Someone has to manage the queues, file versions/locations, logs, databases, configurations and so on. That is all "data". State management is a tricky problem even in a single-threaded a…

>The other thing I'd emphasize here is dealing with "state". Data is effectively state.

It gets even more complicated. It’s not just the current state that matters, but also the history (sometimes the entire history) up to that state.

Re: We don't need data scientists, we need data engineers

#257

Post by walleeee » Thu, Jan 14, 2021, 10:17 PM UTC

theflyinghorse wrote: post #204

C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

I wonder how: 1. one finds organizations that have data engineering 2. gets hired to said organization with software engineering background.

Nearly any field of computational science likely needs skilled data engineers. You could search for topics that interest you online and contact people accordingly.

I cold-emailed my current lab's P.I. and just asked for work. Search for "research software engineer" or "scientific computing professional" positions. Plenty of data engineering goes on in many fields (environmental science, climate modeling, high energy physics, physical chemistry, etc), and plenty of fields desperately need to develop an engineering culture (e.g., plant biology, my field), whatever interests you. Availability and compensation will vary by discipline.

Re: We don't need data scientists, we need data engineers

#258

Post by lordgrenville » Thu, Jan 14, 2021, 10:18 PM UTC

theflyinghorse wrote: post #208

baron_harkonnen wrote: post #88

Earlier quoted context omitted.

The data science field has been flooded with PhDs with nowhere else to go that have no background in engineering, and sadly often have a very poor understanding of both machine learning and statistics. Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model. I've worked in industry awhile do…

I work at a place with a very high count of PhDs. Some of them write code. All of them view writing code as something menial and unimportant and its shows in the resulting work, which from my experience is atrocious. Of course I understand that YMV, but I will forever be skeptical of anyone writing code with a PhD after working here.

Are they CS/EE PhDs?

Re: We don't need data scientists, we need data engineers

#259

Post by mywittyname » Thu, Jan 14, 2021, 10:21 PM UTC

ageofwant wrote: post #14

zeku wrote: post #7

I'm a data engineer for most of my day right now, and a lot of it is done with ruby/python/shell scripts into postgres DBs. What learning path should I go down? I'm a solo actor at work with a lot of agency to decide my workflows. I see myself building small to medium size data collections over the next year or two at my job. Can someone point me to some learning? I have a CS degree etc. and my title in software engi…

Adopt a good Workflow tool, like apache airflow. Easiest is to rent a service from AWS https://aws.amazon.com/managed-workflows-for-apache-airflow/

Seconding this, Apache Airflow is awesome.

I can't believe how much time is saves during development.

Re: We don't need data scientists, we need data engineers

#260

Post by avs733 » Thu, Jan 14, 2021, 10:22 PM UTC

bonniemuffin wrote: post #53

hn_throwaway_99 wrote: post #5

Earlier quoted context omitted.

It's not a straw man, I've seen it clear as day in several companies. When it comes to data science, it's "garbage in, garbage out". I've seen companies do lots of "data science" with a bunch of data scientists skilled in python and jupyter notebooks, only to discover a ton of work was useless because the incoming event data was tagged incorrectly due to a bug. The actual process of collecting, aggregating, cleaning…

I suspect this may actually be an issue of school vs real world rather than scientist vs engineer. Data in the classroom setting is pristine and beautiful; data in the real world is messy and buggy. You have to get burned by buggy data a few times (or maybe a bunch of times) in the real world to learn to look for bad data smells -- I don't think schools effectively teach this kind of intuition, regardless of whether…

>Data in the classroom setting is pristine and beautiful; data in the real world is messy and buggy.

I got to take over our department's undergraduate statistics course a few years back.

The first change I made was all homework, tests, and projects used real data set. I intentionally have them collect bad data (they don't know its bad before hand). First day of class we collect data using the board game operation...I give basic instructions and then halfway through ask everyone to stop and agree on how they are entering data for the variable of 'success or failure' of the surgery. Oops...

In my experience teaching the course, the reason the students (engineers) find statistical reasoning hard is:

* They have never been given anything 'broken', everything is curated to avoid things not working. The result is they think data has inherent meaning. A right answer.

* Their entire learning experience has been stripped of context and the need to make decisions with information. They can give me a p value but are terrified (not unable, just unwilling) to interpret it or give it meaning.

* They have never encountered the concept of variability...everything is presented as systems with exact inputs and outputs.

When I work with postdocs, I sometimes (less frequently) encounter many of the same challenges. Data is treated as sacred and external and inherent. It's wild to me.