We don't need data scientists, we need data engineers • Hacker News Board

Re: We don't need data scientists, we need data engineers

#181

Post by prionassembly » Thu, Jan 14, 2021, 7:05 PM UTC

I'm late to the comment party, but: this is classic "commoditize your complement".

This guy would have you believe that Pytorch has Solved the entire, vast field of data analysis as inherited from Newton, de Moivre, Laplace, Bayes, Fisher, Neyman, Pearson, Wald, Savage, Jaynes, Breiman, Pearl.

This is a lot like saying that photography has Solved art, and now we need people who can climb ladders and glue the posters on them big billboards. It would be delusional if it didn't have a self-interested angle.

What, we with math degrees are fully confident that the plumbing problem is easier to commoditize than the problem of making sense of data.

Re: We don't need data scientists, we need data engineers

#182

Post by proverbialbunny » Thu, Jan 14, 2021, 7:12 PM UTC

a_zaydak wrote: post #175

My view is from a small startup with little to no room for single purpose employees. When I first started hiring and working with data scientist my view was this: If you can only manipulate data and run it through pipelines to generate models then you can't do enough to be highly valuable. You either need to have a strong enough background in CS to build the pipelines / tools or a strong enough mathematics background…

I would be cautious about that. I've worked in the startup space for over 10 years now as a data scientist, often the first one hired on, working on the pipes.

From my experience, there are two types of data scientists who work who do infrastructure work: 1) Those who do not make the best data scientist because their skill set is too far in engineering land, leaving them weak where it counts. If the startup is relying on the data scientist to be profitable, I'd be cautious with these types. or 2) Someone who is senior, beyond senior really, who has worked both jobs, and doesn't mind doing both jobs. This unicorn is so rare it is mythical. The joke when the terminology was created is they're so rare no one has ever seen one, hence unicorn.

Me, I can not do the work I need to do if I'm on call. That is where I draw the line. That means hiring someone to monitor the infrastructure. Furthermore, I'm an okay architect, but you really do want to hire a specialist if you can help it for that. Do I help them with the infrastructure? Absolutely, but they're on call if a server is on fire. They have the admin login credentials, not me.

I get wearing multiple hats, but keep in mind to be a data scientist you're already wearing multiple hats. Being a data scientist is like double majoring and getting a phd. At what point are they stretched too thin? The consensus in the industry is they're already stretched too thin and should be broken up into different specialized roles.

>My question to HN is this; are there rolls at these larger companies for a Data Scientist who who primarily just crunches data in R and Python without the ability to actually build the pipelines / tools or conduct research?

That is the standard role, even at startups. However, the industry consensus these days is data scientists should have more responsibility when it comes to deploying models than previous standards.[1] So data scientists are being pushed in a more engineering direction, not with hosting sql servers and infrastructure, but with working with engineers to make sure the models are monitored properly. This change comes from model deployment being further automated as time goes on, making it easier for the data scientist to have more responsibility during this stage.

[1] source: https://www.dominodatalab.com/static/gfx/uploads/domino-mana... page 9. Suboptimal organization and incentive structures.

Re: We don't need data scientists, we need data engineers

#183

Post by AndrewKemendo » Thu, Jan 14, 2021, 7:12 PM UTC

darksaints wrote: post #90

AndrewKemendo wrote: post #25

Earlier quoted context omitted.

Correct. If you build your instrumentation correctly, then you don't really need to do any "cleaning." Doesn't mean you might not need to do transformation for different uses but ideally wouldn't need to, for example change data types like turning a bool into an int.

The problem is that data engineers that are geared towards analytics very very rarely control the systems that create the data. If you're lucky, you have the task of hounding a team within your company to get their data management practices in order. And the conversation there is whether they should make their job harder in order to make your job easier. Unfortunately, data engineers rarely deal with purely in-house…

Hence why I said DE need to be involved as early as possible. Aspirational sure, but that's what I've seen work the best and repeatably. It's the only scalable solution IMO otherwise you're perpetually playing catch-up.

On the point about the govt I literally built a completely new contract type and civilian hiring practices for the DoD to bring in Data Engineers so they could do exactly what I describe to make your life easier.

Re: We don't need data scientists, we need data engineers

#184

Post by chadash » Thu, Jan 14, 2021, 7:16 PM UTC

proverbialbunny wrote: post #163

twic wrote: post #147

Earlier quoted context omitted.

I'm not sure about crappy quants. Some people of the "quantitatively inclined trader who has learned Python" variety are never going to be good at the engineering side - it takes years to learn to be a good software engineer, and that's not a good use of time, for them, or for their employer. But they can still do useful work. The trick is to figure out how to work effectively with those people. Build infrastructure…

If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many fi…

> "To solve this problem many firms have their own internal programming language."

Any examples other than Jane Street?

Re: We don't need data scientists, we need data engineers

#185

Post by a_zaydak » Thu, Jan 14, 2021, 7:17 PM UTC

proverbialbunny wrote: post #182

a_zaydak wrote: post #175

My view is from a small startup with little to no room for single purpose employees. When I first started hiring and working with data scientist my view was this: If you can only manipulate data and run it through pipelines to generate models then you can't do enough to be highly valuable. You either need to have a strong enough background in CS to build the pipelines / tools or a strong enough mathematics background…

I would be cautious about that. I've worked in the startup space for over 10 years now as a data scientist, often the first one hired on, working on the pipes. From my experience, there are two types of data scientists who work who do infrastructure work: 1) Those who do not make the best data scientist because their skill set is too far in engineering land, leaving them weak where it counts. If the startup is relyin…

Thanks for the feedback! Seems like you and I both have had a bit of experience being first engineering hires at startups but have had very different experiences when it comes to rolls or a data scientist. I appreciate that.

Re: We don't need data scientists, we need data engineers

#186

Post by oivey » Thu, Jan 14, 2021, 7:23 PM UTC

proverbialbunny wrote: post #177

oivey wrote: post #173

Earlier quoted context omitted.

This is dogmatism swung too far in the other direction, IMO. There are many, many successful production code bases written in dynamic languages. In my own experience as a vision scientist/engineer, there is tremendous value in being able to quickly whip up a concept in Python and then being able to easily visualize the results. Doing this exploration in C++ is wasteful. Implementation takes much longer, the correctne…

This was addressed in the previous comment >On the quant side, *unlike on the data science side*, Vision scientist is on the data science side. You're not dealing with monetary values where floating point error compounds on itself to the point your models become garbage. Quant work is it's own unique field with its own unique prerequisites.

Nothing precludes you from doing integer arithmetic in a dynamic language.

I’m not a quant and this isn’t my area of expertise, but, for example, I’m pretty sure various differential equation solving methods depend on variables taking on continuous values, so floating point basically must be used. Understanding the impact of that is definitely very important. Analogously, I frequently run into numerical precision issues in image processing. Understanding how numbers are represented on a computer isn’t unique to being a quant. Understanding how the choice of representation can impact prod is also not unique to being a quant. The dynamicness of the language isn’t particularly relevant, either.

Re: We don't need data scientists, we need data engineers

#187

Post by nerdponx » Thu, Jan 14, 2021, 7:26 PM UTC

alexpetralia wrote: post #89

nerdponx wrote: post #46

Earlier quoted context omitted.

> You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy), technical skills (application programming, advanced databases, advanced machine learning, Excel modeling) and finally some business domain knowledge. This is pretty much the bare minimum requ…

In my experience, software engineers do not make good business analysts (and data/machine learning engineering is a subset of software engineering). Most business analysts cannot program. However, it's likely that our experiences simply diverge here.

I'm talking specifically about data science, not business analyst or software engineer.

Re: We don't need data scientists, we need data engineers

#188

Post by listenallyall » Thu, Jan 14, 2021, 7:28 PM UTC

superhuzza wrote: post #153

listenallyall wrote: post #115

Earlier quoted context omitted.

> novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own…

>Asking them to clean and scrub the source data set...it's not going to happen, they just don't have that skill I think you've been working with conmen/conwomen. I've never seen a data science project that doesn't involve data cleaning or wrangling of some sort.

Have you read through the comment thread? Did you read the article? Most everyone is in agreement that projects require a lot of cleaning & wrangling and a lot more -- the point is that data scientists are generally not doing that stuff, they expect academic-quality, pre-processed, pristine data, so it's data engineers who are stuck preparing the data, and who are in high demand.

Re: We don't need data scientists, we need data engineers

#189

Post by proverbialbunny » Thu, Jan 14, 2021, 7:29 PM UTC

a_zaydak wrote: post #185

proverbialbunny wrote: post #182

Earlier quoted context omitted.

I would be cautious about that. I've worked in the startup space for over 10 years now as a data scientist, often the first one hired on, working on the pipes. From my experience, there are two types of data scientists who work who do infrastructure work: 1) Those who do not make the best data scientist because their skill set is too far in engineering land, leaving them weak where it counts. If the startup is relyin…

Thanks for the feedback! Seems like you and I both have had a bit of experience being first engineering hires at startups but have had very different experiences when it comes to rolls or a data scientist. I appreciate that.

Np. There is a common trend in the industry where a company hires on a data scientist, doesn't know the data prerequisites (specifically labeled data), the data scientist struggles, after a while the company fires the data scientist. This leaves the company with a bad taste in their mouth. In recent years I tend to get hired on as a specialist to help fix this. (And yes, I've been the first engineer hired on too.)

What's interesting is they tend to struggle in two different ways: 1) The data scientist that is gung ho about infrastructure work, jumps in, and then ends up doing a bad job, because it's not their strength. They end up getting let go for not being ideal at that work. 2) The data scientist who struggles with the idea of infrastructure work at all, jumps into other roles they're good at like data analyst work, helps the company in that way, but ultimately because they did not push to get an infrastructure engineer hired, they end up let go as well.

Me, I go out of my way to get an infrastructure engineer / data engineer hired early on. Also, I have worked as an engineer, so I tend to do a lot of the "hard" stuff most software engineers struggle with early on, if applicable. Eg, at one job I wrote a compression format to reduce battery drain on our devices that were collecting data.

Most data scientists struggle when it comes to CS/engineering skills (4/5th of them), so it's not uncommon for them early one while the pipes are being built to do data analyst and BI work. BI work to automate reports, which management loves, and DA work to show some amazing future service the company might be able provide to its customers. It's selling the sun and the moon really, but it gets management inspired, and helps them know what data to collect. It's not unheard of to need a minimum of two years of collected data before building a model that can be deployed becomes feasible. This can be hard on the data scientist, because there is a lot of down time before that. Many get fired during this time even when they're doing a good job. They have to wear multiple hats, but it's analyst roles (like BI work). Technically a data scientist is a kind of analyst, not engineer, so it makes sense that wearing multiple hats for them tilts in the analyst direction, not the engineering direction.

I've been writing code since I was 8 years old, so I'm one of the unusual ones that tilts in the engineering direction, but I think it is unreasonable to expect that from the average data scientist. Let them do what they do best, and hire someone else who can round everything out and you'll be in a good place. Unicorns aside, you'll need a minimum of two professionals for a data project to succeed.

Re: We don't need data scientists, we need data engineers

#190

Post by oivey » Thu, Jan 14, 2021, 7:29 PM UTC

jimbokun wrote: post #171

oivey wrote: post #166

Earlier quoted context omitted.

Yep. The key is really software skills. If you’re unable to even filter the data yourself, you’re also probably unlikely to be able do implement novel analysis techniques, especially if the analysis algorithm has many complicated steps or is computationally expensive. In all fairness, it’s basically impossible for a new grad to have those skills. 4 years of a bachelors in any field isn’t enough to cover such a wide a…

The hope is that the 4 year degree gave you the ability to quickly pick up those skills on your own. If your four year degree didn't give you the ability to learn and expand your knowledge on your own, its a colossal waste of your time and money.

Sure, but depending on what you’re doing, “quick” might be years. You can get a PhD in understanding the theory, a PhD in designing fast numerical algorithms, or spend many years becoming a strong software engineer. I think the willingness to learn a diverse set of things is much more important than learning narrow areas fast. The short length of a bachelors usually isn’t enough to get this diversity.