Just like with data science, don't we expect data engineering to get to a point where they codify best practices into a tool, so 1x data engineers can be closer to be as effective as 10x data engineers? The amount of data engineering work won't change, but the number of people needed to do it will reduce, reducing the open headcounts.
We don't need data scientists, we need data engineers
221–230 of 367 posts
Re: We don't need data scientists, we need data engineers
#222Earlier quoted context omitted.
Data scientists do not pull that kind of bonus. Today many of them get paid less than the data engineers do.
news to me, and welcome news to hear at that since I'm more in the data plumbing and packaging business, not algo publications. my personal data points are from folks on buyside. trading margins have been downward trending for years
Quant research work isn't data science work which is probably where the mix up is.
On the quant side bonuses are distributed to the team.
Re: We don't need data scientists, we need data engineers
#223This has been a bit of an annoying thing for me for quite a while. There is a huge difference between a data engineer and a data scientist. A data scientist is not and should not be a data engineer. These are 2 different specializations that work together.
A data engineer is more of what we use to refer to as people who wrote utility processes to process data or do system optimizations. At some point the industry decided to do away with a lot of the things we used to do (desktop apps, distirbuted systems, etc) and moved to REST services only. Then people realized oh wait.. we can't process data on a rest/web app. In typical inexperienced fashion, people tried to cram in there, but it doesn't work. (See Javascript neural networks)
What is data engineering? It's all about moving data around efficiently and processing it in a way to is per formant and reactive. A lot of people tie hadoop/spark to being a data engineer. That's a terrible way of going about it. More of the modern approaches to this is using streaming platforms and reacting to events. (Sadly a lot of the ML stuff is tied to either python/tensorflow or spark)
At times data engineering is pushed towards data maintenance and pushing all of the data in a bucket. This isn't a very valuable use of effort.. but people want to be buzzword compliant.
Note: There are use cases for hadoop and spark.. but those rarer now. (They've better for very large datasets and merging for data that you have a much longer timeframe for the answer).
Re: We don't need data scientists, we need data engineers
#224Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
>The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML.
Reading this thread has made me realize just how lucky I am to work very closely with strong a very strong Data Scientist, who is complemented by a very strong Data Engineer. Conversations with the Data Scientist are always about strategy, product alignment, and ensuring we're optimizing what we build for learning. The Data Engineer works very closely to ensure we're actually capturing the data we think we are, getting it to analysis systems, and making sure those data pipelines stay healthy.
Re: We don't need data scientists, we need data engineers
#225My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…
I worked in investment banking (as an analyst, not an engineer), so very different part of finance, but this was my take as well. Companies might love to talk about how important engineers are, but at the end of the day, if you can't directly link someone to revenue, they get viewed as a cost center and take on second tier status in the organization. Then the same companies complain that they can't find enough (or re…
I worked for 15 years as a software engineer at Morgan Stanley where they valued the process of taking a 3 martini lunch idea into a production platform so value of engineers was recognized and rewarded as such ... its somewhat easier to whip up a new financial wrinkle its a whole other level of magic to design and implement that idea when it takes 60 software developers 3 years to get that idea to market before the rest of the street ... of course the IT department was/is the largest budgeted portion at the entire bank and for a good reason
Re: We don't need data scientists, we need data engineers
#226Earlier quoted context omitted.
It sucks being that guy because everyone else ends up hating you. Depending on the work environment it's not a stretch to see software engineers complaining to management, sometimes going as far to create rumors to get the jr data scientist fired. So, no the grass is not greener. It's best to not be that person. This is why I go out of my way to prevent that scenario when I lead a team.
Not really. You just get seen as the product owner/project manager.
That's a really good point.
I tend to be seen as a product lead / owner / stakeholder, so I feel like I'm being called out. lol
I think one difference is the software engineers see me as someone who is helping them by making their life easier. I'm not just throwing work at them blindly. I'm working with them. Also, they like it when I include them in the data science brainstorming sessions to solve difficult problems. I guess it's seen as exotic or something, but whatever the reason, they really love to be apart of it.
Re: We don't need data scientists, we need data engineers
#227Earlier quoted context omitted.
Nothing precludes you from doing integer arithmetic in a dynamic language. I’m not a quant and this isn’t my area of expertise, but, for example, I’m pretty sure various differential equation solving methods depend on variables taking on continuous values, so floating point basically must be used. Understanding the impact of that is definitely very important. Analogously, I frequently run into numerical precision iss…
>Nothing precludes you from doing integer arithmetic in a dynamic language. You would be surprised. The second you use pandas with a custom data type (let alone any other library you'd want to use) it can randomly auto convert it to a float. Furthermore identifying when it randomly converts the type on you is a pain. >so floating point basically must be used. Quants tend to use fixed precision types. It is like a flo…
Quants don't care about floating point precision in research. It's just applied stats
Re: We don't need data scientists, we need data engineers
#228My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…
I worked in investment banking (as an analyst, not an engineer), so very different part of finance, but this was my take as well. Companies might love to talk about how important engineers are, but at the end of the day, if you can't directly link someone to revenue, they get viewed as a cost center and take on second tier status in the organization. Then the same companies complain that they can't find enough (or re…
Engineers get paid well in SV because they are in demand, have lots of employment opportunities, and therefore are more difficult to retain.
Re: We don't need data scientists, we need data engineers
#229My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…
[deleted]
Re: We don't need data scientists, we need data engineers
#230What does CV stand for in this context?
Computer vision (i.e. deep learning on images / video)