Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

231–240 of 367 posts

Re: We don't need data scientists, we need data engineers

#231

Post by rhizome »

proverbialbunny wrote: post #226
urthor wrote: post #212

Earlier quoted context omitted.

Not really. You just get seen as the product owner/project manager.

That's a really good point. I tend to be seen as a product lead / owner / stakeholder, so I feel like I'm being called out. lol I think one difference is the software engineers see me as someone who is helping them by making their life easier. I'm not just throwing work at them blindly. I'm working with them. Also, they like it when I include them in the data science brainstorming sessions to solve difficult problems…

I think it's probably seen more as just being a decent boss.

Re: We don't need data scientists, we need data engineers

#232

Post by greenshackle2 »

proverbialbunny wrote: post #163
twic wrote: post #147

Earlier quoted context omitted.

I'm not sure about crappy quants. Some people of the "quantitatively inclined trader who has learned Python" variety are never going to be good at the engineering side - it takes years to learn to be a good software engineer, and that's not a good use of time, for them, or for their employer. But they can still do useful work. The trick is to figure out how to work effectively with those people. Build infrastructure…

If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal. Researchers do not need to have deep programming experience, but they have to be comfortable enough to use an environment that can lend itself itself to the problem at hand. On the quant side, unlike on the data science side, the barrier of entry on the programming side is a bit higher. To solve this problem many fi…

> If they're using a dynamically typed language to do monetary calculations, it's not going to be ideal.

And yet, Q.

Re: We don't need data scientists, we need data engineers

#233

Post by z3ncyberpunk »

[deleted] wrote: post #229
lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

[deleted]

Then you end up with terrible engineers with zero practical experience who make terrible designs due to said lack of practical experience. Engineers out of college today have no zero clue how to run machines or do anything but create drawings that half work which is pathetic

Re: We don't need data scientists, we need data engineers

#234

Post by khaledh »

theflyinghorse wrote: post #204
C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

I wonder how: 1. one finds organizations that have data engineering 2. gets hired to said organization with software engineering background.

Shopify is hiring 2,021 engineers (not just data engineers) in 2021: https://www.shopify.ca/careers/2021

Re: We don't need data scientists, we need data engineers

#235

Post by khaledh »

pricci wrote: post #144
C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

And where would you recommend someone to start a data engineering path. Any book, learning source?

Work at a company that has a good data engineering discipline. Shopify is hiring: https://www.shopify.ca/careers/2021

Re: We don't need data scientists, we need data engineers

#239

Post by LawrenceHecht »

Building or running data infrastructure is an important part of 55% of 372 data engineers’ jobs, according to the “2020 Kaggle Machine Learning & Data Science Survey.” ...Check out three charts I created showing differences between data scientists, data engineers and machine learning engineers: https://thenewstack.io/software-engineers-use-spreadsheets-d...

Re: We don't need data scientists, we need data engineers

#240

Post by edmundsauto »

pricci wrote: post #144
C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

And where would you recommend someone to start a data engineering path. Any book, learning source?

Long time DE here. I recommend trying to build your own data warehouse around something you're interested in. Don't worry about teh scaling - focus on the core engineering, taking data from different places, combining it into a sensible data model, update it automatically every day. Add in more sources.

It's shockingly difficult, and something that only experience can teach.

Post reply on HN