We don't need data scientists, we need data engineers • Hacker News Board

Re: We don't need data scientists, we need data engineers

#141

Post by hntrader » Thu, Jan 14, 2021, 5:44 PM UTC

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

To add, quants that can't do the data engineering work are always crappy quants. I haven't seen a counter-example to that. Profitable models aren't going to be delivered on a silver platter. They need to be able to process pretty low level data effectively and build ad-hoc custom tools and data pipelines around that to test out their ideas. Otherwise they're constrained to the tools others have built and that massively narrows the search space that they're capable of traversing.

The best quants are 1/3 statistician, 1/3 developer and 1/3 trader, in my view.

Re: We don't need data scientists, we need data engineers

#142

Post by Mauricebranagh » Thu, Jan 14, 2021, 5:47 PM UTC

sidlls wrote: post #66

superbcarrot wrote: post #34

Earlier quoted context omitted.

I think that this is more of a problem with the specific people that you have worked with and it isn't inherent to the role of a data scientist.

It’s becoming more inherent, especially as the field is populated with people who have no experience with the “science” part. That is, with the very real and ubiquitous problem of collecting and cleaning data to make it fit for scientific study. Even theoretical physicists, for example, participate in and rely on empirical data collection, and understand deeply how messy and fraught with error it is. I don’t see the…

I remember working with some one who has PhD in Physics and who worked at CERN - and one comment I loved "a key skill is knowing how to place the legend, so it obscures that annoying outlier data point"

Re: We don't need data scientists, we need data engineers

#143

Post by lamename » Thu, Jan 14, 2021, 5:50 PM UTC

tristor wrote: post #56

Agreed. In my decently long career the types of data problems I've seen be most impactful on the business are not head-in-the-clouds ML issues, but more mundane yet more far-reaching: 1. Appropriately identifying what data needs to be captured from a product to correctly operationalize it. 2. Understanding and modeling data structures in internal applications to identify and tune backend data storage mechanisms (incl…

This sounds reasonable. The trick is to identify companies/cultures of each type at interview time. But is that even possible?

Re: We don't need data scientists, we need data engineers

#144

Post by pricci » Thu, Jan 14, 2021, 5:54 PM UTC

C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

And where would you recommend someone to start a data engineering path. Any book, learning source?

Re: We don't need data scientists, we need data engineers

#145

Post by 6d6b73 » Thu, Jan 14, 2021, 5:54 PM UTC

I would go even farther. We don't need data, we need knowledge.

Re: We don't need data scientists, we need data engineers

#146

Post by bart_spoon » Thu, Jan 14, 2021, 6:09 PM UTC

Ansil849 wrote: post #95

Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.

It tends to happen any time something becomes trendy. Data science has/had a lot of hype over the last decade, and people seem to have an inherent tend to want trendy things to fail. Combined with that, you have a bunch of people hopping on the data science bandwagon, so you get a lot of grifters, snake oil salesmen, or simply individuals whose output is poor quality. Seems to have created a feedback loop, where there is always a new example of some AI solution failing, or a data science initiative that didn't work out that everyone can point at and say "See! I always knew this trend was dumb!".

Reality is that data science is here to stay. It's coming out of the honeymoon period, and things may never be as hyped up as it has been the last decade, but that's probably a good thing for the field. Everyone will probably move on to hating the next up and coming thing. I have a hunch it could be something in data engineering because, while not exactly new, it is absolutely the next "data science" in terms of demand, and with products like Snowflake having so much hype behind them, it seems the backlash will be inevitable.

Re: We don't need data scientists, we need data engineers

#147

Post by twic » Thu, Jan 14, 2021, 6:12 PM UTC

hntrader wrote: post #141

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

To add, quants that can't do the data engineering work are always crappy quants. I haven't seen a counter-example to that. Profitable models aren't going to be delivered on a silver platter. They need to be able to process pretty low level data effectively and build ad-hoc custom tools and data pipelines around that to test out their ideas. Otherwise they're constrained to the tools others have built and that massive…

I'm not sure about crappy quants. Some people of the "quantitatively inclined trader who has learned Python" variety are never going to be good at the engineering side - it takes years to learn to be a good software engineer, and that's not a good use of time, for them, or for their employer. But they can still do useful work.

The trick is to figure out how to work effectively with those people. Build infrastructure that keeps them on the rails, refactor their code, push them in the right direction, tell them when they've fucked up, teach them little things with high leverage. As long as that doesn't turn into being their slave, that's fine.

Re: We don't need data scientists, we need data engineers

#148

Post by nitrogen » Thu, Jan 14, 2021, 6:13 PM UTC

aqme28 wrote: post #87

AndrewKemendo wrote: post #3

Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…

> you have a bunch of Data Scientists just handed a pg_dump or flat file from some ops team. I feel seen. At a previous job, our output after some cleaning and transforming was a pg_dump for the data scientists to load. We had little visibility of what they did to that database once they got it.

I suspect in rare cases this is by design, because engineers would object to the behavior of the Business Intelligence department on ethical grounds.

Re: We don't need data scientists, we need data engineers

#149

Post by tchalla » Thu, Jan 14, 2021, 6:13 PM UTC

listenallyall wrote: post #115

alexfromapex wrote: post #30

Earlier quoted context omitted.

I agree, if anything the data engineers (folks with engineering backgrounds) should be doing the applied work while a department of data scientists works on the theoretical or novel data analysis methods. Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.

> novel data analysis methods Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc. Asking them to develop a "novel" method would be fruitless. Asking them to clean and scrub the source data set is like telling an amateur pie-baker the store was out of pie crusts, you'll have to make your own…

> Many "data scientists" (not all, but many) have little to no ability to do anything other than apply "recipes" of algorithms or classification methods or logistic regressions, etc.

This is because they rarely hire people with scientific thinking ability. They just hire people who can code and program from set recipes. Once you hire such people you can not expect them to do non-recipe work. If you don't want recipe work, don't hire people will recipe skills. Do not have job interviews that select for recipe people. But, that is exactly what most companies do.

Re: We don't need data scientists, we need data engineers

#150

Post by eVoLInTHRo » Thu, Jan 14, 2021, 6:13 PM UTC

Agree with the general idea. Data engineers are like the essential workers of the data world, people who today may not receive the appropriate level of appreciation that they deserve.

I think the glut of data scientist occurs because we clump so many different skills and disciplines under the single term "data scientist". Data scientists today come from so many different backgrounds that the definition means something different to everyone. Because of this, the surface area of possible skills that could be expected of a data scientist is vast, to the point where it's pretty unlikely to be sufficiently competent in all of them, let alone a majority.

I'd like to go back to a world where we had a little more specificity about what kind of data scientist you are (e.g. I had no problem with terms like statistician and data miner), which could help ground expectations that others have of us, and it'd also help clearly define the scope of various career paths for the next generation.

Sadly the individual who coined the term shows no contrition for the degree of confusion that the rest of us have been left to deal with: https://observer.com/2019/11/data-scientist-inventor-dj-pati...