this post is so timely. How do you guys handle hiring? I’m looking to hire more data engineers (Singapore only for now), especially senior ones and it’s not easy.
My gut is to just find good engineers who have good first principles and let them learn on the job. Feedback/experiences welcome!
We don't need data scientists, we need data engineers
91–100 of 367 posts
Re: We don't need data scientists, we need data engineers
#92+1. I generally say that data scientists bring an amazing skillet to the table, but companies can only leverage 10% of it.
Re: We don't need data scientists, we need data engineers
#93Good data is hard. Anyone conducting research in the physical sciences knows this firsthand. It takes painstaking effort to conduct carefully controlled experiments and collect a batch of good data that could then be used for analysis. The promise of ML has always been to churn out good results from not-so-good data. If I now need to sanitize my data carefully, what's the advantage?
What makes you think that was the promise? I would say the promise has been to replace code with data, and to build things with data that would be practically impossible to build with code.
"Garbage in, garbage out" has been the mantra for a long time. Sure, there are tools and techniques to deal with not-so-good data, but those are add-ons, not a core part of the value proposition of ML.
Re: We don't need data scientists, we need data engineers
#94I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…
The other thing I'd emphasize here is dealing with "state". Data is effectively state.
As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Someone has to manage the queues, file versions/locations, logs, databases, configurations and so on. That is all "data".
State management is a tricky problem even in a single-threaded application. It's doubly so in distributed systems, where state can be inconsistent between all the moving pieces. This is the source of endless data integrity issues. I think data engineering is a great way to get some exposure to all of this.
Re: We don't need data scientists, we need data engineers
#95Genuine question: why is there so much pure teeming hatred for data scientists in this comment thread? Almost every comment comes off as full of snark and vitriol against data scientists.
Re: We don't need data scientists, we need data engineers
#96this post is so timely. How do you guys handle hiring? I’m looking to hire more data engineers (Singapore only for now), especially senior ones and it’s not easy. My gut is to just find good engineers who have good first principles and let them learn on the job. Feedback/experiences welcome!
Curious to hear this from the other side. I'm an SE manager interested in pivoting to Data Engineering. I've built modest pipelines in R / Postgres for an operations analyst job I did before I became a programmer, but I'm not experienced with most of the technologies I see listed on DE roles (e.g. Airflow) and my statistics etc. are rusty.
Re: We don't need data scientists, we need data engineers
#97Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
This same sentiment (which I personally agree with) applies to software engineering. As in: engineers deliver more practical value than comp scientists. Now you can down-vote me to oblivion.
I think it would apply if companies were hiring large number of computer scientists and using them to try to build usable software. I don't see many making that mistake. Most recognize that computer scientists belong in a research or academic setting.
Re: We don't need data scientists, we need data engineers
#98What's the career trajectory for data engineers? I enjoy the pipeline building and business stakeholder interfacing, but I'm not sure I want to be a SWE a decade from now...
How is that different from a SWE? I see DE as a specialized SWE; tons of overlap, but DEs focus on different tools and concepts than other SWEs.
Re: We don't need data scientists, we need data engineers
#99I don't find it too surprising. I think more people would want to work with the conceptual work in the data science space, like models. The ETL stuff seems extremely boring and it seems it pays less too.
Re: We don't need data scientists, we need data engineers
#100Disclosure: I'm a just plain scientist, without the data. ;-)
Since "data science" appeared on my radar (in other words, on HN), I've noticed that we fling the "scientist" and "engineer" terms around without asking whether the practitioners have science or engineering backgrounds, or something else such statistics, math, programming, etc.
It strikes me that "do we need scientists or engineers" is not unique to data science/engineering. I think we need both, but of course it's an open question as to how many of each are needed. Also, both "scientist" and "engineer" are loosely defined in practice, with some overlap.
Painting with a broad brush, a scientist wants to learn how things work. An engineer wants to make things that work. If you're in the business of making things that work in order to sell them, then you need lots of engineers, but maybe a few scientists. It's about 10:1 at my workplace.
Overlap? Of course. Engineers use scientific knowledge and methodology. Scientists have to make things work in order to make experiments and theoretical computations work. Also, scientists often show up in emerging fields before they become recognizable engineering disciplines, for many reasons: 1) We need the newest stuff, right away. 2) We are opportunists by nature and necessity. For instance many of the older "software engineers" at my workplace have science degrees, but the younger ones all have computer science degrees. The oldest people I know who did programming, digital logic, and embedded systems, did not have degrees in those areas. The youngest ones all do. At my college, the physics professors had personal computers, and were ripping them apart, while the CS professors used the mainframe. That's part of why I chose to major in physics, though I was interested in programming.
Another reason for overlap is that both "scientist" and "engineer" titles include things that are not strictly within either area. A lot of people with science degrees end up working as technicians, marketeers, salesmen, managers, etc. A lot of people with engineering degrees do little or no quantitative engineering, but work as programmers, designers, marketeers, salesmen, managers, etc.
Something to ponder is if there are differences between how scientists and engineers think and approach problems. Naturally there's a lot of folklore about that, probably little hard evidence. That's where I think you should start if you want to know whether to hire scientists, engineers, or both.