We don't need data engineers, we need great tools and storytellers.
We don't need data scientists, we need data engineers
311–320 of 367 posts
Re: We don't need data scientists, we need data engineers
#312this post is so timely. How do you guys handle hiring? I’m looking to hire more data engineers (Singapore only for now), especially senior ones and it’s not easy. My gut is to just find good engineers who have good first principles and let them learn on the job. Feedback/experiences welcome!
Curious to hear this from the other side. I'm an SE manager interested in pivoting to Data Engineering. I've built modest pipelines in R / Postgres for an operations analyst job I did before I became a programmer, but I'm not experienced with most of the technologies I see listed on DE roles (e.g. Airflow) and my statistics etc. are rusty.
I see these experiences as nice-to haves. It just means you'll be able to hit the ground faster. Personally, I'd value someone who shows they can learn quickly.
That said, it's trivial these days to spin up a few VM's, install Airflow and try some practice scenarios just to get your hands dirty.
Re: We don't need data scientists, we need data engineers
#313Earlier quoted context omitted.
> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately wort…
"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.
Yes I agree, idea guys are bad, but I’d like to make a contrast between two words that are often thought of as synonymous. There’s the ‘idea’ guy and a man/woman with ‘vision’. The difference between these two is that someone with a vision enables others to see and feel what they see and feel about the future of a project in a manner that produces action on everybody’s part. The visionary man/woman would act in accordance with their vision, because there end goal is to create and/or share something truly valuable to others. I could think of a couple of people with visions many of them startup founders: Elon Musk, Steve Jobs, Peter Thiel, Paul Graham, and Sam Altman. Visionary man/woman take risks on their vision and put their plans into action. Sometimes they’re the business manager, or the scientist, and at other times the engineer, but in some way shape or form they lead others toward a goal through their tremendous clarity in their own and other’s collective vision.
Re: We don't need data scientists, we need data engineers
#314Preach! The data lifecycle is waaay overpopulated with Data Scientists who are not empowered or knowledgeable enough to work with product designers and engineers to do everything that empowers Data Science and ML. We need more Data Engineers involved at time zero in projects to help: 1. Plan out what data should be produced/captured by the product 2. Instrument systems to actually generate data consistently and effec…
As a Data Scientist, I do everything you mentioned because I came from a SWE background. I think Data Scientists, even if they are only interesting in the "fun" part of the job, should know how and why data are captured their way, so that they understand which models are better suited, which saves a lot of time.
Re: We don't need data scientists, we need data engineers
#315My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…
this perception of classes within engineering is the greatest frustration of my career. People with a PhD or “scientist” in their title are not more valuable than engineers who end up being the ones to get things to work.
Re: We don't need data scientists, we need data engineers
#316Earlier quoted context omitted.
Thank you for your comments! They are very insightful. To piggyback a bit: Assuming you are a competent data "analyst" who wants to become a data engineer, how would you go about it? Is "go back to school and get a CS degree" the answer? I suppose this question is very broad, but I am curious if a practitioner like you has an opinion. --- To give some context: I recently graduated with a STEM PhD, and looking to move…
I'm confused, do you want to become a data engineer or a data scientist? A data scientist is a type of senior data analyst. Data engineering is farther away to a data scientist than a data analyst is. I'm going to answer both questions just in case: To become a data engineer / infrastructure engineer, there are multiple paths forward. I recommend doing BI work aka Business Intelligence Analyst Engineer. It's typicall…
Thank you for the detailed response. It makes things a lot clearer.
I notice that you mention "Machine Learning Engineer" as a separate role. If in the idealized world, data scientists do analytics and train models, and data engineers take care of data, then what do Machine Leaning Engineers do? Are they, basically, software engineers who specialize in putting other peoples' models into production?
And you are right in sensing my confusion. There seems to be an abundance or data-related titles, which seem to overlap in their functions a lot, but are also very different when you examine them closely. So thank you again for your responses, they are very helpful.
Re: We don't need data scientists, we need data engineers
#317Earlier quoted context omitted.
> Ideas are so cheap and easy. I doubt this.
"Pure" ideas are cheap and easy; good ideas require a very thorough knowledge of implementation which is usually achieved through experience
Right, and to add upon this - that's because all ideas originate from the senses.
To demonstrate: try to imagine a new color you've never seen before that's not in any way associated to any of the colors that you've seen. (it's impossible)
Or think of it this way: You could explain a car to someone who has never seen a car before, but only so long as the ideas used to explain the idea of a car (e.g. wheels, doors, windows), have already been familiarized to the other person. Otherwise if those ideas weren't familiar, such as a wheel, you'd need to also explain what a wheel is. And if the concepts used to explain the wheel wasn't familiar (and so on), you'd eventually hit a point where you must expose the idea(s) directly to their senses (eg show them), otherwise they will never understand what you're talking about.
So all ideas come from the senses, and your minds ability to combine these "pure" ideas that you've sensed. "Pure" ideas are cheap and easy because they're the simplest ideas - they're what you directly sensed. To have good ideas, you need to combine many "pure" ideas together, hence why those who have experience working closely and thoroughly with something, will often have the best ideas associated to that something...
Re: We don't need data scientists, we need data engineers
#318Earlier quoted context omitted.
Ideas are so cheap and easy. Implementation is a long hard road. And where you learn your idea was vague enough that it had almost no value. And only through painstaking iteration can you turn it into something with value.
> Ideas are so cheap and easy. I doubt this.
What's the market price for an idea?
If it's greater than zero, let me know. I have notebooks full of them. Business ideas. Project ideas. Political ideas. Social ideas. I generally can't give 'em away, much less sell them. Why? Everybody has their own ideas, and they like 'em better. And the ones in my notebooks don't have what really matters: validation.
Re: We don't need data scientists, we need data engineers
#319Earlier quoted context omitted.
Good Data Engineering is what enables good Data Science. With a good infrastructure you can go 100 times faster. Getting rid of Data Engineers means killing Data Science.
Sure. So in a few years, an all-in-one 80% solution like Palantir Foundry will come along and there will suddenly be a lot less demand for data engineers. Anecdotally, the former head of QA for Palantir UK is now the head of data engineering for Palantir UK, and Palantir does have an out-of-the-box, end-to-end, it-just-works product that handles 80% of ML workflows. You're betting your career that they won't put it i…
If something like that becomes mainstream, data scientists who just glue together canned ML algorithms from libraries should also be concerned. The kind of automation that enables eliding finding and cleaning the right data probably can also manage running PCA and basic ML.
Re: We don't need data scientists, we need data engineers
#320Oh, but maybe we don't need either? It seems that data science is used primarily for advertising. Local internet communities are dead. Message boards are dying. Everything is either a reddit / discord / Steam forums / Boardgamegeek. In 2021, GOG forums pass for a small forum. Only the biggest can float in the ocean of spam.
Advertising? I would say the biggest sectors for data science are healthcare and finance.