Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We don't need data scientists, we need data engineers

mihaileric.com

Post reply on HN
Print view View original

261–270 of 367 posts

Re: We don't need data scientists, we need data engineers

#261

Post by jsinai »

tchalla wrote: post #155
astrophysician wrote: post #68

Earlier quoted context omitted.

Just want to say that while the data science profession definitely includes a wide range of people and skillsets, a good data scientist should be practical and able to work with the available data in whatever state it's in. No good data scientist should ever expect data to be pristine. And a good data scientist, even if they don't have quite the engineering chops necessary to build a production-quality ETL, should kn…

A good data scientist should also be good at science. Otherwise, you can simply hire people with engineering skills - you don't need scientists. If you hire scientists and then are surprised they aren't good at engineering, the hiring process needs a reality check.

Statistics is a science as well. Unfortunately it’s overloaded in business terms and can mean anything from “knows means and regressions” to “has a copy of _Meyn and Tweedie_ on their shelf”.

Re: We don't need data scientists, we need data engineers

#262

Post by wickedsickeune »

nitrogen wrote: post #162
fock wrote: post #127

Earlier quoted context omitted.

I do an introductory Python lab course at my university. It's targeted at engineers who still create graphs from Excel and then normally level up to MATLAB, if things get complicated (think insets, ...). I guess about 30% of the people previously did at least some of the YT/Udemy "courses" on datascience. It's really horrifying for me (not being an engineer myself, but imo having a relatively engineering-like mindset…

why do I need to do this IMO this is the number one problem of our modern culture around education. Popular culture makes it popular to treat education as pointless, and this even affects students who are pursuing difficult degrees. "Why do I need to study humanities? Why should I learn to code if I think I am born to be someone else's boss?" On the other hand, many teachers in K12 and early university have no abilit…

If the educator cannot explain why the knowledge is useful, then he is unfit to teach it.

Re: We don't need data scientists, we need data engineers

#263

Post by 1vuio0pswjnm7 »

lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic.

Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to claim the blacksmith's value as her own.

If the blacksmith is good enough, she will earn the trust of management and management may consult the blacksmith in the selection of magicians. Management may ask the blacksmith to interview magicians and seek her advice on the final hiring decision.

The blacksmith may not carry the "prestige" of the hustling, bullshitting magician but she can command a high salary and dictate her own working conditions. This is only if management understands her value. What the magician thinks of the blacksmith is irrelevant.

Reliable blacksmiths are hard to find. Magicians are a dime-a-dozen.

Re: We don't need data scientists, we need data engineers

#264

Post by basseq »

nerdponx wrote: post #55
basseq wrote: post #15

The problem that I've seen is often "data scientists" are expected to be the equivalent of full-stack engineers (or maybe more accurately: one-man CTO shops)—to understand data architecture, understand business architecture, ensure data quality, build data into product, build dashboards, derive insights, posit hypotheses, set strategy, and drive business value. Thus many "data scientists" are juiced-up report-builder…

This is uncharitable. In my experience, this is true: > "data scientists" are expected to be the equivalent of full-stack engineers (or maybe more accurately: one-man CTO shops)—to understand data architecture, understand business architecture, ensure data quality, build data into product, build dashboards, derive insights, posit hypotheses, set strategy, and drive business value. But this is not: > Thus many "data s…

Now it's my turn to claim uncharitibility.

Indeed, my context here is that people who wear the data scientist title come from multiple backgrounds, and are often asked to wear too many hats. They are non morons—they may be darn good report-builders, but haven't been trained in insights, for instance.

If you're reacting to my word choice in that last sentence, know that I am frustrated with people who claim to be data scientists but can't derive insight. (And we can argue about "many".) But that's not a broad denouncement against all data scientists, either.

Re: We don't need data scientists, we need data engineers

#265

Post by b0rsuk »

Oh, but maybe we don't need either?

It seems that data science is used primarily for advertising. Local internet communities are dead. Message boards are dying. Everything is either a reddit / discord / Steam forums / Boardgamegeek. In 2021, GOG forums pass for a small forum. Only the biggest can float in the ocean of spam.

Re: We don't need data scientists, we need data engineers

#266

Post by yters »

While we like to pooh pooh at the theoreticians, there are some remarkable results proven just through thought experiments and math, which are only confirmed and used many, many years later. People would not even think to look into such things if theoreticians did not come up with the original abstract proof.

For example, there are more combinations of characters for text messages of this length than there are particles in the entire history of all possible multiverses. Clearly I need to use something beside blind trial and error to write a halfway coherent HN comment. Instead I must rely on a little mental theorizing to even just rant on the internet.

Re: We don't need data scientists, we need data engineers

#267

Post by legerdemain »

1vuio0pswjnm7 wrote: post #263
lordnacho wrote: post #112

My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…

What must be communicated to management: It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic. Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to…

  > It is easy to find other magicians. It is not easy to find another blacksmith. Without the right blacksmith, there can be no magic.
What? That runs counter to my experience at every company where I've either seen data engineers or worked as one. My observations of how management treats the two groups is this:

Data engineers ("blacksmiths"): Blacksmiths are paid less. People think of them as less highly educated. Their work is less creative. When they are successful, their work is mostly invisible. They are interchangeable. People think of what blacksmiths do as more like scripting than writing code. Blacksmiths mostly work on configuring systems they didn't build. Blacksmiths do more troubleshooting than building. Their roles are focused on support.

Data scientists ("magicians"): Magicians are paid more. Much more. People think of them as more highly educated. By definition, what they do is magic. They work on prominent projects. Their successes are highly visible. They build large systems that only they can comprehend. They use support staff to clear away mundane obstacles so they can focus on unique, highly creative aspects of work.

Saying that we need more data engineers than data scientists is like saying that we need more janitors than CEOs. That's true, but it's true because we made it true by structuring projects around one prominent, well-paid person supported by a staff of invisible drudges.

This smacks of the positive self-talk that QA and software testers used to give each other: "We are indispensable! We take pride in our craft! Nothing can ship without our signoff!" And then lots of companies reduced their QA or eliminated it wholesale by focusing on continuous delivery and changing consumer expectations of what "broken" or "acceptable" means. The same fate awaits data engineers.

Re: We don't need data scientists, we need data engineers

#268

Post by Avalaxy »

pricci wrote: post #144
C4stor wrote: post #18

I can't recommend the Data Engineer career enough for junior developers. It's how I started and what I pursued for 6 years (and I would love doing it again), and I feel like it gave me such an incredible foundation for future roles : - Actually big data (so, not something you could grep...) will trigger your code in every possible way. You quickly learn that with trillions of input, the probabily to reach a bug is ei…

And where would you recommend someone to start a data engineering path. Any book, learning source?

The book "designing data-intensive applications" is really really good, and covers all the concepts (although not per sé the tools) you need to understand.

Re: We don't need data scientists, we need data engineers

#270

Post by the-pigeon »

jnwatson wrote: post #206
wpietri wrote: post #201

Earlier quoted context omitted.

> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately wort…

"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.

Ideas are so cheap and easy.

Implementation is a long hard road. And where you learn your idea was vague enough that it had almost no value. And only through painstaking iteration can you turn it into something with value.

Post reply on HN