Earlier quoted context omitted.
I really like your magician/blacksmith analogy. I'm in industrial automation, but it's much the same. Projects where someone developed a strategy but has never been involved in the details of a machine are doomed to failure (or at best to be unreliable and producing low quality parts). Projects built by machine fabricators are over-engineered, frequently late, and sometimes unprofitable, but damn if they don't work w…
The funny thing is, even when it's unintentional, people seem to attribute credit to the magician rather than the blacksmith. At my workplace, I even have situations where I explicitly tell people: "I am familiar with X and how (some) of it works, but not all of it. I did not create it nor was it even my idea; all credit goes to Jim. If you need anything to do with X, you're best off asking Jim. But if Jim's too busy…
We don't need data scientists, we need data engineers
351–360 of 367 posts
Re: We don't need data scientists, we need data engineers
#3524 years ago I moved from a role where I primarily wrote C# as an architect on a web application, to an architect helping to build a data warehouse. The contrast in tooling, discipline and information available to build anything in the data world is so stark it had me questioning my career decisions. Sure, you can read Kimball and Inmon and I'm sure there are a handful of others out there - but there are drastically f…
Why do you think that coded ETL is winning over the click-and-drag variant? I'd say the latter makes things a lot easier no?
Re: We don't need data scientists, we need data engineers
#353Earlier quoted context omitted.
> Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a way that's detrimental to the rest of the team. My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately wort…
"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.
Idea people that just blurb out thoughts that go no where are all over the place but there are a few that convince others that their idea is useful. They don't necessarily need to do the work to make their ideas a reality but they need to convince others that their ideas have value.
We need both dreamers and workers.
If you are an idea person figure out how to get others to believe in what you are dreaming and the idea will become a reality.
Think Steve Jobs, he was the idea guy that made his dreams a reality. People want to believe that he was some kind of super engineer or programmer but he was the one that was able to get all the super engineers to do their best to develop his ideas.
Re: We don't need data scientists, we need data engineers
#354Earlier quoted context omitted.
We have a sister company with many data scientists, and very few (actually I don't think that they ever hired any with the specific title) data engineers. And, their production alleged "machine learning" (it's pretty much standard linear regression, but calling it ML is sexy) systems are slow motion train-wrecks. If the string and duct tape holds, then it works, but it's unfortunately continually breaking. Hell, in S…
> Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production. Could you go into more details on what their struggles are? We had many problems as a company doing machine learning projects, and we built our internal platform ( https://iko.ai ) to keep our sanity. I'm always interested in problems others may be having.
Re: We don't need data scientists, we need data engineers
#3554 years ago I moved from a role where I primarily wrote C# as an architect on a web application, to an architect helping to build a data warehouse. The contrast in tooling, discipline and information available to build anything in the data world is so stark it had me questioning my career decisions. Sure, you can read Kimball and Inmon and I'm sure there are a handful of others out there - but there are drastically f…
Why do you think that coded ETL is winning over the click-and-drag variant? I'd say the latter makes things a lot easier no?
Coming over to the data world, I noticed the same type of problems click-and-drag app development had appearing in tools like IBM's DataStage and Informatica's Powercenter. There's only so much you can do by dragging and dropping items on a screen, eventually you need to take their respective escape hatches and do some programming - and when you do it's almost never ideal. I've also yet to see a visual coding tool produce readable concise diffs in any source control provider. Most of these tools also require some sort of centralized server infrastructure and a thick client making it so much more challenging to bootstrap new ETL developers.
I do hear others in the data world who have migrated to Spark or DBT share the same sentiments - but that could just be confirmation bias.
Re: We don't need data scientists, we need data engineers
#356Earlier quoted context omitted.
"I'm the idea guy" out of someone's mouth is the stark red-flag warning that their net contribution is 0.
It's very prevalent in amateur gamedev communities. Every day there's someone who played some game, and has some ideas how to make it better. All he needs is a few programmers and artists to make his vision in reality. Usually the kind of projects he wants to make are big AAA games in whatever is the trending genre at the moment (used to be MMORPG, now it's about battle royale). When confronted they often get defensi…
What really bothers me, though, is not the randos with this attitude. It's that some of them will grab enough money or power that they'll be able to live out their fantasy. And woe be unto any who hop on board. Quibi being the latest big example.
Re: We don't need data scientists, we need data engineers
#357Earlier quoted context omitted.
> Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production. Could you go into more details on what their struggles are? We had many problems as a company doing machine learning projects, and we built our internal platform ( https://iko.ai ) to keep our sanity. I'm always interested in problems others may be having.
Python hell, basically. Getting the right Python version, right dependency versions etc.
Our belief was that there are some odd behaviors in every tool and we had to figure out a way around.
Re: We don't need data scientists, we need data engineers
#358My experience is in quant hedge funds, where sometimes you get some guys who develop the strategy and some guys who put it into production. Yes, I do admit there can be some specialization in terms of time spent on science vs engineering. But you really need people who understand both. Particularly if you have a strategist who thinks his job is just to dream up profitable models, he ends up carving that role out in a…
The (repetitive) blacksmith role is not an interesting one, digital revolution needs to come into place. Architects that build tools, self service systems are much more interesting.
Re: We don't need data scientists, we need data engineers
#359Earlier quoted context omitted.
> Ideas are so cheap and easy. I doubt this.
What's the market price for an idea? If it's greater than zero, let me know. I have notebooks full of them. Business ideas. Project ideas. Political ideas. Social ideas. I generally can't give 'em away, much less sell them. Why? Everybody has their own ideas, and they like 'em better. And the ones in my notebooks don't have what really matters: validation.
Re: We don't need data scientists, we need data engineers
#360Earlier quoted context omitted.
So I think that the delineation between the scientist working with the content, and the Engineers who actually provide the mechanics for it is very fair. If there is a question mark here - it's really how much value are we deriving from all of these data people? Where is all the ML that's changing our lives? Search, Alexa and TikTok, I can see it. In the future obviously vision systems for autonomous cars etc.. But I…
I used to work at a legacy automaker and you’d be shocked at how much ML has changed certain areas of the business. It used to take an entire department to sort warranty claims and it’s now mostly automated. Aluminum part defects are now spotted automatically on the plant floor. Don’t even get me started with telematics data. Most software isn’t consumer facing but just because you don’t see it doesn’t mean it’s not…
However, finding defects in aluminum parts that involves using computer vision, would absolutely be a ML solution.