On the flip side you used to have statisticians writing code that is frankly unusable in a Production environment. You would weep at the R code I've seen and had to turn into something to actually produce business value.
Yeah but in the end it’s just code. And even better, just R. The business value comes from the stats guy.
When the R/stats guy quits and you have to figure out which of his 7 notebooks to run in which order and which local files need to be in which local directories to run correctly and which versions of each package are now broken and which code you need to rewrite to fix it you start to realize the value he produced was clicking a lot of buttons in the right order and that overall this doesn't scale at all.
I think the qualifying term here is "good". I've worked with a surprising number of MLEs that don't really understand gradient descent or how most models really work under the hood. They certainly couldn't implement most things from scratch if they needed to (neither could most data scientists). I used to think an MLE was a solid engineer who also had a strong quantitative and numerical computing background. The kind…
>They certainly couldn't implement most things from scratch if they needed to (neither could most data scientists). Could most CS folks actually implement Linux or Chromium from scratch?
I'm clearly talking about quantitative modeling tools.
That said, while Linux and Chromium are massive projects each with years of development with thousands of engineers behind them, so of course it would be ridiculous to expect a single engineer to build such a thing. I also wouldn't expect an MLE to build SKLearn entirely as is from scratch on their own.
However, I do certainly hope most CS folks could implement an OS or Web browser from scratch.
Where are these jobs where you can interview this badly and still get hired because in my experience DS interviews are extremely hard and often expect people to have very high Stats skills as well as Data Structures/Algo skills at FAANG level.
The ML interviews at FAANG are absurdly simple. Design YouTube recommendations for which canned answers are readily available. A simple stats question. If I double the number of samples, how much will the confidence interval change? Most FAANG ML engineers can't answer this question.
In a recent past life, I was a HPC (high performance computing) administrator for a mid size company (just barely S&P400) who was in the transportation industry, so I had a lot of interactions with the "data science" team and it was just a fascinating delusion to watch. Our CTO did the "Quick, this is the future! I'll be fired if I don't hop on this trend" panic thing and picked up a handful of recent grads and gave…
Honestly, I can't tell you how many jobs ads I saw where I was wondering: "What would they expect me to bring to the table here?" Some companies just don't have the data, or heck even the need, for data scientist yet try and hire them anyway. Give smart people a fundamentally ill-posed problem and they won't get anywhere anyway.
It’s a great skill to walk in to a job and say “hey I’m the expert, that’s not a reasonable proposal, here’s the problem we can solve and here’s what we’ll do”. Much more value to the company, but hard to do.
Great post. A few random comments... > The median data scientist is horrible at coding and engineering in general. The few who are remotely decent at coding are often not good at engineering in the sense that they tend to over-engineer solutions, have a sense of self-grandeur, and want to waste time building their own platform stuff (folks, do not do this). > It was obvious that there is a general industry-wide need…
> 23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college.
Ageism is disgusting and I cannot believe such blatant discriminatory language is seen as OK for a link posted to hackernews. How would you all say if he wrote that 40+ year old programmers should xx?
> One of the things I don't like about statements like this said in a Data Science context, is that they are true outside of Data Science as well. Executives make big decisions, managers make smaller decisions, nobody can evaluate how good/bad they really were for months or years. Engineers build something amazing, or build a house of cards, nobody cares as long as the money people are happy, even if the business use…
IMO the reason behind this is that a lot of "data science" driven decisions are short term decisions. So you can look at something on a PowerPoint, not really care if it's wrong unless you personally will get fired if it turns out to be wrong, and back out of it a quarter later when it turns out to be wrong. IME there's no shortage of justifications or pivoting when it comes to a decision you made a quarter ago. The…
About 10 years ago, I worked at a company that really wanted to use Hadoop for some reason, so I was forced to use it for a project. The amount of data we were processing was minuscule (a few hundred megabytes per run) It could've been done with a simple script on a single EC2 instance for the entire duration of the project without any scalability issues. Instead, I had to provision Hadoop clusters (dev, staging, production), fit the script into the map-reduce paradigm, write another script to kick off the job and process the results, etc. At least we were using Hadoop.
This hit all the same high notes I was feeling when I quit Data Science to become a software engineer. It's an infinitely better gig and I encourage all my colleagues with enough chops to make the same switch.
How did you do the transition from DS to SWE?
Another comment here mentioned the on the job steps you can take, and that mirrors my experience. I also enrolled in Georgia Tech's OMSCS after a year or so of self study. About 1 year in I took a role using Python for network topology analysis software. I went from there to using go and C to develop a distributed database product. It's been incremental steps lower on the stack and towards more "pure" dev work. I'm now where I wanted to be and will keep doing this kind of work for as long as I can get away with it.
yep, exact same feeling here. I had several years as a "data scientist" and it was a an almost totally bullshit job. the org bought into the hype and hired a cohort of us straight out of university, but then couldn't find anything data-science-y for us to actually do. what I actually ended up doing 95% of the time was taping together dodgy excel-based workflows using python scripts. it gave me a visceral appreciation…
Could you elaborate on what kind of "software engineering" you now do? For someone who also would like to get out of data science, mentions of "I became a software engineer" don't really help to clarify what kind of SWE is feasible for a data scientist with decent programming chops to get into.
Another comment here mentioned the on the job steps you can take, and that mirrors my experience. I also enrolled in Georgia Tech's OMSCS after a year or so of self study. About 1 year in I took a role using Python for network topology analysis software. I went from there to using go and C to develop a distributed database product. It's been incremental steps lower on the stack and towards more "pure" dev work. I'm now where I wanted to be and will keep doing this kind of work for as long as I can get away with it.
Great post. A few random comments... > The median data scientist is horrible at coding and engineering in general. The few who are remotely decent at coding are often not good at engineering in the sense that they tend to over-engineer solutions, have a sense of self-grandeur, and want to waste time building their own platform stuff (folks, do not do this). > It was obvious that there is a general industry-wide need…
> 23 year-old data scientists should probably not work in start-ups, frankly; they should be working at companies that have actual capacity to on-board and delegate work to data folks fresh out of college. Ageism is disgusting and I cannot believe such blatant discriminatory language is seen as OK for a link posted to hackernews. How would you all say if he wrote that 40+ year old programmers should xx?
Eh, there's some stereotyping going on here that I don't 100% agree with, but I'm not offended by the notion that people fresh out of college are generally inexperienced in the working world and lack skills that are needed in industry more than in academic work.
This is how I feel about software development. The more people that get into it, the more it seems to be more that you're just a code monkey pushing out shitty code.