Forgot who said it but it was great: a "Data scientist" is a programmer better at stats than any 'normal' programmer and better at programming than any 'normal' statistician." :P
[deleted]
Is “data scientist” the new “programmer”?
211–220 of 246 posts
Re: Is “data scientist” the new “programmer”?
#212Earlier quoted context omitted.
Why is software engineering not a valid engineering? I worked on both software and hardware engineering and general principles seem to be the same. You deal with complexity and simplify it by making abstractions. You make calculations to make sure your project is feasible. It's not like EE and Aerospace engineering are literally the same field but there are some principles shared in those fields, and with software en…
When software engineers stop disclaiming all liability for their products failing, we can talk about them like we talk about engineers (sidenote: some are already take responsibility)
It's foolish for anyone to take responsibility for any software written under the current prevailing industry practices.
It would be funny if real engineers were able to get away with making crumbling messes that can't hold their own weight because their middle managers don't believe in concepts like stress and strain like ours don't believe in refactoring or abstractions.
Re: Is “data scientist” the new “programmer”?
#213Earlier quoted context omitted.
I'm a scientist (wet lab) by training, a programmer (back end) by profession, and a data scientist by hobby (I have a machine learning project that I'm working on), and most of "data science" is not really stats... There will be a bit of stats at the end product but really the bulk of the necessary work is data curation. Annoying stuff like making sure my data fit into the right buckets. I did have to debug a memory…
N == 22 specifically? Or N >= 22? Interesting threshold value
My box has 24 cores. By default I deploy on 22.
Actually it fails at 10 cores, but it gets 3/4 of the way through the dataset. At 22 it dies about 1/4 of the way, at 5 cores it makes it all the way through.
Error is in a string tokenizer, which I wrote as a recursive call. Usually it's fine but I made a code change which absolutely killed it. Also I'm writing in Julia, which does not TCO, the back end stuff I do is in elixir, which does.
Re: Is “data scientist” the new “programmer”?
#214The take down on abstraction and software engineers (by using Java as an example) is similar to saying "back in the day to find a prime number we would simply use a sieve, but today it is a tedium, what with all the pi's and e's and thetas that get in the way, and what are geometry and polynomials doing here, and what in the god's name is this i , I just want to count the prime numbers which are nice round whole numb…
> who otherwise had never dealt with statistics before. And why is statistics required? Let's face it most companies who need "Data Scientists" are looking for regular BI guys with fancy terms. Most of the problems are solvable using out of box functionalities in python/keras etc. Sure there are places and problems which require hard mathematics and stats but those are few and far between.
Most companies that think they need a team of data scientists just need a SELECT with a WHERE clause and maybe GROUP BY.
Re: Is “data scientist” the new “programmer”?
#215Earlier quoted context omitted.
Personally, I am increasingly convinced that a lot of this hate comes from programmers with weak abstract thinking who simply cant do it. Instead of admitting that there is learning cure involved, they will claim the system is bad and everyone else is bad. Compounding factor is difficulty dealing with system that was written by different people who holded different opinions. Yes, there are badly designed large system…
Badly designed systems aside, for most systems that currently exist or have existed in the past, there is little or no documentation that is worthy of existing. Most comments in code and the associated documentation in manuals fails to provide the reasoning as to why the code exists, why it is written that way, what underlying assumptions have been made, etc, etc, etc. I am going through a process at the moment of do…
I meant documentation on whatever tools and libraries project uses. Specifically, a lot of lines that do nothing are 95% result of frameworks integration.
You should not need to look at those particular lines that much.
I agree that the other issues you described here make work on large long running projects harder. It is challenge and somethings fight for every inch. Which is why I am increasingly sick of people who can't do it kicking those who can and kicking tools that make it possible (I have better chance to figure out the system with oral history only you just described then in javascript or python. Not easy, but tools make it less hard.)
Re: Is “data scientist” the new “programmer”?
#216As someone who has been looking for Data Scientist jobs in the past few months, I can reliably say that the term can mean everything from software engineer for big data systems, SQL guy or a person that builds complex machine learning models.
It is just as vague as the job profile of a "programmer". In that sense, the title is right. But, in the context of the article's content, I disagree.
The job done by a data scientist in demanding roles, requires a strong grasp on undergrad level statistics. But because of the recent trends towards ML, the person also needs to have a strong grasp of linear algebra, vectorization and software engineering / undergrad algorithms.
While it is unlikely that one data scientist may need to summon the whole skill set, an interviewee will never know which subset of these skills you will be asked to demonstrate to get hired.
Modern software jobs have figured out distinct subset of skills needed to differentiate between different software roles for experienced employees. Junior level employees are barely even expected to know anything other than algorithms, data structures and high level system design (at least during interviews)
Another funny observation (anecdotal) is there seem to be more openings for "senior data scientist" (who is expected to know everything), than "junior data scientists" whom the company is willing to mentor.
As of now, I find myself scrambling to decide which skills I need to prioritize, often feeling like I am being pulled in opposite directions. Almost of which require formal instruction (the maths), and can't be picked like software skills through youtube and online projects. This isn't a knock against software, just different type of subject matter.
Companies interviewing for these roles may ask everything from leetcode algorithms questions to statistics to questions about modern ML algorithms and domain specific models (in NLP, Vision, finance, recommenders)
I personally find a "junior" Data Scientist's role (in expectations) to be harder than that of a junior SDE. There is a reason many these jobs will put phD into preferred qualifications. It is ironic that there has been such a massive surge of people without the necessary background, who do a couple of MOOCs and crown themselves data scientists. Being good at any software & math heavy domain is hard. Data Science is no exception.
Re: Is “data scientist” the new “programmer”?
#217Forgot who said it but it was great: a "Data scientist" is a programmer better at stats than any 'normal' programmer and better at programming than any 'normal' statistician." :P
This is the best short form definition of a data scientist I've heard yet.
Re: Is “data scientist” the new “programmer”?
#218Earlier quoted context omitted.
In Python 0.1 + 0.2 is not equal to 0.3 because the result is 0.30000000000000004. In R 0.1 + 0.2 is equal to 0.3.
Not on my R: > (0.1 + 0.2) == 0.3 [1] FALSE
The result for 0.1 + 0.2 is 0.3. https://imgur.com/xWpx1Cg
Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points.
But if you want to compare use all.equal(0.1 + 0.2, 0.3)
Re: Is “data scientist” the new “programmer”?
#219Earlier quoted context omitted.
In Python 0.1 + 0.2 is not equal to 0.3 because the result is 0.30000000000000004. In R 0.1 + 0.2 is equal to 0.3.
R: > 0.1+0.2 == 0.3 [1] FALSE ?
Why do you compare point to point, it is meaningless? The result for 0.1 + 0.2 shows 0.3. It doesn't show you wrong result. If you print it prints 0.3, nothing else.
Why do you compare point to point? I have never once in my Statistics education compared point to point. You always need to see the probability of the result if it is within 2 points.
But if you want to compare use all.equal(0.1 + 0.2, 0.3)
Re: Is “data scientist” the new “programmer”?
#220Earlier quoted context omitted.
> To be nitpicky, in the US, engineer means you graduated from an ABET accredited program in something like: Chemical engineering, mechanical engineering, civil engineering, electrical engineering, industrial engineering, computer engineering....etc. Do you happen to have a reference for this? At first glance, it seems to be incorrect rather than nitpicky. Anecdotally, I know plenty of people who do not have ABET acc…
https://motherboard.vice.com/en_us/article/vvapy4/man-fined-... (There are lots of other articles about that case, that one sums it up mostly in the url)
The OP says "the US". That article is one state in the US. Most states don't have restrictions on the use of Engineer in job titles. Canada does though.