Is “data scientist” the new “programmer”? • Hacker News Board

Re: Is “data scientist” the new “programmer”?

#181

Post by noelsusman » Wed, Sep 19, 2018, 12:56 PM UTC

EdwardDiego wrote: post #42

danso wrote: post #36

This is such a bizarre post. The reason why people use a language like R is because it is easy to learn and use (and install, via RStudio) for data analysis without having to be a well-trained programmer. I can’t recall ever hearing from anyone who has relied on R, doing so because it was computationally efficient. The point of the language is convenience — particularly with how easy it is to create attractive graphi…

The fact that R has such buy-in despite being a rather awful programming language (a friend of mine worked on the next Lisp-like version of R under Ross Ihaka, and the next version is based on the fact that current R is a bit awful) is precisely because it offers such convenience to non-programmers. In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the…

>The data scientists write their algorithms in the language they're most comfortable with (typically JS)

I'm sorry, what? You know data scientists who use JavaScript to implement their algorithms?

Re: Is “data scientist” the new “programmer”?

#182

Post by dekhn » Wed, Sep 19, 2018, 12:59 PM UTC

httpz wrote: post #27

denzil_correa wrote: post #18

Data Scientist has two terms in it : Data + Science. More often than not, people ignore the "Science" part of that equation.

Someone said any field with "Science" in the name isn't really a science. Computer science, data science, political science, social science, etc. Physics, chemistry, biology don't have science in their name.

Physics, chemistry, and biology are all part of the Natural Sciences.

Re: Is “data scientist” the new “programmer”?

#183

Post by dekhn » Wed, Sep 19, 2018, 1:01 PM UTC

darkhorn wrote: post #71

stochastic_monk wrote: post #10

Earlier quoted context omitted.

Even the Stanford NLP Java implementations don’t always match the code they’re meant to be a translation of. I think R is a worse offender than Python.

In Python 0.1 + 0.2 is not equal to 0.3 because the result is 0.30000000000000004. In R 0.1 + 0.2 is equal to 0.3.

Not only is this technically wrong, but Python has a decimal module for performing these kinds of calculations. It uses floats/doubles for the native float type which produces exactly the type of results you see above.

Re: Is “data scientist” the new “programmer”?

#184

Post by mollusk » Wed, Sep 19, 2018, 1:01 PM UTC

I don't know, the number of hoops you need to jump through to use a trendy data science tool like Hadoop, Spark etc. is way bigger than that of a simple Java program. From my (limited) experience I'd say they the data science (or big data) way is the bloated and convoluted one.

Re: Is “data scientist” the new “programmer”?

#185

Post by arthurcolle » Wed, Sep 19, 2018, 1:13 PM UTC

jhowell wrote: post #174

arthurcolle wrote: post #170

Earlier quoted context omitted.

N == 22 specifically? Or N >= 22? Interesting threshold value

Why is 22 particularly interesting?

It's not. The fact that it doesn't readily have any real-world significance is what makes it an 'interesting' (read: odd, curious) threshold value, which is why I asked OP whether it would only fail at that core number (N == 22) or whether it effected all processor counts higher than the value. I can see that my use of interesting was colloquial and not literal. My bad for any confusion this may have caused ;)

Re: Is “data scientist” the new “programmer”?

#186

Post by JohnCohorn » Wed, Sep 19, 2018, 1:24 PM UTC

YawningAngel wrote: post #65

garyclarke27 wrote: post #51

Earlier quoted context omitted.

They’re all Turing complete inc SQL with case and recursion. I meant density, 1 line of a code, a SQL window function with a filter clause would probably take a page of Java to achieve the same result.

Nope, Java has map and filter just fine. Eg ``` Words.stream() .map(word -> String.toUpperCase(word)) .filter(word -> word.startsWith('A')) ```

SQL window functions aren't rocket science(not that I've used them much, cause ORMs and popular stripped down DBs like MySQL tend to not support them very well), but they do a lot more than you think if you're comparing them to trivial map/filter operations.

Re: Is “data scientist” the new “programmer”?

#187

Post by romaniv » Wed, Sep 19, 2018, 1:29 PM UTC

thidr0 wrote: post #41

Something about this article strikes me as a thinly-veiled complaint about poorly designed object-oriented systems. Take, for example this comment by the author: >Even if the money were half of what today’s coder gets paid it might still be a better job because one is spared the tedium of looking at millions of lines of Java that do almost nothing! What all those millions of lines of code are is abstractions, decoupl…

>But the problem is the design and not the philosophy.

If the philosophy prescribes dozens of tools for managing complexity and no tools at all for reducing it than it is the problem.

"Abstractions, decoupling, and modularization of logic/responsibilty" are not some kind of universal good. They are only useful within specific contexts. A lot of software engineers do not understand this and routinely engage in premature abstraction. As a result they produce systems that are 10 times more complicated than they need to be for absolutely no reason.

Java definitely encourages this kind of mentality, because the language itself and its standard library lack in some fundamental areas. Introduction of lambdas and streams helped, a lot, but the overall mentality is still well-entrenched.

Re: Is “data scientist” the new “programmer”?

#188

Post by hef19898 » Wed, Sep 19, 2018, 1:32 PM UTC

karmakaze wrote: post #173

vinayms wrote: post #127

The take down on abstraction and software engineers (by using Java as an example) is similar to saying "back in the day to find a prime number we would simply use a sieve, but today it is a tedium, what with all the pi's and e's and thetas that get in the way, and what are geometry and polynomials doing here, and what in the god's name is this i , I just want to count the prime numbers which are nice round whole numb…

"data monkey" is even more of a fitting name than the corresponding "code monkey". Much of so called data science leans more on the side of data engineer where one fits existing solutions to your specific data. The split of data scientist and data engineer is the most unfortunate. It's like splitting programming into program design and development (opposite of devops) in a specific language. That's done too but usual…

The risk I see is over using data science in circumstances where it is just a product feature. Risk is then to over emphasize the data science part and forgetting the relevant context. Like getting lost in the data itself.

A tendency I saw is that math graduates have a tendency to put everything in probability functions. That reality is composed of people that cannot be predicted is sometimes beyond their horizon. As a result everybody believes the solution is mathatically correct and thus suitable to reality while it is quite the opposite.

EDIT: Typos, again...

Re: Is “data scientist” the new “programmer”?

#189

Post by WhompingWindows » Wed, Sep 19, 2018, 1:33 PM UTC

3chelon wrote: post #156

I've interviewed a few "data scientists". Some of them were pretty arrogant. Their idea of a "close to the metal" language was Numerical Python. I don't think these guys are going to be writing the next generation of OS anytime soon.

Why would we expect a statistician/data analyst to write an OS? I'd expect them to write reports, white papers, articles, and functional domain-specific packages for R or Python. Especially in my field of healthcare, where over 50% of the analysts use SAS, I doubt we'll see any groundbreaking innovation, at best it's incremental changes via papers or sharing code.

Re: Is “data scientist” the new “programmer”?

#190

Post by collyw » Wed, Sep 19, 2018, 1:34 PM UTC

raducu wrote: post #147

itronitron wrote: post #123

Earlier quoted context omitted.

Yes, "data scientist" is the new "70's programmer"... write some code in one file that runs within a hosting system (mainframe, spark). Regarding the complexity and tedium of many production code bases I think they got there because many developers don't have the ability (experience) or opportunity (iterations) to do things simply.

True, but the reverse is also applicable: academia most often than not does not get the chance to do anything really complex in a tight schedule; this explains what the author says: "Consider that the “data scientist” uses compact languages such as SQL and R. An entire interesting application may fit in one file. " I have seen horrendous, multi-page SQL queries in very large systems.

SQL is still one of the best languages for readability in my opinion.