Earlier quoted context omitted.
The fact that R has such buy-in despite being a rather awful programming language (a friend of mine worked on the next Lisp-like version of R under Ross Ihaka, and the next version is based on the fact that current R is a bit awful) is precisely because it offers such convenience to non-programmers. In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the…
a friend of mine worked on the next Lisp-like version of R under Ross Ihaka Did anything come of this? I seem to remember seeing a paper or article where he proposed doing this, but I’ve never seen an implementation. Edit: paper I was thinking of is https://www.stat.auckland.ac.nz/%7Eihaka/downloads/Compstat-...
Is “data scientist” the new “programmer”?
171–180 of 246 posts
Re: Is “data scientist” the new “programmer”?
#172The take down on abstraction and software engineers (by using Java as an example) is similar to saying "back in the day to find a prime number we would simply use a sieve, but today it is a tedium, what with all the pi's and e's and thetas that get in the way, and what are geometry and polynomials doing here, and what in the god's name is this i , I just want to count the prime numbers which are nice round whole numb…
> who otherwise had never dealt with statistics before.
And why is statistics required? Let's face it most companies who need "Data Scientists" are looking for regular BI guys with fancy terms. Most of the problems are solvable using out of box functionalities in python/keras etc. Sure there are places and problems which require hard mathematics and stats but those are few and far between.
Re: Is “data scientist” the new “programmer”?
#173The take down on abstraction and software engineers (by using Java as an example) is similar to saying "back in the day to find a prime number we would simply use a sieve, but today it is a tedium, what with all the pi's and e's and thetas that get in the way, and what are geometry and polynomials doing here, and what in the god's name is this i , I just want to count the prime numbers which are nice round whole numb…
"data monkey" is even more of a fitting name than the corresponding "code monkey". Much of so called data science leans more on the side of data engineer where one fits existing solutions to your specific data. The split of data scientist and data engineer is the most unfortunate. It's like splitting programming into program design and development (opposite of devops) in a specific language. That's done too but usually the spec is functional (behavior not FP sense) and not algorithmic.
If this pattern works for anyone, great keep running find it he limitations. I just believe that there will be a better structuring and selective application movement.
A true data scientist would be doing research into new solutions or high level improvements. This can't happen at typical sized companies unless it's the core product and not a feature of one.
The big data, data scientist/engineer bandwagon is a little like blockchain. Everyone wants to leverage it, there are places where it is suitable but not everywhere where it's applied.
Re: Is “data scientist” the new “programmer”?
#174Earlier quoted context omitted.
I'm a scientist (wet lab) by training, a programmer (back end) by profession, and a data scientist by hobby (I have a machine learning project that I'm working on), and most of "data science" is not really stats... There will be a bit of stats at the end product but really the bulk of the necessary work is data curation. Annoying stuff like making sure my data fit into the right buckets. I did have to debug a memory…
N == 22 specifically? Or N >= 22? Interesting threshold value
Why is 22 particularly interesting?
Re: Is “data scientist” the new “programmer”?
#175Earlier quoted context omitted.
All of today's software is built on millions of lines of code. This isn't much of a problem when it's hidden away behind a good abstraction. Being written in another language forces the API to be documented well enough so you don't need to go deeper: "native code", "kernel code", "part of the browser". Crappy million-line Java apps are generally crappy not due to raw line count but rather due to leaky abstractions an…
I sometimes wonder if programmers instinctively overcomplicate things in interest of collective job security. Some of the stuff I've seen in (particularly awful) Java code bases is perplexing to the point where it seems intentional.
Writing clear and as-simple-as-possible code is one of the most difficult parts of programming, one of the hardest skills to learn. In fact, many programmers don't ever learn it. Some do not even realize it's a problem.
I could talk about people unprepared for the jobs they do, and companies and managers setting unrealistic deadlines, having unrealistic expectations, etc. But I think most often than not, the issue is that to really build good code, you need to have a lot of experience and plan and design really well from the start. And simply, most companies and products start as they can. If you start with a deficient codebase, trying to fix that later under heavy budget and time constraints is pretty much impossible, and it's so accepted that some people believe it's the normal way to work.
And even with proper design, clear perspective, and good programmers, you still have to be lucky that no one higher up the ladder imposes decisions that f*ck up all the good work done. There are a few factors more.
What I mean is that you might find all kinds of people, but in general, for those of us who care about what we are doing it's the exact opposite. We consciously try really hard to keep code as simple as possible.
Re: Is “data scientist” the new “programmer”?
#176Sure, we’ve stolen the term “engineer” for long enough, let’s bother the scientists now.
Why is software engineering not a valid engineering? I worked on both software and hardware engineering and general principles seem to be the same. You deal with complexity and simplify it by making abstractions. You make calculations to make sure your project is feasible. It's not like EE and Aerospace engineering are literally the same field but there are some principles shared in those fields, and with software en…
Yes, while in many countries Software Engineering is actually a degree that needs accreditation and in some cases even a professional exam if you need to sign off projects as the responsible person, on US apparently you can call yourself software engineer after a three week bootcamp.
Re: Is “data scientist” the new “programmer”?
#177Earlier quoted context omitted.
Why is software engineering not a valid engineering? I worked on both software and hardware engineering and general principles seem to be the same. You deal with complexity and simplify it by making abstractions. You make calculations to make sure your project is feasible. It's not like EE and Aerospace engineering are literally the same field but there are some principles shared in those fields, and with software en…
When software engineers stop disclaiming all liability for their products failing, we can talk about them like we talk about engineers (sidenote: some are already take responsibility)
Depends on the country, as some do require a certain level for people to call themselves software engineers to start with.
As for the liability, I fully agree.
Re: Is “data scientist” the new “programmer”?
#178The take down on abstraction and software engineers (by using Java as an example) is similar to saying "back in the day to find a prime number we would simply use a sieve, but today it is a tedium, what with all the pi's and e's and thetas that get in the way, and what are geometry and polynomials doing here, and what in the god's name is this i , I just want to count the prime numbers which are nice round whole numb…
A good programmer is someone who can communicate with the problem domain experts and provide an a solution to fit their problem. Someone who understands the limitations of the computing environment and can engineer a solution that is adequate for the problem space problem. Many who consider themselves programmers produce solution space solutions that just don't get to the core of the problem space problems. This is a…
Good points, if a bit verbose. :)
Tangent: regarding "problem space vs solution space" issues, I find that many projects suffer needlessly from too much focus on one of these over the other. Learning to balance them isn't easy, but is critically important.
Re: Is “data scientist” the new “programmer”?
#179I thought “engineer” is the new “programmer.”
Only in countries that don't have certification of job titles.
Re: Is “data scientist” the new “programmer”?
#180Similar story: once, when Googlers were talking about full stack engineers, Rob Pike piped up to say they weren't really full stack engineer; he said he was a full stack engineer back when he worked on the Voyager project because he had to know everything from silicon quantum mechanics up to interplanetary astrophysics.
Sure, there are parallels between modern programmers and the scientist-engineer-calculators of the past.