191–200 of 246 posts
I've interviewed a few "data scientists". Some of them were pretty arrogant. Their idea of a "close to the metal" language was Numerical Python. I don't think these guys are going to be writing the next generation of OS anytime soon.
This post states that a data scientist uses compact languages such as SQL and R. Genuine question - do people really believe that being able to write and understand complex SQL makes you a data scientist? I ask because, I've been writing some of the nastiest, most difficult looking SQL around for probably at least 15 years. And yet, I would NOT call myself a data scientist because I know and can work with data and us…
Earlier quoted context omitted.
They’re all Turing complete inc SQL with case and recursion. I meant density, 1 line of a code, a SQL window function with a filter clause would probably take a page of Java to achieve the same result.
Nope, Java has map and filter just fine. Eg ``` Words.stream() .map(word -> String.toUpperCase(word)) .filter(word -> word.startsWith('A')) ```
Something about this article strikes me as a thinly-veiled complaint about poorly designed object-oriented systems. Take, for example this comment by the author: >Even if the money were half of what today’s coder gets paid it might still be a better job because one is spared the tedium of looking at millions of lines of Java that do almost nothing! What all those millions of lines of code are is abstractions, decoupl…
I've been hearing the same complaint between Cobol and Java for years : it was simpler before, more efficient, etc. Of course it was, but you were tied to one system (no application server), security was login/pw, database had no constraint, typing systems were ultra limited, everybody has its own way of writing batches (no Spring), business code was mixed with tons of technical code (no JPA). Now, sure, if you glue…
IMHO there's far to little attention paid to how data might be valuable in an economic sense when storage strategies are being designed by database designers. I recently gave a talk at a developer conference and was really surprised at the level of pushback to adding more data elements or higher precision data "just in case it might be useful".
The preconception that you have to be maximally efficient with storage has led to huge quantities of valuable data being lost.
This post states that a data scientist uses compact languages such as SQL and R. Genuine question - do people really believe that being able to write and understand complex SQL makes you a data scientist? I ask because, I've been writing some of the nastiest, most difficult looking SQL around for probably at least 15 years. And yet, I would NOT call myself a data scientist because I know and can work with data and us…
That said, the term data scientist itself is a bit frustrating. It gets thrown around a lot as if it is a well-defined role, and it is anything but. In my experience, the role of a "data scientist" is about as well defined as the role of an "engineer": it has connotations about the type of work and maybe a few shared skills, but the specifics of what an "engineer" does and their skillset varies widely depending on if they are a software engineer, an electrical engineer, or a civil engineer.
So while I think that most data scientists know SQL or use SQL frequently, I don't think that all data scientists use it, nor do I think that everyone who uses SQL works in a role that would probably be considered that of a data scientist.
Deming, from Out of the Crisis (1986): People with master's degrees in statistical theory accept jobs in industry and government to work with computers. It is a vicious cycle. Statisticians do not know what statistical work is, and are satisfied to work with computers. People that hire statisticians likewise have no knowledge about statistical work, and somehow suppose that computers are the answer. Statisticians and…
Earlier quoted context omitted.
A good programmer is someone who can communicate with the problem domain experts and provide an a solution to fit their problem. Someone who understands the limitations of the computing environment and can engineer a solution that is adequate for the problem space problem. Many who consider themselves programmers produce solution space solutions that just don't get to the core of the problem space problems. This is a…
Good points, if a bit verbose. :) Tangent: regarding "problem space vs solution space" issues, I find that many projects suffer needlessly from too much focus on one of these over the other. Learning to balance them isn't easy, but is critically important.
It takes a lot of effort to actually elucidate what the actual problem is that needs solving. Which is why I have made the comment earlier that programmers need to get out and see what the end user (client/customer/whatever you might want to call them) is actually doing and experiencing. When all you have is some design documents, functional specifications and technical specifications, the actual working environment for the solution is then missing.
We need to get out and face the complaints, observations, ire and suggestions of those who use the software we write.
Edit: as for verbosity, my mother has made the statement for many decades that, of her children, I was the one who could talk the legs off a cast iron stove. As my sons and daughters, grandsons and granddaughters, nephews and nieces have all had to learn, to shut me up, they have to talk.
I've interviewed a few "data scientists". Some of them were pretty arrogant. Their idea of a "close to the metal" language was Numerical Python. I don't think these guys are going to be writing the next generation of OS anytime soon.
The rest of 'developers' just get shit done.
Earlier quoted context omitted.
I've been working as a data scientist for several years and have written some pretty gnarly looking SQL myself. I have a background in math and hard science so I have some understanding of the scientific method as well. While I respect our DBAs I wouldn't call any of them qualified to be data scientists. While I have been able to hold my own in this job I went back to school to pursue a graduate degree (partly) becau…
I'll preface this with I've not had a look at any Kaggle competition, but I always assumed Kaggle competitions was on par with programming competitions in terms of how the skills transfer professionally. A great programmer is not necessarily great at programming competitions after all. Am I way off here?
Earlier quoted context omitted.
The fact that R has such buy-in despite being a rather awful programming language (a friend of mine worked on the next Lisp-like version of R under Ross Ihaka, and the next version is based on the fact that current R is a bit awful) is precisely because it offers such convenience to non-programmers. In my sister company, they have data scientists, and data engineers. The data scientists write their algorithms in the…
> ... despite being a rather awful programming language (...) it offers such convenience to non-programmers I've heard people say similar things about MATLAB - that it's a poorly designed language, but many that people (mostly non-CS folk) use it out of convenience. Can someone with experience using R explain what makes it so appealing to non-programmers? It seems like these two factors, "poorly designed" and "easy t…
By that definition, I would be a non-programmer, as I come from a statistics background, and though I have lots of experience in C++ and Python, most of my experience and work is in R. But that is by choice.
If I'm trying to create an application or build a website, I wouldn't use R. But when it comes to ingesting data, transforming and cleaning data, and modeling data, R is second to none. Yes, its syntax looks ugly and bizarre if you are used to object-oriented programming, software development, etc. In the context of working with data, I have never found anything in R to be even remotely confusing or strange.
On the contrary, the next best option to R would almost certainly be Python, and the gulf between the two is massive in my opinion. Python is a great general purpose programming language, but its data analysis capabilities, using packages like Pandas and sci-kit learn, feel like poorly designed, bolted on, and unwieldy. R is better for virtually every aspect of data analysis than Python.
So it isn't that R is poorly designed. Conversely, its very well designed, for its purpose as a data analysis-focused programming language. It only seems to be poorly designed to "programmers" because programmers work on problems that R isn't meant for. But that is like complaining that a screwdriver looks poorly designed for hammering nails.