Live data from Hacker News

Is “data scientist” the new “programmer”?

blogs.harvard.edu

241–246 of 246 posts

Re: Is “data scientist” the new “programmer”?

#241

The take down on abstraction and software engineers (by using Java as an example) is similar to saying "back in the day to find a prime number we would simply use a sieve, but today it is a tedium, what with all the pi's and e's and thetas that get in the way, and what are geometry and polynomials doing here, and what in the god's name is this i , I just want to count the prime numbers which are nice round whole numb…

> statistics is a three year long grueling applied maths degree, and condensing it to three weeks is silly.

I agree. I was in a very prestige organization and they didn't know what a statistician really do and just hired CS machine learning PHDs. Even those people don't even know what statistician does. One person gave me ISLR when I ask for advice to get hire at this prestige place (I did an equivalent of this over several graduate courses in statistic program).

Another person proudly told me that in his project, he was using GLM stating he knows GLM. I asked what was the link function and that person stated he didn't know and it's somewhere in the code...

I've since then double down on statistic and will be going into Biostatistic field instead of data science. It feels like there are a lot of impostors especially in start ups and government organizations in data science. I have no clue why but there is just this culture in tech industry that have made me left it for better field. I've intern in the biostat field it is much better. CNN and other have listed with high quality of life.

Re: Is “data scientist” the new “programmer”?

#242

Earlier quoted context omitted.

What I would consider the difference between an engineer and not engineer (I am not an engineer) is delivering a qos or sla driven by measurement of tolerances and either empirical or imputed information from existing qos or tolerance information... Not necessarily the optimization part. Everyone does optimization to some degree (possibly negative) but not everyone is an engineer.

I think of engineering as the practical counterpart to science. Science is finding patterns/uncovering truths/building and testing models; engineering is the deployment of technology to fulfill an objective.

Engineering is more than just application. In some places you need to be certified as an engineer, for better or worse, which says that there is an understanding of how to calculate and communicate product tolerances and service level guarantees that go beyond just applying science. Basically be the bare minimum of applying science I would call hacking - which is a great thing, but there is value in the distinction between hacking a solution and engineering

Re: Is “data scientist” the new “programmer”?

#243

Earlier quoted context omitted.

It's more likely the natural entropy of code - it's easy to add stuff to a system in a way that makes it more messy; and if the system already is a big mess, then it's much harder to do non-messy additions and the bigger mess it is, the harder it is to start cleaning it up.

I refer to this as the "spaghetti law of attraction". The burden of refactoring things gets higher and higher and no one wants to touch it. So they just add another try-catch block and do some side effect and get the PR merged.

It is also a question of management, it is easier to motivate adding new features instead of keeping thing nice and tidy. Especially as finding good abstractions are very hard and time-consuming.

Re: Is “data scientist” the new “programmer”?

#244

Earlier quoted context omitted.

Yeah, the author's take is myopic. What they call bloat, people from the 70s would call wondrous: ubiquitous networking with and without wires, beautiful graphical interfaces, encryption everywhere (and expanding), far more open systems than proprietary re-engineered ones, the list goes on and on.

The author is Philip Greenspun, who in the 1980s worked with the people that created all of the things you listed: https://en.wikipedia.org/wiki/Philip_Greenspun There is nothing myopic about his perspective.

It's fair to mention that he is well known, though in fact I'm one of the old guard that remembers when he had a higher profile.

But, as with a new Paul Graham essay, surely we can critique the blog post on its merits instead of falling back on an assessment based on some kind of appeal to authority/"expertise by association". Philip Greenspan doesn't need to be treated with kid gloves as if he was the pope.

John Ousterhout made comments that touch on some similar (though not identical) distinctions in programming practices. That was years ago, and he was then a much more credible figure in software than Greenspun. All the same, his essay was heavily criticised. That's what serious intellectual discussion should involve.

https://en.wikipedia.org/wiki/Ousterhout%27s_dichotomy

http://www.tcl.tk/doc/scripting.html

Re: Is “data scientist” the new “programmer”?

#245

Earlier quoted context omitted.

I’m not referring to transient research prototypes, but to robust long-lived systems needed for experimentation and reproducible results tracking, and services that are directly customer facing. We are often required to create new services and functionality because it is how our company can grow, and we have to have ease of access to experimental working space, with freedom to do things like custom compilations of ML…

We may be talking past each other. I am on the research side of academic computing/informatics and have faced these struggles my whole career, encountering some very reluctant IT divisions. We have had to bite the bullet and use colo facilities to self-host internet-facing deployments that the overhead-funded IT groups would not touch with a ten foot pole. From these experiences, I also acquired a more nuanced perspe…

> “I think anybody trying to toss integrations over the fence to an ops team needs to have an extended tour of duty trying to operate their own solutions in production WITHOUT assistance before they form bold opinions about operations constraints.”

I think this is one of the worst possible attitudes to have. It’s a petty way to feel, desiring some type of “I’ve seen some shit” tough guy credential more than supporting the stuff needed to actually solve business problems.

If you hire people whose value add to your company is inventing completely new things, including deployment, ops, scaling, etc., that goes along with that, then it is the job of infrastructure on the other side of that fence to happily and eagerly accept whatever is tossed over the fence, to understand why developer teams made the choices they made, and to take an attitude of supporting as much as possible.

> “You need a plan for how all the components will be maintained, patched, and upgraded. You need contingency planning when some of those bleeding edge components are going to become deprecated. You need to consider what staff capabilities are assigned to do that maintenance work or what will happen when the institutional knowledge used to form the original integration is not on-call to reintegrate it in the face of unexpected events.”

Yes, of course. But all this is already what dev teams are doing. Ops / infra is not taking a hare-brained plan and adding these robustness aspects into it. Not at all. Instead they take plans from application teams and try to use policy to minimize their own maintenance burden, even when that optimization is antithetical to what the company requires at a more fundamental level.

A lot of companies languish and die because of sociological dysfunction in the policy interface between dev teams and infrastructure. The more that infrastructure has political control of that interface, the closer to death is that company.

It’s like a body that is disallowed from generating white blood cells in response to a new immune challenge. Even if the bleeding edge integrations are really hard, the alternative world where you slow them down with policy is death and attrition.

Re: Is “data scientist” the new “programmer”?

#246

Earlier quoted context omitted.

Good points, if a bit verbose. :) Tangent: regarding "problem space vs solution space" issues, I find that many projects suffer needlessly from too much focus on one of these over the other. Learning to balance them isn't easy, but is critically important.

It was one of my former managers/mentors that introduced me to the concepts of problem space vs solution space. As the decades have passed since then, what I have seen is that most computing solutions that have been offered for the problems people have experienced do not really consider what the problem is that is being faced. It takes a lot of effort to actually elucidate what the actual problem is that needs solvin…

wrt verbosity: Haha, I'm the same way, as in: "Sorry this [email|message|comment|...] is so long, I didn't have time to write a short one."

wrt problem space, yes! In contrast to all the focus on product development and engineering methodologies, somehow customer development generally suffers from a lack of rigor and attention. Ditto marketing -- in the sense of identifying or growing a market for the goods or services on offer.

Post reply on HN