Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: As a data scientist, what should be in my toolkit in 2018?

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 177 posts

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#101

Post by ms013 »

Aardwolf wrote: post #67
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

R is not present in your list, did you ever try it and what's your opinion about it?

I know R and have used it in the past. I just don’t like the language. I keep RStudio around though because on rare occasions I do look around in it to see if it has something I need. So rarely though that I forgot to list it...

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#102

Post by in9 »

anarchimedes wrote: post #92
swarchal wrote: post #88

Earlier quoted context omitted.

You're not the only one. Though I've found there seems to be a bit of a cult surrounding the tidyverse, a mere hint of criticism usually results in outrage and attacking other tools/packages (by users, not the authors).

I like the Tidyverse. My only complaint is that it presents another headache when moving from one language (R) to another (usually Python/SQL). Using the base R functions while integrating loops and functions lessens the fatigue of moving between R and Python.

I tend to think that, if I am working with other people, python is the best choice. But if I'm working alone, R is the way to go.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#103

Post by minimaxir »

fsloth wrote: post #62
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

Do people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.

During my recent data science job hunt, I received a lot of resistance due to my lack of a Master's/PhD: https://twitter.com/minimaxir/status/951117788835278848

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#104

Post by biswaroop »

p33p wrote: post #56
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do. I don't necessarily agree with this. Yes, a sound understanding of the domain and knowledge of…

I think what he/she means is poorly designed visualizations. Just because a plot is grayscale and not interactive doesn't mean it's worse than a cluttered poorly-designed super interactive web widget. It's a poor choice of wording, but I think by "overuse" they might mean "unclear but eye-catching". Besides "overuse" is literally the quantity that is excessive.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#105

Post by in9 »

nonbel wrote: post #80
elsherbini wrote: post #74

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the powe…

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

Dependency management, in my opinion, is one of the problems in the R ecosystem. The lack of name spaces when calling functions has made the community have many little packages that only do one thing on you are not really sure where it was actually used, unless you know the code and the package.

An example is the janitor::clean_names function I like to use for standardizing the column names on a data.frame.

However, the tidyverse is really serious in terms of api consistency and functional style, with pipes and purrr's functionalities. The unixy style of base R is unproductive in terms of fast iterating an analysis. Also, the idea of "everything in a data frame" (or tibble, with list columns and whatnot) together with the tidy data principles really takes the cognitive load off to just get things started.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#106

Post by kmax12 »

One crucial skill you will need is feature engineering. Formal methods for it aren’t typically in data science classes. Still, it’s worth understanding in order to build ML applications. Unfortunately, there aren't many available tools today, but I expect that to change this year.

Deep learning addresses it to some extent, but isn’t always the best choice if you don’t have image / text data (eg tabular datasets from databases, log files) or a lot of training examples.

I’m the developer of a library called Featuretools (https://github.com/Featuretools/featuretools) which is a good tool to know for automated feature engineering. Our demos are also a useful resource to learn using some interesting datasets and problems: https://www.featuretools.com/demos

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#107

Post by dermybaby »

nrjames wrote: post #60

There are two "poles" in data science: math/modeling and backend/data-wrangling. Most of the time, the backend/data-wrangling piece is a prerequisite to the math/modeling. The vast majority of small and medium sized companies have not set up the systems they would need to support a data scientist who knows only math/modeling. Depending on the domain, it's not uncommon to find that a small/medium company outsourced an…

So basic DBA skills + expert programming skills + very good math/stats?

Also - your model of asking questions before starting a new gig is very relevant to nearly every programming job. Could also be some of the questions a candidate asks in an interview.

Have you ever needed any Microsoft skills(MSSQL/C#) so far?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#108

Post by in9 »

I saw a simple tool somewhere a while ago (maybe a month or so ago) of a simple cli for data inspection in the terminal. It seemed very useful for inspecting data ssh'ed into a machine.

However, I can't seem to recall the name. Has any one seen what I'm talking about?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#109

Post by dsacco »

red_hare wrote: post #69
booleandilemma wrote: post #64

Earlier quoted context omitted.

I think data scientist, much like software engineer, is something you can call yourself without having any credentials whatsoever. It’s why technical interviews can be so brutal, unfortunately. There are a lot of frauds out there. Money attracts frauds. What’s the fizzbuzz test for data scientists anyway?

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

> I studied math in undergrad so one of my requirements is "knows more math than me".

What kind of questions are you asking to ensure that they’re correct when they’re speaking about math you don’t know?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#110

Post by dsacco »

jerednel wrote: post #86
red_hare wrote: post #69

Earlier quoted context omitted.

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

At least with fizzbuzz you are working through how to logically solve a problem. This is just regurgitating a formula. I don't see how this is helpful.

No, using numpy would be analogous to a simple formula. Doing it without numpy requires actually understanding what’s going on.

It’s a filter that theoretically allows false positives (which is why you continue with other questions), but it really shouldn’t have any false negatives.

Post reply on HN