Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: As a data scientist, what should be in my toolkit in 2018?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 177 posts

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#91

Post by shepardrtc »

jerednel wrote: post #86
red_hare wrote: post #69

Earlier quoted context omitted.

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

At least with fizzbuzz you are working through how to logically solve a problem. This is just regurgitating a formula. I don't see how this is helpful.

Its designed to quickly weed out people who don't know the underlying math, just as FizzBuzz is designed to quickly weed out people who don't know programming.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#92

Post by anarchimedes »

swarchal wrote: post #88
nonbel wrote: post #80

Earlier quoted context omitted.

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

You're not the only one. Though I've found there seems to be a bit of a cult surrounding the tidyverse, a mere hint of criticism usually results in outrage and attacking other tools/packages (by users, not the authors).

I like the Tidyverse. My only complaint is that it presents another headache when moving from one language (R) to another (usually Python/SQL). Using the base R functions while integrating loops and functions lessens the fatigue of moving between R and Python.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#93

Post by sampo »

mr_overalls wrote: post #84
fsloth wrote: post #62

Earlier quoted context omitted.

Do people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.

I'd call myself more of a "data plumber", with a double-BS in physics and computer science, but I'm considering returning for a MS so I'm more qualified to do interesting work. Is there a consensus about what kind of Master's would be most useful data-sciency stuff? Computer science? Stats?

> I'd call myself more of a "data plumber"

I think the actual term is Data Engineer.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#94

Post by chasedehan »

nonbel wrote: post #80
elsherbini wrote: post #74

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the powe…

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

I really enjoy the tidyverse, especially dplyr. I do most of my work in python now and find myself moving more and more of time in python.

There are definitely some issues if you have to reliably run scripts (not to mention the difficulties of putting into production)

The thing I really like about R over python is for SPECIFIC tasks like inspecting data and trying to get an answer out quickly, there really isn't a quicker or better tool to use. The ONLY reason I still even use R is because of the ease to get answers with the tidyverse

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#95

Post by squigs25 »

fsloth wrote: post #62
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

Do people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.

Unless you do a masters degree in Data Science, AI, CS, Statistics, or a related field, a masters degree only serves as proof that you are intelligent and work hard. Someone without a masters degree can still have those attributes, but they would just have to prove it some other way.

For example, if you have a bachelors degree from a top engineering school (MIT, Cal Tech, Stanford, Berkeley, etc.) you have proven that you are intelligent and can work hard.

People without a masters degree, but more business experience, bring a different perspective, and are often more business results focused, and potentially work more collaboratively than an individual who just graduated from a masters program.

Source: I am a Data Science hiring manager, and have interviewed 100+ candidates at several companies

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#96

Post by chubot »

Xcelerate wrote: post #51

As a data scientist who has been using the language for 5 years now, Julia is by far the best programming language for analyzing and processing data. That said, it’s common to find many Julia packages that are only half-maintained and don’t really work anymore. (I still don’t know how to connect to Postgres in a bug-free way using Julia.) And you’d be hard pressed to find teams of data scientists that use Julia. So i…

Has Julia converged on a solution for data frames? I watched some JuliaCon videos and got the impression that they hadn't. There seem to be a lot of different overlapping efforts.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#97

Post by Jeff_Brown »

Static typing lets you catch errors before running the code.

Pattern matching helps you write code faster (that is, spending less human time).

Algebraic data types, particularly sum types, let you represent complicated kinds of data concisely.

Coconut is an extension of Python that offers all of those.

Test driven development also helps you write more correct code.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#98

Post by Jeff_Brown »

mr_overalls wrote: post #84
fsloth wrote: post #62

Earlier quoted context omitted.

Do people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.

I'd call myself more of a "data plumber", with a double-BS in physics and computer science, but I'm considering returning for a MS so I'm more qualified to do interesting work. Is there a consensus about what kind of Master's would be most useful data-sciency stuff? Computer science? Stats?

I have needed this term for years. Thanks!

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#99

Post by jballanc »

Xcelerate wrote: post #51

As a data scientist who has been using the language for 5 years now, Julia is by far the best programming language for analyzing and processing data. That said, it’s common to find many Julia packages that are only half-maintained and don’t really work anymore. (I still don’t know how to connect to Postgres in a bug-free way using Julia.) And you’d be hard pressed to find teams of data scientists that use Julia. So i…

> That said, it’s common to find many Julia packages that are only half-maintained and don’t really work anymore.

On this specific point, it's worth noting that up until now there's been a single massive repository of every Julia package ever published, regardless of its current state or utility. Starting with the upcoming 0.7 release, Julia will introduce the concept of "curated" repositories so that, going forward, if you stick just with the default curated repository of packages you should have much less chance of running into a broken or unmaintained package.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#100

Post by ms013 »

anewhnaccount2 wrote: post #41
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

Out of interest, can you give an example of a problem you've solved using Z3?

One data problem boiled down to being an instance of the set cover problem (https://en.m.wikipedia.org/wiki/Set_cover_problem). Pretty easy to pose as an integer constraint problem, and Z3 solved it in about 20 minutes for me.
Post reply on HN