Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: As a data scientist, what should be in my toolkit in 2018?

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 177 posts

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#111

Post by dsacco »

arca_vorago wrote: post #82
red_hare wrote: post #69

Earlier quoted context omitted.

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

I have to admit this scares me just a little bit. I'm a senior sysadmin who is trying to lateral transition into data science, but I'm no math whiz, I'm just good at pragmatic use of tech stacks and have a generally analytical mind. If you are a math undergrad how could I ever expect to know more math than you? Of course a standard deviation should be easy, but your comment on math just stuck out to me.

> If you are a math undergrad how could I ever expect to know more math than you?

Read through, and do all the exercises in, one textbook each for:

1. Calculus

2. Linear Algebra

3. Abstract Algebra

4. Analysis

5. Topology

6. Probability Theory

7. Number Theory

...more or less in that order. Make sure your calculus book covers single variable and multivariable calculus. Supplement with applied mathematical statistics. Do that, and you have the equivalent of a mathematics undergrad (as far as relevant courses are concerned).

You could even do this with something like UIllinois’ NetMath program, or some courses on Coursera. You can swap out Number Theory for Complex Analysis or deeper Probability Theory and it’d be more relevant.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#112

Post by ZeroCool2u »

chewxy wrote: post #44
ZeroCool2u wrote: post #43

Earlier quoted context omitted.

I'm a big Go fan, but this is the first time I've seen someone recommend Go for data science. After looking at this cheat sheet you've got me convinced though. Would you mind pointing me to any other less cheat sheet style and more in depth examples that you particularly like?

Working on it. Part of my goal for 2018 is to write a lot more soft documentation - tutorials etc. Go is quite straightforwards though - WYSIWYG for the most parts, hence you probably won't find a lot of sexy tutorials. Almost everything is just a loop away, and in the next version of Gorgonia, even more native looping capability is coming

Awesome, thank you!

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#113

Post by zamazingo »

nonbel wrote: post #80
elsherbini wrote: post #74

I'm a scientist (PhD student in microbiolgy) that works with lots of data. My data is on the order of hundreds of gigabytes (genome collections and other sequencing data) or megabytes (flat files). I use the `tidyverse` from R[0] for everything people use `pandas` for. I think the syntax is soooo much more pleasant to use. It's declarative and because of pipes and "quosures" is highly readable. Combined with the powe…

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

I agree on these reservations, especially in terms of silent errors (which get compounded through minor ways in which backwards incompatibility can sneak in to the existing scripts) and dependencies.

As a half-solution, I ended up restricting myself to a very few libraries in this family (mainly dplyr, lubridate, stringr, broom) and to using packrat to consistently freeze the library versions for these.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#114

Post by peatmoss »

red_hare wrote: post #69
booleandilemma wrote: post #64

Earlier quoted context omitted.

I think data scientist, much like software engineer, is something you can call yourself without having any credentials whatsoever. It’s why technical interviews can be so brutal, unfortunately. There are a lot of frauds out there. Money attracts frauds. What’s the fizzbuzz test for data scientists anyway?

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

I think regurgitation of math formulas is a terrible way to hire for most data science positions. I've seen a breakdown of data scientists into two categories:

1) People who are great at the mathematics behind the statistical tooling

2) People who are great at conceptualizing a relevant question, operationalizing it, and then using a computer to apply appropriate models.

I think in most cases, for businesses needing to solve business problems, the latter kind is probably more useful. There are applications where the former is required, but you probably know if you need this kind of data scientist.

I should also add that these traits aren't mutually exclusive, but that individual data scientists typically are stronger or weaker along approximately those axes.

In general, I still dislike the term "data science" because it obfuscates meaningful distinctions between math nerds, computer science nerds, and research nerds who happen to do some applied stats.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#115

Post by Seanny123 »

chasedehan wrote: post #94
nonbel wrote: post #80

Earlier quoted context omitted.

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

I really enjoy the tidyverse, especially dplyr. I do most of my work in python now and find myself moving more and more of time in python. There are definitely some issues if you have to reliably run scripts (not to mention the difficulties of putting into production) The thing I really like about R over python is for SPECIFIC tasks like inspecting data and trying to get an answer out quickly, there really isn't a qu…

I would love some examples where R makes it easier to get answers than Python. Probably would make good Pandas2 issue too!

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#116

Post by telchar »

fsloth wrote: post #62
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

Do people get careers as 'data scientists' without masters degrees? I'm heavily biases towards experts... but still would call it fair in the general case to call anyone doing data science without at least a masters degree or the equivalent mental toolkit more like a 'data quack'.

I'm in that boat. I think a technical bachelor's plus work experience and self-study allows me to get along, but similarly to mr_overalls in the sibling comment I'm going back to school for a master's degree. It seems that getting into a data science career can be done without at least a master's but it seems that it would be hard to advance without one.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#117

Post by peatmoss »

xitrium wrote: post #79

If you care about quantifying uncertainty, knowing about Bayesian methods is a good idea I don't see represented here yet. I care so much about uncertainty quantification and propagation that I work on the Stan project[0] which has an extremely complete manual (600+ pages) and many case studies illustrating different problems. Full Bayesian inference such as that provided by Stan's Hamiltonian Monte Carlo inference a…

I think this is an important point. Having worked in / proximate to public policy kinds of problems, Bayesian methods have some really great properties:

1. easier interpretation of results than frequentist methods for lay people (business strata, elected officials, or other decision makers)

2. Uncertainty can be quantified and visualized reasonably well, which helps decision makers not think of stats as a magic box that produces a single answer.

3. Sensitivity analysis can be placed right up front: selection of priors representative of the beliefs of differing opinions / ideologies can inform decision makers of when they should consider changing their minds, and when they might still hold out.

Downsides of Bayesian methods:

1) Conceptually more involved than typical maximum likelihood estimation methods

2) Computationally expensive

3) Methods might not be as well known to a nominally stats-savvy audience.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#118

Post by kirillseva »

in9 wrote: post #105
nonbel wrote: post #80

Earlier quoted context omitted.

Sometimes I think I'm the only one who isn't really a fan of the tidyverse. I've found it slower, more prone to dependency issues, more prone to silent errors, and less well documented than most R packages (ie most of what you find on CRAN).

Dependency management, in my opinion, is one of the problems in the R ecosystem. The lack of name spaces when calling functions has made the community have many little packages that only do one thing on you are not really sure where it was actually used, unless you know the code and the package. An example is the janitor::clean_names function I like to use for standardizing the column names on a data.frame. However,…

You should try https://github.com/robertzk/lockbox for dependency management

It's like bundler or cargo for R

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#119

Post by geebee »

bitL wrote: post #52

Spark + MLlib, Python + Pandas + NumPy + Keras + TensorFlow + PyTorch, R, SQL, top placement in some Kaggle competitions. This would get you long way.

Good tool set recommendations (+1 for mentioning SQL, immensely helpful), and I enjoy Kaggle. Not sure how critical top placement is, though.

It seems like getting into the upper echelons of Kaggle is a matter of refining your model, and I do wonder how much value these refinements offer over a more basic and general approach in a real world scenario. To be clear, when I say I wonder, I'm not saying I'm rejecting the value, I really do mean it, I'm uncertain about the value. I think it's probably very scenario specific.

Think of it this way - a predictive value of 90% vs 95% could be the difference between placing in the top 10% and the bottom third. Now, 5% isn't nothing, it could be very valuable. It really depends.

But Kaggle is an environment where the question is already posed, the data has been collected, the test and train sets are already split apart for you, and winning model is the one that scores best on a hidden test set by a predefined goodness of fit score.

In a real world scenario, suppose someone does a great job figuring out the question to ask, gathering the data, and determining the most effective way to act on the results, but uses a fairly basic, unrefined model. Someone else does a middling job on those things, but builds a very accurate model as measured by the data that has been collected. I'd say the first scenario is likely to be more valuable, but again, it depends of course.

A couple other things, since I am a fan of Kaggle and do highly recommend it. First, these things aren't necessarily exclusive - you can have a particularly well conceived and refined model as well as a thorough and excellent businesss and data collection process (though you may have to decide where to put your time and resources).

Also, refining a model with Kaggle can be an exceptional training opportunity to really understand what drives these things. So go for it! (I also find these things kinda fun).

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#120

Post by jeffreyrogers »

dsacco wrote: post #111
arca_vorago wrote: post #82

Earlier quoted context omitted.

I have to admit this scares me just a little bit. I'm a senior sysadmin who is trying to lateral transition into data science, but I'm no math whiz, I'm just good at pragmatic use of tech stacks and have a generally analytical mind. If you are a math undergrad how could I ever expect to know more math than you? Of course a standard deviation should be easy, but your comment on math just stuck out to me.

> If you are a math undergrad how could I ever expect to know more math than you? Read through, and do all the exercises in, one textbook each for: 1. Calculus 2. Linear Algebra 3. Abstract Algebra 4. Analysis 5. Topology 6. Probability Theory 7. Number Theory ...more or less in that order. Make sure your calculus book covers single variable and multivariable calculus. Supplement with applied mathematical statistics.…

That doesn't seem like a good use of time. I've tried reading through and doing the exercises in an abstract algebra textbook. It's a lot of work and the applicability to real world problems is virtually non-existent. I think a more targeted approach would give you a better return on your time.
Post reply on HN