Ask HN: As a data scientist, what should be in my toolkit in 2018?

For powerful GPU viz, have you considered vispy? Four authors of four independent Python science visualization libs got together to build it.

Any monkey can make nice plots with R or python. Using a tool will not make you efficient. What matters is the brain of the one using the tools. In fact for the same reasons astronauts are scientific trained to pilot and not pilots trained in science, data scientists/analyst are scientist trained to use tools and thus the science part is the only one that matters.

Major problem I have encountered in IT is that monkey hate science, they just go on internet, see a blog post and reproduce nice biased manipulation (the more obscure, the deeper it will be perceived). It is basically cargo cult. Mimicking science instead of doing science.

Science is boring and is all about being a data janitor, and cleaning data so much, it sparkles with simplicity.

How to clean data is the most vital part of data "science".

IT data are often plagued with a lot of errors typo, poor encoding translations, heterogeneous sources which quality and relevance varies a lot due to IRL bias.

Having a background in being a DBA, a coal faced coder helps assessing fast the quality of data and transforming them.

Also, IT is plagued with floats. Where science matters, the uncertainty inherent with float representation can introduce artefacts that are a pain in the a.. when it comes to phenomenon that are sensitive to initial conditions or when measure instruments are put at the edge of their resolutions.

So at my opinion, the "measure theory" (Lebegue) is a must. It helps you understand the confidence you can have from your data, the margin error you have, and assess if data are faked.

In IT a lot of cheating is involved, so being able to detect fraud is required, for this you have to know the shape of "randomness" (Poisson, Student...) and know your estimators.

Also, signal processing basics are important for seasonal/recurrent activities (trend analysis).

However, some phenomeon are non linear, hence using fourier transform, moving average sometimes are the wrong (overused) tool, thus a little knowledge of complex systems (stochastic) is a plus.

Then after a LOT of work (the last 5%) you may have to decide if you face a correlation, a causation, or just noise.

For this there are 2 schools : bayesians and frequentists.

I am a frequentist because the tools are far more robusts, easy to understand and to use.

The market want bayesian because it is so obscure you can easily make your data say whatever you want. My intuition is bayesian are at the core of the problem of reproducibility in science.

And last but not least, your work is useless if you cannot communicate. There are standards : removing all useless elements from plots, quoting your sources the right way...

For the tools ?

It depends. They are domain specific. Some use matlab, mathematica, excel, R, python, proprietary software. The scientific background is so harder to find than a programming background no serious companies will turn you down for not knowing their tools. But clearly having a factual knowledge of accessing, transforming, cleaning data is a must and this can a combination of languages and tools (C, python, Perl, bash, sed, awk, grep, C++, pandoc, imagemagick) since every languages have their strengths and weaknesses and you don't know where you data will come from (CSV, DBs (SQL or not), web pages, scans, pictures, instruments, simulations, text files, structured documents, syscalls....).

Thus, being a data analyst is first of all knowing the intellectual and computer tools to assess, gather, transform, clean the datas and methodic about their source and how you did it so that someone else can try to reproduce your claims.