Hey everyone, I'm not a data scientist or a developer but I work with a lot of them. My company, Introspective Systems, recently released xGraph, an executable graph framework for intelligent and collaborative edge computing that solves big problems: those that have massive decision spaces, tons of data, are highly distributed, dynamically reconfigure, and need instantaneous decision making. It's great for the modeling work that data scientists do. Comment if you want more info.
Ask HN: As a data scientist, what should be in my toolkit in 2018?
161–170 of 177 posts
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#162Earlier quoted context omitted.
I'd call myself more of a "data plumber", with a double-BS in physics and computer science, but I'm considering returning for a MS so I'm more qualified to do interesting work. Is there a consensus about what kind of Master's would be most useful data-sciency stuff? Computer science? Stats?
Lots of universities have Data Science Master's Programs, which may ultimately fall under the Computer Science, Mathematics or Stats department. So, it depends on the university.
Data Science degrees seem to be such a hodge-podge of topics, with very uneven quality between programs. For long-term benefit, I'll admit I'm more inclined to tackle a traditional MS like CS, math, or stats.
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#163Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…
I think visualization can be a helpful tool to understand the data. I have seen some DS's get caught up in visualization for visualization's sake which I think can be wasteful.
I definitely think a solid mathematical understanding helps to build quantitative and critical thinking skills which are very key in data science.
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#164Earlier quoted context omitted.
Good tool set recommendations (+1 for mentioning SQL, immensely helpful), and I enjoy Kaggle. Not sure how critical top placement is, though. It seems like getting into the upper echelons of Kaggle is a matter of refining your model, and I do wonder how much value these refinements offer over a more basic and general approach in a real world scenario. To be clear, when I say I wonder, I'm not saying I'm rejecting the…
Top placement in Kaggle attracts recruiters for higher positions; i.e. I observed a top 10 person getting a job of Head/VP of analytics in a large European company even if let's say formal education wasn't top 100. I agree real-world it is often useless, but people are drawn to proven winners.
I'm not too surprised to hear that. In fact, I'd say a top score on Kaggle is probably a pretty positive indicator. Yeah, refining the model probably isn't as big a deal in a real project as it is on Kaggle, but it still takes some decent chops to get a good score like that.
My best was somewhere in the top third, so I'm not an especially strong Kaggle competitor. But even that took a lot of data parsing, piping, cleaning, moving some things to a database, populating a model, and parallelizing the processing so I could things on a cloud in an hour rather than 100 hours on my laptop. I learned a lot from it.
If you can score high on Kaggle, you definitely have some skill. And it's hardly like people who can do this never have the other skills necessary to manage the other stages of a data science project.
I probably wouldn't hire someone purely on Kaggle scores, but sure, it's a positive indicator of programming and data management ability.
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#165Earlier quoted context omitted.
I make the claim that you can go very far in the SciPy ecosystem without ever touching R. It is worth understanding the concepts of numpy and pandas. Furthermore, try out IPython/Jupyter, especially for rapid publishing (people run their blogs on jupyter notebooks). I think certain libraries depend very much on where you focus. Machine learning? Native language processing? Visualization? Something in economics? Funda…
Agree, I would drop R, Python has you mostly covered now. Julia is also worth learning.
R, python and Julia are all Turing-complete languages, so of course you can drop any two and get by with just the third.
The real selection happens when you consider what's available in opensource world. What code you don't have to write? What high-quality libraries are available vs which ones you will have to write yourself?
On this topic, R has vast advantage over python in some domains, such as bioinformatics for example, while python definitely shines when it comes to deep learning (and using for loops).
You can't just claim that one shouldn't look at R because you personally know one language better the other, quite likely because in your domain it's not being used as much.
I do prefer the deep learnin, NLP and production serving story in python, but you will have to pry dplyr+ggplot from my cold dead hands for quick analysis and charting. Not to mention that pandas's API is a clusterfuck compared to R's native data frames.
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#166Earlier quoted context omitted.
> less well documented than most R packages I on the other hand, find most R packages provide barely readable documentation. I can just hope that the vignette exists and actually explains the inputs/outputs.
Here it is for one of the most often used functions: https://www.rdocumentation.org/packages/ggplot2/versions/2.2... You think this is better than barely readable?
I am looking here and it's great: http://ggplot2.tidyverse.org/reference/aes.html
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#167Earlier quoted context omitted.
Here it is for one of the most often used functions: https://www.rdocumentation.org/packages/ggplot2/versions/2.2... You think this is better than barely readable?
I am looking here and it's great: http://ggplot2.tidyverse.org/reference/aes.html
It is the exact same thing except the examples are run. So ok, it seems some people consider this great documentation.
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#168Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…
I really like to get a degree in Mathematics but I simply don't have the time to throw at it (work, children, etc). What do you suggest I should do to have something on my resume? MOOC maybe?
You could work thru one of the ML in R/python books e.g. Géron or Raschka/Mirjalili and then dig into the lin alg, prob, stats, calculus/analysis you see there with the books everybody recommends, LA by Axler, Strang or Proabilty by Bertsekas /Tsitsiklis, Real Analysis by Pugh, Abbott, Strichartz etc
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#169You need to buckle up on Mathematics. Do not look at fancy stuff like PyTorch, Keras, Tensorflow, etc. Libraries and languages are a means to an end. Good, intuitive understanding of mathematical concepts behind a certain ML algorithm will go a long way. As mentioned in one of the comments, statistics shows up everywhere.
Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?
#170If you care about quantifying uncertainty, knowing about Bayesian methods is a good idea I don't see represented here yet. I care so much about uncertainty quantification and propagation that I work on the Stan project[0] which has an extremely complete manual (600+ pages) and many case studies illustrating different problems. Full Bayesian inference such as that provided by Stan's Hamiltonian Monte Carlo inference a…
I have also used Bayesian quantification of uncertainty in pricing forecast models. Decision makers love a measure of uncertainty when one recommends a pricing scenario that can have significant impact on revenue. Also, you get the chance to build multilayer models to combine knowledge from independent samples. PyMC3 is fantastic for building these models within Jupyter and Gelman's Bayesian Data Analysis is a great introduction for different Bayesian model applications.