Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: As a data scientist, what should be in my toolkit in 2018?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 177 posts

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#21

Post by jonathankoren »

Ultimatt wrote: post #15
trevz wrote: post #2

A couple of thoughts, off the top of my head: Programming languages: - python (for general purpose programming) - R (for statistics) - bash (for cleaning up files) - SQL (for querying databases) Tools: - Pandas (for Python) - RStudio (for R) - Postgres (for SQL) - Excel (the format your customers will want ;-) ) Libraries: - SciPy (ecosystem for scientific computing) - NLTK (for natural language) - D3.js (for renderi…

I'd gently suggest basic CLI Perl over BASH for cleaning up files, as it combines grep/sed/awk in a language thats more generally useful.

Agreed. Perl was designed for text munging, and is superior to pretty much everything for this task.

WRT bash, where to begin? In the past 40 years, there’s pretty much a better tool for everything someone tries to do with bash. It lives on pretty much through inertia and pride.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#22

Post by ms013 »

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis.

Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you can go surprisingly far with few and relatively simple ones. Knowing how, when, and where to apply them is the hard part: and that often boils down to understanding the mathematics and domain you are working in.

And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do.

My arsenal of tools: python, scipy/matplotlib, Mathematica, Matlab, various specialized solvers (eg, CPLEX, Z3). Mathematical arsenal: stats, probability, calculus, Fourier analysis, graph theory, PDEs, combinatorics.

(Context: Been doing data work for decades, before it got its recent “data science” name.)

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#23

Post by trollied »

What does "Data Scientist" actually mean these days? Does it mean "Write 10 lines of Python or R, and not fully understand what it actually does"? Or something else?

I just see the term flinged around so much recently, and applied to so many different roles, it has all become a tad blurred.

Maybe we need a Data Scientist to work out what a Data Scientist is?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#25

Post by jxub »

trollied wrote: post #23

What does "Data Scientist" actually mean these days? Does it mean "Write 10 lines of Python or R, and not fully understand what it actually does"? Or something else? I just see the term flinged around so much recently, and applied to so many different roles, it has all become a tad blurred. Maybe we need a Data Scientist to work out what a Data Scientist is?

That's a good meta reflection. Let's make an Y Combinator of Data Scientist a and Data Scientist b (recursive data scientist) to prove they can support recursion if Data Scientists a and b are first class functions, just because we can:

  const Y = a => (b => b(b))(b => a(x => b(b)(x)));

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#26

Post by gaius »

ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

Indeed. Too many people when asked about their skills or experience just rattle off a list of tools or libraries. Usually the same ones as everyone else!

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#27

Post by threeseed »

albertgoeswoof wrote: post #19
ktpsns wrote: post #5

Earlier quoted context omitted.

I make the claim that you can go very far in the SciPy ecosystem without ever touching R. It is worth understanding the concepts of numpy and pandas. Furthermore, try out IPython/Jupyter, especially for rapid publishing (people run their blogs on jupyter notebooks). I think certain libraries depend very much on where you focus. Machine learning? Native language processing? Visualization? Something in economics? Funda…

Agree, I would drop R, Python has you mostly covered now. Julia is also worth learning.

I wouldn't be recommending to drop R at all.

Very few enterprise data science teams are 100% Python (in fact none I've heard of). R is still very heavily used (and in fact all data science teams I've worked in it has been the dominant technology).

There is a reason Microsoft purchased Revolution.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#29

Post by threeseed »

trollied wrote: post #23

What does "Data Scientist" actually mean these days? Does it mean "Write 10 lines of Python or R, and not fully understand what it actually does"? Or something else? I just see the term flinged around so much recently, and applied to so many different roles, it has all become a tad blurred. Maybe we need a Data Scientist to work out what a Data Scientist is?

I hire data scientists so can tell you.

It means someone who can work with business stakeholders to break down a problem e.g. "we don't know why customers are churning", produce a machine learning model or some adhoc analysis (usually the former) and either communicate the results back or assist in deploying the model into production.

Typically there will be data engineers who will be doing acquisition and cleaning and so the data scientists are all about (a) understanding the data and (b) liaising with stakeholders.

As for technologies it is typically R/Python with Spark/H20 on top of a data lake i.e. HDFS, S3. Every now and again on top of an SQL store e.g. EDW, Presto or a Feature store e.g. Cassandra.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#30

Post by threeseed »

eggie5 wrote: post #4

a lot of people using spark?

Absolutely

Every single large scale data science team e.g. Google, Spotify, AirBnb will be using Spark for most of their work. It is by far the defacto standard for working with large datasets. Especially since it integrates so well with machine learning (H2O) and different languages (Scala, Python, R).

Post reply on HN