Ask HN: As a data scientist, what should be in my toolkit in 2018?

#1

Post by mxgr » Tue, Feb 20, 2018, 7:51 AM UTC

No post body was provided.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#2

Post by trevz » Tue, Feb 20, 2018, 8:56 AM UTC

A couple of thoughts, off the top of my head:

Programming languages:

  - python (for general purpose programming)
  - R (for statistics)
  - bash (for cleaning up files)
  - SQL (for querying databases)

Tools:

  - Pandas (for Python)
  - RStudio (for R)
  - Postgres (for SQL)
  - Excel (the format your customers will want ;-) )

Libraries:

  - SciPy (ecosystem for scientific computing)
  - NLTK (for natural language)
  - D3.js (for rendering results online)

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#3

Post by jxub » Tue, Feb 20, 2018, 9:00 AM UTC

trevz wrote: post #2

A couple of thoughts, off the top of my head: Programming languages: - python (for general purpose programming) - R (for statistics) - bash (for cleaning up files) - SQL (for querying databases) Tools: - Pandas (for Python) - RStudio (for R) - Postgres (for SQL) - Excel (the format your customers will want ;-) ) Libraries: - SciPy (ecosystem for scientific computing) - NLTK (for natural language) - D3.js (for renderi…

Maybe SpaCy for NLP. Way more intuitive and fast too. Good list.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#4

Post by eggie5 » Tue, Feb 20, 2018, 9:05 AM UTC

a lot of people using spark?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#5

Post by ktpsns » Tue, Feb 20, 2018, 9:07 AM UTC

trevz wrote: post #2

A couple of thoughts, off the top of my head: Programming languages: - python (for general purpose programming) - R (for statistics) - bash (for cleaning up files) - SQL (for querying databases) Tools: - Pandas (for Python) - RStudio (for R) - Postgres (for SQL) - Excel (the format your customers will want ;-) ) Libraries: - SciPy (ecosystem for scientific computing) - NLTK (for natural language) - D3.js (for renderi…

I make the claim that you can go very far in the SciPy ecosystem without ever touching R.

It is worth understanding the concepts of numpy and pandas. Furthermore, try out IPython/Jupyter, especially for rapid publishing (people run their blogs on jupyter notebooks).

I think certain libraries depend very much on where you focus. Machine learning? Native language processing? Visualization? Something in economics? Fundamental sciences? For instance, I never need NLTK in theoretical astrophysics ;-) Instead, I need powerful GPU based visualization, which is however very old school with VTK and Visit/Amira/Paraview (also very much pythonic).

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#6

Post by greyman » Tue, Feb 20, 2018, 9:09 AM UTC

If you will work in some bigger company doing data analytics, you can also come across Tableau instead of Excel. Apart from SQL, if there is more data, you might want to use Bigquery or something similar.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#7

Post by latenightcoding » Tue, Feb 20, 2018, 9:17 AM UTC

If you use Python: scikit-learn, Pandas, NumPy, Tensorflow or PyTorch

Language agnostic: XGBoost, LibLinear, Apache Arrow, MXNet

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#8

Post by ever1 » Tue, Feb 20, 2018, 9:22 AM UTC

Python: Jupyter, pandas, numpy, scipy, scikit-learn

Numba for custom algorithms.

Dataiku (amazing tool for preprocessing and complex flows)

Amazon RDS (postgress), but thinking about redshift.

Spark

Tableau or plotly/seaborn

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#9

Post by sandGorgon » Tue, Feb 20, 2018, 9:24 AM UTC

eggie5 wrote: post #4

a lot of people using spark?

same question that i have. Anyone using pyspark in production ?

Would you use pyspark mllib in a webservice instead of scikit ?

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#10

Post by eps » Tue, Feb 20, 2018, 9:27 AM UTC

You probably mean "data analyst".

"Data scientist" title would apply only if you are applying scientific method to discover new fact about natural world exclusively through data analysis (as opposed to observation and experiments).