31–40 of 177 posts
IPython/Jupyter, Pandas/Numpy and Python will get you everywhere you need to go. Currently, until maybe Go gets decent DataFrame support, in terms of the total time to get to your solution, I'd be amazed if any other setup got you there quicker.
No it won't.
That combination can't handle large datasets that are typical for most data science teams i.e. maybe include PySpark. And then it's very limited so far as ML/DL technologies.
You probably mean "data analyst". "Data scientist" title would apply only if you are applying scientific method to discover new fact about natural world exclusively through data analysis (as opposed to observation and experiments).
The analysis part is usually quite simple, often if it gets really complex then that's a sign that the data is being tortured. Sometimes the marginal gains that complex methods create (vs simple but good approaches) are not worthwhile even if they are valid - simply in terms of time spent and difficulty in communications.
IPython/Jupyter, Pandas/Numpy and Python will get you everywhere you need to go. Currently, until maybe Go gets decent DataFrame support, in terms of the total time to get to your solution, I'd be amazed if any other setup got you there quicker.
> get you everywhere you need to go No it won't. That combination can't handle large datasets that are typical for most data science teams i.e. maybe include PySpark. And then it's very limited so far as ML/DL technologies.
Pandas and Spark are both DataFrame libraries, and seem to offer very similar functionality to me. Why do you prefer Spark over Pandas?
> very limited so far as ML/DL technologies
I mean, getting Tensorflow up and running with GPU support isn't trivial, but it's not exactly hard, and Keras[1] provides excellent support for a wide variety of other backends. What, in your experience, is less limited?
[1]: https://keras.io/
A couple of thoughts, off the top of my head: Programming languages: - python (for general purpose programming) - R (for statistics) - bash (for cleaning up files) - SQL (for querying databases) Tools: - Pandas (for Python) - RStudio (for R) - Postgres (for SQL) - Excel (the format your customers will want ;-) ) Libraries: - SciPy (ecosystem for scientific computing) - NLTK (for natural language) - D3.js (for renderi…
I make the claim that you can go very far in the SciPy ecosystem without ever touching R. It is worth understanding the concepts of numpy and pandas. Furthermore, try out IPython/Jupyter, especially for rapid publishing (people run their blogs on jupyter notebooks). I think certain libraries depend very much on where you focus. Machine learning? Native language processing? Visualization? Something in economics? Funda…
If you're doing a lot of work with matrices, model fitting in production, then python seems fine. However, a lot of data scientists I see are more like scrappy data analysis / visualization types, who are churning out small dashboards. In that case R's tidy verse and shiny are just incredibly fast to develop with.
1. You need research skills that will allow you to ask the right questions, define the problem and put it in a mathematical framework.
2. Familiarity with math (which? depends on what you are doing) to the point where you can read articles that may have a solution to your problem and the ability to propose changes, creating proprietary algorithms.
3. Some scripting language (Python, R, w/e)
4. (optional) Software Engineering skills. Can you put your model into production? Will your algorithm scale? Etc.
* statistical methods (more math)
* big, in-production model fitting (more python)
* quick, scrappy data analyses for internal use (more R)
For example, I would feel weird writing a robust web server in R, but it's straightforward in python. On the other hand R's shiny lets you put up quick, interactive web dashboards (that I wouldn't trust in exposing to users).
Libs: - Dask for distributed processing - matplotlib/seaborn for graphing - IPython/Jupyter for creating shareable data analyses
Environment: - S3 for data warehousing, I mainly use parquet files with pyarrow/fastparquet - EC2 for Dask clustering - Ansible for EC2 setup
My problems usually can be solved by 2 memory-heavy EC2 instances. This setup works really well for me. Reading and writing intermediate results to S3 is blazing fast, especially when partitioning data by days if you work with time series.
Lots of difficult problems require custom mapping functions. I usually use them together with dask.dataframe.map_partitions, which is still extremely fast.
The most time-consuming activity is usually nunique/unique counting across large time series. For this, Dask offers hyperloglog based approximations.
To sum it up, Dask alone makes all the difference for me!
A couple of thoughts, off the top of my head: Programming languages: - python (for general purpose programming) - R (for statistics) - bash (for cleaning up files) - SQL (for querying databases) Tools: - Pandas (for Python) - RStudio (for R) - Postgres (for SQL) - Excel (the format your customers will want ;-) ) Libraries: - SciPy (ecosystem for scientific computing) - NLTK (for natural language) - D3.js (for renderi…
Any programming language that you are proficient in. A solid understanding how a computer works. Solid basis of statistics. Anything else is just sprinkles, trends and field-specific.