Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Data scientists, what tools do you use the most on a daily basis?

news.ycombinator.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Ask HN: Data scientists, what tools do you use the most on a daily basis?

#11

Post by syllogism »

usgroup wrote: post #10

Out of interest, is everyone using Spark/Hadoop/etc because you need to and/or you chose to or another reason? IMO, legit use cases seem to me relatively sparse.

Let's say you're developing your analysis script, and it takes 45 minutes to run. That doesn't sound like much of a problem, but it means you only really get to try 10 things a day. You can kick something off and then start trying something different, but that's actually quite a burden. It's cognitively easier to run a series of parallel tasks instead of parallel serial tasks. Even if a job is taking 15-20 minutes, that can really wear you down, and give you a pretty unproductive week. It's hard to keep your place.

That said: I think often these 45 minute tasks should be much faster, and folks would actually benefit from optimising their code a bit more. If I have to do a word count in 30gb of text, I'd rather implement carefully or use Hyper Log Log. I find it weird that people would rather use Spark, which I always find quite painful. I guess it comes down to familiarity.

Re: Ask HN: Data scientists, what tools do you use the most on a daily basis?

#12

Post by dagw »

Much of the data I analyse has a spatial/geometric/geographic component and for that I wouldn't want to do my job without FME. In fact it's probably may one of my all time favorite pieces of software.

Otherwise jupyter with numpy and all the other great python libraries is where I spend much of my time.

Re: Ask HN: Data scientists, what tools do you use the most on a daily basis?

#13

Post by lovelearning »

laingc wrote: post #6

An answer in two parts: The software tools: For me, PyCharm and Jupyter for development, Dask for data preprocessing, Tensorflow, Theano, and various scipy libraries for ML, as well as helpers such as PyMC3 and Edward for probabilistic programming. The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to…

Thank you for a very interesting and informative answer.

When you say "PhD in applied mathematics taught me how to abstract and codify with math, and to approach numerically and computationally", do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch?

I too would like to have such skills one day, but I don't have the opportunities or time for a PhD. Do you feel it can be self taught to some extent?

Re: Ask HN: Data scientists, what tools do you use the most on a daily basis?

#14

Post by achompas »

laingc wrote: post #6

An answer in two parts: The software tools: For me, PyCharm and Jupyter for development, Dask for data preprocessing, Tensorflow, Theano, and various scipy libraries for ML, as well as helpers such as PyMC3 and Edward for probabilistic programming. The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to…

The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to approach them numerically and computationally.

What is a first approximation to this experience, particularly the creative thinking portion? As someone with a MS but without a PhD, I suspect you get to flex this ability during the research phase of your studies.

Re: Ask HN: Data scientists, what tools do you use the most on a daily basis?

#15

Post by laingc »

achompas wrote: post #14
laingc wrote: post #6

An answer in two parts: The software tools: For me, PyCharm and Jupyter for development, Dask for data preprocessing, Tensorflow, Theano, and various scipy libraries for ML, as well as helpers such as PyMC3 and Edward for probabilistic programming. The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to…

The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to approach them numerically and computationally. What is a first approximation to this experience, particularly the creative thinking portion? As someone with a MS but without a PhD, I suspect you get to flex this ability during the research phase of…

I'm note really sure what a good first approximation would be, but maybe I can break down the elements of it a bit, and we can see whether there is some other experience that mimics it.

* Spending several years, uninterrupted and undistracted, thinking deeply about a mathematical topic.

* Being advised and supported, but not led or coddled, by an experienced researcher.

* Being surrounded by clever people, who do research full-time.

* Working in an environment where not knowing the answer. immediately, or even quickly, is not just accepted by expected.

* Being "left in the dark", with the understanding that the process of being in that position and finding your way out is the very thing that you are trying to learn.

* Having rigorous expectations for the "final result" (to the extent that one can be found).

I can't think of a great approximation to that, but perhaps someone else can chime in.

EDIT0: Formatting

EDIT1: Also worth noting that by far the smartest person I know - a deep thinker, also nominally a data scientist, but foremost a truly original and creative mathematical thinker - didn't complete his PhD and left with a Masters. Intellectually, this guy makes me look like a geriatric Corgi on opium, and he didn't need a PhD to make him that way.

Re: Ask HN: Data scientists, what tools do you use the most on a daily basis?

#16

Post by laingc »

lovelearning wrote: post #13
laingc wrote: post #6

An answer in two parts: The software tools: For me, PyCharm and Jupyter for development, Dask for data preprocessing, Tensorflow, Theano, and various scipy libraries for ML, as well as helpers such as PyMC3 and Edward for probabilistic programming. The important tool: a PhD in applied mathematics that taught me how to think originally and creatively about problems, to abstract and codify them with mathematics, and to…

Thank you for a very interesting and informative answer. When you say "PhD in applied mathematics taught me how to abstract and codify with math, and to approach numerically and computationally", do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch? I too would like to have such skills one day, but I don't have…

> do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch?

I really think that the PhD experience was the major factor in me developing this, though I think I showed a certain promise earlier on. I think it's certainly true that, without the PhD, I wouldn't have known how to go about the process of learning and thinking in the way that I do now.

> Do you feel it can be self taught to some extent?

I think you'd have to be a much smarter and more dedicated person than I am to achieve the same results without the environment, and as I mentioned in a cousin comment, I think that environment might be tough to replicate. I am a big proponent of formal education for exactly this reason. Academia certainly has many flaws, and I'm a harsh critic of them, but I personally experienced what I consider to be the best side of it, and it's this side that formed the opinion that I'm expressing here.

EDIT0: Repeating what I posted in a cousin comment: Also worth noting that by far the smartest person I know - a deep thinker, also nominally a data scientist, but foremost a truly original and creative mathematical thinker - didn't complete his PhD and left with a Masters. Intellectually, this guy makes me look like a geriatric Corgi on opium, and he didn't need a PhD to make him that way.

Re: Ask HN: Data scientists, what tools do you use the most on a daily basis?

#17

Post by lovelearning »

laingc wrote: post #16
lovelearning wrote: post #13

Earlier quoted context omitted.

Thank you for a very interesting and informative answer. When you say "PhD in applied mathematics taught me how to abstract and codify with math, and to approach numerically and computationally", do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch? I too would like to have such skills one day, but I don't have…

> do you feel you already had good mathematical intuition earlier too and the PhD made it better? Or did the PhD by itself help you build such skills from scratch? I really think that the PhD experience was the major factor in me developing this, though I think I showed a certain promise earlier on. I think it's certainly true that, without the PhD, I wouldn't have known how to go about the process of learning and th…

Thank you, laingc. Due to my circumstances, I wish the answer was not formal education. But I'll remember your experience and advice, and see if I can go that route atleast in the future. Thank you very much for replying, have a great day!

Re: Ask HN: Data scientists, what tools do you use the most on a daily basis?

#18

Post by eshvk »

usgroup wrote: post #10

Out of interest, is everyone using Spark/Hadoop/etc because you need to and/or you chose to or another reason? IMO, legit use cases seem to me relatively sparse.

I run models on large scale user data (for recommendations). A couple of hundred gigs is what "cleaned" data looks like. This might need to be joined with metadata. This cleaning, joining is easier to do on Map Reduce than wait a few days on a beefy computational machine.
Post reply on HN