Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being a Data Scientist: My Experience and Toolset

jeffersonheard.github.io

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Being a Data Scientist: My Experience and Toolset

#11

Post by thinkr42 »

curiousgal wrote: post #7
lacampbell wrote: post #5

Earlier quoted context omitted.

I'm wondering how useful deriving functions yourself is in the age of computers. I feel like knowing axioms about the mathematical structure you're dealing with and how to do proofs is very important, but it always struck me as odd that were still stepping through complex applied maths functions manually in pen and paper. Programmers don't bother say, writing our own hashtable implementation more than a handful of ti…

>stepping through complex applied maths functions manually in pen and paper. We do that because: A it helps us understand them better B it teaches us how to think, the way Feynman said "Know how to solve every problem that has been solved". Granted, it seems pointless to work through what is easily accessible through machine BUT it teaches how to solve new problems. I wouldn't consider using NumPy or Matlab as the fi…

Completely agree. There's a lot of nuance in these algorithms, they're not as cut and dry as simply calling a package method and oftentimes they aren't optimized to your use case. I work in Machine Learning, specifically on NLP, and it is really obvious when interviewing potential employees who knows what SVD means and who just know the NumPy function. Most "data scientists" I've interviewed fall in the latter category.

edit-This is of course completely anecdotal experience.

Re: Being a Data Scientist: My Experience and Toolset

#12

Post by ianai »

paulgb wrote: post #10
mordant wrote: post #2

'Data scientist' is just title inflation by statisticians.

There are cases where this may be the case, but did you look at the tools in the blog post? Can statisticians be expected to write mongoDB code, create a web scraper, and make interactive visualizations in D3? Title inflation exists, but there is a real-world role here that isn't really captured by "statistician" at all.

If you're in a statistics program you're going to learn to code. That's been my experience anyway.

Re: Being a Data Scientist: My Experience and Toolset

#13

Post by thinkr42 »

laughfactory wrote: post #9
lacampbell wrote: post #5

Earlier quoted context omitted.

I'm wondering how useful deriving functions yourself is in the age of computers. I feel like knowing axioms about the mathematical structure you're dealing with and how to do proofs is very important, but it always struck me as odd that were still stepping through complex applied maths functions manually in pen and paper. Programmers don't bother say, writing our own hashtable implementation more than a handful of ti…

I agree. A smart data scientist doesn't waste their time reinventing the wheel: they build off the hard work of others. When necessary they can create what is needed, but they don't do so typically. They are both more and less, in my experience, than statisticians (more flexible and solution-oriented, less rigorous and classical), than analysts (they can do more, in general, but a great analyst will be better at anal…

I respectfully disagree. While I understand where you're coming from, I don't agree with your distinction between an analyst and a scientist. Given the data scientist's typical compensation and expected experience, there should be a higher bar set for them that does include developing solutions from base. I understand the use of utilities, but far too frequently I find people who rely on packages to do their work don't really understand what they're working on (they often don't realize the underlying assumptions that the package writers made for them either). With your description of the tasks for a data scientist, I would label this as a Data Analyst's work if I was hiring one.

I could of course be wrong and have a bit too narrow of a view from my particular subfield.

Re: Being a Data Scientist: My Experience and Toolset

#14

Post by codr4life »

Am I the only one who came here looking for someone's experience as a tool set? For a second there I thought I might have stumbled over real honesty, a rare treat these days. Maybe, if we stop putting each other in stupid labeled boxes to please our bullshit peddling masters, we would get somewhere...

Re: Being a Data Scientist: My Experience and Toolset

#15

Post by jeffheard »

ianai wrote: post #12
paulgb wrote: post #10

Earlier quoted context omitted.

There are cases where this may be the case, but did you look at the tools in the blog post? Can statisticians be expected to write mongoDB code, create a web scraper, and make interactive visualizations in D3? Title inflation exists, but there is a real-world role here that isn't really captured by "statistician" at all.

If you're in a statistics program you're going to learn to code. That's been my experience anyway.

I think it's great that students and young professors in the sciences are taught to code now. I've even taught some of them.

To me, data science is more than understanding statistics, it's been essential to know how to scale them up and out.

If you're a domain scientist, you won't necessarily learn how to write reusable tools that are performant (or runnable) on data that is different from your initial model data. I once worked with a group whose model had grown so unwieldy that their config file was in NetCDF.

I found my niche was often in doing things that were slightly (or completely) outside the comfort zone of most domain scientists who were competent coders themselves, but who didn't have the funded time nor the inclination to learn things like database, visualization, and networking technologies that became necessary either to share their work with other research groups or to operate on larger datasets.

One project had me take a big model that was normally run twice a day and on a 4km grid and help write something that could run and visualize the results of the same thing on a 0.5km grid over a larger area and hourly. And then devise something that could help them visually explore the timeseries as it evolved, sometimes over months.

Designing the pipeline that can handle that is outside the scope of most scientists, even the ones who are good coders.

Re: Being a Data Scientist: My Experience and Toolset

#16

Post by searine »

thinkr42 wrote: post #3
mordant wrote: post #2

'Data scientist' is just title inflation by statisticians.

More like 'analyst' in how easily it is thrown around. Calling a built in function in python or R is just about equivalent to calling one in Excel. Sure, you can claim that folks need to know more about what is going on, but honestly, how many have actually gone through the work of deriving the functions they're calling to begin with?

>how many have actually gone through the work of deriving the functions they're calling to begin with?

Why would you waste your time re-inventing a wheel.

A good data scientist isn't good because he/she can ace shitty trivia, he/she is good because they know the right question to ask.

Re: Being a Data Scientist: My Experience and Toolset

#17

Post by lacampbell »

curiousgal wrote: post #7
lacampbell wrote: post #5

Earlier quoted context omitted.

I'm wondering how useful deriving functions yourself is in the age of computers. I feel like knowing axioms about the mathematical structure you're dealing with and how to do proofs is very important, but it always struck me as odd that were still stepping through complex applied maths functions manually in pen and paper. Programmers don't bother say, writing our own hashtable implementation more than a handful of ti…

>stepping through complex applied maths functions manually in pen and paper. We do that because: A it helps us understand them better B it teaches us how to think, the way Feynman said "Know how to solve every problem that has been solved". Granted, it seems pointless to work through what is easily accessible through machine BUT it teaches how to solve new problems. I wouldn't consider using NumPy or Matlab as the fi…

I suppose my real question is - how many times do we need to do it? Once we have stepped through it by pen and paper once, or derived the result, how many times do we need to keep doing it? My experience in that mathematicians will do this again and again and again.

Re: Being a Data Scientist: My Experience and Toolset

#18

Post by Volt »

I'm not sure this is what a data scientist is. It was supposed to be a research scientist (which is where the scientist part came from) that wrangles data and code. This individual should have both domain knowledge and coding chops while knowing how to conduct research.

Re: Being a Data Scientist: My Experience and Toolset

#19

Post by 131012 »

Volt wrote: post #18

I'm not sure this is what a data scientist is. It was supposed to be a research scientist (which is where the scientist part came from) that wrangles data and code. This individual should have both domain knowledge and coding chops while knowing how to conduct research.

That would make me a data scientist, but I do not think I am and still have to learn a few tricks from this guy (and others).

Re: Being a Data Scientist: My Experience and Toolset

#20

Post by achompas »

These types of posts validate my concern about the people entering my field right now.

Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics. This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" companies).

Implicit in this definition is avoiding the destruction of business value by misapplying ML/statistics. In that sense, I am concerned about blog posts like these (which list 50 libraries and zero textbooks or papers) and those who comment arguing the relevance of "real math" in the era of computers.

Speaking bluntly: if you are a "data scientist" that can't derive a posterior distribution or explain the architecture of a neural network in rigorous detail, you're only going to solve easy problems amenable to black-box approaches. This is code for "toss things into pandas and throw sklearn at it". I would look for a separate line of work.

Post reply on HN