Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being a Data Scientist: My Experience and Toolset

jeffersonheard.github.io

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Being a Data Scientist: My Experience and Toolset

#41

Post by mastazi »

From the article:

> Machine learning and data mining are not well distinguished, but machine learning techniques increasingly favor “unsupervised” learning algorithms.

The statement above puzzles me because it does not align with what I can see in the news. Maybe I'm just uninformed, so please let me know if I'm wrong.

According to what I can read in the news:

1 - Almost all of the recent ML developments that I can think of are in the field of supervised learning / reinforcement learning

2 - the only field that I can think of where unsupervised learning techniques are prevalent is data mining, which is precisely why I see it as a very specific field.

Am I missing something?

Re: Being a Data Scientist: My Experience and Toolset

#42

Post by tangue »

SatvikBeri wrote: post #32
achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

I think the "Data Scientist" job title is overloaded–I see several clusters of skills being useful, and in my ideal world they would have similar but slightly different job titles: –Medium Stats/ML, medium Engineering ("Data Scientist" or "Data Engineer") –High Engineering on very large datasets, low/medium Stats/ML ("Data Engineer" or "Backend Engineer") –High Analysis, medium Stats/ML, low Engineering ("Analyst") –…

One of the lessons of the web (in the 1990s everyone was a webmaster until the field mature.) is that after the coders, specialists emerged in fields like design, management, UX, seo and content. For data science the most obvious is data visualization but I guess there's plenty of new jobs ahead in addition to core data science jobs.

Re: Being a Data Scientist: My Experience and Toolset

#43

Post by cityhall »

mastazi wrote: post #41

From the article: > Machine learning and data mining are not well distinguished, but machine learning techniques increasingly favor “unsupervised” learning algorithms. The statement above puzzles me because it does not align with what I can see in the news. Maybe I'm just uninformed, so please let me know if I'm wrong. According to what I can read in the news: 1 - Almost all of the recent ML developments that I can t…

No, you're right. Nothing about this blog post/resume inspires confidence.

Re: Being a Data Scientist: My Experience and Toolset

#44

Post by di4na »

tangue wrote: post #42
SatvikBeri wrote: post #32

Earlier quoted context omitted.

I think the "Data Scientist" job title is overloaded–I see several clusters of skills being useful, and in my ideal world they would have similar but slightly different job titles: –Medium Stats/ML, medium Engineering ("Data Scientist" or "Data Engineer") –High Engineering on very large datasets, low/medium Stats/ML ("Data Engineer" or "Backend Engineer") –High Analysis, medium Stats/ML, low Engineering ("Analyst") –…

One of the lessons of the web (in the 1990s everyone was a webmaster until the field mature.) is that after the coders, specialists emerged in fields like design, management, UX, seo and content. For data science the most obvious is data visualization but I guess there's plenty of new jobs ahead in addition to core data science jobs.

the second most obvious is cleaner. 90% of time spent when i try to do any analysis is just spent trying to get data that are usable, or in an usable shape.

And let's not talk about the different CSV you can encounter... or how to get data out of databases that are literally the only reason some "guardian DBA" still have a job. It can take months to get any access, if you ever get it...

Re: Being a Data Scientist: My Experience and Toolset

#46

Post by nonbel »

jeffheard wrote: post #31
achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

You know, I really should add a post soon about algorithms, papers, and textbooks. You make an important point which the first responder highlighted, "avoiding the destruction of business value by misapplying ML/statistics." I understand the math behind what I do, but it's not a fair assumption to think that everyone reading my post will be motivated to pick up and understand the math before they start applying the t…

I propose you can basically monte carlo yourself to a decent understanding.

The disadvantage is: You never know you are right for sure, plus there is extra time spent on applying your experience to each new type of problem.

The advantage is: You can easier relax assumptions once it is set up, and learned heuristics to deal with new problems quicker than the perfect way.

Re: Being a Data Scientist: My Experience and Toolset

#47

Post by achompas »

lacampbell wrote: post #34
achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

and those who comment arguing the relevance of "real math" in the era of computers. Is this related to my comment? I used "age of computers", but close enough. It's really not a fair representation of what I said at all. I stressed the importance of knowing theorems and deriving proofs - arguably "realer" math than learning an equation by rote. I did some applied maths in undergrad, and in my experience a lot of my t…

> Is this related to my comment?

It was initially related, yeah, but I realized I had uncharitably read your point. I edited my comment, but not enough. Sorry about that.

To be fair, this point is often raised in these threads as "why do math when computers do it for us?" so the criticism wasn't specifically levied against you.

We agree that repeated derivation when working on a new problem can be useless. It would be silly to work out OLS assumptions from first principles upon any import of sklearn.linear_model! I believe understanding those assumptions, though, or (say) how backpropagation works is important, since (1) it can help you debug issues and (2) explain modifications to the core models (GLMs or LSTMs, in the above examples).

Re: Being a Data Scientist: My Experience and Toolset

#48

Post by q_revert »

marketforlemmas wrote: post #36
achompas wrote: post #21

Earlier quoted context omitted.

That's only part of it. A good data scientist is also good because they know how to answer hard questions. In those situations math isn't "shitty trivia," but instead a tool to be leveraged against those hard questions. You can consider the derivation of SVD to be shitty trivia while throwing np.linalg.svd around while engineering features. That's fine! Good luck visualizing that data in a meaningful way, or dealing…

> dealing with non-linear data What is non-linear data?

Using a term like nonlinear science is like referring to the bulk of zoology as the study of non-elephant animals.

— Stanislaw Ulam

https://en.wikipedia.org/wiki/Nonlinear_system

Re: Being a Data Scientist: My Experience and Toolset

#49

Post by milliondollar »

Declanomous wrote: post #26
achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

> Implicit in this definition is avoiding the destruction of business value by misapplying ML/statistics This is an incredibly important point. I'm working as a fundraising and marketing analyst for a non-profit, but my background is in biology. The skill-set needed for analysis is pretty similar between marketing and population ecology. If you ask someone in either field what the biggest barrier to analysis is, gett…

Speaking as a business person: often the biggest challenge is to make ANY decision and actually DO something. The perfect is the enemy of the good. So to continue the cliches the business critique of your objections would be "analysis paralysis."

I tell you this just to help you understand what you describe. But in my observations of failure modes in business, it is rarely because one follows the wrong analysis, but more because most are unwilling to make any changes unless confronted with overwhelming evidence. (And that hurdle always gets higher no matter how much evidence you give.)

Re: Being a Data Scientist: My Experience and Toolset

#50

Post by Declanomous »

milliondollar wrote: post #49
Declanomous wrote: post #26

Earlier quoted context omitted.

> Implicit in this definition is avoiding the destruction of business value by misapplying ML/statistics This is an incredibly important point. I'm working as a fundraising and marketing analyst for a non-profit, but my background is in biology. The skill-set needed for analysis is pretty similar between marketing and population ecology. If you ask someone in either field what the biggest barrier to analysis is, gett…

Speaking as a business person: often the biggest challenge is to make ANY decision and actually DO something. The perfect is the enemy of the good. So to continue the cliches the business critique of your objections would be "analysis paralysis." I tell you this just to help you understand what you describe. But in my observations of failure modes in business, it is rarely because one follows the wrong analysis, but…

>most are unwilling to make any changes unless confronted with overwhelming evidence.

That's probably the second most common problem. I'd say 80% of my job is just fighting confirmation bias. So if someone thinks something needs to be changed, they'll take any sign that it should be changed. If someone thinks something should stay the same way, they'd argue with god about it.

I probably propose changes more often than I propose keeping things the same way, if only because testing an idea and gathering information requires making a change somewhere. I have a lot of conversations with people who are pressuring me to make a conclusion that the current way is best as soon as possible, so they can throw a lot of money at their pet project.

I'd say that most of the claims I'm being asked to make with limited evidence would be supporting the status quo, which is in line with your assessment.

Post reply on HN