Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Being a Data Scientist: My Experience and Toolset

jeffersonheard.github.io

Post reply on HN
Print view View original

31–40 of 50 posts

Re: Being a Data Scientist: My Experience and Toolset

#31

Post by jeffheard »

achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

You know, I really should add a post soon about algorithms, papers, and textbooks. You make an important point which the first responder highlighted, "avoiding the destruction of business value by misapplying ML/statistics."

I understand the math behind what I do, but it's not a fair assumption to think that everyone reading my post will be motivated to pick up and understand the math before they start applying the tools.

Especially with tools like scikit-learn and orange, it's especially easy to misapply ML and statistics or simply approach a problem without understanding the tools and come out with something that looks plausible to the untrained eye.

Key to the reason that you should understand your tools, including the math that underlies them, is that you should be able to look at the results of your work and know if there's something "off". And beyond that the underlying understanding of the math involved gives you the tools you need to debug.

Re: Being a Data Scientist: My Experience and Toolset

#32

Post by SatvikBeri »

achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

I think the "Data Scientist" job title is overloaded–I see several clusters of skills being useful, and in my ideal world they would have similar but slightly different job titles:

–Medium Stats/ML, medium Engineering ("Data Scientist" or "Data Engineer")

–High Engineering on very large datasets, low/medium Stats/ML ("Data Engineer" or "Backend Engineer")

–High Analysis, medium Stats/ML, low Engineering ("Analyst")

–High traditional Stats, High Analysis, low ML/Engineering ("Statistician")

–High ML, medium Stats, medium Analysis ("Data Scientist")

–High ML, medium Engineering ("Machine Learning Engineer")

Re: Being a Data Scientist: My Experience and Toolset

#33

Post by jordz »

Cassandra is mentioned, I agree it's great for storing metadata and can be used to build efficient graph implementations but it's cited for Graphs and Relationships? I think that can be misleading as Cassandra is a a distributed column based key-value store.

Re: Being a Data Scientist: My Experience and Toolset

#34

Post by lacampbell »

achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

and those who comment arguing the relevance of "real math" in the era of computers.

Is this related to my comment? I used "age of computers", but close enough. It's really not a fair representation of what I said at all.

I stressed the importance of knowing theorems and deriving proofs - arguably "realer" math than learning an equation by rote. I did some applied maths in undergrad, and in my experience a lot of my time was devoted to solving large and complex equations using fairly mechanical rules, and comparatively little of my time was spent on axioms and proofs. I wonder whether this focus is justified in the age of computers - might we derive the complex formulas just once or twice as an exercise, and not step through them ourselves again and again? Might we focus more on what the computer can't do well for us - rigour and intuition?

Re: Being a Data Scientist: My Experience and Toolset

#35

Post by wenc »

jordz wrote: post #33

Cassandra is mentioned, I agree it's great for storing metadata and can be used to build efficient graph implementations but it's cited for Graphs and Relationships? I think that can be misleading as Cassandra is a a distributed column based key-value store.

I noticed that too. I don't want to gainsay the author's experiences, but it sounds like the author is describing the job of a data analyst who happens to dabble with various software. I don't get the sense the author has in-depth knowledge about the tools he lists.

Also, I don't know about putting Mongo and Cassandra under "Tools for working with unusual datasets".

Re: Being a Data Scientist: My Experience and Toolset

#36

Post by marketforlemmas »

achompas wrote: post #21
searine wrote: post #16

Earlier quoted context omitted.

>how many have actually gone through the work of deriving the functions they're calling to begin with? Why would you waste your time re-inventing a wheel. A good data scientist isn't good because he/she can ace shitty trivia, he/she is good because they know the right question to ask.

That's only part of it. A good data scientist is also good because they know how to answer hard questions. In those situations math isn't "shitty trivia," but instead a tool to be leveraged against those hard questions. You can consider the derivation of SVD to be shitty trivia while throwing np.linalg.svd around while engineering features. That's fine! Good luck visualizing that data in a meaningful way, or dealing…

> dealing with non-linear data

What is non-linear data?

Re: Being a Data Scientist: My Experience and Toolset

#37

Post by jupiter90000 »

achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

Part of the issue as I see it (for me, unrelated to the article), is that companies are willing to use the data scientist term for positions that need none of the rigor you mention. However, the people were hired and are now called a data scientist.

The same type of thing seems to happen in other fields, too. Software engineers who don't engineer, data scientists who don't 'science', project managers who don't manage. Are they top in their field? No, they somehow have a job with the title though and so far have managed to not become unemployable. Do they care if they are rigorous in what their title is expected to be by top practitioners? Probably not, they get paid still and have the title, and can probably get hired at the next similar place.

Kind of sad that these positions may 'cheapen' the title, so what can be done about that? Not much I guess, since companies can use position titles as they'd like it seems...

Re: Being a Data Scientist: My Experience and Toolset

#38

Post by avn2109 »

jupiter90000 wrote: post #37
achompas wrote: post #20

These types of posts validate my concern about the people entering my field right now. Data science, as a line of work, is distinct from other technical roles in its focus on creating business value using machine learning and statistics . This quality is easily observed in the most successful data scientists I've worked with (whether at unicorn startups, big companies like my current employer, or "mission-driven" com…

Part of the issue as I see it (for me, unrelated to the article), is that companies are willing to use the data scientist term for positions that need none of the rigor you mention. However, the people were hired and are now called a data scientist. The same type of thing seems to happen in other fields, too. Software engineers who don't engineer, data scientists who don't 'science', project managers who don't manage…

In my (admittedly short) experience as a data scientist, "solving the wrong problem"/"working on irrelevant things" and "inadequately cleaned/prepped training data" are vastly, overwhelmingly more common failure modes than "building the right thing with good data inputs but misunderstanding the algos." Probably more common by an order of magnitude or two.

Then again, maybe I'm just working at companies with problems that are amenable to easily-understood algos but have plenty of data-and-product-themed problems.

Re: Being a Data Scientist: My Experience and Toolset

#39

Post by jupiter90000 »

avn2109 wrote: post #38
jupiter90000 wrote: post #37

Earlier quoted context omitted.

Part of the issue as I see it (for me, unrelated to the article), is that companies are willing to use the data scientist term for positions that need none of the rigor you mention. However, the people were hired and are now called a data scientist. The same type of thing seems to happen in other fields, too. Software engineers who don't engineer, data scientists who don't 'science', project managers who don't manage…

In my (admittedly short) experience as a data scientist, "solving the wrong problem"/"working on irrelevant things" and "inadequately cleaned/prepped training data" are vastly, overwhelmingly more common failure modes than "building the right thing with good data inputs but misunderstanding the algos." Probably more common by an order of magnitude or two. Then again, maybe I'm just working at companies with problems…

Great points about issues faced before dealing with models/algos directly. Understanding something about the models/algos can help guide data prep too.

Re: Being a Data Scientist: My Experience and Toolset

#40

Post by bigger_cheese »

marketforlemmas wrote: post #36
achompas wrote: post #21

Earlier quoted context omitted.

That's only part of it. A good data scientist is also good because they know how to answer hard questions. In those situations math isn't "shitty trivia," but instead a tool to be leveraged against those hard questions. You can consider the derivation of SVD to be shitty trivia while throwing np.linalg.svd around while engineering features. That's fine! Good luck visualizing that data in a meaningful way, or dealing…

> dealing with non-linear data What is non-linear data?

Data derived from non linear inputs.

That is to say problems that can't be expressed by linear functions.

I.e. Y= mx + B is a linear function.

Y= ax^2 + bx + C is a polynomial (non linear) function.

Linear Programming (LP) involves solving a series of linear equations (something like Excel's Solver can do this).

When you are dealing with non linear functions you need to use a method such as Sequential Quadratic Programming (SQP).

Post reply on HN