Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: As a data scientist, what should be in my toolkit in 2018?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 177 posts

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#151

Post by ms013 »

fhk wrote: post #144
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

@ms013 interested to know how you are using the solvers, are you willing to share any further details?

Responded to someone else earlier about this. Used solvers for problems that end up requiring solutions to problems like minimum set cover or schedule optimization problems. Basically, problems where a naïve or brute force approach will take forever to run and you need to use a real solver to attack it. These usually are data problems that end up looking like what would traditionally be considered under the umbrella of operations research.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#152

Post by nrjames »

dermybaby wrote: post #107
nrjames wrote: post #60

There are two "poles" in data science: math/modeling and backend/data-wrangling. Most of the time, the backend/data-wrangling piece is a prerequisite to the math/modeling. The vast majority of small and medium sized companies have not set up the systems they would need to support a data scientist who knows only math/modeling. Depending on the domain, it's not uncommon to find that a small/medium company outsourced an…

So basic DBA skills + expert programming skills + very good math/stats? Also - your model of asking questions before starting a new gig is very relevant to nearly every programming job. Could also be some of the questions a candidate asks in an interview. Have you ever needed any Microsoft skills(MSSQL/C#) so far?

Yep, I’ve used MS SQL products and I write C# sometimes and read and write code to parse it very often because it is the primary language of the products I support.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#153

Post by CoVar »

edem wrote: post #85
CoVar wrote: post #65

Earlier quoted context omitted.

Usually MOOC for resume don't help as everyone does them. The advice that I found useful for resume building is working on projects that you can catalog in a portfolio. With regards to gaining math skills, this upcoming MOOC from Microsoft on EdX looks promising[1]. [1] https://www.edx.org/course/essential-mathematics-for-artific...

So you suggest that I should learn from MOOC then go on and work on some projects so that I can prove I really know it.

Exactly. And to take it one step further, choose one industry you are interested in. That way you will gain invaluable domain experience as you add relevant portfolio projects.

If you don't have an industry in mind, you can use a site like glassdoor.com and search for data scientist positions by city and industry to get a feel for demand.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#154

Post by gesman »

p33p wrote: post #56
ms013 wrote: post #22

Mathematics. Which branch of math is domain dependent. Stats come up everywhere. Graphs do too. In addition to baseline math, you really need to understand the problem domain and goals of the analysis. Languages and libraries are just tools: knowing APIs doesn’t tell you at all how to solve a problem. They just give you things to throw at a problem. You need to know a few tools, but to be honest, they’re easy and you…

And don’t over use viz. Pictures do effectively communicate, but often people visualize without understanding. The result is pretty pictures that eventually people realize communicate little effective domain insight. You’d be surprised that sometimes simple and ugly pictures communicate more insight than beautiful ones do. I don't necessarily agree with this. Yes, a sound understanding of the domain and knowledge of…

Good viz is what connects non-ML/AI users to the "magical" results of ML/AI

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#155

Post by edem »

CoVar wrote: post #153
edem wrote: post #85

Earlier quoted context omitted.

So you suggest that I should learn from MOOC then go on and work on some projects so that I can prove I really know it.

Exactly. And to take it one step further, choose one industry you are interested in. That way you will gain invaluable domain experience as you add relevant portfolio projects. If you don't have an industry in mind, you can use a site like glassdoor.com and search for data scientist positions by city and industry to get a feel for demand.

Full disclosure: I'm in the industry for 10+ years as a programmer. I just realized that if I want to move in the AI direction I'll need some math education. I don't want to become a data scientist.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#156

Post by gkya »

jonathankoren wrote: post #21
Ultimatt wrote: post #15

Earlier quoted context omitted.

I'd gently suggest basic CLI Perl over BASH for cleaning up files, as it combines grep/sed/awk in a language thats more generally useful.

Agreed. Perl was designed for text munging, and is superior to pretty much everything for this task. WRT bash, where to begin? In the past 40 years, there’s pretty much a better tool for everything someone tries to do with bash. It lives on pretty much through inertia and pride.

FreeBSD sh(1) (not bash(1)) man page. That's just how I understood how to shell. Nowadays I'm running Debian and my $SHELL is /bin/bash, but when I was on FreeBSD I really learnt tools like make(1), sh(1); the man pages were pieces of art. Having read sh(1), I do have a nice grasp of how shell works in general, to which knowledge I can add anytime the higher-level goodies bash has to offer (though I generally prefer keeping it POSIX, and using an actual programming language when it doesn't cut it).

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#157

Post by charliej2 »

red_hare wrote: post #69
booleandilemma wrote: post #64

Earlier quoted context omitted.

I think data scientist, much like software engineer, is something you can call yourself without having any credentials whatsoever. It’s why technical interviews can be so brutal, unfortunately. There are a lot of frauds out there. Money attracts frauds. What’s the fizzbuzz test for data scientists anyway?

I'm a data engineer for a startup that's trying to hire its first data scientist. The range of candidates that apply with this title is massive. Defining our expectations has been challenging. My phone screen "fizzbuzz" is having them calculate a standard deviation from an array of data w/out with only basic operators (no numpy.std). Then explain why they choose population/sample and explain the difference. I studied…

This was useful info for a noob, thanks. It makes sense to me. If you understand SD in principle, you don't need to memorize the formula for a simple exercise like this, so it's a good filter for candidates who can demonstrate they do understand some basic math.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#158

Post by throwaway7645 »

throwaway7645 wrote: post #73
Aardwolf wrote: post #67

Earlier quoted context omitted.

R is not present in your list, did you ever try it and what's your opinion about it?

If you have Mathematica, you might not need R as both are like Swiss Army Chainsaws for Data Analytics.

Not sure y I was downvoted here. I've used both products in this problem area. Mathematica is definitely more than a CAS. Both are great in their own ways.

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#159

Post by petemir »

CoVar wrote: post #65
edem wrote: post #49

Earlier quoted context omitted.

I really like to get a degree in Mathematics but I simply don't have the time to throw at it (work, children, etc). What do you suggest I should do to have something on my resume? MOOC maybe?

Usually MOOC for resume don't help as everyone does them. The advice that I found useful for resume building is working on projects that you can catalog in a portfolio. With regards to gaining math skills, this upcoming MOOC from Microsoft on EdX looks promising[1]. [1] https://www.edx.org/course/essential-mathematics-for-artific...

Link didn't work for me. This one [1] did.

[1] https://www.edx.org/course/essential-mathematics-for-artific...

Re: Ask HN: As a data scientist, what should be in my toolkit in 2018?

#160

Post by 0kto »

chubot wrote: post #96
Xcelerate wrote: post #51

As a data scientist who has been using the language for 5 years now, Julia is by far the best programming language for analyzing and processing data. That said, it’s common to find many Julia packages that are only half-maintained and don’t really work anymore. (I still don’t know how to connect to Postgres in a bug-free way using Julia.) And you’d be hard pressed to find teams of data scientists that use Julia. So i…

Has Julia converged on a solution for data frames? I watched some JuliaCon videos and got the impression that they hadn't. There seem to be a lot of different overlapping efforts.

Well, only the DataFrames.jl package comes to my mind. However, there exist a few packages that extend this package (DataFramesMeta.jl or Query.jl; these overlap to some extend, but the newer Query package seems to go beyond DataFrames and offers some piping capabilities to interface with plotting packages). In general: During the three years of my PhD some language / package upgrades broke some of my scripts (during 0.4 -> 0.5 and -> 0.6), but the language (and its extensive documentation, online and from the source code of the packages) is very pleasant to use - the deprecation warnings usually help you to adjust your code in time. I have been relaying heavily on said DataFrame package, and am quite happy - the community is usually responsive and helpful in case of problems or questions.
Post reply on HN