Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data science tips and tricks from the developer community

blog.algorithmia.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Data science tips and tricks from the developer community

#51

Post by kolbe »

gipp wrote: post #50
kolbe wrote: post #45

Earlier quoted context omitted.

The whole time I was sarcastically thinking, "yeah I'm sure you get 10x by choosing consistent naming conventions. That will make up for the months of tearing your hair out trying to learn how ANNs work without a very solid understanding of math/stats."

90% of data scientists do not use neural networks. Of those who do, 90% shouldn't be, and are letting what's fun/interesting get in the way of actually producing value. The fact of the matter is that if you're not FB/GOOG/AMAZ, the vast majority of what companies need from their data scientists actually requires very little advanced mathematics, and much more focus on rigor, reproducibility, and good deployment/engin…

The job you're describing may have the title of Data Scientist, but it isn't data science if it doesn't involve advanced methods.

Re: Data science tips and tricks from the developer community

#52

Post by flaviuspopan »

chestervonwinch wrote: post #18
flaviuspopan wrote: post #11

Earlier quoted context omitted.

Nothing feels cleaner than storing everything (notebook, raw data, cleansed data, misc scripts, etc.) in a docker image when you're finished with the project. Data science and docker are meant to be besties.

I've never used Docker. Searching "reproducible research with Docker" yields lots of results. Any stand-out resource suggestions?

I would simply familiarize with just the basics because you don't have to go much further than that to make use of it for research purposes. My usual process involves breaking the process down into multiple stages (cleansing, conformance, testing, reporting), including a data dir, and finally creating a dockerfile that simply adds the data/source to a simple hierarchy and includes all dependencies. As long as you know how to build a dockerfile, you're golden. You can then upload the image to dockerhub, and have somebody else pull the image and run it to reproduce your entire environment. Helps a ton for online DS courses and MOOCs.

Re: Data science tips and tricks from the developer community

#53

Post by gipp »

kolbe wrote: post #51
gipp wrote: post #50

Earlier quoted context omitted.

90% of data scientists do not use neural networks. Of those who do, 90% shouldn't be, and are letting what's fun/interesting get in the way of actually producing value. The fact of the matter is that if you're not FB/GOOG/AMAZ, the vast majority of what companies need from their data scientists actually requires very little advanced mathematics, and much more focus on rigor, reproducibility, and good deployment/engin…

The job you're describing may have the title of Data Scientist, but it isn't data science if it doesn't involve advanced methods.

That's a semantic battle you've lost already.

Re: Data science tips and tricks from the developer community

#54

Post by nerdponx »

gipp wrote: post #50
kolbe wrote: post #45

Earlier quoted context omitted.

The whole time I was sarcastically thinking, "yeah I'm sure you get 10x by choosing consistent naming conventions. That will make up for the months of tearing your hair out trying to learn how ANNs work without a very solid understanding of math/stats."

90% of data scientists do not use neural networks. Of those who do, 90% shouldn't be, and are letting what's fun/interesting get in the way of actually producing value. The fact of the matter is that if you're not FB/GOOG/AMAZ, the vast majority of what companies need from their data scientists actually requires very little advanced mathematics, and much more focus on rigor, reproducibility, and good deployment/engin…

Agreed. But you could add "domain expertise" and "basic statistics" to that list as well.

Re: Data science tips and tricks from the developer community

#55

Post by carlmr »

CJefferson wrote: post #36
NumberCruncher wrote: post #35

Earlier quoted context omitted.

In my experience if you develop a "data science pipeline" forcing the data scientist to build - reproducible - validated - back-tested - easy to deploy models, they are going to hate it. It just kills the fun and/or makes obvious if they made a mistake.

I blame software. I don't understand why we couldn't have some system, perhaps using strace and friends, which tracks everything I ever do, and how every file was created. Then I could just say "how did I make X?"

Make it happen!

Re: Data science tips and tricks from the developer community

#56

Post by carlmr »

gipp wrote: post #49
NumberCruncher wrote: post #35

Earlier quoted context omitted.

In my experience if you develop a "data science pipeline" forcing the data scientist to build - reproducible - validated - back-tested - easy to deploy models, they are going to hate it. It just kills the fun and/or makes obvious if they made a mistake.

So we should sacrifice all the things that actually make a Data Scientist's work valuable in the name of fun and obscuring mistakes? Fun I almost get, obviously good for productivity (though I think you'd really be sacrificing productive output for non-productive output), but I just don't get where you're even coming from with the "making mistakes more obvious" angle.

I think he was being facetious. Of course we need all these things, but data science right now is still not that mature I guess.

Re: Data science tips and tricks from the developer community

#57

Post by carlmr »

KirinDave wrote: post #26
mattmanser wrote: post #19

A 10x developer is not ten times more productive than the average developer, they're 10x more productive than the worst developer. Wish this myth would stop perpetuating, they're very clear in the original study. Here's some more details from the horses mouth, Steve McConnell who popularized the concept in Code Complete: http://www.construx.com/10x_Software_Development/Origins_of_...

But also, even this data is questionable in the extreme. It may simply be that "10x" people who do exist do so in ways that are challenging to observe. As an example, not making difficult-to-detect mistakes early in the software lifecycle that cause major problems later (classic real world example: mongodb). Or that their influence on a software org causes overall productivity improvements. In any case, it's a toxic…

The real 10x developers in my experience are the hardest to measure. Because they pull the whole team by always being helpful and improving things where they see potential. But that doesn't necessarily show up in their results, but in the whole teams results.

Which is why metrics driven organizations in my experience with their disincentive to help others, slow everything down.

Re: Data science tips and tricks from the developer community

#58

Post by carlmr »

JPKab wrote: post #33

The first step to being a 10x data scientist: Know how to actually write code, and also understand a broad range of modeling approaches and the math behind them. The majority of people passing themselves off as data scientists in the traditional corporate world these days are at best unqualified and at worst outright frauds.

I.e. hire engineers and computer scientists who actually have the right math background and know how to build software.

Re: Data science tips and tricks from the developer community

#59

Post by mattmanser »

mikeyanderson wrote: post #44
mattmanser wrote: post #19

A 10x developer is not ten times more productive than the average developer, they're 10x more productive than the worst developer. Wish this myth would stop perpetuating, they're very clear in the original study. Here's some more details from the horses mouth, Steve McConnell who popularized the concept in Code Complete: http://www.construx.com/10x_Software_Development/Origins_of_...

Sure from a "task" stand point, but take quality, reusability, unique approaches, business sense, etc. and the best devs add easily 10x value if not more. Right?

Most of that comes from exposure to the code base, not innate skill. Experience.

Re: Data science tips and tricks from the developer community

#60

Post by mattmanser »

escribmac wrote: post #21
mattmanser wrote: post #19

A 10x developer is not ten times more productive than the average developer, they're 10x more productive than the worst developer. Wish this myth would stop perpetuating, they're very clear in the original study. Here's some more details from the horses mouth, Steve McConnell who popularized the concept in Code Complete: http://www.construx.com/10x_Software_Development/Origins_of_...

The worst developers I've worked with take 2 weeks for tickets that should be simple. That would mean doing 1 easy ticket every day or two makes you a 10x.... I've always hated this term and the mindset around it. I think organizational practices, intelligent engineering strategy, etc are far more important to the output of a team than hiring one genius dev.

Did it ever occur to you they might not be bad developers, they're just goofing off because there's no consequence for being slow?

Like when my old work actually started measuring ticket closure times, our best developers were only 2x more productive than our worst ones. But suddenly a lot more tickets were getting closed.

I mean,I know that some complicated tasks needed the best developers, as the worst ones literally were incapable of understanding the code, but then again doesn't that say something about the code itself and how poorly it communicates its intent? Perhaps clever code is simply confusing code...

Post reply on HN