Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data science tips and tricks from the developer community

blog.algorithmia.com

Post reply on HN
Print view View original

31–40 of 69 posts

Re: Data science tips and tricks from the developer community

#31

Post by KirinDave »

delazeur wrote: post #17
gilbetron wrote: post #15

Earlier quoted context omitted.

Is there a good solution to that problem, though? (Serious question). I recently did a laptop refresh and am using it as an opportunity to solidify my approach to ML development, and would love to hear if there is a good solution to long-term reproducibility. I'm currently leaning towards Docker, but maybe Vagrant or another "pure" VM approach is better...

Carve it onto stone tablets.

You joke it, but it's a major problem that our tech for very stable WORM media has lagged behind demand.

Our use of data has grown so much faster than our network capacity (and indeed, it seems like we're going to hit a series of physical laws and practical engineering constraints here). "Data has gravity" but the only way to "sustainable" hold a non-trivial volume of data for 20 years right now is to run a data center with a big dht that detects faults and replicates data.

Re: Data science tips and tricks from the developer community

#32

Post by beagle3 »

mattmanser wrote: post #19

A 10x developer is not ten times more productive than the average developer, they're 10x more productive than the worst developer. Wish this myth would stop perpetuating, they're very clear in the original study. Here's some more details from the horses mouth, Steve McConnell who popularized the concept in Code Complete: http://www.construx.com/10x_Software_Development/Origins_of_...

The worst programmers in a team sometimes have a negative contribution....

Re: Data science tips and tricks from the developer community

#33

Post by JPKab »

The first step to being a 10x data scientist:

Know how to actually write code, and also understand a broad range of modeling approaches and the math behind them.

The majority of people passing themselves off as data scientists in the traditional corporate world these days are at best unqualified and at worst outright frauds.

Re: Data science tips and tricks from the developer community

#34

Post by drvdevd »

minimaxir wrote: post #2

It's worth nothing that the article does not discuss the reproducibility of results (e.g. with a Jupyter Notebook) and the implementation of said results (e.g. deploying/validating models), both of which matter much more than any code style conventions for data-related projects.

Thus making you a 1x Data Scientist because your result can only be demonstrated once? ;)

Re: Data science tips and tricks from the developer community

#35

Post by NumberCruncher »

minimaxir wrote: post #2

It's worth nothing that the article does not discuss the reproducibility of results (e.g. with a Jupyter Notebook) and the implementation of said results (e.g. deploying/validating models), both of which matter much more than any code style conventions for data-related projects.

In my experience if you develop a "data science pipeline" forcing the data scientist to build

- reproducible

- validated

- back-tested

- easy to deploy

models, they are going to hate it. It just kills the fun and/or makes obvious if they made a mistake.

Re: Data science tips and tricks from the developer community

#36

Post by CJefferson »

NumberCruncher wrote: post #35
minimaxir wrote: post #2

It's worth nothing that the article does not discuss the reproducibility of results (e.g. with a Jupyter Notebook) and the implementation of said results (e.g. deploying/validating models), both of which matter much more than any code style conventions for data-related projects.

In my experience if you develop a "data science pipeline" forcing the data scientist to build - reproducible - validated - back-tested - easy to deploy models, they are going to hate it. It just kills the fun and/or makes obvious if they made a mistake.

I blame software.

I don't understand why we couldn't have some system, perhaps using strace and friends, which tracks everything I ever do, and how every file was created. Then I could just say "how did I make X?"

Re: Data science tips and tricks from the developer community

#37

Post by cjslep »

arcanus wrote: post #10
minimaxir wrote: post #2

It's worth nothing that the article does not discuss the reproducibility of results (e.g. with a Jupyter Notebook) and the implementation of said results (e.g. deploying/validating models), both of which matter much more than any code style conventions for data-related projects.

^ this. I cannot describe how many times I've been shown results and when asking how to reproduce them, after several notes (and sometimes complaints to higher ups) I eventually get a series of command line arguments or a barely functioning R-script. These conclusions are too important to be so sloppily produced. We need verification, validation and uncertainty quantification for any result provided to decision maker…

Learning uncertainty propagation in engineering statistics was one of those concepts that seemed to be immediately useful and have far more implications than any textbooks emphasized.

I was very happy to have that background when I took part developing statistical models used for wind hazard analysis on nuclear powerplants in my first job out of college.

Re: Data science tips and tricks from the developer community

#38

Post by wayfarer2s »

the_af wrote: post #4

Is writing docstrings with argument types a thing in Python? If so, wouldn't these developers benefit from using actual type annotations (or a language with static types)? This is one area where types actually help a great deal with rapid prototyping! Also, I disagree with the Scala examples and the argument against brevity, but I guess this is the stuff of flamewars. Not only do I not find his more verbose examples…

I was thinking the same thing about those sortBy code snippets. I use shorthand for lambdas all the time because the individual item names are implied by the collection.

Re: Data science tips and tricks from the developer community

#40

Post by gghyslain »

I highly recommend "Best Practices for ML Engineering from Google" [1], which contains one of the best piece advice on the topic:

> "To make great products: do machine learning like the great engineer you are, not like the great machine learning expert you aren’t."

[1] Previous HN discussion: https://news.ycombinator.com/item?id=13414776

Post reply on HN