Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data science tips and tricks from the developer community

blog.algorithmia.com

Post reply on HN
Print view View original

61–69 of 69 posts

Re: Data science tips and tricks from the developer community

#61

Post by mooneater »

mikeyanderson wrote: post #47
mooneater wrote: post #39

Most of this nit-picky advice will not increase my leverage. What I think would: occasional access to domain experts in specific niches of data science.

What would that look like? Data-scientist AMAs? Or in person? What would you like to see?

- a marketplace for getting small doses of top-level expert advice - more written about real-world, messy, data-science and machine learning implementations - the vast majority of writing about ML/DS involves the elements. There is a lack of writing about how full systems integrate.

Re: Data science tips and tricks from the developer community

#62

Post by [deleted] »

gipp wrote: post #49
NumberCruncher wrote: post #35

Earlier quoted context omitted.

In my experience if you develop a "data science pipeline" forcing the data scientist to build - reproducible - validated - back-tested - easy to deploy models, they are going to hate it. It just kills the fun and/or makes obvious if they made a mistake.

So we should sacrifice all the things that actually make a Data Scientist's work valuable in the name of fun and obscuring mistakes? Fun I almost get, obviously good for productivity (though I think you'd really be sacrificing productive output for non-productive output), but I just don't get where you're even coming from with the "making mistakes more obvious" angle.

[deleted]

Re: Data science tips and tricks from the developer community

#63

Post by ontouchstart »

Angostura wrote: post #13
flaviuspopan wrote: post #11

Earlier quoted context omitted.

Nothing feels cleaner than storing everything (notebook, raw data, cleansed data, misc scripts, etc.) in a docker image when you're finished with the project. Data science and docker are meant to be besties.

Potentially problematic for those who want to check your findings in 30 years time?

Put them in golden record?

http://www.newyorker.com/tech/elements/voyager-golden-record...

Re: Data science tips and tricks from the developer community

#64

Post by princesspea »

minimaxir wrote: post #2

It's worth nothing that the article does not discuss the reproducibility of results (e.g. with a Jupyter Notebook) and the implementation of said results (e.g. deploying/validating models), both of which matter much more than any code style conventions for data-related projects.

Hey thanks for the comment! I'm the author of the talk-turned-post :-) You are completely correct about reproducibility being super important in data science workflows. While I did mention it in the post (and in the talk the post was based on), I mentioned it as a part of version control tools. That said, I think it's not something that is focused on enough (obviously I'm guilty of that too) so I plan on doing a follow up post focused on reproducibility and the tools that can help you recreate your results. Kinda putting the "science" back in data science. Really I want an excuse to play around with tools like https://dataversioncontrol.com/ which looks super useful and I mentioned it in the post, but haven't had a chance to use.

Re: Data science tips and tricks from the developer community

#65

Post by mattmanser »

carlmr wrote: post #57
KirinDave wrote: post #26

Earlier quoted context omitted.

But also, even this data is questionable in the extreme. It may simply be that "10x" people who do exist do so in ways that are challenging to observe. As an example, not making difficult-to-detect mistakes early in the software lifecycle that cause major problems later (classic real world example: mongodb). Or that their influence on a software org causes overall productivity improvements. In any case, it's a toxic…

The real 10x developers in my experience are the hardest to measure. Because they pull the whole team by always being helpful and improving things where they see potential. But that doesn't necessarily show up in their results, but in the whole teams results. Which is why metrics driven organizations in my experience with their disincentive to help others, slow everything down.

Do they? Or are they just wasting time refactoring instead of getting stuff done?

It's a matter of perspective.

Re: Data science tips and tricks from the developer community

#66

Post by mdekkers »

webnrrd2k wrote: post #43
joe_the_user wrote: post #41

Earlier quoted context omitted.

To put it cynically, if you other people can reproduce your results, you might not be demonstrating that you are x10 more productive than them. Which is to say I think starting with the idea that you're aim for x10 is pernicious and tends to create dysfunctional teams. The claim some developers in some circumstances are ten time more productive than others may or may not be true but software development needs process…

Yes, it's better to have an x10 team, rather than an x10 developer. Developers should strive to better themselves, but it's important not to fool yourself, too. Having a strong team is almost always better from a business point of view.

Indeed. I have seen several teams with a bunch of (self-styled) "10x" devs, and found that the productivity and quality of the team decreases in direct proportion to the amount of "10x" devs on the team.

I shun "rockstar" and "10x" (and whatever other bullshit moniker they will come up with next) team members. Give me a group of smart people that gel well together, and are highly self-confident without egos getting into the way, and we can move mountains.

Re: Data science tips and tricks from the developer community

#67

Post by escribmac »

mattmanser wrote: post #60
escribmac wrote: post #21

Earlier quoted context omitted.

The worst developers I've worked with take 2 weeks for tickets that should be simple. That would mean doing 1 easy ticket every day or two makes you a 10x.... I've always hated this term and the mindset around it. I think organizational practices, intelligent engineering strategy, etc are far more important to the output of a team than hiring one genius dev.

Did it ever occur to you they might not be bad developers, they're just goofing off because there's no consequence for being slow? Like when my old work actually started measuring ticket closure times, our best developers were only 2x more productive than our worst ones. But suddenly a lot more tickets were getting closed. I mean,I know that some complicated tasks needed the best developers, as the worst ones literal…

I agree, I think that falls under organizational practices

Re: Data science tips and tricks from the developer community

#68

Post by kolbe »

gipp wrote: post #53
kolbe wrote: post #51

Earlier quoted context omitted.

The job you're describing may have the title of Data Scientist, but it isn't data science if it doesn't involve advanced methods.

That's a semantic battle you've lost already.

My point is that this 'data scientist' may be called one at fartapp.io, but not at Google/Microsoft/DARPA/MIT.

Re: Data science tips and tricks from the developer community

#69

Post by dmpetrov »

princesspea wrote: post #64
minimaxir wrote: post #2

It's worth nothing that the article does not discuss the reproducibility of results (e.g. with a Jupyter Notebook) and the implementation of said results (e.g. deploying/validating models), both of which matter much more than any code style conventions for data-related projects.

Hey thanks for the comment! I'm the author of the talk-turned-post :-) You are completely correct about reproducibility being super important in data science workflows. While I did mention it in the post (and in the talk the post was based on), I mentioned it as a part of version control tools. That said, I think it's not something that is focused on enough (obviously I'm guilty of that too) so I plan on doing a foll…

'Kinda putting the "science" back in data science. ' Exactly! This is the primary goal in DVC project.
Post reply on HN