Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Data Science Team at a Startup - An Engineering Perspective

kurt.karmalab.org

Post reply on HN
Print view View original

11–18 of 18 posts

Re: Building a Data Science Team at a Startup - An Engineering Perspective

#11

Post by dangoldin »

adamio wrote: post #4
dangoldin wrote: post #3

Something more interesting to me is how do you convince a company to build out a true data science. Every company can use one but very few have one. Has anyone had any experience convincing management of the value?

In my experience if management isn't asking the question, then its a dead issue. At a large company I worked at - data analysis was flat out rejected. I've been told to stop doing and return to standard procedure using heuristics . Even something as simple as introducing a new data model - at a large company I've been told that "my future replacement" wouldn't understand it. Management looking for continuity will not…

Yea - that's what I'm afraid of. I've been trying pretty hard to get people to view data as an asset and it's very hit or miss. Some people embrace it, most will agree with me but not do anything more than that, others don't care.

Re: Building a Data Science Team at a Startup - An Engineering Perspective

#12

Post by PaulHoule »

kreilly wrote: post #5

This is pretty spot on with what I've seen up close. Specifically: 1) "Throw [y]our normal engineering practices out of the window." - We treat Data Science very much like the "R" in R&D. We point them generally towards a problem and give them time and latitude to solve it. Trying to fit that into our normal scrum process is impossible. 2) "Data scientists are going to end up building things that need to be translate…

yeah, my experience is that big data + agile planning = fail.

in the typical agile process, calendar time is ignored, and people pretend that you can just manage punchclock time. the result of that is that you end up with two days to go to the end of the sprint and a four day job that needs to run.

Re: Building a Data Science Team at a Startup - An Engineering Perspective

#13

Post by etrain »

I'll chime in here as a member of a 'Data Science' team at an early stage company. A lot of this really resonated with me. In particular - I think several of his points are consequences of the others:

1) We sit on an island, quasi-separated from engineering. We have our own processes, codebase, and tools, that are mostly different from what engineering has deployed in the wild.

2) This leads to 'throw normal engineering processes out the window,' which is good and bad . We write plenty of code, but not much of it makes it to source control. Why? so much of what you write is "throwaway code" - this, of course, is no good, but it's very tough to tell apriori what is going to work and what isn't.

3) This makes getting things deployed to production hard. Your code isn't written in the Java that the rest of the backend is. There was never the thought that there'd be a build process attached to it. Deployment? Nightmare.

As far as not scaring the rest of the company with all the math - totally agree, but only to an extent. If your models are so sophisticated that they can't be explained to an engineer, or your customers, in plain (insert locale-specific spoken language), they might be too complicated.

In terms of having an impact immediately - I think this goes for all engineers and data scientists. Someone should be able to put a fresh set of eyes on your problems, and be able to solve a new one with enthusiasm.

Re: Building a Data Science Team at a Startup - An Engineering Perspective

#15

Post by aaronjg »

kreilly wrote: post #5

This is pretty spot on with what I've seen up close. Specifically: 1) "Throw [y]our normal engineering practices out of the window." - We treat Data Science very much like the "R" in R&D. We point them generally towards a problem and give them time and latitude to solve it. Trying to fit that into our normal scrum process is impossible. 2) "Data scientists are going to end up building things that need to be translate…

There is a big gap between the disciplines of Data Science and engineering, that makes it difficult to translate the code.

When you are trying to discover answer to a statistical problem it doesn't make much difference if a task takes 30 minutes or 10 hours. This is especially true if the task can be parallelized, then you can just boot up some EC2 nodes and run it. If a task fails one out of five times, it's not a problem because you can just run it again.

However getting this sort of code into production is another challenge, in terms of building the system, dealing with scale, handling edge cases, etc.

Peter Norvig talks about this in one of his talks. At google, they typically start programs in google research and migrate them over to the development teams. One exception is google translate, which they kept in the research division, and brought in engineers to help bring it into production.

Re: Building a Data Science Team at a Startup - An Engineering Perspective

#16

Post by Drbble »

PaulHoule wrote: post #12
kreilly wrote: post #5

This is pretty spot on with what I've seen up close. Specifically: 1) "Throw [y]our normal engineering practices out of the window." - We treat Data Science very much like the "R" in R&D. We point them generally towards a problem and give them time and latitude to solve it. Trying to fit that into our normal scrum process is impossible. 2) "Data scientists are going to end up building things that need to be translate…

yeah, my experience is that big data + agile planning = fail. in the typical agile process, calendar time is ignored, and people pretend that you can just manage punchclock time. the result of that is that you end up with two days to go to the end of the sprint and a four day job that needs to run.

Where is a two day delay for delivery "fail". There is no incompatibility with agile planning. You don’t go on vacation every time a job runs, you work on another project.
Post reply on HN