Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Science Challenges at Instacart

tech.instacart.com

Post reply on HN
Print view View original

41–50 of 65 posts

Re: Data Science Challenges at Instacart

#41

Post by disgruntledphd2 »

sandGorgon wrote: post #34
jeremystan wrote: post #14

Earlier quoted context omitted.

We've worked hard to open source projects whenever we think they'll be useful broadly: https://www.instacart.com/opensource . There is definitely more of this we can (and I hope will) do in the future.

How do you guys run R in production? Just getting started with R based datascience and it has been a struggle to figure out how to build a production data science stack. Do you snapshot the computed models as RData and stream them to s3, etc

If I were running R in production, then I'd probably fit models on some kind of batch process and then serve up the predictions/output from a DB or something.

In general, R is not well-suited for DB-backed websites in real-time, but you can certainly use the outputs in production.

You can do it, but I'm not sure it's worth the effort. You could probably provide a predict() interface in real-time if it was reasonably quick.

Re: Data Science Challenges at Instacart

#42

Post by sgt101 »

sandGorgon wrote: post #34
jeremystan wrote: post #14

Earlier quoted context omitted.

We've worked hard to open source projects whenever we think they'll be useful broadly: https://www.instacart.com/opensource . There is definitely more of this we can (and I hope will) do in the future.

How do you guys run R in production? Just getting started with R based datascience and it has been a struggle to figure out how to build a production data science stack. Do you snapshot the computed models as RData and stream them to s3, etc

We run udfs in Hive to invoke R models, which is fine for compiling dashboards and reports but I wouldn't run it for something that needed instant responses.

Re: Data Science Challenges at Instacart

#43

Post by sandGorgon »

disgruntledphd2 wrote: post #41
sandGorgon wrote: post #34

Earlier quoted context omitted.

How do you guys run R in production? Just getting started with R based datascience and it has been a struggle to figure out how to build a production data science stack. Do you snapshot the computed models as RData and stream them to s3, etc

If I were running R in production, then I'd probably fit models on some kind of batch process and then serve up the predictions/output from a DB or something. In general, R is not well-suited for DB-backed websites in real-time, but you can certainly use the outputs in production. You can do it, but I'm not sure it's worth the effort. You could probably provide a predict() interface in real-time if it was reasonably…

So I have seen a couple of large data science driven startups (like consumer finance) to throw R on 128gb machines and call it a day. That's reasonably going to be my plan except that I can't make it work very well.

I really wish pandas had a "save workspace" feature - R does that very well. No point in saving to dB if you're going to need the data set in memory anyway.... Or use Hadoop.

Re: Data Science Challenges at Instacart

#44

Post by CPLX »

So we no longer use the word scientist to describe people who do science? What a shame, I think science is really neat and scientists deserve unique respect.

As far as I can tell what these people do every day is called "business" or maybe "logistics"

Re: Data Science Challenges at Instacart

#45

Post by yummyfajitas »

CPLX wrote: post #44

So we no longer use the word scientist to describe people who do science? What a shame, I think science is really neat and scientists deserve unique respect. As far as I can tell what these people do every day is called "business" or maybe "logistics"

What do you believe distinguishes "science" from "data science"? I.e., why can't "logistics" be a subset of "science"?

Re: Data Science Challenges at Instacart

#46

Post by CPLX »

yummyfajitas wrote: post #45
CPLX wrote: post #44

So we no longer use the word scientist to describe people who do science? What a shame, I think science is really neat and scientists deserve unique respect. As far as I can tell what these people do every day is called "business" or maybe "logistics"

What do you believe distinguishes "science" from "data science"? I.e., why can't "logistics" be a subset of "science"?

Because science is an academic pursuit designed to create and test generalizable hypotheses and add to our collective knowledge, while the people in the article are trying to figure out how to optimize the act of underpaying someone to go grab some cans off a supermarket shelf and bring them to me.

They're not scientists, they're engineers perhaps, or business analysts.

Re: Data Science Challenges at Instacart

#47

Post by yummyfajitas »

CPLX wrote: post #46
yummyfajitas wrote: post #45

Earlier quoted context omitted.

What do you believe distinguishes "science" from "data science"? I.e., why can't "logistics" be a subset of "science"?

Because science is an academic pursuit designed to create and test generalizable hypotheses and add to our collective knowledge, while the people in the article are trying to figure out how to optimize the act of underpaying someone to go grab some cans off a supermarket shelf and bring them to me. They're not scientists, they're engineers perhaps, or business analysts.

So a person doing basic biology research for Monsanto isn't a scientist? And whether or not Kantorovich qualifies as a scientist depends on whether he was working for the military or a university at the time he came up with linear programming?

That's an interesting definition.

Re: Data Science Challenges at Instacart

#48

Post by jeremystan »

sandGorgon wrote: post #34
jeremystan wrote: post #14

Earlier quoted context omitted.

We've worked hard to open source projects whenever we think they'll be useful broadly: https://www.instacart.com/opensource . There is definitely more of this we can (and I hope will) do in the future.

How do you guys run R in production? Just getting started with R based datascience and it has been a struggle to figure out how to build a production data science stack. Do you snapshot the computed models as RData and stream them to s3, etc

We use R in production in two ways:

1. For batch processes that run daily, hourly or minutely, where the models are rebuilt on every run, and outputs (often predictions) are written to a database 2. For computation of coefficients in large sparse regularized models, where the coefficients are written to a database and scoring is done in another language in real-time

For situations where we want real-time predictions, recommendations or optimizations, we tend to setup Python services instead. For batch processes, you can definitely store models in S3 to re-use them, and I've done that at other companies. But in general I've found it better to rebuild models frequently and cache them for short periods of time only if they are cost-prohibitive to rebuild.

Re: Data Science Challenges at Instacart

#49

Post by jeremystan »

grayclhn wrote: post #38

> Many of our best data science ideas have come from Instacart employees in the field – working directly with our shoppers in our stores, or interacting directly with our customers. I have no idea what this could mean. Either you're getting algorithm suggestions from your shoppers and customers, or (more likely) "data science" means "user interface."

Agreed, that wasn't worded well - i'll try to explain. By 'employees in the field' we mean people who work in operations and management roles in the cities we operate in. They work with shoppers in stores and respond to shopper and customer feedback. They have ideas about how to improve our logistics and our apps - and while many ideas will be about user interfaces, many others will relate to how the algorithms operate behind the UIs.

Re: Data Science Challenges at Instacart

#50

Post by JasonCEC »

asdfologist wrote: post #35
JasonCEC wrote: post #27

Earlier quoted context omitted.

It's a 3 month paid internship, and 99% of the students have been from Princeton. Did you see the work we linked to? That's intern work here - we treat our interns as full members of the team, and they've delivered.

So you had 100 interns and 99 of them were from Princeton??

That's now how percentages work....
Post reply on HN