Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

logicalclocks.com

Post reply on HN
Print view View original

11–20 of 57 posts

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#11

Post by xapata »

blauditore wrote: post #7

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…

Nah, it's a messy pile of data that you want to understand rather than throw away.

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#12

Post by maycotte »

moritzmeister wrote: post #9
maycotte wrote: post #5

We have been building a feature first data store for seven years and it feels like feature store is about to become one of the more exciting ways to extract value from data. We see feature stores doing much more than becoming just another silo for ML, but instead a way to get a real-time, centralized view of fragmented data that has to either be federated or put in a data lake to to be queried together. I share more…

This looks like a highly specialised tool. How is it going to integrate with a Data Scientist's favourite tools, such as Jupyter notebooks, Pandas or Spark and especially ML frameworks like TensorFlow, SkLearn etc.?

Great question. Right now we have pushed hard to make SQL the primary interface to our feature store and can output Pandas and other formats at query time, however we are working on integrated/hosted Jupyter notebooks and excited to continue collaborating with the community on better feature first/oriented endpoints/interfaces. There is so much room to innovate here.

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#13

Post by LexSiga »

blauditore wrote: post #7

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…

You are not wrong on the buzzword, but you are not absolutely right if you suggest that it is merely that; this is a recurring question and interrogation in that specific area.

The fact that it happens to be kinda buzz worthy is a collateral aspect: everything that answers what some people wonder and that is not yet answered plainly, is.

(and I mean, the first on the front page at this very second has : "We hacked apple" in the title.)

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#14

Post by PeterisP »

blauditore wrote: post #7

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…

The reason for data lakes appears in large enough organizations where it becomes exceedingly likely that there is some data that may be useful to you that's maintained by people you'll never meet in a department you don't know about, where it's impractical or even impossible to get them involved in your project that would consume this data.

It's not so much about data itself as an attempt to solve a communications and coordination organizational problem; you decouple sources of data and consumers of data (not the technical systems/databases, but the people and organizational units) to a 'hub-and-spoke' model where the providers of data just supply raw data without getting into a multinational project that takes a year just to identify the potential stakeholders for that data throughout a distributed organization with tens of thousands of employees.

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#15

Post by dcolkitt »

blauditore wrote: post #7

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…

With the caveat that "data lake" means different things to different people, I think it's important to have a repository where the raw data exists un-manipulated in the form that it was ingested.

From the end-user standpoint that's not very useful. But that's why you have data marts that normalize the raw data into a standardized format. Ultimately though the raw data needs to remain the single source of truth. If you skip that step, and only store the post-normalized format, you're likely to run into problems down the road. This could either be because you want to change the normalization format. Or you want to use some aspect of the data that isn't captured in the normalized form. Or even you discover a pre-existing bug that affected all the previous post-ETL data.

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#16

Post by dkarl »

blauditore wrote: post #7

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…

> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name.

That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the data you have time to nicely structure and validate. We want all of it. It doesn't matter if a column is getting truncated to three characters or columns are mislabeled or there are amounts in dollars and euros mixed together; we can still get signal out of it."

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#17

Post by zekrioca »

dkarl wrote: post #16
blauditore wrote: post #7

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…

> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name. That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the d…

More or less how a proof-of-concept for new type of wine turns out to be sold as Champagne..

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#18

Post by streetcat1 »

jamesblonde wrote: post #3

Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.

Can you please explain how the ONLINE feature store works? I.e. if the prediction request contain, let say, user id, and the user record is not in the ONLINE feature store (e.g. Redis), than you would need to go to the OFFLINE store and do a join, or a select?

To sum up, assuming that the online feature store is some sort of a cache, how do you know which objects to place in the cache?

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#19

Post by notsuoh »

jamesblonde wrote: post #3

Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.

I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?
Post reply on HN