Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

logicalclocks.com

Post reply on HN
Print view View original

51–57 of 57 posts

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#51

Post by sjg007 »

blauditore wrote: post #7

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…

Data is the new oil right so you want to decouple your data consumers from the data generators. This means you dump all the data into a lake. That way a reporting team or analytics team doesn’t need to connect to your production database.

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#52

Post by cpard »

jamesblonde wrote: post #50
cpard wrote: post #48

Earlier quoted context omitted.

I think a reason that leads to the need of something like a 'data lake' or anything that looks like a messy pile of incoherent data, is the difference between how data scientists and traditionally data analysts deal with noise in the data. Most BI tasks require the data to be as clean as possible, it's important to be aware of the quality of the data before you calculate your MRR for example. On the other side, data…

Noise yes, null or missing values will crash your training and out-of-distribution values (which your schema will not help you with!) will be even worse, hurting performance but impossible to find.

Totally agree on that, both have still to work with data quality but in different ways and with a different mindset.

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#53

Post by dmpetrov »

freewilly1040 wrote: post #42
dkarl wrote: post #16

Earlier quoted context omitted.

> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name. That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the d…

> we can still get signal out of it Today you can, but then when the app owner drops a column (or worse, stop populating it!) in a month that signal will break, and the data lake maintainer will be in the unenviable position of navigating the completely undocumented dependency.

That's might be fine. Anyway, you need to retrain your models quite often due to the concept drift. Solving data problems like this is just a part of model retraining (and concept drift).

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#54

Post by hodgesrm »

jamesblonde wrote: post #33
notsuoh wrote: post #19

Earlier quoted context omitted.

I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?

Hi. No, I don't think it is. Because your online applications that need low latency access to features won't tolerate the latencies provided by existing data warehouses. The online app that has an operation model that makes predictions is one client of the feature store. For the other client - a data scientist who is browsing features and creating train/test datasets - yes, that is similar to a data warehouse, except…

Which existing data warehouses do you mean? ClickHouse and Druid can return answers in millisecond. Data warehouses are starting to optimize for low latency response.

Disclaimer: My company supports ClickHouse.

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#55

Post by ipiz0618 »

blauditore wrote: post #7

I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…

In large organizations it is common to have multiple databases maintained by many different people. When a DS needs some data he/she needs to talk to all these different people, wait for them or their managers to grant access, and join the inconsistent labels. This takes a long time and some people are extremely protective of their (messy anyways) tables, requiring higher ups to straight up demand them to grant access.

Having nicely organized data is perfect, but I'd rather fetch the data myself from a pile of messy data instead of dealing with all these organizational nightmare.

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#56

Post by jascii »

notsuoh wrote: post #21
jascii wrote: post #20

Oh, this article is not about ML (the language) It is about Machine Learning... Can we just call Machine Learning Machine Learning, to avoid confusion?

Maybe you're being facetious. I had to look up what ML Language is, apparently it's a fifty year old programming language that hasn't had a stable release in 23 years, in case anyone else was wondering. Respectfully, let's keep ML to be Machine Learning. ;)

Thanks for engaging instead of what seems now to be HN's standard response: downvote anything they don't know/understand... The standard has been stable for 23 years however various implementations are under active use and development (MLton's latest release was 10 days ago.). On top of that, a fair bit of founding research into Machine Learning was done in ML, adding to potential confusion.

Allas, I'll stop shouting "get off my lawn" and let the young ones make their own mistakes...

Re: Machine Learning Engineer Guide: Feature Store vs. Data Warehouse

#57

Post by cpdoughe »

ScoutOrgo wrote: post #44
supercanuck wrote: post #25

Earlier quoted context omitted.

it absolutely is. This company is trying to make a distinction between Online Data (real time streaming with low latency), no joins, key/store and a more traditional batch processing, OLAP type configurations. but modern data warehouses can support both. https://www.snowflake.com/streaming-data/ I think this is an effort to segment the data warehousing market and provide new names for things that already exist and pr…

As a data scientist using snowflake and in the market for a feature store, the snowflake streaming is only for data ingestion, not serving. It doesn't solve the problem of serving data for a low latency app.

hi, i'm the co-founder and CTO of a feature store startup that is building this on top of snowflake, can we chat? my name is Patrick, website is rasgoml.com, and e-mail is patrick@rasgoml.com. Thanks!
Post reply on HN