21–30 of 57 posts
Oh, this article is not about ML (the language) It is about Machine Learning... Can we just call Machine Learning Machine Learning, to avoid confusion?
Respectfully, let's keep ML to be Machine Learning. ;)
Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.
Can you please explain how the ONLINE feature store works? I.e. if the prediction request contain, let say, user id, and the user record is not in the ONLINE feature store (e.g. Redis), than you would need to go to the OFFLINE store and do a join, or a select? To sum up, assuming that the online feature store is some sort of a cache, how do you know which objects to place in the cache?
Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates over some commonly used time windows) to facilitate efficiency in the end to end ML pipeline.
I’d also argue a feature store is not complete unless there are tools and infrastructure to make it relatively easy to provision and access new features: that is, it should cover the issues you raise.
Earlier quoted context omitted.
Can you please explain how the ONLINE feature store works? I.e. if the prediction request contain, let say, user id, and the user record is not in the ONLINE feature store (e.g. Redis), than you would need to go to the OFFLINE store and do a join, or a select? To sum up, assuming that the online feature store is some sort of a cache, how do you know which objects to place in the cache?
That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things. Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates…
My question is about the operational aspect of online vs offline stores.
I.e. what happen if the features needed for a prediction are not in the ONLINE store?
Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.
I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?
Quoting some from the article: "Data warehouses are used primarily by business analysts for interactive querying and for generating historical reports/dashboards on the business. Feature stores are used by both data scientists and by the online/batch applications, and they are fed data by feature pipelines, typically written in Python or Scala/Java. Also, Data warehouses mostly stores data in relational tables, whereas a Feature Store stores it as numerical and categorical features and outputs tensors and/or vectors for training or serving.
Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.
I mean, maybe you're leaving this intentionally open ended to garner comments to get your post higher on the HN page, but perhaps you could answer the question you posted: Isn't this just a data warehouse?
This company is trying to make a distinction between Online Data (real time streaming with low latency), no joins, key/store and a more traditional batch processing, OLAP type configurations.
but modern data warehouses can support both.
https://www.snowflake.com/streaming-data/
I think this is an effort to segment the data warehousing market and provide new names for things that already exist and providing a vocabulary to users who may not be familiar with a company's existing datawarehouse solutions
Earlier quoted context omitted.
> Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy name. That's kind of what I understand as well, but the data science folks pitched it in a slightly more positive way, like, "Please don't limit us just to the d…
More or less how a proof-of-concept for new type of wine turns out to be sold as Champagne..
If it's something else, it's the antithesis of data science.
I get a strong buzzword bingo vibe from this post. On a related note, is there a good reason to ever have something like a "data lake" (and call it like that)? Whenever I've encountered someone bringing up the idea to "build a data lake", a few questions later it became clear they just had a messy pile of incoherent, poorly-understood data and wanted to twist it into something positive by giving that pile a fancy nam…
The reason for data lakes appears in large enough organizations where it becomes exceedingly likely that there is some data that may be useful to you that's maintained by people you'll never meet in a department you don't know about, where it's impractical or even impossible to get them involved in your project that would consume this data. It's not so much about data itself as an attempt to solve a communications an…
I would challenge the assumption that "the Data Warehouse is an input to the Feature Store" though. I'm more inclined towards having a first stage of data cleanup/modeling that could be reused (as input) for both DWH and FS instead.
Earlier quoted context omitted.
That depends on the organizational structure of the teams, what they’re trying to accomplish, and the budget for resources to name a few things. Generally I don’t view a “feature store” as being a simple lake of raw data. To me a feature store is a place where features known to have signal in a ML model are stored. It has structure, “clean” data (to the extent possible) and some pre-computed elements (e.g. aggregates…
I am not sure how this answer my question. I understand that a feature store contain the features that have signal. My question is about the operational aspect of online vs offline stores. I.e. what happen if the features needed for a prediction are not in the ONLINE store?
Author here. I wrote this article because I keep getting the question from prospects - isn't this just a data warehouse? If I missed out on anything or got anything wrong, please let us know here.
> isn't this just a data warehouse [DW]?
My understanding is that the addition of a RowStore to the DW/ColumnStore addresses the training phase of Machine Learning.
I come from a Data Engineering background. I struggled with the Data Science centric terminology. Uber's 2017 post [1] was helpful in establishing the motivations and terminology of their Michelangelo machine learning (ML) platform. The main distinguishing feature of a Feature Store seems to be that it supports efficient batch downloading of row data that is used as the training dataset. The discussion made more sense once I figured out that feature refers to a column or data field.
Figure 4 in the Logical Clocks whitepaper Hopsworks Feature Store [2] helped me understand the architecture better. The architecture appears to be what I would call a Hybrid Transactional/Analytical Processing (HTAP) [3] engine with MySQL cluster acting as the RowStore and Apache Hive acting as the ColumnStore. I'm assuming that the Hopsworks Feature Store periodically merges the MySQL updates into Hive and also provides a mechanism to perform federated queries.
The use of Hive seems outdated (vs. say Presto) and I wonder if the use of MySQL is required compared to directly accessing column oriented files like ORC/Parquet/Kudu.
[1] https://eng.uber.com/michelangelo-machine-learning-platform/
[2] (PDF) https://uploads-ssl.webflow.com/5e6f7cd3ee7f51d539a4da0b/5ef...
[3] https://en.wikipedia.org/wiki/Hybrid_transactional/analytica...