Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
1–10 of 122 posts
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#2This is really cool. Vespa was probably first described in this 2007 paper: https://brage.bibsys.no/xmlui/bitstream/handle/11250/251199/...
Next up, I would really like to see Sherpa/PNUTS (their NoSQL operational database) and Everest (their petabyte-scale Postgres data warehouse) open sourced :)
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#3"When machines are lost or new ones added, data is automatically redistributed over the machines, while continuing serving and accepting writes to the data. Changes to configuration and Java components can be made while serving by deploying a changed application package - no down time or restarts required." That sounds pretty impressive.
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#4Wow this project is humongous!
https://github.com/vespa-engine/vespa
I'm really curious how it compares to Lucene/ElasticSearch/ELK, which is currently my tool of choice for (faceted) search and recommendation.
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#5Wow this project is humongous! https://github.com/vespa-engine/vespa I'm really curious how it compares to Lucene/ElasticSearch/ELK, which is currently my tool of choice for (faceted) search and recommendation.
I think that's actually the broadest root folder I've ever seen on a project! It's really hard to know where to start looking. Is anyone familiar with the internals?
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#6This is really cool. Vespa was probably first described in this 2007 paper: https://brage.bibsys.no/xmlui/bitstream/handle/11250/251199/... Next up, I would really like to see Sherpa/PNUTS (their NoSQL operational database) and Everest (their petabyte-scale Postgres data warehouse) open sourced :)
May I ask some stupid questions? :/
I don't quite get the diagram of the Vespa Architecture. Is Vespa a middleware between database engine and query parser? This is what puzzles me.
If so, are there other such middlewares available for ie. PostgresSQL that allow hooking "Query Templating Models" (that is it?) generated via Machine-Learning Models? Is it way more complicated than that, or did they overengineer the problem into a monolith? EDIT: Looking at https://github.com/vespa-engine/vespa it seems that it is overengineered, or maybe it consists of individual micro-components like node.js, hmm more questions :(
Is GraphQL such middleware or lower-level?
Does Vespa replace custom Glue-Code between Backend and Frontend that generates such query-sets for content ranking/positioning?
Or what exactly does Vespa solve? I'm sorry, I've read the article, but can't say, yep that's what it is!
EDIT: How else could you solve what Vespa does using Rust, Go, or C/C++ libraries? A very simple or general direction would be immensely useful to understand Vespa =) The project makes the simultanous impression of an immense engineering feat and at the same time a huge code debt.
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#7Maybe this can be merged with the discussion from yesterday: https://news.ycombinator.com/item?id=15339851
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#8This is really cool. Vespa was probably first described in this 2007 paper: https://brage.bibsys.no/xmlui/bitstream/handle/11250/251199/... Next up, I would really like to see Sherpa/PNUTS (their NoSQL operational database) and Everest (their petabyte-scale Postgres data warehouse) open sourced :)
May I ask some stupid questions? :/ I don't quite get the diagram of the Vespa Architecture. Is Vespa a middleware between database engine and query parser? This is what puzzles me. If so, are there other such middlewares available for ie. PostgresSQL that allow hooking "Query Templating Models" (that is it?) generated via Machine-Learning Models? Is it way more complicated than that, or did they overengineer the pro…
I think at a glance, it's basically a much more scalable version of something like Elasticsearch, optimized for very quick wide fanout to a large number of leaf nodes.
It's a datastore in its own right (just like ES), but I imagine that e.g. you wouldn't use it to handle transactions.
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#9I observe that it was written in Java: https://github.com/vespa-engine/vespa
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#10I observe that it was written in Java: https://github.com/vespa-engine/vespa
The serving and admin/config layers are Java, while the content cluster is C++. Please see the architecture diagram on http://vespa.ai/.