I see it has tensor processing built in - http://docs.vespa.ai/documentation/tensor-intro.html
Can this be used as a spark+tensorflow replacement ?
Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
11–20 of 122 posts
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#12Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#13"When machines are lost or new ones added, data is automatically redistributed over the machines, while continuing serving and accepting writes to the data. Changes to configuration and Java components can be made while serving by deploying a changed application package - no down time or restarts required." That sounds pretty impressive.
Looks impressive!
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#14Earlier quoted context omitted.
May I ask some stupid questions? :/ I don't quite get the diagram of the Vespa Architecture. Is Vespa a middleware between database engine and query parser? This is what puzzles me. If so, are there other such middlewares available for ie. PostgresSQL that allow hooking "Query Templating Models" (that is it?) generated via Machine-Learning Models? Is it way more complicated than that, or did they overengineer the pro…
I think at a glance, it's basically a much more scalable version of something like Elasticsearch, optimized for very quick wide fanout to a large number of leaf nodes. It's a datastore in its own right (just like ES), but I imagine that e.g. you wouldn't use it to handle transactions.
So the upsides of Vespa over Elasticsearch are speeding up the rate at which it scales? Ah, that seems reasonable for a company this size, but is there something in there that's of use for Startups?
This blog post shows how Elasticsearch was used to reindex a 136TB dataset with 36B documents[1], so I'm unsure exactly where except for Google/Yahoo Scale companies Vespa is of use. I would like to understand howto utilize it though without adding an umnanagable complexity.
EDIT: Maybe a Vespa Cloud startup, that abstracts the management and makes "Scalability as a Service" by utilizing other Cloud providers.
--
[1] https://thoughts.t37.net/how-we-reindexed-36-billions-docume...
[2] http://docs.vespa.ai/documentation/vespa-quick-start.html
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#15Wow this project is humongous! https://github.com/vespa-engine/vespa I'm really curious how it compares to Lucene/ElasticSearch/ELK, which is currently my tool of choice for (faceted) search and recommendation.
Found some comparison (on the root page of their project website, my bad).
http://vespa.ai/#featurematrix
Next thing I'd really like to know is what existing software it builds on top of.
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#16I see it has tensor processing built in - http://docs.vespa.ai/documentation/tensor-intro.html Can this be used as a spark+tensorflow replacement ?
It is more like an alternative to tensorflow serving - it does not train models but is good at evaluating them and use the evaluation for ranking.
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#17Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground.
Really goes to show that engg != business and unless you have a firm business model and growth, an amazing engg team can only get you so far.
I know I'm just stating the obvious, but just putting it out there!
Vespa looks super interesting (more so since I'm in a company that provides ecommerce search APIs as a product) and I'm sure I'll play with it more. Thanks Yahoo! :)
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#18Wow...
I'm guessing (hoping) their lawyers made sure to go over their old agreements with a very fine comb to make sure their license for the software allowed them to open source this.
Don't get me wrong, they've added a lot to it, but there's a lot of code in there that could only have come from their purchase of Overture, who had purchased AllTheWeb from FAST (which was itself purchased by Microsoft).
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#19This is really cool. Vespa was probably first described in this 2007 paper: https://brage.bibsys.no/xmlui/bitstream/handle/11250/251199/... Next up, I would really like to see Sherpa/PNUTS (their NoSQL operational database) and Everest (their petabyte-scale Postgres data warehouse) open sourced :)
May I ask some stupid questions? :/ I don't quite get the diagram of the Vespa Architecture. Is Vespa a middleware between database engine and query parser? This is what puzzles me. If so, are there other such middlewares available for ie. PostgresSQL that allow hooking "Query Templating Models" (that is it?) generated via Machine-Learning Models? Is it way more complicated than that, or did they overengineer the pro…
> How else could you solve what Vespa does using Rust, Go, or C/C++ libraries?
Let me try myself answering my own question, I hope someone hops in and tells me where I'm wrong or how else to improve :)
1) Get PostgresSQL exntensions via "package manager" pgxnclient
1.1) pg_bouncer - For connetion pooling
1.2) yoke - As a high-availability cluster manager with auto-failover and automated cluster recovery
1.3) prestodb.io - Distributed SQL query engine for pgsql
1.4) pglogical - Logical streaming replication for using a publish/subscribe model
1.5) pg_lambda - To create your own AWS (meta) Lambda
1.6) pg_strom - To offload tasks to the GPU
1.7) zombodb - To utilize full-text searching via indexes backed by Elasticsearch
2) Put all together with pglogical and presto to seperate GPU/CPU intensive tasks.
2.1) "Build Missing Middleware" - To design/fuse a query visually that combines multiple backends
2.1.1) Create a binary data-stream by integrating pg_lambda, pg_strom, presto and zombodb
2.1.2) "Build Missing Middleware" - A tensor processing extension to use ML Model evaluations
2.1.3) "Use Missing Middleware" - For data-processing via Machine-Learning models
2.1.4) "Use Missing Middleware"- To output ML processed results into the database
2.2) Partition these queries using "pg_lambda + middleware" to create accelerated and fused query results
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#20Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…
IMHO, yahoo has bad investors. One of the proof is the firing of Marissa Meyer. As soon as she was hired, the reputation of yahoo has started to improve. All the fights of investors against her until they make her leave had the opposite effect. It takes years to build a reputation and it takes more years for this reputation to bring dividends. I have a similar feeling for Microsoft. The reputation has (slowly) improved since the arrival of Satya Nadella. I which he succeeds to make it back a big player.