The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on... "Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source. Arguabl…
> Cloudera, Hortonworks, MapR to name three in a market some calculate will be worth $50bn by 2020 Very optimistic calculation...
The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on... "Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source. Arguabl…
What do you mean you run the Open Source process at yahoo? Hopefully Verizon's open source legacy will become even bigger and make it easier for these kinds of innovations to happen. We've had some success a few cities over in Verizon Labs... https://verizon.github.io/
My job is to manage the open source process for Oath (which is essentially Yahoo + AOL). That includes helping ensure we can publish code like this and the hundreds of other projects we publish too. I'm the one who cares about open source licenses, patent clauses, github permissions, etc. Many large tech companies have someone in a comparable role and some of us work together in the todogroup to help manage the way we do opensource. I'm beginning to meet the people in Verizon who do the same. I hope their open source legacy grows too. Heck I celebrate when Google, Amazon, and Comcast publish great code too. It's good for us all. But Vespa is a real treat. It's really really special to Yahoo and we are very hopeful that the Big Data community sees how many things they can do with this, at scale, the way we have.
"Vespa is the single greatest piece of software Yahoo ever built. It's like ElasticSearch but a hundred times better. I am so happy."
Laurie Voss Co-founder/COO of @npmjs
https://twitter.com/seldo/status/912876700542787585
Say I'm using ELK for log aggregation. Would Vespa be a good replacement? One pain point is ingest rate. How many "average" log lines per second can Vespa do per node?
It could be a replacement for the 'E', but the APIs are different enough that there's no drop-in replacement for the 'L' and 'K' and creating or making those compatible would be a significant effort. Would be great if someone did though :-)
Gotcha. On the ingest front, do you have any numbers around that? I see some benchmarks that focus on other (important) aspects like QPS but didn't catch anything on ingest.
Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…
Business is (mostly) about people and relationships. That's probably where Yahoo was lacking - not the engineering part.
Outside of general purpose libraries that are found in most software projects it's not based on any existing software. It's built from the ground up by Oath, and the companies that preceded it: Yahoo, FAST, Overture since early-mid 2000s. This article has some more details about the history: https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for... Disclamer: I work on the Vespa team in Trondheim, Norway.
Is Vespa relevant if you're not into writing Java? I.e., can it be used as a black box similar to Elasticsearch? From the repo, it looks like an absolutely huge, monolithic codebase. (It even bundles its own memory allocator!) Do you know if there are plans to break it up into smaller, more manageable pieces? While I haven't looked at what's required to deploy this beast, operationally speaking, it sounds it might be…
You don't need to plug in any Java code, you can use it with HTTP calls to read and write.
No plans to break it up into pieces (apart from already consisting of modules). It does one thing, it just happens to be a big thing :-)
If you have a mac of Linux box you can have it up and running in 10 minutes. Multi-node production deployments are no different because Vespa manages the nodes, not you directly.
Outside of general purpose libraries that are found in most software projects it's not based on any existing software. It's built from the ground up by Oath, and the companies that preceded it: Yahoo, FAST, Overture since early-mid 2000s. This article has some more details about the history: https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for... Disclamer: I work on the Vespa team in Trondheim, Norway.
Doug Cutting, the creator of Lucene, was an employee of Yahoo. So does Vespa share any technologies with Lucene?
It could be a replacement for the 'E', but the APIs are different enough that there's no drop-in replacement for the 'L' and 'K' and creating or making those compatible would be a significant effort. Would be great if someone did though :-)
Gotcha. On the ingest front, do you have any numbers around that? I see some benchmarks that focus on other (important) aspects like QPS but didn't catch anything on ingest.
Write speed (add or update) is typically between a few thousand to a few tens of thousands operations per second per node sustained, depending on sizeof data etc.
Sustaining throughput over long time is important and often overlooked mentioned in benchmarks.
What do you mean you run the Open Source process at yahoo? Hopefully Verizon's open source legacy will become even bigger and make it easier for these kinds of innovations to happen. We've had some success a few cities over in Verizon Labs... https://verizon.github.io/
My job is to manage the open source process for Oath (which is essentially Yahoo + AOL). That includes helping ensure we can publish code like this and the hundreds of other projects we publish too. I'm the one who cares about open source licenses, patent clauses, github permissions, etc. Many large tech companies have someone in a comparable role and some of us work together in the todogroup to help manage the way w…