Earlier quoted context omitted.
Found some comparison (on the root page of their project website, my bad). http://vespa.ai/#featurematrix Next thing I'd really like to know is what existing software it builds on top of.
Outside of general purpose libraries that are found in most software projects it's not based on any existing software. It's built from the ground up by Oath, and the companies that preceded it: Yahoo, FAST, Overture since early-mid 2000s. This article has some more details about the history: https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for... Disclamer: I work on the Vespa team in Trondheim, Norway.
Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
81–90 of 122 posts
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#82Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#83Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#84Say I'm using ELK for log aggregation. Would Vespa be a good replacement? One pain point is ingest rate. How many "average" log lines per second can Vespa do per node?
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#85Earlier quoted context omitted.
Note: TFoS is also a Yahoo open source project. The teams work together. https://github.com/yahoo/TensorFlowOnSpark
is tfos production-ready right now ? because i thought it was still experimental. is it used inside Yahoo - because Vespa comes with its own tensor processing engine. I wondered who would use one over the other.
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#86Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…
The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on... "Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source. Arguabl…
Very optimistic calculation...
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#87I don't understand how such a move is profitable for a business. Can someone please point me to some articles that discuss this?
https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ The point of Joel's article is that "Smart companies try to commoditize their products’ complements." For what I understand, Yahoo is a media company, and as so it may try to commoditize a natural complement of today's media companies, which is data search.
Smart companies commoditize their products' complements (something that needs to be bought with the product) so that whoever wants buy their product has a large variety of offerings to select from. For instance, MS-DOS's ability to run on any standard PC architecture machine commoditized the PC.
It's not clear how commoditizing data search makes selling media to consumers easier, because consumers of media don't buy data search. In fact they expect it to come for free from the media company.
Might it not have the opposite effect instead? That is, it makes starting up a media company cheaper and allows competitors to spend more money on acquiring media?
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#88Earlier quoted context omitted.
Found some comparison (on the root page of their project website, my bad). http://vespa.ai/#featurematrix Next thing I'd really like to know is what existing software it builds on top of.
Outside of general purpose libraries that are found in most software projects it's not based on any existing software. It's built from the ground up by Oath, and the companies that preceded it: Yahoo, FAST, Overture since early-mid 2000s. This article has some more details about the history: https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for... Disclamer: I work on the Vespa team in Trondheim, Norway.
From the repo, it looks like an absolutely huge, monolithic codebase. (It even bundles its own memory allocator!) Do you know if there are plans to break it up into smaller, more manageable pieces?
While I haven't looked at what's required to deploy this beast, operationally speaking, it sounds it might be daunting to run, and for non-"big data" applications might very well be overkill as an alternative to Elasticsearch.
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#89Earlier quoted context omitted.
Can you elaborate?
Even split between Java / C++, with at least 4 different build systems: 2 of which are effectively the next generation of the other 2. 2 different UNIX shells and 5 other dynamic programming languages. Not the end of the world, and there are probably good reasons for a lot of it (maybe there are bindings for various languages, maybe some of it is misindeitified, maybe they're harnessing 2 large bodies of existing wor…
There is some misidentification in your list (our .def files are nothing to do with Teamcenter). We use 2 languages because we Java and C++ have different strengths which make each suitable at different layers of the architecture. The rest is a combination of "for good reasons" and "leftover scraps" :-)
Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine
#90Earlier quoted context omitted.
Proprietary code is expensive to maintain. Even though there's a dedicated team of a few dozen people that has been working on Vespa over the years there are thousands of developers who have been contributing to ElasticSearch, Lucene, and other projects in the open source world that are in a similar space. Getting contributions to Vespa will help it grow and evolve to make it better -- much like Yahoo did when it evo…
> We know that sharing Hadoop helped our competitors, but we also know that the revenue stream comes from ads. This is not clear to me, can you please explain? I'm still stuck at thinking "if you help your competitors then you give up some of your market share". EDIT: your pie analogy is really nice, I guess it means you grow the whole market by sharing tools like Vespa so you get a smaller slice of the bigger pie. I…
There's a good give and take in the tech world. Sure, our code has helped Facebook and Google eat our lunch. But we don't blame the tech sharing for that -- since they've contributed quite a bit too. We work rather closely with them on a bunch of projects -- which help us all.
Sure, there are some projects that we'd consider the "secret sauce" that really differentiates us from others. We won't open source those. But a lot of code is there 'cuz we need to move bits around quickly. Sharing that code is not going to make or break a multibillion$ enterprise. It's actually going to help make it better in the long run.
To so make money: our sales people to do that, not the tech people. The sales people are given amazing products, and huge audiences to sell to the advertisers. Whereas a podcast might have a million subscribers, a popular radio program have 10 millions listeners, a TV show getting 50 million viewers, or a wireless company have 150 million subscribers, we have over a billion users -- and the advertisers LOVE that. So we all sell ads, and who ever does that better wins. But as tech folks, we're collaborative. It's not really a new thing, it's very much part of the fabric that has helped the internet evolve.