Live data from Hacker News

Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

oath.com

81–90 of 122 posts

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#81
post #37
post #15

Earlier quoted context omitted.

Found some comparison (on the root page of their project website, my bad). http://vespa.ai/#featurematrix Next thing I'd really like to know is what existing software it builds on top of.

Outside of general purpose libraries that are found in most software projects it's not based on any existing software. It's built from the ground up by Oath, and the companies that preceded it: Yahoo, FAST, Overture since early-mid 2000s. This article has some more details about the history: https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for... Disclamer: I work on the Vespa team in Trondheim, Norway.

Doug Cutting, the creator of Lucene, was an employee of Yahoo. So does Vespa share any technologies with Lucene?

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#84
post #82

Say I'm using ELK for log aggregation. Would Vespa be a good replacement? One pain point is ingest rate. How many "average" log lines per second can Vespa do per node?

It could be a replacement for the 'E', but the APIs are different enough that there's no drop-in replacement for the 'L' and 'K' and creating or making those compatible would be a significant effort. Would be great if someone did though :-)

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#85
post #58

Earlier quoted context omitted.

Note: TFoS is also a Yahoo open source project. The teams work together. https://github.com/yahoo/TensorFlowOnSpark

is tfos production-ready right now ? because i thought it was still experimental. is it used inside Yahoo - because Vespa comes with its own tensor processing engine. I wondered who would use one over the other.

TensorFlow on Spark is for learning, Vespa is for serving. Where Vespa excels is in evaluating a learned model very quickly over lots of documents. We're working on providing support for running models learned with TensorFlow directly. For now people make the translation on their own.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#86
post #48

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on... "Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source. Arguabl…

> Cloudera, Hortonworks, MapR to name three in a market some calculate will be worth $50bn by 2020

Very optimistic calculation...

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#87
post #60
post #51

I don't understand how such a move is profitable for a business. Can someone please point me to some articles that discuss this?

https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ The point of Joel's article is that "Smart companies try to commoditize their products’ complements." For what I understand, Yahoo is a media company, and as so it may try to commoditize a natural complement of today's media companies, which is data search.

That article is a classic but I don't really understand how it applies in this case.

Smart companies commoditize their products' complements (something that needs to be bought with the product) so that whoever wants buy their product has a large variety of offerings to select from. For instance, MS-DOS's ability to run on any standard PC architecture machine commoditized the PC.

It's not clear how commoditizing data search makes selling media to consumers easier, because consumers of media don't buy data search. In fact they expect it to come for free from the media company.

Might it not have the opposite effect instead? That is, it makes starting up a media company cheaper and allows competitors to spend more money on acquiring media?

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#88
post #37
post #15

Earlier quoted context omitted.

Found some comparison (on the root page of their project website, my bad). http://vespa.ai/#featurematrix Next thing I'd really like to know is what existing software it builds on top of.

Outside of general purpose libraries that are found in most software projects it's not based on any existing software. It's built from the ground up by Oath, and the companies that preceded it: Yahoo, FAST, Overture since early-mid 2000s. This article has some more details about the history: https://www.cnbc.com/2017/09/26/yahoo-open-sources-vespa-for... Disclamer: I work on the Vespa team in Trondheim, Norway.

Is Vespa relevant if you're not into writing Java? I.e., can it be used as a black box similar to Elasticsearch?

From the repo, it looks like an absolutely huge, monolithic codebase. (It even bundles its own memory allocator!) Do you know if there are plans to break it up into smaller, more manageable pieces?

While I haven't looked at what's required to deploy this beast, operationally speaking, it sounds it might be daunting to run, and for non-"big data" applications might very well be overkill as an alternative to Elasticsearch.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#89
post #64

Earlier quoted context omitted.

Can you elaborate?

Even split between Java / C++, with at least 4 different build systems: 2 of which are effectively the next generation of the other 2. 2 different UNIX shells and 5 other dynamic programming languages. Not the end of the world, and there are probably good reasons for a lot of it (maybe there are bindings for various languages, maybe some of it is misindeitified, maybe they're harnessing 2 large bodies of existing wor…

No, this is run by a single, very cohesive, very remote team.

There is some misidentification in your list (our .def files are nothing to do with Teamcenter). We use 2 languages because we Java and C++ have different strengths which make each suitable at different layers of the architecture. The rest is a combination of "for good reasons" and "leftover scraps" :-)

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#90
post #65
post #57

Earlier quoted context omitted.

Proprietary code is expensive to maintain. Even though there's a dedicated team of a few dozen people that has been working on Vespa over the years there are thousands of developers who have been contributing to ElasticSearch, Lucene, and other projects in the open source world that are in a similar space. Getting contributions to Vespa will help it grow and evolve to make it better -- much like Yahoo did when it evo…

> We know that sharing Hadoop helped our competitors, but we also know that the revenue stream comes from ads. This is not clear to me, can you please explain? I'm still stuck at thinking "if you help your competitors then you give up some of your market share". EDIT: your pie analogy is really nice, I guess it means you grow the whole market by sharing tools like Vespa so you get a smaller slice of the bigger pie. I…

Some companies make money by licensing software. They are less willing to publish code since, to them, code is revenue and they don't want to give it away. Internet-media companies view code more like a required expense. Giving code away 1. helps reduce carrying costs 2. attracts developers 3. forces out of dependency debt, 4. encourages developers to make their code better 5. builds skills that transfer to other companies 6. make it easier for us to find people who already know the tech we use. etc. etc. There are TONS of upsides. Case in point: we can hire a Hadoop developer. Had we never open sourced the code, we'd have to keep an army of developers in house for a decade. Instead, they can leave to form startup companies (if that's what they want to do) and we still get the benefit of their creative effort. We also invested in one of those companies, so when they make money, we do too. Developers want to work at Yahoo (really) since it will help them build skills they can take elsewhere, or they can stick around and use those skills internally. Either way, why would a developer work on proprietary code when they can work on open source code which will give them more options.

There's a good give and take in the tech world. Sure, our code has helped Facebook and Google eat our lunch. But we don't blame the tech sharing for that -- since they've contributed quite a bit too. We work rather closely with them on a bunch of projects -- which help us all.

Sure, there are some projects that we'd consider the "secret sauce" that really differentiates us from others. We won't open source those. But a lot of code is there 'cuz we need to move bits around quickly. Sharing that code is not going to make or break a multibillion$ enterprise. It's actually going to help make it better in the long run.

To so make money: our sales people to do that, not the tech people. The sales people are given amazing products, and huge audiences to sell to the advertisers. Whereas a podcast might have a million subscribers, a popular radio program have 10 millions listeners, a TV show getting 50 million viewers, or a wireless company have 150 million subscribers, we have over a billion users -- and the advertisers LOVE that. So we all sell ads, and who ever does that better wins. But as tech folks, we're collaborative. It's not really a new thing, it's very much part of the fabric that has helped the internet evolve.

Post reply on HN