Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

oath.com

Post reply on HN
Print view View original

111–120 of 122 posts

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#111

Post by yesiamyourdad »

KGIII wrote: post #104
abiox wrote: post #59

i wonder if this is able to squeeze something useful out of Yahoo Answers.

I am not trying to be snarky when I say this, so I'll make it longer than a single word. No. That is, Y! Answers was/is a great idea with terrible results. I'm not sure if it should have been moderated better, or if it should have been marketed better. Hell, maybe it should have had a basic literacy test prior to being allowed to post and answer? I could actually come up with a few hundred ways to have made it better…

Yep, the old maxim "garbage in, garbage out". Yahoo! was the biggest internet company out there (maybe AOL was bigger). You had the whole population in all its warty glory using Yahoo as the default search engine, landing page, email, and who knows what else.

Plus answers was done at a time when we were extremely naive about the wisdom of crowds and open source knowledge. Later efforts learned from those early 2000s failures. I'm pretty sure Joel Spolsky specifically called out Yahoo Answers when he was talking about what Stack Overflow was going to do right in his early announcements.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#112

Post by gyehuda »

KGIII wrote: post #104
abiox wrote: post #59

i wonder if this is able to squeeze something useful out of Yahoo Answers.

I am not trying to be snarky when I say this, so I'll make it longer than a single word. No. That is, Y! Answers was/is a great idea with terrible results. I'm not sure if it should have been moderated better, or if it should have been marketed better. Hell, maybe it should have had a basic literacy test prior to being allowed to post and answer? I could actually come up with a few hundred ways to have made it better…

FWIW, there was a research team at Yahoo who would actually get insights about the way people used language (relevant for contextual search) from Yahoo Answers. They were more active during the earlier years when the content quality was much higher. I don't know if they used Vespa in their mining process, but it would not surprise me if they did. Vespa is used in many projects because it's just that good. So if you are thinking of using it -- for Yahoo Answers, for Quora, whatever, go for it. I know a cancer researcher/engineer who wants to use Vespa for serving clinical reports and trial outcomes.

As for the current Answers site, well, we'll see what happens. I know the PM, a delightful person. I don't know the plan. But if there is a plan to make something useful from it, apply. It apparently makes money (otherwise it would have been killed long ago), and that means there's something to work with. When I started at Yahoo I was hoping to get onto the Groups team for the same reason -- a huge challenge to fix something that could be made cool again.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#113

Post by eip »

weego wrote: post #36
relyio wrote: post #31

Earlier quoted context omitted.

I haven't had the longest career, but the little I have see empirically confirmed to me that there's actually a strong connection between engineering culture and business success. Of course, a great engineering culture alone is not enough, but it can act as multiplicative factor.

In an engineering centric product. Why would a mom and pop email / news / casual gaming portal succeed directly because of good engineering? Time and time again we massively over-play our role in many business stories.

Yahoo mail was garbage. News was not useful and mostly ads. Search was not great.

If you build crappy tools don't be surprised when people don't find a lot of value in them.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#114

Post by eip »

talad wrote: post #22
reacweb wrote: post #20

Earlier quoted context omitted.

IMHO, yahoo has bad investors. One of the proof is the firing of Marissa Meyer. As soon as she was hired, the reputation of yahoo has started to improve. All the fights of investors against her until they make her leave had the opposite effect. It takes years to build a reputation and it takes more years for this reputation to bring dividends. I have a similar feeling for Microsoft. The reputation has (slowly) improv…

Meyer basically destroyed a healthy, but not growing, company in the span of 4 years through short-term number-massaging actions at the cost of their competent engineers and core product development. Nadella isn't making a particularly strong case for MS either with his seeming distaste for any product of theirs that isn't either mobile or cloud-based (like the ones where they actually have a monopoly to build off an…

>Meyer basically destroyed a healthy, but not growing, company in the span of 4 years

So she did exactly what Google paid her to do. Eliminate a competitor.

How dumb do you have to be to hire a CEO from high in the ranks of your biggest rival?

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#115

Post by lobster_johnson »

RealJon wrote: post #97
lobster_johnson wrote: post #88

Earlier quoted context omitted.

Is Vespa relevant if you're not into writing Java? I.e., can it be used as a black box similar to Elasticsearch? From the repo, it looks like an absolutely huge, monolithic codebase. (It even bundles its own memory allocator!) Do you know if there are plans to break it up into smaller, more manageable pieces? While I haven't looked at what's required to deploy this beast, operationally speaking, it sounds it might be…

You don't need to plug in any Java code, you can use it with HTTP calls to read and write. No plans to break it up into pieces (apart from already consisting of modules). It does one thing, it just happens to be a big thing :-) If you have a mac of Linux box you can have it up and running in 10 minutes. Multi-node production deployments are no different because Vespa manages the nodes, not you directly.

From what I can tell from the documentation, schemas ("search definitions" in Vespa parlance) are stored as files that are part of the "application package" (which I don't know what is yet); it doesn't seem like you can make dynamic schema changes through a REST API, and some schema changes actually require search nodes to be restarted?(!) Doesn't Vespa have anything similar to the Elasticsearch mappings API for dynamically updating a schema?

This may be a difference in mindset, but we generally let apps define the schema, so it's in the developer's realm of responsibility, not an operational concern. We also have apps (currently using Elasticsearch) that manage the schema automatically, derived from a high-level application definition. With ES, the app creates a new index with new mappings, shovels data into the new index, and then activates the new index. But it uses the ES APIs to do this, and restarting anything is not on the table.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#116

Post by gyehuda »

mbrookes wrote: post #78
gyehuda wrote: post #48

Earlier quoted context omitted.

The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on... "Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source. Arguabl…

Congrats on the Vespa release, it looks phenomenal! Generally when you disclose something (like the fact that you run the open source process at Yahoo) that's a disclosure, rather a disclaimer. A disclaimer might be considered the opposite "I don't run the open source process at Yahoo, but...", or, more commonly, "IANAL".

oops, thanks. I meant to disclose my affiliation, not disclaim it.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#117

Post by toast0 »

vidarh wrote: post #110
toast0 wrote: post #47

Earlier quoted context omitted.

Yahoo's largest problem while I was there (2004-2011) was that nobody could articulate what y! is and what it does. Actually, at employee orientation in 2004, they had a clear mission: to be a part of everything you do online: either by doing it, or cobranding it, with a goal of being in the top 3 of every internet vertical. Somehow, over time, this message was lost, and it was no longer ok to be #2 or #3 in some ver…

That sounds like an incredibly unclear mission to me. It also sounds like the mission of a media company, not a tech company, but without making that distinction clear...

What's unclear about it?

And/or what do you propose that fits what Yahoo! did (or should have been doing) at any point in its life? Alternatively, provide a clear mission statement than encompasses what any large company with lots of products does -- GE, IBM, Google, etc.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#118

Post by hboon »

tallanvor wrote: post #18

Wow... I'm guessing (hoping) their lawyers made sure to go over their old agreements with a very fine comb to make sure their license for the software allowed them to open source this. Don't get me wrong, they've added a lot to it, but there's a lot of code in there that could only have come from their purchase of Overture, who had purchased AllTheWeb from FAST (which was itself purchased by Microsoft).

I was with FAST. Were you there, by any chance?

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#119

Post by vidarh »

toast0 wrote: post #117
vidarh wrote: post #110

Earlier quoted context omitted.

That sounds like an incredibly unclear mission to me. It also sounds like the mission of a media company, not a tech company, but without making that distinction clear...

What's unclear about it? And/or what do you propose that fits what Yahoo! did (or should have been doing) at any point in its life? Alternatively, provide a clear mission statement than encompasses what any large company with lots of products does -- GE, IBM, Google, etc.

Well for starters it is clearly hyperbole. There is no way Yahoo particularly wanted to be part of everyones porn surfing for example - Tumblr notwithstanding. And you'd find plenty of other niches where Yahoo made no attempt whatsoever to enter.

And outside of investment in Alibaba, their involvement in online shopping was marginal. In fact, Yahoo's premium services was always marginal across the board - I spent years trying to push product teams to find services they could potentially charge for in Europe, and from what I can tell my replacements had no luck in that respect either.

Instead Yahoo actually divested a number of services in that respect. Yahoo! Personals was sold to Match.com, and while it may have been co-branded for a while, that's long since lapsed in most markets.

So either Yahoo dramatically failed to follow this up in any reasonable way, or the actual intent was a lot narrower. Or both.

But what does "be a part of" even mean? Have an ad on the page? Be recognised by the user as providing the service? Providing content? Provide the tech? It really says nothing. It doesn't say anything about the purpose either. Is it to sell more ads? To grow the brand? To drive people to premium services?

It's a non-statement.

To me it's a statement that's basically carte blanche for whatever management happens to want right now, but it's certainly not providing focus.

And while I would interpret it as directed towards media, I suspect it did nothing to e.g. clarify to the company whether Yahoo was a media company or a tech company. Maybe intentionally, because there were a lot of people in engineering when I was there who did not want to accept that what mattered to Yahoo was connecting eyeballs to content, and that.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#120

Post by ksec »

madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

And they used to be a BSD shop as well, then Marrisa changed it to Linux I think.
Post reply on HN