Live data from Hacker News

Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

oath.com

61–70 of 122 posts

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#61
Here is a snippet to decodeURIComponent for all the yql examples in the documentation. Makes it a bit easier to see the yql syntax.

  $('pre:contains("yql=")').each((i, el) => { el.innerText = el.innerText.replace(/\+/g, ' ').replace(/yql=(.+%3B)/, (m, p1) => 'yql=' + decodeURIComponent(p1)) })

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#62
post #58

I see it has tensor processing built in - http://docs.vespa.ai/documentation/tensor-intro.html Can this be used as a spark+tensorflow replacement ?

Note: TFoS is also a Yahoo open source project. The teams work together. https://github.com/yahoo/TensorFlowOnSpark

is tfos production-ready right now ? because i thought it was still experimental.

is it used inside Yahoo - because Vespa comes with its own tensor processing engine. I wondered who would use one over the other.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#63

$ cloc-git https://github.com/vespa-engine/vespa.git http://cloc.sourceforge.net v 1.60 T=64.10 s (224.5 files/s, 28276.3 lines/s) -------------------------------------------------------------------------------- Language files blank comment code -------------------------------------------------------------------------------- Java 6573 106215 102720 537097 C++ 3209 76542 19178 504855 C/C++ Header 2985 42731 57087 1583…

As an ex-employee, there could not be a better description of Yahoo! development than this.

Hey, there is no yinst/buildyblocks stuff here. That's kind of a must-have.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#64

$ cloc-git https://github.com/vespa-engine/vespa.git http://cloc.sourceforge.net v 1.60 T=64.10 s (224.5 files/s, 28276.3 lines/s) -------------------------------------------------------------------------------- Language files blank comment code -------------------------------------------------------------------------------- Java 6573 106215 102720 537097 C++ 3209 76542 19178 504855 C/C++ Header 2985 42731 57087 1583…

As an ex-employee, there could not be a better description of Yahoo! development than this.

Can you elaborate?

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#65
post #57
post #54

Earlier quoted context omitted.

2nd point seems valid. 1st one however suggests that people will try to use it to their own benefit - some of them to compete with Yahoo. Why else would they contribute to the project?

Proprietary code is expensive to maintain. Even though there's a dedicated team of a few dozen people that has been working on Vespa over the years there are thousands of developers who have been contributing to ElasticSearch, Lucene, and other projects in the open source world that are in a similar space. Getting contributions to Vespa will help it grow and evolve to make it better -- much like Yahoo did when it evo…

> We know that sharing Hadoop helped our competitors, but we also know that the revenue stream comes from ads.

This is not clear to me, can you please explain? I'm still stuck at thinking "if you help your competitors then you give up some of your market share".

EDIT: your pie analogy is really nice, I guess it means you grow the whole market by sharing tools like Vespa so you get a smaller slice of the bigger pie. I still don't get how the "revenue stream comes from ads" part relates to everything else.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#66
post #47
post #42

Earlier quoted context omitted.

When I was there (2003-2005), the big problem already seemed to be that a lot of engineering saw Yahoo as a tech company, while a lot of the business side saw Yahoo as a media company. Especially so after Yahoo conceded Search. So one side wants to pour money into getting technical leverage. The other side largely just wanted more effective ways of publishing and monetizing content. It doesn't really matter which sid…

Yahoo's largest problem while I was there (2004-2011) was that nobody could articulate what y! is and what it does. Actually, at employee orientation in 2004, they had a clear mission: to be a part of everything you do online: either by doing it, or cobranding it, with a goal of being in the top 3 of every internet vertical. Somehow, over time, this message was lost, and it was no longer ok to be #2 or #3 in some ver…

Interesting "to be a part of everything you do online: either by doing it, or cobranding it, with a goal of being in the top 3 of every internet vertical." Could you say that part of the reason they lost was a mission/purpose that from the above seems to be purely focused on internal value creation? No word on how they would benefit their customers? It does not say anything about the value it creates.

Put another way should your purpose be something customer focused?

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#67

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

See: DEC, Sun Microsystems. Engineering != Ability to make money. While none of us may like it, you almost always need a solid marketing and sales force to have a successful business.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#68

Earlier quoted context omitted.

Not sure that Juicero is a good example good engineering, "Do You Need a $400 Juicer?" https://m.youtube.com/watch?v=5lutHF5HhVA

I think you and the parent comment are both partially right - from having watched a number of teardowns, the Juicero was (if anything) massively over-engineered in terms of number of parts, cost of those parts, how they were machined, etc. It was also kind of clear that it went too far, that a more experienced hardware design and engineering team would have found (not a cheaper), but a more effective way to handle th…

This reminds me all too much of Coffee Equipment Company which back around 2005 built the world's greatest coffee machine, Clover. Unfortunately, at $11k per machine even Starbucks, after buying the company, wasn't able to find a profitable niche for it. Clover machines were, are, beautiful machines that are able to repeatedly and dependably turn out genuinely great cup after great caffeinated cup; the problem is that it was a solution in search of a problem, much like Jucero.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#69
post #60
post #51

I don't understand how such a move is profitable for a business. Can someone please point me to some articles that discuss this?

https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/ The point of Joel's article is that "Smart companies try to commoditize their products’ complements." For what I understand, Yahoo is a media company, and as so it may try to commoditize a natural complement of today's media companies, which is data search.

Good point, thanks for the excellent article!

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#70
post #48

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on... "Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source. Arguabl…

What do you mean you run the Open Source process at yahoo?

Hopefully Verizon's open source legacy will become even bigger and make it easier for these kinds of innovations to happen.

We've had some success a few cities over in Verizon Labs... https://verizon.github.io/

Post reply on HN