Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

oath.com

Post reply on HN
Print view View original

41–50 of 122 posts

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#41

Post by wonderous »

overcast wrote: post #40
madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

Juicero had an amazing engineering culture. Look where that went. You need a proper management team to wrangle it in. Clearly Yahoo has made mistakes in that aspect.

Not sure that Juicero is a good example good engineering, "Do You Need a $400 Juicer?"

https://m.youtube.com/watch?v=5lutHF5HhVA

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#42

Post by vidarh »

madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

When I was there (2003-2005), the big problem already seemed to be that a lot of engineering saw Yahoo as a tech company, while a lot of the business side saw Yahoo as a media company.

Especially so after Yahoo conceded Search.

So one side wants to pour money into getting technical leverage. The other side largely just wanted more effective ways of publishing and monetizing content.

It doesn't really matter which side was right, only that they were at times pulling in wildly different directions in terms of what they believed it was important to invest in.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#43

Post by michaelbuckbee »

wonderous wrote: post #41
overcast wrote: post #40

Earlier quoted context omitted.

Juicero had an amazing engineering culture. Look where that went. You need a proper management team to wrangle it in. Clearly Yahoo has made mistakes in that aspect.

Not sure that Juicero is a good example good engineering, "Do You Need a $400 Juicer?" https://m.youtube.com/watch?v=5lutHF5HhVA

I think you and the parent comment are both partially right - from having watched a number of teardowns, the Juicero was (if anything) massively over-engineered in terms of number of parts, cost of those parts, how they were machined, etc.

It was also kind of clear that it went too far, that a more experienced hardware design and engineering team would have found (not a cheaper), but a more effective way to handle those same challenges.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#44

Post by andreer »

sanxiyn wrote: post #35

Vespa, using its own malloc: https://github.com/vespa-engine/vespa/tree/master/vespamallo...

The reason is simply, for performance. To avoid having to go to the kernel every time we need to allocate memory for a query, and avoid having to clear memory on free/reuse. It is made for Vespa, but also used for other programs.

Similar in purpose to Google's TCMalloc: http://goog-perftools.sourceforge.net/doc/tcmalloc.html

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#45

Post by sanxiyn »

andreer wrote: post #44
sanxiyn wrote: post #35

Vespa, using its own malloc: https://github.com/vespa-engine/vespa/tree/master/vespamallo...

The reason is simply, for performance. To avoid having to go to the kernel every time we need to allocate memory for a query, and avoid having to clear memory on free/reuse. It is made for Vespa, but also used for other programs. Similar in purpose to Google's TCMalloc: http://goog-perftools.sourceforge.net/doc/tcmalloc.html

How is it better than TCMalloc? (If it isn't, it probably should be replaced by TCMalloc.)

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#46

Post by andreer »

sanxiyn wrote: post #45
andreer wrote: post #44

Earlier quoted context omitted.

The reason is simply, for performance. To avoid having to go to the kernel every time we need to allocate memory for a query, and avoid having to clear memory on free/reuse. It is made for Vespa, but also used for other programs. Similar in purpose to Google's TCMalloc: http://goog-perftools.sourceforge.net/doc/tcmalloc.html

How is it better than TCMalloc? (If it isn't, it probably should be replaced by TCMalloc.)

In our tests, vespamalloc has simply been faster. I don't know how in-depth the analysis has been as to why, but obviously vespamalloc is written and tuned for Vespa so that is a likely factor.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#47

Post by toast0 »

vidarh wrote: post #42
madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

When I was there (2003-2005), the big problem already seemed to be that a lot of engineering saw Yahoo as a tech company, while a lot of the business side saw Yahoo as a media company. Especially so after Yahoo conceded Search. So one side wants to pour money into getting technical leverage. The other side largely just wanted more effective ways of publishing and monetizing content. It doesn't really matter which sid…

Yahoo's largest problem while I was there (2004-2011) was that nobody could articulate what y! is and what it does. Actually, at employee orientation in 2004, they had a clear mission: to be a part of everything you do online: either by doing it, or cobranding it, with a goal of being in the top 3 of every internet vertical.

Somehow, over time, this message was lost, and it was no longer ok to be #2 or #3 in some verticals (like search), and having very healthy profit margins wasn't enough either. The Microsoft search deal was a bad deal, poorly executed, as well: Microsoft couldn't meet the monetization and performance requirements from the start (Microsoft paid out of pocket for a while for this), y! lost control of an important product, and the staffing reductions to search that were supposed to be enabled never came.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#48

Post by gyehuda »

madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on...

"Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source.

Arguably Yahoo!’s greatest legacy once it is a division of Verizon will be big data, after one of its engineers – Doug Cutting – wrote an open-source implementation of Google’s MapReduce that became Hadoop. What followed was an entire ecosystem of startups and projects crunching data at scale – Cloudera, Hortonworks, MapR to name three in a market some calculate will be worth $50bn by 2020."

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#49

Post by hypertexthero »

madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

Yes.

Steve Jobs responding to a question about OpenDoc in 1997:

> You’ve got to start with the customer experience and work backwards to the technology.

https://mikecanex.wordpress.com/2011/06/08/wwdc-1997-video-s...

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#50

Post by RealJon »

aidos wrote: post #5
cies wrote: post #4

Wow this project is humongous! https://github.com/vespa-engine/vespa I'm really curious how it compares to Lucene/ElasticSearch/ELK, which is currently my tool of choice for (faceted) search and recommendation.

I think that's actually the broadest root folder I've ever seen on a project! It's really hard to know where to start looking. Is anyone familiar with the internals?

Sorry about that - we haven't really optimized the module structure for newcomer comprehension. If you tell me what you want to are looking for I can point you to the right place.
Post reply on HN