Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

oath.com

Post reply on HN
Print view View original

71–80 of 122 posts

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#71

Post by coredog64 »

tw04 wrote: post #67
madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

See: DEC, Sun Microsystems. Engineering != Ability to make money. While none of us may like it, you almost always need a solid marketing and sales force to have a successful business.

Sun was making money, their problem was cash flow. Their primary customers were on Wall Street. They borrowed a bunch of money prior to the GFC, and right when they needed their Wall Street customers to be placing orders to cover payments on the debt, their customers were tightening their belts.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#72

Post by toast0 »

ismail wrote: post #66
toast0 wrote: post #47

Earlier quoted context omitted.

Yahoo's largest problem while I was there (2004-2011) was that nobody could articulate what y! is and what it does. Actually, at employee orientation in 2004, they had a clear mission: to be a part of everything you do online: either by doing it, or cobranding it, with a goal of being in the top 3 of every internet vertical. Somehow, over time, this message was lost, and it was no longer ok to be #2 or #3 in some ver…

Interesting "to be a part of everything you do online: either by doing it, or cobranding it, with a goal of being in the top 3 of every internet vertical." Could you say that part of the reason they lost was a mission/purpose that from the above seems to be purely focused on internal value creation? No word on how they would benefit their customers? It does not say anything about the value it creates. Put another way…

The customer focused version of this missionis 'to help users do everything they want to do online'. Part of doing everything is providing some form of discoverability, which takes the form of linking to other products -- big traffic drivers are links on the front page to major products, and integration with search, but also relevant cross linking within the more specific verticals -- at Y! Travel, we would use flickr photos, events from upcoming, share restaurant data with local, we would have done things with yahoo calendar if it was more usable, etc.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#74

Post by afandian »

madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

What does "engg" stand for? A particular field of engineering?

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#75

Post by TallGuyShort »

zymhan wrote: post #64
zimpenfish wrote: post #28

Earlier quoted context omitted.

As an ex-employee, there could not be a better description of Yahoo! development than this.

Can you elaborate?

Even split between Java / C++, with at least 4 different build systems: 2 of which are effectively the next generation of the other 2. 2 different UNIX shells and 5 other dynamic programming languages. Not the end of the world, and there are probably good reasons for a lot of it (maybe there are bindings for various languages, maybe some of it is misindeitified, maybe they're harnessing 2 large bodies of existing work, 1 in Java and 1 in C++), but it may suggest a lot of people just doing things their own way trying to carve out their own niche, without a cohesive philosophy across the system, which mirrors my experience with Yahoo's engineering org.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#76

Post by fasquoika »

michaelbuckbee wrote: post #43
wonderous wrote: post #41

Earlier quoted context omitted.

Not sure that Juicero is a good example good engineering, "Do You Need a $400 Juicer?" https://m.youtube.com/watch?v=5lutHF5HhVA

I think you and the parent comment are both partially right - from having watched a number of teardowns, the Juicero was (if anything) massively over-engineered in terms of number of parts, cost of those parts, how they were machined, etc. It was also kind of clear that it went too far, that a more experienced hardware design and engineering team would have found (not a cheaper), but a more effective way to handle th…

>from having watched a number of teardowns, the Juicero was (if anything) massively over-engineered in terms of number of parts, cost of those parts, how they were machined, etc.

What you mean to say is that the Juicero was overbuilt and underengineered. As the saying goes, "anyone can build a bridge, it takes an engineer to barely build it".

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#78

Post by mbrookes »

gyehuda wrote: post #48
madmax108 wrote: post #17

Yahoo time and again releases open source software which is super super helpful to the community at large. But it always makes me wonder why an org with such an amazing engg culture (multiple anecdotes from friends who were at Yahoo, plus the amazing experiences at Yahoo OpenHack each year as a testament to this) could be run into the ground. Really goes to show that engg != business and unless you have a firm busine…

The Register did a good job describing the dichotomy between the product business and the tech side of the business here, related to a previous open source project Yahoo published (disclaimer, I run the open source process at Yahoo) https://www.theregister.co.uk/2017/03/23/yahoo_tensorflow_on... "Over the decades Yahoo! has contributed substantially to the greater good, publishing its own code as open source. Arguabl…

Congrats on the Vespa release, it looks phenomenal!

Generally when you disclose something (like the fact that you run the open source process at Yahoo) that's a disclosure, rather a disclaimer.

A disclaimer might be considered the opposite "I don't run the open source process at Yahoo, but...", or, more commonly, "IANAL".

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#79

Post by FractalNerve »

FractalNerve wrote: post #19
FractalNerve wrote: post #6

Earlier quoted context omitted.

May I ask some stupid questions? :/ I don't quite get the diagram of the Vespa Architecture. Is Vespa a middleware between database engine and query parser? This is what puzzles me. If so, are there other such middlewares available for ie. PostgresSQL that allow hooking "Query Templating Models" (that is it?) generated via Machine-Learning Models? Is it way more complicated than that, or did they overengineer the pro…

> How else could you solve what Vespa does using Rust, Go, or C/C++ libraries? Let me try myself answering my own question, I hope someone hops in and tells me where I'm wrong or how else to improve :) 1) Get PostgresSQL exntensions via "package manager" pgxnclient 1.1) pg_bouncer - For connetion pooling 1.2) yoke - As a high-availability cluster manager with auto-failover and automated cluster recovery 1.3) prestodb…

I thik basing this on PostgresSQL was wrong now and believe that a meaningful approach at creating a Vespa alternative yourself is basing this on a Content-Adressable-Storage[1] and adding a DB-Layer ontop (ie. using AUFS).

It would have following properties: decentralized, distributed, resilient, highly-available, software-defined storage & retrieval system.

According to http://vespa.ai/#featurematrix:

        FEATURE	                    VESPA	ELASTIC SEARCH	RELATIONAL DATABASES
        ACID transactions			                •••
        Optimized for analytics		        •••	        ••
        Optimized for serving	    •••	        •	        ••
        Scalable	            •••	        ••	        •
        Easy to operate at scale    ••	                        •
        Text search	            •••	        ••	        •
        Machine learned ranking	    •••	        •               2.1.2) - 2.1.4)	
        Middleware logic container  •••		                1.4)
        Live reconfiguration	    •••	                        1.2)
And yet I've to admit that even if the Github repository looks quite chaotic, making an alternative, even using existing technologies would be big feat.

Initially I would've chosen PostgresSQL as a base, but the "HA-Layer" is something that shouldn't be decoupled and not a later thought. That's why CAS is a much better form of integration. Also integrating the PostgresSQL Engine into a zfs kernel extension ie. would be a mess. And integrating the database engine into a a distributed p2p algorithm would only add compatability issues an no real advantages.

[1] https://en.wikipedia.org/wiki/Content-addressable_storage#Op...

PS: Clever aquisition by Docker! "Infinit.sh is a content-addressable and decentralized (peer-to-peer) storage platform that was acquired by Docker Inc." And in my eyes one of the best implementations and easiest targets that allow adding a database-layer ontop.

Re: Open-Sourcing Vespa, Yahoo’s Data Processing and Serving Engine

#80

Post by tedd4u »

At Flickr, we worked closely with the Vespa team from 2011 through 2016 on a wide range of advancements:

   * partial document refeeding (i.e. expedite indexing a new field to 20+ billion documents without refeeding everything and staying online handling 100M+ free text queries a day)
   * visual similarity search - check out the tensor ranking features [1] [2]
   * online elasticity - add/remove replicas / shards online. A must when it could take weeks+ to re-feed from scratch. This is non-trivial to make work smoothly at scale. 
   * latency / tail-latency on complex queries. p90 reduction from 3,000 to 30 ms.
This is a major gift to the open-source community of a battle-tested search engine that works reliably without babysitting with very large datasets, and simultaneous high query / high feed volumes. Huge debt of gratitude to the team in Trondheim and Verizon/Oath/Yahoo legal & management teams for making this happen. :+1:

[1] http://docs.vespa.ai/documentation/tensor-intro.html [2] http://docs.vespa.ai/documentation/tensor-user-guide.html

Post reply on HN