Live data from Hacker News

The Architecture of a Large-Scale Web Search Engine, Circa 2019

0x65.dev

21–23 of 23 posts

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#21
post #17

Earlier quoted context omitted.

> There is plenty of documentation on data collected (see first posts regarding Human Web on the tech blog), how anonymization works, why record-linkability on data collected is prevented (and forbidden), etc. Furthermore, source code can be inspected, as well as traffic in the case documentation is not enough. Question is, is it opt-in data collection or do you make the choice for me? If it's opt-in, great. Otherwis…

You can apparently opt-out. On mobile but I think it was in one of their blog posts.

I really rather opt-in:-)

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#22
post #14

If I remember correctly, Yahoo Open Source their current and next generation Search Engine Vespa [1], why wasn't that used, and instead starting from scratch? [1] https://vespa.ai

Vespa is a very interesting project. But, it came quite late for us (Sept. 2017 [1]). Work on Cliqz Search started way earlier ~2013. Our work on Kubernetes and modernizing our architecture was also started around year 2016. [1] https://www.verizonmedia.com/press/open-sourcing-vespa-yahoo...

Did you guys choose Go over Java?

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#23
Thank you for great article. I've been writing my own search engine during last 3 years and it's funny how similar my setup is to yours with K8S/Kafka/Streams/Go/RocksDB. Actually about RocksDB - are you using it from Go(via gorocksdb?). Now and then I have hard time optimising RocksDB and still have very loose understanding how much RAM it will consume
Post reply on HN