Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Architecture of a Large-Scale Web Search Engine, Circa 2019

0x65.dev

Post reply on HN
Print view View original

11–20 of 23 posts

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#11

Post by solso »

leeoniya wrote: post #8
pythux wrote: post #5

Earlier quoted context omitted.

The "last time Cliqz came up" can be found here: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... . We have been posting multiple articles on our tech blog, explaining what we do and how we do it in great details. Your link points to an old thread of more than two years ago. There were more recent discussions about Cliqz no latter than this month, in particular here: https://news.ycombinator.com/item?i…

yeah sorry, should have said "last i remember". > We have been posting multiple articles on our tech blog, explaining what we do and how we do it in great details. it's possible to have both great tech and loose morals - the two are not mutually exclusive, and one does not absolve the other (e.g. facebook's social experiments) has there been a followup to any of the points brought up in the reddit thread?

[Disclaimer: I do work at Cliqz]

There is plenty of documentation on data collected (see first posts regarding Human Web on the tech blog), how anonymization works, why record-linkability on data collected is prevented (and forbidden), etc. Furthermore, source code can be inspected, as well as traffic in the case documentation is not enough. I believe that is a better proxy to assess "morality" than random accusations on reddit or opinions formed solely on a half-baked press releases.

Do we need to refute all miss-conceptions and FUD that might arise due to the fact that 1) we collect data to build our services (search) and 2) we are funded by a media company (VCs seem to be more pure for an unknown reason).

The answer is no. Cannot recall who said that it takes much more effort to refute BS than to generate it. (That does not go for your comment in particular, that's why we replied, but for many of the comments and some of content of the subredit that you mention.)

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#12

Post by ssubu »

ThePhysicist wrote: post #10
ssubu wrote: post #7

Earlier quoted context omitted.

[Disclaimer: work at Cliqz] We do not crawl the web in the traditional sense, our search was bootstrapped on query logs. It is the very reason we could succeed in building a search engine with minimal resources, in comparison to our competitors.We have written about this in a lot more detail here : How we collect data : https://www.0x65.dev/blog/2019-12-03/human-web-collecting-da... How we build the search using this…

Does really all of your data come from the human web project or do you also buy clickstream data from data brokers?

We speak about this is much more detail in this post (https://0x65.dev/blog/2019-12-05/a-new-search-engine.html), but in short, we prototyped our search initially with data we purchased from data-brokers. Once the concept was proven and HumanWeb was deployed (2015/2016), we rely only on our data.

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#13

Post by petra »

ssubu wrote: post #7
prox wrote: post #6

How does a new engine find webpages at start? Does it work from the Dns system and indexes every domain name? At a certain point it will follow links I presume, but how does it start?

[Disclaimer: work at Cliqz] We do not crawl the web in the traditional sense, our search was bootstrapped on query logs. It is the very reason we could succeed in building a search engine with minimal resources, in comparison to our competitors.We have written about this in a lot more detail here : How we collect data : https://www.0x65.dev/blog/2019-12-03/human-web-collecting-da... How we build the search using this…

What about tools for power searchers ? Google have abandoned us.

Are you planning to create strong tools in that area ?

For example: custom search engines, the NEAR operator, limit search to sites that don't update that often or aren't linked to very strong sites(against SEO), etc

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#15

Post by netankit »

ksec wrote: post #14

If I remember correctly, Yahoo Open Source their current and next generation Search Engine Vespa [1], why wasn't that used, and instead starting from scratch? [1] https://vespa.ai

Vespa is a very interesting project. But, it came quite late for us (Sept. 2017 [1]).

Work on Cliqz Search started way earlier ~2013. Our work on Kubernetes and modernizing our architecture was also started around year 2016.

[1] https://www.verizonmedia.com/press/open-sourcing-vespa-yahoo...

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#16

Post by streetcat1 »

yowlingcat wrote: post #9
streetcat1 wrote: post #3

Just be careful. Kubeflow is heading toward GCP only features (for example, they are dropping cert-manager), while you are betting on AWS.

Would it be fair to say you're saying that Kubeflow may become eventually incompatible with EKS?

Yes. Or more likely there will be a version of kubeflow for EKS. Alas amazon is pushing sage maker.

Note that kubeflow R&D is all done by google cloud.

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#17

Post by neiman »

solso wrote: post #11
leeoniya wrote: post #8

Earlier quoted context omitted.

yeah sorry, should have said "last i remember". > We have been posting multiple articles on our tech blog, explaining what we do and how we do it in great details. it's possible to have both great tech and loose morals - the two are not mutually exclusive, and one does not absolve the other (e.g. facebook's social experiments) has there been a followup to any of the points brought up in the reddit thread?

[Disclaimer: I do work at Cliqz] There is plenty of documentation on data collected (see first posts regarding Human Web on the tech blog), how anonymization works, why record-linkability on data collected is prevented (and forbidden), etc. Furthermore, source code can be inspected, as well as traffic in the case documentation is not enough. I believe that is a better proxy to assess "morality" than random accusation…

> There is plenty of documentation on data collected (see first posts regarding Human Web on the tech blog), how anonymization works, why record-linkability on data collected is prevented (and forbidden), etc. Furthermore, source code can be inspected, as well as traffic in the case documentation is not enough.

Question is, is it opt-in data collection or do you make the choice for me? If it's opt-in, great. Otherwise, I don't want to read your "plenty of documentation" and so on and so forth.

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#19

Post by markpapadakis »

I like how the blog post title is, likely, based on Google's Page and Brin seminal paper "The Anatomy of a Large-Scale Hypertextual Web Search Engine". :)

This blog post, and other in the series, mention RocksDB is used for the index, but it's not explicitly described how and to what end. I 'd love to know the details.

Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019

#20

Post by rewq4321 »

neiman wrote: post #17
solso wrote: post #11

Earlier quoted context omitted.

[Disclaimer: I do work at Cliqz] There is plenty of documentation on data collected (see first posts regarding Human Web on the tech blog), how anonymization works, why record-linkability on data collected is prevented (and forbidden), etc. Furthermore, source code can be inspected, as well as traffic in the case documentation is not enough. I believe that is a better proxy to assess "morality" than random accusation…

> There is plenty of documentation on data collected (see first posts regarding Human Web on the tech blog), how anonymization works, why record-linkability on data collected is prevented (and forbidden), etc. Furthermore, source code can be inspected, as well as traffic in the case documentation is not enough. Question is, is it opt-in data collection or do you make the choice for me? If it's opt-in, great. Otherwis…

You can apparently opt-out. On mobile but I think it was in one of their blog posts.
Post reply on HN