The Architecture of a Large-Scale Web Search Engine, Circa 2019
1–10 of 23 posts
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#2I hadn't heard of it, but apparently Cliqz is a search engine [1] and browser [2] built by a German media company [3].
[1] https://0x65.dev/blog/2019-12-01/the-world-needs-cliqz-the-w...
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#3Just be careful. Kubeflow is heading toward GCP only features (for example, they are dropping cert-manager), while you are betting on AWS.
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#4last time Cliqz came up on here was not in the best of contexts...
https://old.reddit.com/r/firefox/comments/74yo19/cliqz_and_m...
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#5last time Cliqz came up on here was not in the best of contexts... https://old.reddit.com/r/firefox/comments/74yo19/cliqz_and_m...
The "last time Cliqz came up" can be found here: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu.... We have been posting multiple articles on our tech blog, explaining what we do and how we do it in great details. Your link points to an old thread of more than two years ago.
There were more recent discussions about Cliqz no latter than this month, in particular here: https://news.ycombinator.com/item?id=21676252
[disclaimer: I work at Cliqz]
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#6How does a new engine find webpages at start? Does it work from the Dns system and indexes every domain name? At a certain point it will follow links I presume, but how does it start?
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#7How does a new engine find webpages at start? Does it work from the Dns system and indexes every domain name? At a certain point it will follow links I presume, but how does it start?
[Disclaimer: work at Cliqz]
We do not crawl the web in the traditional sense, our search was bootstrapped on query logs. It is the very reason we could succeed in building a search engine with minimal resources, in comparison to our competitors.We have written about this in a lot more detail here :
How we collect data : https://www.0x65.dev/blog/2019-12-03/human-web-collecting-da...
How we build the search using this data: https://www.0x65.dev/blog/2019-12-06/building-a-search-engin...
Feel free to peruse these posts and ask questions!
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#8last time Cliqz came up on here was not in the best of contexts... https://old.reddit.com/r/firefox/comments/74yo19/cliqz_and_m...
The "last time Cliqz came up" can be found here: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... . We have been posting multiple articles on our tech blog, explaining what we do and how we do it in great details. Your link points to an old thread of more than two years ago. There were more recent discussions about Cliqz no latter than this month, in particular here: https://news.ycombinator.com/item?i…
yeah sorry, should have said "last i remember".
> We have been posting multiple articles on our tech blog, explaining what we do and how we do it in great details.
it's possible to have both great tech and loose morals - the two are not mutually exclusive, and one does not absolve the other (e.g. facebook's social experiments)
has there been a followup to any of the points brought up in the reddit thread?
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#9Just be careful. Kubeflow is heading toward GCP only features (for example, they are dropping cert-manager), while you are betting on AWS.
Would it be fair to say you're saying that Kubeflow may become eventually incompatible with EKS?
Re: The Architecture of a Large-Scale Web Search Engine, Circa 2019
#10How does a new engine find webpages at start? Does it work from the Dns system and indexes every domain name? At a certain point it will follow links I presume, but how does it start?
[Disclaimer: work at Cliqz] We do not crawl the web in the traditional sense, our search was bootstrapped on query logs. It is the very reason we could succeed in building a search engine with minimal resources, in comparison to our competitors.We have written about this in a lot more detail here : How we collect data : https://www.0x65.dev/blog/2019-12-03/human-web-collecting-da... How we build the search using this…
Does really all of your data come from the human web project or do you also buy clickstream data from data brokers?