Do you use Common Crawl? It seems like a pretty big corpus, reasonably up-to-date and free. It seems like a good way to supplement the page data from the browser extension.
Building a Search Engine from Scratch
21–30 of 151 posts
Re: Building a Search Engine from Scratch
#22Cliqz still hasn't given a good answer as to why they use what's seemingly a typo in 'analysis' to get around user tracker blockers. https://anolysis.privacy.cliqz.com/ As you can see in this subthread, they claim it's "anonymized" analytics, https://news.ycombinator.com/item?id=21718694 which, as 99% of research suggests, isn't anonymous at all: https://www.fastcompany.com/90278465/sorry-your-data-can-sti...
[Disclaimer: I work at Cliqz] Hi, I read the thread and thought the answer was good enough, but it seems that you are not yet convinced. Let me try: 1) Here there is a list of publications regarding privacy by Cliqz (including published scientific papers). It should have fairly easy to find it using a search engine :-) https://0x65.dev/pages/dissemination-cliqz.html Hopefully, the paper will convince you that Cliqz p…
[deleted]
Re: Building a Search Engine from Scratch
#23Definitely interesting. But...my searching has gotten to the point where well over half the time I am no longer looking at Google's conventional search results.
could you explain that a bit more? What are you using instead?
Re: Building a Search Engine from Scratch
#24Do you use Common Crawl? It seems like a pretty big corpus, reasonably up-to-date and free. It seems like a good way to supplement the page data from the browser extension.
From the article:
"It may seem like Common Crawl would suffice for this purpose, but it has poor coverage outside of the US and its update frequency is not realistic for use in a search engine."
Re: Building a Search Engine from Scratch
#25We've had at least 2 posts from cliqz in the past few days. I have genuine issues with my short-term memory after having recovered from a coma so I don't know whether this is a glitch where I keep seeing the same posts or they keep getting reposted.
Re: Building a Search Engine from Scratch
#26Cliqz still hasn't given a good answer as to why they use what's seemingly a typo in 'analysis' to get around user tracker blockers. https://anolysis.privacy.cliqz.com/ As you can see in this subthread, they claim it's "anonymized" analytics, https://news.ycombinator.com/item?id=21718694 which, as 99% of research suggests, isn't anonymous at all: https://www.fastcompany.com/90278465/sorry-your-data-can-sti...
[Disclaimer: I work at Cliqz] Hi, I read the thread and thought the answer was good enough, but it seems that you are not yet convinced. Let me try: 1) Here there is a list of publications regarding privacy by Cliqz (including published scientific papers). It should have fairly easy to find it using a search engine :-) https://0x65.dev/pages/dissemination-cliqz.html Hopefully, the paper will convince you that Cliqz p…
I'm one of those people who remains skeptic about the anonymity of the tracking Cliqz does in general. Obviously people have a hard time believing any company that is in the advertising space and preaches about privacy, mainly because they have been burned several times before.
For me it was the data proxying through FoxyProxy that made me uncomfortable. I have also remained unconvinced about the motivation for not using Tor: because it is hard to integrate into extensions. Cliqz also has its own browser, not just an extension, where they could opt to use Tor, but they route through FoxyProxy.
You must find a way to make it technically impossible to identify users, a legal or business structure is not enough. It wouldn't be unheard of to secretly own the proxy company and relink user data.
Re: Building a Search Engine from Scratch
#27Cliqz still hasn't given a good answer as to why they use what's seemingly a typo in 'analysis' to get around user tracker blockers. https://anolysis.privacy.cliqz.com/ As you can see in this subthread, they claim it's "anonymized" analytics, https://news.ycombinator.com/item?id=21718694 which, as 99% of research suggests, isn't anonymous at all: https://www.fastcompany.com/90278465/sorry-your-data-can-sti...
[Disclaimer: I work at Cliqz] Hi, I read the thread and thought the answer was good enough, but it seems that you are not yet convinced. Let me try: 1) Here there is a list of publications regarding privacy by Cliqz (including published scientific papers). It should have fairly easy to find it using a search engine :-) https://0x65.dev/pages/dissemination-cliqz.html Hopefully, the paper will convince you that Cliqz p…
A quick skimming of those PDFs found no mentions of "anolysis." Your colleague claimed that papers were going to be released "soon" on it. It's been at least two years since you started using it, so why hasn't there been yet?
It does not take long to find references of the name on the source code.
No references in any of your published source code, and your search engine isn't free software:
https://github.com/search?q=org%3Acliqz-oss+anolysis
Honestly, if someone tells you that anolysis means anonymous + analysis, why do you not believe it?
Cliqz has done unsavory things in the past (like the Firefox fiasco a few years back, for example, which I can't fault Cliqz entirely for: Mozilla is just as guilty).
On a separate note, as a company (Cliqz) that offers anti-tracking and ad-blocking, I can tell you that blocklists are a bit more sophisticated than that.
"anolysis" gets around both uBlock Origin and uMatrix, despite both of them automatically blacklisting any URL with "analytics" in it, as an example. Getting around the most popular content filterers on the internet is a pretty strong signal.
Re: Building a Search Engine from Scratch
#28We've had at least 2 posts from cliqz in the past few days. I have genuine issues with my short-term memory after having recovered from a coma so I don't know whether this is a glitch where I keep seeing the same posts or they keep getting reposted.
Your short-term memory is fine in this case, and I hope that is part of a complete recovery.
They're doing an "Advent Calendar" series where they're posting one a day:
https://news.ycombinator.com/item?id=21676252
https://news.ycombinator.com/item?id=21684708
https://news.ycombinator.com/item?id=21694980
and https://news.ycombinator.com/item?id=21716860 (not even a day ago).
This is a problem for HN because users here are not used to this sort of repetition—indeed, we moderate HN explicitly to dampen repetition, because the point of the site is curiosity and curiosity withers under it (https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...). The more these posts show up with what for HN is a crazy frequency, the more likely users here are to experience it as a barrage and start to complain.
On the other hand, these articles are well-crafted, contain a lot of information, and would normally be fine HN submissions. The topic of building a new search engine is intrinsically interesting. It also resonates with a lot of themes that get discussed a lot on HN (concerns about big tech and so on). So this is a different situation than the usual marketing onslaughts that HN gets subjected to, where the content is crappy, users flag it away, and moderators squash what users missed.
I'm not sure what to do about this yet.
Re: Building a Search Engine from Scratch
#29That Cliqz is trying to actually build a new search stack is commendable. This is way more exciting to me than DuckDuckGo and other services that just package up Bing search results under different branding.
I'm skeptical that they'll be successful, but I wish them the best. They should market (and engineer) strongly on privacy since that's where Google is weak.
Re: Building a Search Engine from Scratch
#30I don’t see how this will ultimately be successful. They’re basically reverse engineering Google by looking at user logs. Google will always have a leg up here because they have all the Google data. And even if it does work for a while, there still needs to be the original signal to copy. Someone will have to crawl the web and index content. I’m super eager to find new approaches to search, but another Google clone i…
The article describes techniques used by all search engines, not just Google. Search engines have existed before Google, and despite Google's monopoly on search, "Google clone" is a poor term to describe all search engines when alternatives with unique features (e.g. DuckDuckGo) exist.
They are literally rebuilding Google by tracking how users use Google and rebuilding the SERPs.
If that’s not a Google clone I don’t know what is.