Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building a Search Engine from Scratch

0x65.dev

Post reply on HN
Print view View original

41–50 of 151 posts

Re: Building a Search Engine from Scratch

#41

Post by pythux »

kick wrote: post #27
solso wrote: post #15

Earlier quoted context omitted.

[Disclaimer: I work at Cliqz] Hi, I read the thread and thought the answer was good enough, but it seems that you are not yet convinced. Let me try: 1) Here there is a list of publications regarding privacy by Cliqz (including published scientific papers). It should have fairly easy to find it using a search engine :-) https://0x65.dev/pages/dissemination-cliqz.html Hopefully, the paper will convince you that Cliqz p…

A quick skimming of those PDFs found no mentions of "anolysis." Your colleague claimed that papers were going to be released "soon" on it. It's been at least two years since you started using it, so why hasn't there been yet? It does not take long to find references of the name on the source code. No references in any of your published source code, and your search engine isn't free software: https://github.com/search…

> No references in any of your published source code, and your search engine isn't free software:

You looked at the wrong tab, check in "Code" to find "code" related to Anolysis: https://github.com/search?q=org%3Acliqz-oss+anolysis&type=Co...

> Cliqz has done unsavory things in the past (like the Firefox fiasco a few years back, for example, which I can't fault Cliqz entirely for: Mozilla is just as guilty).

Not sure how this is Cliqz' fuck-up. We are not hiding anything. On the contrary we are very transparent and detailed about how everything we do is designed to not track users. All of this is on our new tech blog: https://0x65.dev, feel free to have a look between two comments on HN and give us some feedback!

> "anolysis" gets around both uBlock Origin and uMatrix, despite both of them automatically blacklisting any URL with "analytics" in it, as an example. Getting around the most popular content filterers on the internet is a pretty strong signal.

It's not called "getting around it" when there is no tracking or ads going on (if you want to see how smart the "most popular content filterers are" check out this link and see that the image is blocked because it contains the substring: "analytics": https://whotracks.me/blog/private_analytics.html. Wicked smart!).

Anolysis is not a typo, it's a project name, people tend to do that when they care and spend a lot of time on projects: give them names. So, at the risk of repeating myself, Anolysis = Analysis + Anonymous (at the time we thought it was a pretty neat name!).

Anolysis does not operate outside of Cliqz products (no websites analytics here and we do not rely on a third-party, we built it in-house for this reason) and we put a lot of work into it to make sure it does not use a unique ID (like virtually every other analytics out there) but allows to by-design not track any single user (in fact the system does not even have the concept of a user). Sure, we did not write extensively about it but I guess we have to start somewhere (in December we are writing on 24 different things we do, we will be sure to consider Anolysis as a good candidate for a technical blog post in the future).

What you attribute to malice is simply a lack of time, as you probably noticed Cliqz is working on solving a lot of very hard problems (search, browsers, antitracking, adblocking, privacy-preserving telemetry and so much more) and writing a paper about the new system you designed and implemented is not always the priority :)

Re: Building a Search Engine from Scratch

#42

Post by adventskalender »

commoner wrote: post #17
nostromo wrote: post #8

I don’t see how this will ultimately be successful. They’re basically reverse engineering Google by looking at user logs. Google will always have a leg up here because they have all the Google data. And even if it does work for a while, there still needs to be the original signal to copy. Someone will have to crawl the web and index content. I’m super eager to find new approaches to search, but another Google clone i…

The article describes techniques used by all search engines, not just Google. Search engines have existed before Google, and despite Google's monopoly on search, "Google clone" is a poor term to describe all search engines when alternatives with unique features (e.g. DuckDuckGo) exist.

How many real alternatives exist, though? DuckDuckGo is based on Bing, afaik.

So there are Google and Bing - any others?

Re: Building a Search Engine from Scratch

#44

Post by [deleted] »

dessant wrote: post #36
tschakkaMarc wrote: post #31

Earlier quoted context omitted.

Well, we released our beta search for Tor yesterday: search4tor7txuze.onion/ (works obviously only in the Tor browser). That’s as good as it gets regarding making it technically impossible, isn’t it? More complex obviously for a browser - but the answer can simply not be „only no data at all is good“, because that’s a destructive approach that only favors the worst privacy intruders; no one would then be able to buil…

We appreciate your openness about how you collect data, but that's still not enough because literally every other advertising company is deceptive when they talk about privacy. The openness must be paired with privacy that is guaranteed under all circumstances, and the most common way to achieve that would be to route the anonymized data through Tor. Your search engine being also available on Tor has nothing to do wi…

[deleted]

Re: Building a Search Engine from Scratch

#45

Post by kkm »

dessant wrote: post #36
tschakkaMarc wrote: post #31

Earlier quoted context omitted.

Well, we released our beta search for Tor yesterday: search4tor7txuze.onion/ (works obviously only in the Tor browser). That’s as good as it gets regarding making it technically impossible, isn’t it? More complex obviously for a browser - but the answer can simply not be „only no data at all is good“, because that’s a destructive approach that only favors the worst privacy intruders; no one would then be able to buil…

We appreciate your openness about how you collect data, but that's still not enough because literally every other advertising company is deceptive when they talk about privacy. The openness must be paired with privacy that is guaranteed under all circumstances, and the most common way to achieve that would be to route the anonymized data through Tor. Your search engine being also available on Tor has nothing to do wi…

Hi dessant,

Disclaimer: I work for Cliqz.

Would be really interesting to know your concerns with FoxyProxy.FoxyProxy is legally bound to not log the IP or share it.

From the HPN protocol's perspective, data can be routed via any trusted party - in our case it's FoxyProxy.

Right now, there is no way to configure this in the Browser, but should be doable. It's actually one of the motivations to move to the newer version of HPN[1].

We do agree that sending data through Tor network is the gold standard for anonymity.

- We did a lot of work on getting Tor running in Cliqz browsers. It's a hard problem but definitely do-able, something we might pursue again in future[2]. - We also have experimented with WebAssembly version of Tor client to make it compatible for web extension[3].

Having the ability to use the Tor network in Cliqz products is also good, because we can actually leverage the anonymity guarantees by sending data via .onion services. You can also check more details under evaluation section of the paper[4].

In case you wish to check the network traffic you can also check the debugging section[5].

References:

1. https://www.0x65.dev/blog/2019-12-04/human-web-proxy-network... 2. https://github.com/cliqz-oss/browser-f/commit/12fcef8479d9c3... 3. https://github.com/cliqz-oss/browser-f/commit/12fcef8479d9c3... 4. https://arxiv.org/pdf/1812.07927.pdf 5. https://www.0x65.dev/blog/2019-12-03/human-web-collecting-da...

Re: Building a Search Engine from Scratch

#46

Post by mrb »

bla3 wrote: post #29

That Cliqz is trying to actually build a new search stack is commendable. This is way more exciting to me than DuckDuckGo and other services that just package up Bing search results under different branding. I'm skeptical that they'll be successful, but I wish them the best. They should market (and engineer) strongly on privacy since that's where Google is weak.

«DuckDuckGo and other services that just package up Bing search results»

Had to look that up. I found https://help.duckduckgo.com/results/sources/ Bing is just one of "hundreds of vertical sources delivering" results to DuckDuckGo.

Re: Building a Search Engine from Scratch

#47

Post by mda »

mrb wrote: post #46
bla3 wrote: post #29

That Cliqz is trying to actually build a new search stack is commendable. This is way more exciting to me than DuckDuckGo and other services that just package up Bing search results under different branding. I'm skeptical that they'll be successful, but I wish them the best. They should market (and engineer) strongly on privacy since that's where Google is weak.

« DuckDuckGo and other services that just package up Bing search results » Had to look that up. I found https://help.duckduckgo.com/results/sources/ Bing is just one of "hundreds of vertical sources delivering" results to DuckDuckGo.

Bing is the primary source though.

Re: Building a Search Engine from Scratch

#48

Post by macintux »

jermaustin1 wrote: post #39
otterley wrote: post #37

Earlier quoted context omitted.

If Google went back to the original algorithms it would return even worse results. Much of the noise can be attributed to two factors: (1) the explosive growth of the Internet itself, which was much much smaller and had more focused and authoritative content 20 years ago; and (2) aggressive SEO tactics today that would easily fool the early versions of PageRank.

I kind of miss the early directories for sites. It probably can't happen now, since there are billions of websites, but it was a simpler time, and finding something you needed wasn't THAT hard.

I would strongly disagree. I was at a dinner party in the mid-90s and we had a long-running discussion about how impossible it was to find anything using Altavista and the other search engines. Yahoo's directory was even worse.

The best idea anyone had, which I thought wouldn't scale but didn't have better ideas, was implementing a keyword registry à la AOL.

Re: Building a Search Engine from Scratch

#49

Post by pcmaffey »

I really just wish exact match search still worked. But now, words are all vectorized as every search engine tries to determine "my intent", resulting in a deluge of fuzzy matches.

Maybe I'm old school, but I don't want software that fixes my spelling mistakes. I want software that fails when I make a mistake.

Re: Building a Search Engine from Scratch

#50

Post by riazrizvi »

otterley wrote: post #37
riazrizvi wrote: post #35

Earlier quoted context omitted.

Google search worked well enough for me when it was launched 22 years ago. So the patents to that early version should be expired. I'd be happy to use a competitor's service if they would reintroduce a Search API, and PageRank, just give me a programmatic interface that doesn't throw up Captcha after a few searches. I'd pay a reasonable price for the compute time + profit margin.

If Google went back to the original algorithms it would return even worse results. Much of the noise can be attributed to two factors: (1) the explosive growth of the Internet itself, which was much much smaller and had more focused and authoritative content 20 years ago; and (2) aggressive SEO tactics today that would easily fool the early versions of PageRank.

The point about a Search API is that it opens up the problem to a world of 3rd party developers, as opposed to Google putting up competitive defenses by restricting programmatic access. A few years ago I built a web directory that organized and filtered Alexa's top 1MM websites, using PageRank, semantic clustering, and natural language processing. It worked really well, gave me a holistic picture of the internet that was quite different to how Yahoo presented it through their directory. However building it was time-consuming because Google would keep shutting out the ipaddress I was using, so I'd have to keep driving to the next Starbucks up the street to resume my scripts.

Now compare the world of search to the world of mobile phones. Imagine if mobile phones only came with proprietary apps, and there were no app stores. That's where search is right now.

Post reply on HN