21–30 of 107 posts
What I really want, is not another search engine for contextless queries. Except for really basic queries (which Google/etc already do a good job at), I'm trying to answer a question, perhaps open-ended, and it will take multiple queries to resolve. And it's not a linear process of narrowing down with + or - keywords. It's establishing a context: I'm searching for something relevant to "go" the language, not "go" the…
Ah, yes. The old "hiybbprqag" method. Worked great for Bing.
Earlier quoted context omitted.
Hi, Interesting, could you tell what's the error to see. Other ways you can reach the blog: If you use Tor browser can you try opening: http://cliqzdevxo33b4h6.onion/ Or if you use Beaker browser: dat://ee172d7cd9235b2cf86ea9481e8a40e48cea29c743036621edc79a4765aa0281 Disclaimer: I work for Cliqz.
I get the following error: This site can’t be reached0x65.dev refused to connect. Try: Checking the connection Checking the proxy and the firewall ERR_CONNECTION_REFUSED This happens on Chrome, Firefox, Safari, and Opera on my Mac.
Seems like you have some mapping for .dev TLD. Assuming based on your mention of Safari, that you are using Mac.
Could you check if you have some setting in your /etc/resolver for dev TLD, or if you are using some service like dnsmasq which is trying to resolve .dev to a non-existent location.
Earlier quoted context omitted.
An excerpt from the 1st post of this series: "Why would a team be motivated to build another search engine? Why would Hubert Burda Media finance this over several years (they continued to back us especially in times when things got tough)?" https://0x65.dev/blog/2019-12-01/the-world-needs-cliqz-the-w...
Yes, they mention it, which is a good move regarding transparency, but they still don't answer the question as to why the finance them. They mostly push a narrative of privacy and censorship, when in the end the answer is probably close to "we want a piece of the pie" or "we want to be that monopoly".
Why are costs so high for crawling?
They talk about using query logs to optimize their search results: >Queries performed by people, if associated to a web page, serve as even cleaner summaries than anchor text. This is because all the logic put in place by the search engine, who resolved the query with a list of web pages, and all human understanding and experience that led one to select the best page from the offered result list end up embedded in th…
[Disclaimer, I work at Cliqz] Your point is spot on. Old pages tend to have more association to seen queries, which does not play in favor for new pages. That said, however, there are a couple of things to consider: 1) seen queries is not the only way to create queries, we are pretty good creating synthetic queries based on the content, descriptions, etc. This queries are more noisy that the seen queries of course, b…
It is still a little fuzzy to me. What is a "synthetic query"? Is this basically generating queries that would match the content (i.e. essentially reversing the process)?
Novelty, freshness are interesting but can lead back to the noise problem mentioned in the blog. If many pages are created that may match the query (e.g. "best new movies") many young pages will match this. Popularity would be useful but difficult to establish and then there's the clickbait and other gaming problems.
> The experts, who chose to answer, suggested that we should first start with crawling the whole web. We were told that this would take between 1 and 2 years to complete, and would cost a minimum of $1 billion Why are costs so high for crawling?
The bulk of the content ended up being index pages - i.e. large list of links taking you to the content - pagination, other breadcrumbs etc.
You can exhaust a lot of resources without getting anything useful.
It is no longer a point and go kind of thing unfortunately, you need a good understanding of page structure, estimate what kinds of links are vital, etc. else there's a ton of crap you'll pick up.
Or maybe I was doing something wrong.
Anybody knows Cliqz's database stack? Curious to see what powers a large scale information retrieval index of this sort.
We will have a blog post tomorrow on this very topic, but in short, we use a combination of Keyvi, Granne (both in-house) along with Cassandra and RocksDB.
Though our approach mentioned in this blogpost significantly reduces the storage needed to host the index, we still have an index of around 50 TB of data.