Earlier quoted context omitted.
Bing is the primary source though.
How does ddg do that without running afoul of bing's ToS?
Presumably they're using Bing's API directly, and have a suitable legal agreement with Microsoft in place.
Building a Search Engine from Scratch
121–130 of 151 posts
Re: Building a Search Engine from Scratch
#122Earlier quoted context omitted.
« DuckDuckGo and other services that just package up Bing search results » Had to look that up. I found https://help.duckduckgo.com/results/sources/ Bing is just one of "hundreds of vertical sources delivering" results to DuckDuckGo.
If I recall, in the very beginning they were massively using Yahoo and to some extent Bing. It might be that over the years Bing has become a more reliable source.
Yahoo Search was based on Bing since 2009.
Re: Building a Search Engine from Scratch
#123Earlier quoted context omitted.
Certainly it can be done. Cuil did it with about 30 people a decade ago. No business model, they ran out of funding, and no one wanted to acquire them, but they did get the crawler and search engine running and publicly available. It might be worthwhile to do the search part in-house and outsource the question-answering functions. Wolfram Alpha and IBM Watson could be used for answering common questions.
Doesn’t that imply they weren’t successful?
I’m guessing successful in the sense that they had a product come to fruition
Re: Building a Search Engine from Scratch
#124Earlier quoted context omitted.
By selling ads solely against the query, and figuring out a way to track conversions against generated unique referring URL instead of using cookies. Maybe it's less profitable, but it might still be profitable, and it gives you a foothold.
Or a sponsored and unsponsored section. The only thing you pay for is for being higher up in the sponsored pane. No need for tracking. No conversion clicks tracked. Only higher on the sponsored listing.
Wouldn’t that prevent people with less money from buying their way up to the top? In other words, the ones with the most money (not startups, mom&pop shops, etc) would be the only ones at the top.
Re: Building a Search Engine from Scratch
#125Earlier quoted context omitted.
Being successfully might not mean being at the top of the pyramid. DDG is a an example of a successful search engine. Further, they don't have to be better than google in the quality of their search results. As soon as the results have good relevance, it's good competition. What matters is that they are 'good enough', that there is a hint of competition to the Google-Bing monopoly. Offering privacy centric 'competiti…
DDG is not a real search engine, I think this project is way more impressive than DDG.
DDG absolutely is a search engine; It’s a system (engine) that lets you search. From Wikipedia[0]:
> A web search engine or Internet search engine is a software system that is designed to carry out web search (Internet search), which means to search the World Wide Web in a systematic way for particular information specified in a textual web search query.
Just because they don’t do the crawling like Google and Bing doesn’t mean they aren’t a search engine.
Re: Building a Search Engine from Scratch
#126Earlier quoted context omitted.
How does ddg do that without running afoul of bing's ToS?
Presumably they're using Bing's API directly, and have a suitable legal agreement with Microsoft in place.
I don't see how such a deal could possibly work. Microsoft's entire history and culture has been devoted to squashing nascent competitors whenever possible. So why in the world would MS license Bing to DDG? If DDG can monetize that traffic more effectively than Bing by not tracking (which seems unlikely) then why does MS just stop tracking on Bing? And if they can't, then how would they afford to pay the license fee, which surely must be at least as high as the lost revenue from the diverted traffic?
Re: Building a Search Engine from Scratch
#127Earlier quoted context omitted.
Presumably they're using Bing's API directly, and have a suitable legal agreement with Microsoft in place.
I don't see how such a deal could possibly work. Microsoft's entire history and culture has been devoted to squashing nascent competitors whenever possible. So why in the world would MS license Bing to DDG? If DDG can monetize that traffic more effectively than Bing by not tracking (which seems unlikely) then why does MS just stop tracking on Bing? And if they can't, then how would they afford to pay the license fee,…
I imagine few people using DDG would use Bing instead, so there is little income loss from people moving Bing->DDG. If the choice is "provide DDG with a search API, get increased reach for your ad-program through an audience that otherwise wouldn't touch bing" vs "do not provide DDG with a search API, either a competitor does or DDG is way worse and everyone uses Google", why is the latter the better choice for MS?
Re: Building a Search Engine from Scratch
#128>> "Our model of a web page is based on queries only. ... This is a crucial differentiating factor – it is the reason we are able to build a search engine with dramatically less resources in comparison to our competitors."
>> "The total size of our index currently is around 50 TB."
Could you share your current index size (number of pages, size of raw text) to put those 50 TB into perspective in order to get an idea how much less resources in comparison to your competitors you need? This would help to compare your approach to Elasticsearch, Solr, Lucene
Re: Building a Search Engine from Scratch
#129Earlier quoted context omitted.
I don't see how such a deal could possibly work. Microsoft's entire history and culture has been devoted to squashing nascent competitors whenever possible. So why in the world would MS license Bing to DDG? If DDG can monetize that traffic more effectively than Bing by not tracking (which seems unlikely) then why does MS just stop tracking on Bing? And if they can't, then how would they afford to pay the license fee,…
I imagine few people using DDG would use Bing instead, so there is little income loss from people moving Bing->DDG. If the choice is "provide DDG with a search API, get increased reach for your ad-program through an audience that otherwise wouldn't touch bing" vs "do not provide DDG with a search API, either a competitor does or DDG is way worse and everyone uses Google", why is the latter the better choice for MS?
Even if that were true, why would not MS just re-skin Bing as a tracker-free search engine? Or acquire DDG?
Re: Building a Search Engine from Scratch
#130Is the Human Web dataset available for download?
Data release, it's not possible, but if people want to come and do experiments on the data or try to test it for privacy, we are more than welcome to host them. There is no formal process in any way, best effort, we have done several times in the past. If you are very interested contact us and we will see if we can accommodate you. [Disclaimer: I work at Cliqz]
(Disclaimer: I work at Cliqz) Extending on that, let me elaborate why we cannot open the data, not even a subset of it. We had the discussion in the past, but for two reasons it is not an option.
Although it is anonymous data - currently we are not aware of any de-anonymization attacks - it is still data that came from real persons. We have a responsibility: once the data is out, we have to guarantee that no-one will ever be able to identity a single person in the data. Take also in account that attackers can combine multiple data sets (Background Knowledge Attacks); that even includes data sets that will be published (or leaked) in the future.
You should never be too confident when it comes to security, neither should you underestimate the creativity of attackers. What we can do - and did in the past - is to simulate the scenario in a controlled environment by hiring pen testing companies. If they would find an attack, they will not use that knowledge to harm the persons behind the identities that they could reveal.
That is the main reason. We don't want to end up in a situation as AOL or Netflix when they published their data. By the way, Netflix is an example of a background attack where they needed to combine data sources.
There is also another argument. Skeptics will most likely remain skeptics, as we cannot proof that we did not filter out data before publishing. In other words, there is nothing to gain for us, we can only loose. Trust is important, but for building trust, it is better to be transparent about the data that gets sent on the client. You can verify that part yourself and do not have to rely on trust alone. That is the core idea behind our privacy by design approach.
Those are the arguments that I'm aware of why we will not open the data. However, getting access in controlled environments is possible. If you doing security/privacy research, you can reach out to us. In my opinion, having more people that will try to find flaws in our heuristics is useful. That gives us a chance to fix it before it can be used for attacks.
One notable exception: https://whotracks.me is built from Human Web and all its underlying data can be freely downloaded. We know that it has been already used for research.