91–100 of 151 posts
I feel like the web parsing / indexing, perhaps rather than the search algorithm itself, is the hardest part of rolling a new search engine (largely due to the associated computing and storage costs).
Honestly, what would be interesting is if there was an open source database of crawled webpages, available for anyone to search / use with their own algorithms. That would make it possible for... a lot of things, really. I feel like the web parsing / indexing, perhaps rather than the search algorithm itself, is the hardest part of rolling a new search engine (largely due to the associated computing and storage costs)…
Earlier quoted context omitted.
> They should market (and engineer) strongly on privacy since that's where Google is weak. How can you build a privacy oriented search engine and still make money?
If one didn't mind breaking even rather than making money one could create a search engine which is free software and have groups/people host their own instances.
Honestly, what would be interesting is if there was an open source database of crawled webpages, available for anyone to search / use with their own algorithms. That would make it possible for... a lot of things, really. I feel like the web parsing / indexing, perhaps rather than the search algorithm itself, is the hardest part of rolling a new search engine (largely due to the associated computing and storage costs)…
There is https://commoncrawl.org/ , but it would be really cool if there were a more well-lit path towards building the rest of a simple search engine. For example, another commentator wanted “like Google, but without the spelling correction”, well, spin up one of these and just stub out the spelling module :)
50TB on localhost is awesome! I wonder what the amount of new content created was between 2018 and 2019.
The 10PB of disk is also quite reachable given its possible to buy bulk 10T disks at $150 each.
Bottom line, I did some of these calculations a couple years ago because I was interested in a topic based search engine that only indexed for certain topics and basically tossed any crawler results that didn't appear to fit the subject matter.
So, while the web is a lot bigger than when google started, storage and compute is also a lot cheaper. A web search engine that specialized in say cooking recipes might be entirely doable on a fairly limited budget.
Earlier quoted context omitted.
Bing is the primary source though.
I'm not sure that's true. If you search the same query on both Bing and DDG, you'll see pretty different results.
Cliqz bought Ghostery to acquire a pool of privacy-conscious users. The goal is to show them ads. Not sure how excited they will be about that.
If Cliqz really is a search engine, can a user submit a query to the database using her own choice of tcp/http client. It looks like submitting requires first downloading and installing software from Cliqz.
That Cliqz is trying to actually build a new search stack is commendable. This is way more exciting to me than DuckDuckGo and other services that just package up Bing search results under different branding. I'm skeptical that they'll be successful, but I wish them the best. They should market (and engineer) strongly on privacy since that's where Google is weak.
It might be worthwhile to do the search part in-house and outsource the question-answering functions. Wolfram Alpha and IBM Watson could be used for answering common questions.