I searched "meta programming c++" and the top returns are all about java.
I'm curious, is it expensive to run a search site like this?
Show HN: Open-source search engine with 2bn-page index
101–110 of 148 posts
Re: Show HN: Open-source search engine with 2bn-page index
#102You get really good performance on not much hardware. Can you share some technical details? - file formats, particularly the postings - query evaluation strategy - update strategy I poked around in the source code a bit, but couldn't find these things.
File formats will be documented when I publish the data-files in a few weeks.
What do you mean with postings?
The main index is split into 32 shards (there is also an additional news-index which is updated about every 5-10 minutes). Each shard is updated and queried seperately. The query actually runs 2/3 on a Windows server and 1/3 on a Linux server. The latter in Docker containers. I want to move everything to Linux over time.
Query has two phases. First only a rough - but fast - ranking is done. Then the top results of all shards are combined and completely re-ranked. This is basically a meta search engine hidden within.
First query phase is in src/searchservernew.dpr, and the second phase is in src/cgi/PostProcess.pas.
Re: Show HN: Open-source search engine with 2bn-page index
#103Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…
I always had this, maybe stupid idea, of vouching referrals search engine. You pick a few sites you know are good and they (their 'webmasters') would vouch for new ones, and those could vouch for new ones, etc. Catch is, if one or two (whatever) of your child or grandchild vouched sites screws up, then they're toast, out of index, but so are you. That way you would pick wisely.
Same idea would probably work for online commenting. Vouch with a chain of responsibility. That's essentially how pagerank did its thing, but with no repercussions and vouching was automatic based on links from initial seed of what they thought was good. I'd do it with humans.
Re: Show HN: Open-source search engine with 2bn-page index
#104Earlier quoted context omitted.
There were a few attempts at that in the past, one being http://omgili.com/ that now seems to return pretty much garbage. BTW About 12 years ago I was building this search engine, and I was toying with the idea of building a classifier that classifies web pages based on their "genre" rather than category, so you can limit your search for shopping websites, forums, blogs, news sites, social media, etc. It was a bitch…
Heh, classifying by "genre" is exactly what I was thinking of doing. Had some debate with myself if I should start by focusing on training for shopping pages (product pages & product reviews) - because that might make some money; or start by training for forums - which I'd enjoy a lot more. Or build a more general system which would definitely never work and never get finished. Google actually let you filter by "disc…
Iirc Gigablast had such a feature.
Re: Show HN: Open-source search engine with 2bn-page index
#105Earlier quoted context omitted.
Some issues that appeared over the years: Block outgoing connects to local IP nets in your firewall. Otherwise your hosting provider might think you are trying to hack them. Apparently there are a lot of links out there that point to hosts which resolve to private IP ranges. Another problem with following links is that you are bound to run across some that are malware command & control servers. Had several complaints…
> Run your own DNS server and even cache the hostname/IP pairs yourself. This[1] might be a useful resource to get started: [1] https://scans.io/ (Register and download the IPv4 Address Space data file to use as an initial cache and then append/update as you go.)
Bookmarked. Thanks!
Re: Show HN: Open-source search engine with 2bn-page index
#106I searched "meta programming c++" and the top returns are all about java. I'm curious, is it expensive to run a search site like this?
Currently €300/month. More details on https://deusu.org/donate.html
Re: Show HN: Open-source search engine with 2bn-page index
#107Is the two billion page index open source? I've been thinking a lot about days recently. Seems to me like Pandora's box is open. Google knows where you live, where you eat, what your fetishes are, all of your sexual partners. Facebook knows most of those things to, via different methods. And if you run Windows Microsoft probably has access to most of that as well. Apple will too, because if they don't they won't be a…
If you're looking for an open source web crawl, commoncrawl.org has billions of pages.
Re: Show HN: Open-source search engine with 2bn-page index
#108Alternative general purpose search engines are an exciting idea. It seems a lot like we're about the time when yahoo was dominant and searching was sort of awful. When you searched, what ranked highest was market-driven sorts of stuff. Right now, for topics normal people search for - not techies -all you get are content farm sites with js-popups asking for your email address. Try searching for anything health related…
I would love to have a search engine that would allow you supply your own ranking function.
How would this work? You could boost the query term for instance, like it possible to boost the column score in postgresql but that is all. Otherwise allowing user to provide their own ranking function (which is itself an art) would not be pratical performance wise. It should be noted that search engine interface, the search box is already a DSL for the underlying algorithm that support OR/AND and NOT.
Re: Show HN: Open-source search engine with 2bn-page index
#109You get really good performance on not much hardware. Can you share some technical details? - file formats, particularly the postings - query evaluation strategy - update strategy I poked around in the source code a bit, but couldn't find these things.
File formats will be documented when I publish the data-files in a few weeks. What do you mean with postings? The main index is split into 32 shards (there is also an additional news-index which is updated about every 5-10 minutes). Each shard is updated and queried seperately. The query actually runs 2/3 on a Windows server and 1/3 on a Linux server. The latter in Docker containers. I want to move everything to Linu…
Thank you. "Postings" is another word for the format of the doc ids and related information in the inverted file. A google for "inverted index postings" will turn up a bunch of references.
Re: Show HN: Open-source search engine with 2bn-page index
#110Earlier quoted context omitted.
DuckDuckGo's approach of !bang searches, making duckduckgo the place[1] I go when I want to search another site, is really useful. [1] It's my default search engine in Chrome, so I use bang searching in the address bar.
Same here. The problem is that I find myself using `!g` way too often... I guess I'm not used to the DDG results page.
ddg is my primary search engine, it takes time but you get use to it. If what you are looking for is mostly on HN, SO or wikipedia it works quite well.