Viewing profile — deusu

You can do your own search-index with 2.3 billion pages for about €300/month: https://deusu.org

It's alive and well. The TIOBE index still lists it ahead of Ruby, Swift, Objective-C, GoLang... And I started this software 20 years ago. Granted, a LOT of the software has change…

Thank you!

I don't know. But I do know that the end-of-year statistics from search-engines about what people searched for, are complete BS. I have such a list for the German DeuSu page: https…

Yes. I have downloaded several data dumps, but haven't gotten around to import them yet.

Currently €300/month. More details on https://deusu.org/donate.html

Bookmarked. Thanks!

File formats will be documented when I publish the data-files in a few weeks. What do you mean with postings? The main index is split into 32 shards (there is also an additional ne…

I have the filter implemented now. It's not perfect yet, but it already filters out a lot of the NSFW stuff. Unless you explicitly search for it. I'm gonna further improve this ove…

I don't know sphinx at all, and my knowledge of lucene is very limited. Which means I don't know how they would compare to DeuSu.

Thank you! Depending on who you are (there were 2 bitcoin donations today), you funded either about 18 or 28 hours of operations. :)

The software is already open-source. A free search API will be fully available probably next week. It's in testing already. It's just a matter of putting the finishing touches on t…

Only ASCII and German umlauts (äöüß) at the moment. The parser needs rewriting. It was originally written in pre-unicode times. :)

Originally it was written in Delphi. But I now use FreePascal for the development. I'm even compiling both Windows and Linux versions on my Linux machine.

Yes, it would be better. The snippets are currently the first 255 characters of the page's text. For snippets to be customized to the search term, I would have to store all the tex…

Some issues that appeared over the years: Block outgoing connects to local IP nets in your firewall. Otherwise your hosting provider might think you are trying to hack them. Appare…

4 servers in total. 2 are used for crawling, index-building and raw-data storage. Quadcore, 32gb RAM, 4tb HDD and 1gbit/s internet connection on each of these. They are rented and …

It's all open-source. So, yes.

I will publish the index for download in a few weeks. I'm currently working on the documentation. Oh, and I will publish the raw crawl-data too. Everything together is about 2.5tb.…

Thx. But all the traffic from here is currently driving the servers to their limit. Queries are already slowing down a bit because of imminent overload. Usually the average query t…

And why should it? You are already at the destination. No need to find it. :)

A fresh recrawl is currently running. Should take about 2-3 months. Newly crawled data will gradually replace older data during that time.

In my experience this is usually caused by the fact that even 2bn pages aren't that many nowadays. The index needs to get bigger to better find (and rank) long-tail results like qu…

I hadn't even thought about that. But it should be pretty easy to do in post-processing. I just have to take a list of "porn" keywords. If none of them occurs in the query, but in …