> After a month I had most newsgroups - excepting binaries - and it came to 800GB. > Trying to index THAT lot was impossible Stupid question, but why would indexing 800GB of newsgroup postings be impossible?
I'm trying again on that 800GB with KISS DB (append-only hashtable), and Elasticsearch. Doesn't matter if GPL because it's a website.