ah yes, the old 're-implement the search engine inside the database' project, undoubtedly put up on the board because someone is tired of their get-the-data-from-the-database-to-the-search-engine process breaking constantly. next on the map: discovering how shitty dictionary management is, the joys of NLP, and abandoning the project entirely because you realize all this stuff has already been solved in 3 or 4 differe…
the joys of NLP ... this is a search index for Gitlab, that's used primarily to store code , not natural language. Looking at its competitor, Github's search engine clearly has a base in NLP, and ditches many punctuation characters ( https://help.github.com/articles/searching-code/ ) - which are way more important in code than they are in English - for example, I can't search for code containing "$/" when it should h…
since developers tag, reference, and document their code with comments, they're going to expect the full suite of NLP treatments with all indexed content, including code.
which basically means: you'll have to tokenize out the special characterize for NLP, but retain them for literal code searches, thereby increasing the size of the inverted index for every permutation of the desired search criteria.
maybe you can detect and filter out all the comments, and index them separately, or maybe have some kind of dual system where the NL indexing system exists separately from the code indexing system ... you see where this is going?