Code search is hard
blog.val.town
Code search is hard
1–10 of 164 posts
Re: Code search is hard
#2Be careful with trigram indexes. At least in the postgres 10 era they caused severe index bloat for frequently updated tables.
Re: Code search is hard
#3When a val is deployed on val town, my understanding is that it's parsed/compiled. At that point, can you save the parts of the program that people might search for? Names of imports, functions, variables, comments, etc.
Re: Code search is hard
#4any nuggets here?
https://github.blog/2023-02-06-the-technology-behind-githubs...
Re: Code search is hard
#5It seems like some of their gists have documentation attached and maybe that’s enough? I’m not sure I’m all that interested in seeing undocumented gists in search results.
Re: Code search is hard
#6Surprised not to see Livegrep [0] on the list of options. Very well-engineered technology; the codebase is clean (if a little underdocumented on the architecture side) and you should be able to index your code without much difficulty. Built with Bazel (~meh, but useful if you don't have an existing cpp toolchain all set up) and there are prebuilt containers you can run. Try that first.
By the way, there's a demo running here for the linux kernel, you can try it out and see what you think: https://livegrep.com/search/linux
EDIT: by the way, "code search" is deeply underspecified. Before trying to compare all these different options, you really would benefit from writing down all the different types of queries you think your users will want to ask, including why they want to run that query and what results they'd expect. Building/tuning search is almost as difficult a product problem as it is an engineering problem.
Re: Code search is hard
#7It indeed is hard, and a good code search platform makes life so much easier. If I ever leave Google, the internal code search is for sure going to be the thing I miss the most. It's so well integrated into how everything else works (blaze target finding, guice bindings etc), I can't imagine my life without it.
I remember to appreciate it even more every time I use Github's search. Not that it's bad, it's just inherently so much harder to build a generalized code search platform.
Re: Code search is hard
#8Be careful with trigram indexes. At least in the postgres 10 era they caused severe index bloat for frequently updated tables.
Interesting, do you know anywhere I can easily read more about this? (I will do my own research, too.)
Re: Code search is hard
#9Oracle has USER/ALL/DBA_SOURCE views, and all of the PL/SQL (SQL/PSM) code that has been loaded into the database is presented there. These are all cleartext visible unless they have been purposefully obfuscated.
It has columns for the owner, object name, LINE[NUMBER] and TEXT[VARCHAR2(4000)] columns and you can use LIKE or regexp_like() on any of the retained source code.
I wonder if EnterpriseDB implements these inside of Postgres, and/or if they are otherwise available as an extension.
Since most of SQL/PSM came from Oracle anyway, these would be an obvious desired feature.
Re: Code search is hard
#10I suppose using something like tree sitter to get a consistent abstract syntax tree to work with would be a good starting point. And then try building a custom analyzer (if using elasticsearch lingo) with that?