Live data from Hacker News

Code search is hard

blog.val.town

131–140 of 164 posts

Re: Code search is hard

#131
post #125

Earlier quoted context omitted.

Also Github is a fantastic tool for searching code across repos, ones you may not even have cloned yet! Either public ones or org ones.

The new GitHub CS is pretty great indeed. Still not on par with it's role model, but getting closer.

What’s its role model?

Re: Code search is hard

#133
>Lemmatization: some search indexes are even fancy enough to substitute synonyms for more common words, so that you can search for “excellent” and get results for documents including “great.”

This isn't what lemmatization is about.

Stemming the word ‘Caring‘ would return ‘Car‘. Lemmatizing the word ‘Caring‘ would return ‘Care‘.

Re: Code search is hard

#134
post #49

Earlier quoted context omitted.

> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .

I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close. (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)

AWS has services around pipeline and deploy, right ?

Re: Code search is hard

#135

Basic code searching skills seems like something new developers are never explicitly taught, but which is an absolutely crucial skill to build early on. I guess the knowledge progression I would recommend would look something kind this: - Learning about Ctrl+F, which works basically everywhere. - Transitioning to ripgrep https://github.com/BurntSushi/ripgrep - I wouldn't even call this optional, it's truly an incredi…

Also Github is a fantastic tool for searching code across repos, ones you may not even have cloned yet! Either public ones or org ones.

GitHub's code search functionality is only available to people who are logged in.

It used to be possible to perform global/multi-org/multi-repo/single-repo code searches without being logged in but over time they removed all code search functionality for people who are not logged in.

It is completely stupid that it's not possible for a non-logged-in person to code search even within a single repo[0].

It is textbook enshittification by a company with an monumental amount of leverage over developers.

(The process will presumably continue until the day when being logged in is required to even view code from the myriad Free and Open Source projects who find themselves trapped there.)

[0] Which is why I, somewhat begrudgingly[1], use Sourcegraph for my non-local code search needs these days.

[1] Primarily because Sourcegraph are susceptible to the same forces that lead to enshittification but given they also have less leverage I've left that as a problem for future me to worry about. (But also the site is quite "heavy" for when one just wants to do a "quick" search in a single repo...)

Re: Code search is hard

#136
post #15

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…

I hope this and SCIP becomes a standard and we have more programming languages emitting symbols in SCIP format.

Re: Code search is hard

#137
post #15

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…

Any opinions on mozilla's DXR?

Re: Code search is hard

#138

Earlier quoted context omitted.

Also Github is a fantastic tool for searching code across repos, ones you may not even have cloned yet! Either public ones or org ones.

GitHub's code search functionality is only available to people who are logged in. It used to be possible to perform global/multi-org/multi-repo/single-repo code searches without being logged in but over time they removed all code search functionality for people who are not logged in. It is completely stupid that it's not possible for a non-logged-in person to code search even within a single repo[0]. It is textbook e…

You make it sound as if being logged in to github is somehow a big hurdle. It's free and it's easy, so why should one care if it's only available to logged in users?

Re: Code search is hard

#139
post #55

Code search is indeed hard. Stop words, stemming and such do rule out most off the shelf indexing solutions but you can usually turn them off. You can even get around the splitting issues of things like a.toString() With some pre-processing of the content. However were you really get into a world of pain is allowing someone to search for ring in the example. You can use partial term search, prefix, infix, or suffix b…

The preprocessing that you need is (in Lucene nomenclature, but it's the same principle for search in general) an Analyzer (the component, which knows to prepare the plain text that gets inside for storing it in an index and the corresponding component for a search query) made for code search. That's not different from analyzers for other languages (Stemming sucks for almost everything but English). Thinking about it .. the frontend of most compilers for a language could maybe make a pretty good Analyzer. It already knows language specific components and can split them into parts it needs for further processing, which is basically what an analyzer does.

Re: Code search is hard

#140
post #15

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…

I hope this and SCIP becomes a standard and we have more programming languages emitting symbols in SCIP format.

I thought SCIP got promoted into https://lsif.dev/ but chasing the https://github.com/sourcegraph/lsif-java link resolves to https://github.com/sourcegraph/scip-java so maybe I had the evolution relationship backward. Anyway, I'm thankful at least that code is still Apache 2

https://github.com/topics/lsif may interest this audience, too, since the scip topic tag seems to clash with something else

Also, I learned last night that GitLab embraces LSIF, too https://docs.gitlab.com/ee/topics/autodevops/stages.html#aut...

Post reply on HN