Is "hard" a bit of an overstatement for problems like "I'm using a library that mangles the query"? Couldn't you search for the literal text the user inputs? Maybe let them use regex?
Code search is hard
91–100 of 164 posts
Re: Code search is hard
#92Earlier quoted context omitted.
> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .
I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close. (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)
NixOS/nixpkgs is about the closest thing you'll find in the wild. You have to squint a bit, I'll admit.
Re: Code search is hard
#93Earlier quoted context omitted.
I've used both Code Search and Livegrep. No, Livegrep does not even come close to what Code Search can do. Sourcegraph is the closest thing I know of.
Is there like a summary of what's missing from public attempts and what makes it so much better?
The short summary is: It's a suite of stuff that someone actually thought about making work together well, instead of a random assortment of pieces that, with tons of work, might be able to be cobbled together into a working system.
All the answers about the technical details or better/worseness mostly miss the point entirely - the public stuff doesn't work as well because it's 1000 providers who produce 1000 pieces that trade integration flexibility for product coherence. On purpose mind you, because it's hard to survive in business (or attract open source users if that's your thing) otherwise.
If you are trying to do something like make "code review" and "code search" work together well, it's a lot easier to build a coherent, easy to use system that feels good to a user if you are trying to make two things total work together, and the product management directly talks to each other.
Most open source doesn't have product management to begin with, and the corporate stuff often does but that's just one provider.
They also have a matrix of, generously, 10-20 tools with meaningful marketshare they might need to try to work with.
So if you are a code search provider are trying to make a code search tool integrate well with any of the top 20 code review tools, well, good luck.
Sometimes people come along and do a good enough job abstracting a problem that you can make this work (LSP is a good example), but it's pretty rare.
Now try it with "discover, search, edit, build, test, release, deploy, debug", etc. Once you are talking about 10x10x10x10x10x10x10x10 combinations of possible tools, with nobody who gets to decide which combinations are the well lit path, ...
Also, when you work somewhere like Google or Amazon, it's not just that someone made those specific things work really well together, but often, they have both data and insight into where you get stuck overall in the dev process and why (so they can fix it).
At a place like Google, I can actually tell you all the paths that people take when trying to achieve a journey. So that means I know all the loops (counts, times, etc) through development tools that start with something like "user opens their editor". Whether that's "open editor, make change, build, test, review, submit" or "open editor, make change, go to lunch", or "open editor, go look at docs, go back to editor, go back to docs, etc".
So i have real answers to something like "how often do people start in their IDE, discover they can't figure out how to do X, leave the IDE to go find the answer, not find it, give up, and go to lunch". I can tell you what the top X where that happens is, and how much time is or is not wasted through this path, etc.
Just as an example. I can then use all of this to improve the tooling so users can get more done.
You will not find this in most public tooling, and to the degree telemetry exists that you could generate for your own use, nobody thinks about how all that telemetry works together.
Now, mind you, all the above is meant as an explanation - i'm trying to explain why the public attempts don't end up as "good". But myself, good/bad is all about what you value.
Most tradeoffs here were deliberate.
But they are tradeoffs.
Some people value the flexibility more than coherence. or whatever. I'm not gonna judge them, but I can explain why you can't have it all :)
Re: Code search is hard
#94Earlier quoted context omitted.
Amazon internal tools for building codes are _amazing_. Brazil is their internal dependency management tool. It handles building and versioning software. It introduced the concept of version sets which essentially allows you to group up related software, e.g. version 1.0 of my app needs version 1.1 of library x and 2.0 of runtime y. This particular set of software versions get its own version number. Everything from…
so what I'm hearing is that app-1.0 needs app-1.0-runtime-build-20240410 which was, itself, built from a base of runtime-y-2.0 and layering library-x-1.11 upon it, kind of like # in some "app-runtimes" project, they assemble your app's runtime cat > Dockerfile Dockerfile .gitlab-ci.yml and then, yadda, yadda, blue-green, incremental rollout https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/... >, feature fl…
No, not even close. You might even have it exactly backwards.
Re: Code search is hard
#95Earlier quoted context omitted.
Just left Google a few months ago. My take is that there's a difference between a company that is willing to invest money into EngProd endeavors, and a company that uses SaaS for everything. While I can understand that most companies don't have the financial means to invest heavily into EngProd, the outcome is that the tightly integrated development experience in the former is far superior. Code Search is definitely…
What's #1? Memegen?
Cheesy answer but, the people.
Re: Code search is hard
#96Earlier quoted context omitted.
so what I'm hearing is that app-1.0 needs app-1.0-runtime-build-20240410 which was, itself, built from a base of runtime-y-2.0 and layering library-x-1.11 upon it, kind of like # in some "app-runtimes" project, they assemble your app's runtime cat > Dockerfile Dockerfile .gitlab-ci.yml and then, yadda, yadda, blue-green, incremental rollout https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/... >, feature fl…
No, not even close. You might even have it exactly backwards.
which is why, as I originally asked GP: what have you already tried and what features were they missing
I presume by "exactly backwards" you mean that one should have absolutely zero knobs to influence anything because the Almighty Jeff Build System does all the things, which GitLab also supports but is less amusing to look at on an Internet forum because it's "you can't modify anything, it just works, trust me"
Or, you know, if you have something constructive to add to this discussion feel free to use more words than "lol, no"
Re: Code search is hard
#97A feature I'd appreciate from Val Town is the ability to point it to a GitHub repo that I own and have it write the source code for all of my Vals to that repo, on an ongoing basis. Then I could use GitHub code search, or even "git pull" and run ripgrep.
I've actually built a tool in Val Town that could be used as the basis for something like this: https://www.val.town/v/nbbaier/valToGH
Right now it only commits one val, but it would be trivial to write it into a loop and then use a scheduled val to have it run over all your vals as a cron job!
Re: Code search is hard
#98Hey! I'm a val.town fanboy and I immediately thought about a workaround while reading the blog post: What if I dumped every publics vals in Github, in order to be able to user their (awesome) search ? So here is my own "Val Town Search": https://val-town-search.pomdtr.me And here is the repo containing all vals, updated hourly thanks to a github action: https://github.com/pomdtr/val-town-mirror
This is great!
Re: Code search is hard
#99Earlier quoted context omitted.
My experience has been that any of these in-house things do not adapt well to the high chaos of external environments, as if there are 3 companies one will find 9 systems and processes in use thus making "one size fits all" a fantasy But, I'll bite: what made the CD system so dreamy, and what have you evaluated thus far that fall short?
Amazon internal tools for building codes are _amazing_. Brazil is their internal dependency management tool. It handles building and versioning software. It introduced the concept of version sets which essentially allows you to group up related software, e.g. version 1.0 of my app needs version 1.1 of library x and 2.0 of runtime y. This particular set of software versions get its own version number. Everything from…
How did you avoid version hell? At Google, almost everything just shipped from master (except for some things that had more subtle bugs, those did their work on a dev branch and merged into master after testing).
Re: Code search is hard
#100Earlier quoted context omitted.
No, not even close. You might even have it exactly backwards.
which is why, as I originally asked GP: what have you already tried and what features were they missing I presume by "exactly backwards" you mean that one should have absolutely zero knobs to influence anything because the Almighty Jeff Build System does all the things, which GitLab also supports but is less amusing to look at on an Internet forum because it's "you can't modify anything, it just works, trust me" Or,…
I don't work at Amazon, and haven't for a long time, and this format is insufficient to fully express what they're doing, so I won't try.
You're better off searching for how Brazil and Apollo work.
That being said, the short of it is that: imagine when you push a new revision to source control, you (you) can run jobs testing every potential consumer of that new revision. As in, you push libx-1.1.2 and anyone consuming libx >= 1.1 (or any variety of filters) is identified. If the tests succeed, you can update their dependencies on your package and even deploy them, safely and gradually, to production without involving the downstream teams at all. If they don't, you can choose your own adventure: pin them, fork, fix them, patch the package, revise the versioning, whatever you want.
It's designed to be extremely safe and put power in the hands of those updating dependencies to do so safely within reason.
Imagine you work on a library and you can test your PR against every consumers.
It's not unlike what Google and other monorepos accomplish but it's quite different also. You can have many live versions simultaneously. You don't have to slog it out and patch all the dependents -- maybe you should, but you have plenty of options.
It all feels very simple. I'm glossing over a lot.