Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code search is hard

blog.val.town

Post reply on HN
Print view View original

61–70 of 164 posts

Re: Code search is hard

#61

Post by klysm »

sqs wrote: post #15

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…

I apply this thinking to lots of problems. Do the dumb thing that involves the least state and prove we need to lean more towards memory for speed. It’s much simpler to keep things correct when nothing is cached

Re: Code search is hard

#62

Post by hinkley »

sqs wrote: post #15

I'm at Sourcegraph (mentioned in the blog post). We obviously have to deal with massive scale, but for anyone starting out adding code search to their product, I'd recommend not starting with an index and just doing on-the-fly searching until that does not scale. It actually will scale well for longer than you think if you just need to find the first N matches (because that result buffer can be filled without needing…

There was someone doing temporal databases that was compressing blocks on disk and doing streaming decompress and search on them. Things in L2 cache go very very fast.

Re: Code search is hard

#63

Post by humanrebar »

jeffbee wrote: post #48
birktj wrote: post #37

Earlier quoted context omitted.

I see most replies here ar mentioning the the build integration is what is mainly missing in the public tools. I wonder if nix and nixpkgs could be used here? Nix is a language agnostic build-system and with nixpkgs it has a build instructions for a massive amount of packages. Artifacts for all packages are also available via hydra. Nix should also have enough context so that for any project it can get the source cod…

Build integration is not the main thing that is missing between Livegrep and Code Search. The main thing that is missing is the semantic index. Kythe knows the difference between this::fn(int) and this::fn(double) and that::fn(double) and so on. So you can find all the callers of the nullary constructor of some class, without false positives of the callers of the copy constructor or the move constructor. Livegrep sim…

The build system coherence provided by a monorepo with a single build system is what makes you understand this::fn(double) as a single thing. Otherwise, you will get N different mostly compatible but subtly different flavors of entities depending on the build flavor, combinations of versioned dependencies, and other things.

Re: Code search is hard

#64

Post by jeffbee »

humanrebar wrote: post #63
jeffbee wrote: post #48

Earlier quoted context omitted.

Build integration is not the main thing that is missing between Livegrep and Code Search. The main thing that is missing is the semantic index. Kythe knows the difference between this::fn(int) and this::fn(double) and that::fn(double) and so on. So you can find all the callers of the nullary constructor of some class, without false positives of the callers of the copy constructor or the move constructor. Livegrep sim…

The build system coherence provided by a monorepo with a single build system is what makes you understand this::fn(double) as a single thing. Otherwise, you will get N different mostly compatible but subtly different flavors of entities depending on the build flavor, combinations of versioned dependencies, and other things.

Sure. Also, if you eat a bunch of glass, you will get a stomach ache. I have no idea why anyone uses a polyrepo.

Re: Code search is hard

#65

Post by boyter »

herrington_d wrote: post #42

Is it possible to combine n-gram and AST to dump a better indexing? Take `sourceCode.toString()` as an example, the AST can dump it to `sourceCode` and `toString`. A further indexer can break `sourceCode` to `source` and `code`. For ast dumping, project like https://github.com/ast-grep/ast-grep can help.

You could, but I don't know what you gain out of it. The underlying index would be almost the same size, and n-gram would also allow you to search for e.t for example which you are losing in this process.

Re: Code search is hard

#66

Post by humanrebar »

jeffbee wrote: post #64
humanrebar wrote: post #63

Earlier quoted context omitted.

The build system coherence provided by a monorepo with a single build system is what makes you understand this::fn(double) as a single thing. Otherwise, you will get N different mostly compatible but subtly different flavors of entities depending on the build flavor, combinations of versioned dependencies, and other things.

Sure. Also, if you eat a bunch of glass, you will get a stomach ache. I have no idea why anyone uses a polyrepo.

The problem with monorepos is that they're so great that everyone has a few.

Re: Code search is hard

#68

Post by morgante »

simonw wrote: post #57
worldsayshi wrote: post #56

Earlier quoted context omitted.

What does 'on the fly' entail here?

I'm going to guess brute force - scan everything for the search term, rather than trying to use an index. I'm always amazed at how fast ripgrep (rg) can brute force it's way through hundreds of MBs of source code.

Yes, exactly. When doing a search, we parse and search every file without any indexing.

Of course, it could still be sped up considerably with an index but brute force is surprisingly effective (we use some of the same techniques/crates as ripgrep).

Re: Code search is hard

#69

Post by boyter »

peter_l_downs wrote: post #8
semiquaver wrote: post #2

Be careful with trigram indexes. At least in the postgres 10 era they caused severe index bloat for frequently updated tables.

Interesting, do you know anywhere I can easily read more about this? (I will do my own research, too.)

Its a result of trigrams themselves. For example turning searchcode (please ignore plug, this is just the example I had to hand) goes from 1 thing you would need to index into 8.

    "searchcode"   -> [sea, ear, arc, rch, chc, hco, cod, ode]
As a result the index rapidly becomes larger than you would expect.

Re: Code search is hard

#70

Post by simonw »

A feature I'd appreciate from Val Town is the ability to point it to a GitHub repo that I own and have it write the source code for all of my Vals to that repo, on an ongoing basis.

Then I could use GitHub code search, or even "git pull" and run ripgrep.

Post reply on HN