Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code search is hard

blog.val.town

Post reply on HN
Print view View original

71–80 of 164 posts

Re: Code search is hard

#71

Post by sdesol »

bevekspldnw wrote: post #52
sdesol wrote: post #25

> It’s hard to find any accounts of code-search using FTS I'm actually going to be doing this soon. I've thought about code search for close to a decade, but I walked away from it, because there really isn't a business for it. However, now with AI, I'm more interested in using it to help find relevant context and I have no reason to believe FTS won't work. In the past I used Lucene, but I'm planning on going all in w…

Mmm, it’s not that straight forward: indexes can vastly slow down large scale ingest, so it’s really about when to index as well. I work with a lot of multi billion row datasets and a lot of my recent focus has been on developing strategies to avoid the slow down with ingest, and then enjoying the speed up for indexed on search. I’ve also gotten some mjnd boggling speed increases by summarizing key searchable data in…

Yeah I agree. I've had a lot of practice so far with coordinating between hundreds of thousands of tables to ensure ingestion/lookup is fast. Everything boils down to optimizing for your query patterns.

I also believe in using what I call "compass tables" (like your summarization tables), which I guess are indexes of indexes.

Re: Code search is hard

#73

Post by pomdtr »

Hey! I'm a val.town fanboy and I immediately thought about a workaround while reading the blog post:

What if I dumped every publics vals in Github, in order to be able to user their (awesome) search ?

So here is my own "Val Town Search": https://val-town-search.pomdtr.me

And here is the repo containing all vals, updated hourly thanks to a github action: https://github.com/pomdtr/val-town-mirror

Re: Code search is hard

#74

Post by simonw »

pomdtr wrote: post #73

Hey! I'm a val.town fanboy and I immediately thought about a workaround while reading the blog post: What if I dumped every publics vals in Github, in order to be able to user their (awesome) search ? So here is my own "Val Town Search": https://val-town-search.pomdtr.me And here is the repo containing all vals, updated hourly thanks to a github action: https://github.com/pomdtr/val-town-mirror

Well this is fun...

    git clone https://github.com/pomdtr/val-town-mirror
    cd val-town-mirror
    rg news.ycombinator.com
Now I can ripgrep search public Vals, e.g. to see who's hitting Hacker News from a Val.

Re: Code search is hard

#75

Post by pomdtr »

simonw wrote: post #74
pomdtr wrote: post #73

Hey! I'm a val.town fanboy and I immediately thought about a workaround while reading the blog post: What if I dumped every publics vals in Github, in order to be able to user their (awesome) search ? So here is my own "Val Town Search": https://val-town-search.pomdtr.me And here is the repo containing all vals, updated hourly thanks to a github action: https://github.com/pomdtr/val-town-mirror

Well this is fun... git clone https://github.com/pomdtr/val-town-mirror cd val-town-mirror rg news.ycombinator.com Now I can ripgrep search public Vals, e.g. to see who's hitting Hacker News from a Val.

Yeah, and you can finally run/debug vals locally (kind of, the version query param is not yet supported)

Re: Code search is hard

#76

Post by shepherdjerred »

mdaniel wrote: post #54
scubbo wrote: post #49

Earlier quoted context omitted.

I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close. (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)

My experience has been that any of these in-house things do not adapt well to the high chaos of external environments, as if there are 3 companies one will find 9 systems and processes in use thus making "one size fits all" a fantasy But, I'll bite: what made the CD system so dreamy, and what have you evaluated thus far that fall short?

Amazon internal tools for building codes are _amazing_.

Brazil is their internal dependency management tool. It handles building and versioning software. It introduced the concept of version sets which essentially allows you to group up related software, e.g. version 1.0 of my app needs version 1.1 of library x and 2.0 of runtime y. This particular set of software versions get its own version number.

Everything from the CI/CD to the code review tool to your local builds use the same build configuration with Brazil. All software packages in Brazil are built from source on Amazon's gigantic fleet of build servers. Builds are cached, so even though Amazon builds its own version of Make, Java, etc., these are all built and cached by the build servers and downloaded.

A simple Java application at Amazon might have hundreds of dependencies (because you'll need to build Java from scratch), but since this is all cached you don't have to wait very long.

Lastly, you have Pipelines which is their internal CI/CD tool which integrates naturally with Brazil + the build fleet. It can deploy to their internal fleet with Apollo, or to AWS Lambda, S3 buckets, etc.

In all, everything is just very well integrated. I haven't seen anything come close to what you get internally at Amazon.

Re: Code search is hard

#77

Post by sdesol »

scubbo wrote: post #49
keybored wrote: post #38

Earlier quoted context omitted.

> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .

I suspect you're being sarcastic - but can confirm that being nearly two years out of Amazon, I still miss its in-house CD system nearly every day. I've actively looked around for OSS replacements and very few come anywhere close. (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)

> (I would be _delighted_ for someone to "Umm actually" me by providing a great product!)

I think the issue is, nobody would be willing to pay for a good solution since they usually come with a steep maintenance cost. I wouldn't be surprised if the in-house CD team at Amazon were putting out fires every week/month behind the scene.

Re: Code search is hard

#78

Post by MatthiasPortzel »

pomdtr wrote: post #73

Hey! I'm a val.town fanboy and I immediately thought about a workaround while reading the blog post: What if I dumped every publics vals in Github, in order to be able to user their (awesome) search ? So here is my own "Val Town Search": https://val-town-search.pomdtr.me And here is the repo containing all vals, updated hourly thanks to a github action: https://github.com/pomdtr/val-town-mirror

That is, uh, one solution, to say the least.

There’s a HN comment I’ll never forget where the commenter suggests that Discord move their search infrastructure to a series of text file searched with ripgrep, but Val.town’s scale is small enough that they could actually consider it.

Re: Code search is hard

#79

Post by refulgentis »

humanrebar wrote: post #66
jeffbee wrote: post #64

Earlier quoted context omitted.

Sure. Also, if you eat a bunch of glass, you will get a stomach ache. I have no idea why anyone uses a polyrepo.

The problem with monorepos is that they're so great that everyone has a few.

God that is good.

Re: Code search is hard

#80

Post by fizx »

There's a million paths, but here's one I like.

Use ElasticSearch. It will scale more than Postgres. Three hosted options are AWS, Elastic, Bonsai. I founded Bonsai and retired (so am partial), but they will provide the best human support for you, and you won't have to worry about java Xmx.

Your goal with ES is to use the Regex PatternAnalyzer to split the code into reasonable exact code-shaped tokens (not english words).

Here's a rough GPT4 explanation with sample config that I'd head towards: https://chat.openai.com/share/e4d08586-b7ef-48f2-9de1-7f82ea...

Post reply on HN