Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Code search is hard

blog.val.town

Post reply on HN
Print view View original

81–90 of 164 posts

Re: Code search is hard

#81

Post by bevekspldnw »

sdesol wrote: post #71
bevekspldnw wrote: post #52

Earlier quoted context omitted.

Mmm, it’s not that straight forward: indexes can vastly slow down large scale ingest, so it’s really about when to index as well. I work with a lot of multi billion row datasets and a lot of my recent focus has been on developing strategies to avoid the slow down with ingest, and then enjoying the speed up for indexed on search. I’ve also gotten some mjnd boggling speed increases by summarizing key searchable data in…

Yeah I agree. I've had a lot of practice so far with coordinating between hundreds of thousands of tables to ensure ingestion/lookup is fast. Everything boils down to optimizing for your query patterns. I also believe in using what I call "compass tables" (like your summarization tables), which I guess are indexes of indexes.

Scaling databases both oddly frustrating and also rewarding. Getting that first query that executes at 10x of the old one feels great. The week of agony that makes it possible…less so.

Re: Code search is hard

#82

Post by sdesol »

bevekspldnw wrote: post #81
sdesol wrote: post #71

Earlier quoted context omitted.

Yeah I agree. I've had a lot of practice so far with coordinating between hundreds of thousands of tables to ensure ingestion/lookup is fast. Everything boils down to optimizing for your query patterns. I also believe in using what I call "compass tables" (like your summarization tables), which I guess are indexes of indexes.

Scaling databases both oddly frustrating and also rewarding. Getting that first query that executes at 10x of the old one feels great. The week of agony that makes it possible…less so.

Fully agree. I do have to give hardware a lot of credit though. With SSD and now NVME, fast random read/write speed is what makes a lot of things possible.

Re: Code search is hard

#83

Post by alienchow »

keybored wrote: post #38
peter_l_downs wrote: post #14

Earlier quoted context omitted.

If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs. [0] https://github.com/livegrep/livegrep

> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .

Just left Google a few months ago.

My take is that there's a difference between a company that is willing to invest money into EngProd endeavors, and a company that uses SaaS for everything. While I can understand that most companies don't have the financial means to invest heavily into EngProd, the outcome is that the tightly integrated development experience in the former is far superior. Code Search is definitely #2 on the list of things I miss the most.

Re: Code search is hard

#85

Post by fragmede »

alienchow wrote: post #83
keybored wrote: post #38

Earlier quoted context omitted.

> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .

Just left Google a few months ago. My take is that there's a difference between a company that is willing to invest money into EngProd endeavors, and a company that uses SaaS for everything. While I can understand that most companies don't have the financial means to invest heavily into EngProd, the outcome is that the tightly integrated development experience in the former is far superior. Code Search is definitely…

What's #1? Memegen?

Re: Code search is hard

#86

Post by bytefish »

fizx wrote: post #80

There's a million paths, but here's one I like. Use ElasticSearch. It will scale more than Postgres. Three hosted options are AWS, Elastic, Bonsai. I founded Bonsai and retired (so am partial), but they will provide the best human support for you, and you won't have to worry about java Xmx. Your goal with ES is to use the Regex PatternAnalyzer to split the code into reasonable exact code-shaped tokens (not english wo…

GitLab is also using ElasticSearch, so one could recreate the ElasticSearch Indices they came up with. [1]

They also share some of the challenges, they faced along the way. It also discusses interesting challenges, like implementing the authorization model. [2], [3]

When GitHub removed its most useful Search feature, which is sorting results by date, I wrote a small “Search Engine” with ElasticSearch to selectively index Microsoft repositories. It works good enough for my needs. [4]

[1] https://gitlab.com/gitlab-org/gitlab/-/blob/7bbbc00bd871aeb6...

[2] https://about.gitlab.com/blog/2019/07/16/elasticsearch-updat...

[3] https://about.gitlab.com/blog/2020/04/28/elasticsearch-updat...

[4] https://github.com/bytefish/ElasticsearchCodeSearch

Re: Code search is hard

#87

Post by johnthescott »

the rum index has worked well for us on roughly 1TB of pdfs. written by postgrespro, same folks who wrote core text search and json indexing. not sure why rum not in core. we have no problems.

   https://github.com/postgrespro/rum

Re: Code search is hard

#88

Post by elevatedastalt »

keybored wrote: post #38
peter_l_downs wrote: post #14

Earlier quoted context omitted.

If you ever leave you can use Livegrep, which was based on code-search work done at Google. I personally don't use it right now but it's great and will probably meet all your needs. [0] https://github.com/livegrep/livegrep

> If you ever leave you can use Livegrep, which was based on code-search work done at Google. If I’ve learned anything from the fainting spells that I-work-at-X have over their internal tools on HN: no, whatever the public/OSS variant is always a mere shadow of the real thing .

You would be better off actually trying to understand those sentiments instead of posting sarcastic replies on HN.

The sort of tight knit integration and developer-focus that internal tools at developer-friendly companies like Google has cannot be matched by clobbering together 50 different SaaS products, half of which will probably run out of funding in 3 years.

You literally have entire companies just LARPing internal tools that Google has because they are just that good. Glean is literally Moma. There's really nothing like Critique or Buganizer.

Re: Code search is hard

#89

Post by mdaniel »

shepherdjerred wrote: post #76
mdaniel wrote: post #54

Earlier quoted context omitted.

My experience has been that any of these in-house things do not adapt well to the high chaos of external environments, as if there are 3 companies one will find 9 systems and processes in use thus making "one size fits all" a fantasy But, I'll bite: what made the CD system so dreamy, and what have you evaluated thus far that fall short?

Amazon internal tools for building codes are _amazing_. Brazil is their internal dependency management tool. It handles building and versioning software. It introduced the concept of version sets which essentially allows you to group up related software, e.g. version 1.0 of my app needs version 1.1 of library x and 2.0 of runtime y. This particular set of software versions get its own version number. Everything from…

so what I'm hearing is that app-1.0 needs app-1.0-runtime-build-20240410 which was, itself, built from a base of runtime-y-2.0 and layering library-x-1.11 upon it, kind of like

  # in some "app-runtimes" project, they assemble your app's runtime
  cat > Dockerfile  Dockerfile  .gitlab-ci.yml
and then, yadda, yadda, blue-green, incremental rollout https://gitlab.com/gitlab-org/gitlab/-/blob/v16.10.2-ee/lib/...>, feature flags https://docs.gitlab.com/ee/operations/feature_flags.html>, error capture https://docs.gitlab.com/ee/operations/error_tracking.html#in...>, project-managed provisioning https://docs.gitlab.com/ee/user/infrastructure/iac/#integrat...>, on call management https://docs.gitlab.com/ee/operations/incident_management/>, on call runbooks https://docs.gitlab.com/ee/user/project/clusters/runbooks/in...>

you can orchestrate all that from ~~Slack~~ Chime :-D if you're into that kind of thing https://docs.gitlab.com/ee/ci/chatops/

Re: Code search is hard

#90

Post by voiceblue »

fragmede wrote: post #85
alienchow wrote: post #83

Earlier quoted context omitted.

Just left Google a few months ago. My take is that there's a difference between a company that is willing to invest money into EngProd endeavors, and a company that uses SaaS for everything. While I can understand that most companies don't have the financial means to invest heavily into EngProd, the outcome is that the tightly integrated development experience in the former is far superior. Code Search is definitely…

What's #1? Memegen?

The business end of gThanks
Post reply on HN