Live data from Hacker News

Don’t underestimate grep-based code scanning

littlemaninmyhead.wordpress.com

31–40 of 122 posts

Re: Don’t underestimate grep-based code scanning

#31

Still never going to beat AST-integrated searching like VS has for C#. Which has a regex search too.

Are there any stand-alone AST based search tools?

I'm going to plug my own here: astpath, which is AST-based search for Python. https://github.com/hchasestevens/astpath/

Re: Don’t underestimate grep-based code scanning

#32

Random idea: maybe you could supercharge this by introducing to grep some constructs from programming languages. Now you have things like "word character", "whitespace", "start of line". In supercharged version you would have "function", "identifier"

https://github.com/mvdan/gogrep does something like this for Go :)

Re: Don’t underestimate grep-based code scanning

#33
post #22
post #16

Earlier quoted context omitted.

fgrep -r takes about six times longer than rg on the repository that I'm currently working on.

Real fgrep does not implement -r because that would be implementing tools within tools, which is against the UNIX®️ philosophy. Try /usr/bin/find . -depth -type f -print | /usr/bin/xargs -i /usr/bin/fgrep string '{}' and run it several times so that the filesystem cache is primed.

rg searches my repo in under a second. Your command takes over a minute, with a warm cache. It also takes thirty seconds to type. And it doesn't work for files with a space in them. Even on single files rg is faster for me than fgrep. For example my tags file is 2GB large. rg takes 0.4 seconds to search it, fgrep takes 0.6 seconds.

Re: Don’t underestimate grep-based code scanning

#35

Random idea: maybe you could supercharge this by introducing to grep some constructs from programming languages. Now you have things like "word character", "whitespace", "start of line". In supercharged version you would have "function", "identifier"

There's an old project from Facebook that does something like this, called Pfff[0]

It provides a tool called sgrep (syntactical grep) that lets you do some cool tricks, for instance:

`sgrep "some_func(X,X)"` returns all calls to some_func with the same argument repeated.

`sgrep "some_func(X,Y,...)"` returns all calls to some_func with 3 or more arguments.

It's come in very handy for refactoring some troublesome codebases.

[0]: https://github.com/facebookarchive/pfff

Re: Don’t underestimate grep-based code scanning

#36
post #27
post #22

Earlier quoted context omitted.

Real fgrep does not implement -r because that would be implementing tools within tools, which is against the UNIX®️ philosophy. Try /usr/bin/find . -depth -type f -print | /usr/bin/xargs -i /usr/bin/fgrep string '{}' and run it several times so that the filesystem cache is primed.

> Real fgrep does not implement -r That's BS. The fgrep on my system – GNU grep 3.1 – provides recursive search (-r). What now, are you claiming that's not "real fgrep"? [1] > that would be implementing tools within tools, which is against the UNIX®️ philosophy Even more BS. Or are you telling me that "rm -r" is also against the "UNIX philosophy"? > /usr/bin/find . -depth -type f -print | /usr/bin/xargs -i /usr/bin/f…

This is why Unix is great. It gives you enough tools to shoot yourself in the foot.

Re: Don’t underestimate grep-based code scanning

#37
post #6

Earlier quoted context omitted.

Why is it better than grep?

It's way faster, which is great when you're working with big repos. It's designed for recursively searching through a lot of files.

GNU grep is fast as well but by default it doesn't ignore anything. Granted it's handy to have this configured out of the box, but I prefer to know and use the flags and perhaps write a shell script wrapper, and I find that practically it feels just as fast as rg or others. My main point in doing this is to avoid the situation where I'm on a different machine to my laptop and can just get going right away without having to install anything. There's a trade off in all things, I just prefer it this way.

Re: Don’t underestimate grep-based code scanning

#38
post #23
post #21

Earlier quoted context omitted.

Illumos is also a supported platform (SPARC and x86-64).

You obviously haven't tried it on either of those. They are "second tier", which means one is completely on one's own. What in your opinion would have to be the size of the source code to warrant jumping through the hoops to get this software running, as opposed to a combination of find + xargs + egrep,fgrep,awk?

> one is completely on one's own

That's not accurate.

The test suites are not run on CI for tier 2, but they are at least guaranteed to build.

Tier 2 platforms have binary builds, are supported by rustup and often work just fine.

Re: Don’t underestimate grep-based code scanning

#39
post #3

One thing which was not immediately obvious to me for a while: the stricter your language’s formatting is, the easier it will be to grep source code. I work a lot with Go, where all code in our repository is gofmt'ed. You can get quite far with regular expressions for finding/analyzing Go code. (And when regexps don’t cut it anymore, Go has excellent infrastructure for working with it programmatically. http://golang.…

> the stricter your language’s formatting is, the easier it will be to grep source code

Well that makes sense, the less reliable your input text, the more complex the regexp.

Re: Don’t underestimate grep-based code scanning

#40
post #4
post #2

Just a small note that I would highgly recommend ripgrep[0] over standard grep. It's another modern tool that has been created by leveraging Rust and it's from BurntSushi[1] who is excellent. 0: https://github.com/BurntSushi/ripgrep 1. https://github.com/BurntSushi

In general I think that's very good advice. In this particular instance however a dumb old grep might be superior because it could catch potential security vulnerabilities that are not explicitly hardcoded in the source code by greping through the compilation artifacts for instance. Sure you'll get a bunch of false positives that way but at least you know that nothing is slipping through the cracks.

To be clear, you can disable all smart filtering in ripgrep. e.g., `rg -uuu foo` should be equivalent to `grep -r foo`. And if you want to exhaustively search binary files, then you need to add the `-a` flag to both commands.
Post reply on HN