Live data from Hacker News

Don’t underestimate grep-based code scanning

littlemaninmyhead.wordpress.com

51–60 of 122 posts

Re: Don’t underestimate grep-based code scanning

#51

The post's core message seems to be lost on HN. It's about screening sources for supposedly insecure and/or injection-prone funcs using simple text scanning (such as strcat, which however is considered in iOS apps when it is a C std API func); supposedly grepability is also about quickly finding code locations of messages and variables. But comments are all about Rust or Go superiority, irrelevant grep implementation…

Man, n-gate.com is going to have a field day.

Re: Don’t underestimate grep-based code scanning

#52
Don't use grep. Use ag[0], which is specifically designed for searching code. It's much faster, honors .gitignore, and the output can be piped back through grep if you like.

    ag FooBar | grep -v Baz
It's in brew/apt/yum etc as `the_silver_searcher` (although brew install ag works fine too).

0: https://github.com/ggreer/the_silver_searcher

Re: Don’t underestimate grep-based code scanning

#53
post #8
post #6

Earlier quoted context omitted.

It's way faster, which is great when you're working with big repos. It's designed for recursively searching through a lot of files.

Grep is really fast at that at the actual search (gnu grep at least), the gain there is mostly that "smarter" tools will ignore e.g. VCS data or binary files by default whereas grep will trawl through your PNGs and git packfiles.

If I remember that giant post of benchmarks correctly, there are some big exceptions, particularly around non-ASCII searches.

Re: Don’t underestimate grep-based code scanning

#54

The post's core message seems to be lost on HN. It's about screening sources for supposedly insecure and/or injection-prone funcs using simple text scanning (such as strcat, which however is considered in iOS apps when it is a C std API func); supposedly grepability is also about quickly finding code locations of messages and variables. But comments are all about Rust or Go superiority, irrelevant grep implementation…

Or maybe the core message just resonates and people have additional discussion in the comments?

Re: Don’t underestimate grep-based code scanning

#55
post #29

Earlier quoted context omitted.

.. which falls over as soon as you have a file with a space in the name. Edit: this highlights the big weakness in the "UNIX philosophy", in which the only record delimiter that's conventionally recognized in pipelines is the newline but the shell recognizes characters as filename delimiters that are also allowed in filenames . Causing a cascade of delimiter bugs. Sometimes you really do need a bit more structure to…

And that’s why PowerShell is awesome :)

Powershell falls over in the other direction: the objects flowing down the pipeline are "magic" and can't be serialised, or even necessarily inspected with normal tools. For most unix operations you can replace

    foo | sort
with

    foo > file
    sort 
I like the idea of powershell, but every time I try to do something complicated with it I'm disappointed.

Re: Don’t underestimate grep-based code scanning

#56
post #29
post #22

Earlier quoted context omitted.

Real fgrep does not implement -r because that would be implementing tools within tools, which is against the UNIX®️ philosophy. Try /usr/bin/find . -depth -type f -print | /usr/bin/xargs -i /usr/bin/fgrep string '{}' and run it several times so that the filesystem cache is primed.

.. which falls over as soon as you have a file with a space in the name. Edit: this highlights the big weakness in the "UNIX philosophy", in which the only record delimiter that's conventionally recognized in pipelines is the newline but the shell recognizes characters as filename delimiters that are also allowed in filenames . Causing a cascade of delimiter bugs. Sometimes you really do need a bit more structure to…

> .. which falls over as soon as you have a file with a space in the name.

Yeah, spaces are nasty. find has -print0 and xargs has -0 to handle this gracefully, but one needs to know to use it.

Re: Don’t underestimate grep-based code scanning

#57
post #52

Don't use grep. Use ag[0], which is specifically designed for searching code. It's much faster, honors .gitignore, and the output can be piped back through grep if you like. ag FooBar | grep -v Baz It's in brew/apt/yum etc as `the_silver_searcher` (although brew install ag works fine too). 0: https://github.com/ggreer/the_silver_searcher

It's not much faster as in "over 50% faster". It's faster to invoke as you have much less to type to scan recursively with ignore list. However I find that in many projects .gitignore is too extensive, because it includes generated code, that many times is quite informative. Then it's still nice to use those alternative grep-likes, but not by much. Besides, when you can't install easily it's hard to beat something that it's already there and everywhere else.

Re: Don’t underestimate grep-based code scanning

#58
post #57
post #52

Don't use grep. Use ag[0], which is specifically designed for searching code. It's much faster, honors .gitignore, and the output can be piped back through grep if you like. ag FooBar | grep -v Baz It's in brew/apt/yum etc as `the_silver_searcher` (although brew install ag works fine too). 0: https://github.com/ggreer/the_silver_searcher

It's not much faster as in "over 50% faster". It's faster to invoke as you have much less to type to scan recursively with ignore list. However I find that in many projects .gitignore is too extensive, because it includes generated code, that many times is quite informative. Then it's still nice to use those alternative grep-likes, but not by much. Besides, when you can't install easily it's hard to beat something th…

I second ag too. Not because it's faster, but because it is developer-oriented so it has sane defaults for searching into code.

Re: Don’t underestimate grep-based code scanning

#59
post #37
post #6

Earlier quoted context omitted.

It's way faster, which is great when you're working with big repos. It's designed for recursively searching through a lot of files.

GNU grep is fast as well but by default it doesn't ignore anything. Granted it's handy to have this configured out of the box, but I prefer to know and use the flags and perhaps write a shell script wrapper, and I find that practically it feels just as fast as rg or others. My main point in doing this is to avoid the situation where I'm on a different machine to my laptop and can just get going right away without hav…

Portability is key. There’s a reason these gnu tools have such staying power. It’s not necessarily because they’re the best, but because they’re ubiquitous.

Re: Don’t underestimate grep-based code scanning

#60
post #3

One thing which was not immediately obvious to me for a while: the stricter your language’s formatting is, the easier it will be to grep source code. I work a lot with Go, where all code in our repository is gofmt'ed. You can get quite far with regular expressions for finding/analyzing Go code. (And when regexps don’t cut it anymore, Go has excellent infrastructure for working with it programmatically. http://golang.…

Related to this, it is generally a very good idea to be strict when naming functions, parameters, variables, etc. so that each concept has exactly one name throughout the codebase.
Post reply on HN