Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Don’t underestimate grep-based code scanning

littlemaninmyhead.wordpress.com

Post reply on HN
Print view View original

21–30 of 122 posts

Re: Don’t underestimate grep-based code scanning

#21

Post by dmit »

Annatar wrote: post #17
adrianN wrote: post #15

Earlier quoted context omitted.

It "reinvented" it and made it dramatically faster. Rust is also available on FreeBSD. I don't think you actually need Rust to run ripgrep. It's not like it's an interpreted language.

Since when do FreeBSD executables run on the illumos family of operating systems, as far as I know, there is no freebsd-branded zone yet in illumos? Dramatically faster? Is there any scientific evidence to that? If it were me, I wouldn't rush to make assumptions.

Illumos is also a supported platform (SPARC and x86-64).

Re: Don’t underestimate grep-based code scanning

#22

Post by Annatar »

adrianN wrote: post #16
Annatar wrote: post #13

Earlier quoted context omitted.

However fast it is, it's going to have a tough time beating /usr/bin/fgrep.

fgrep -r takes about six times longer than rg on the repository that I'm currently working on.

Real fgrep does not implement -r because that would be implementing tools within tools, which is against the UNIX®️ philosophy.

Try

/usr/bin/find . -depth -type f -print | /usr/bin/xargs -i /usr/bin/fgrep string '{}'

and run it several times so that the filesystem cache is primed.

Re: Don’t underestimate grep-based code scanning

#23

Post by Annatar »

dmit wrote: post #21
Annatar wrote: post #17

Earlier quoted context omitted.

Since when do FreeBSD executables run on the illumos family of operating systems, as far as I know, there is no freebsd-branded zone yet in illumos? Dramatically faster? Is there any scientific evidence to that? If it were me, I wouldn't rush to make assumptions.

Illumos is also a supported platform (SPARC and x86-64).

You obviously haven't tried it on either of those. They are "second tier", which means one is completely on one's own.

What in your opinion would have to be the size of the source code to warrant jumping through the hoops to get this software running, as opposed to a combination of find + xargs + egrep,fgrep,awk?

Re: Don’t underestimate grep-based code scanning

#24

Post by semi-extrinsic »

Annatar wrote: post #17
adrianN wrote: post #15

Earlier quoted context omitted.

It "reinvented" it and made it dramatically faster. Rust is also available on FreeBSD. I don't think you actually need Rust to run ripgrep. It's not like it's an interpreted language.

Since when do FreeBSD executables run on the illumos family of operating systems, as far as I know, there is no freebsd-branded zone yet in illumos? Dramatically faster? Is there any scientific evidence to that? If it were me, I wouldn't rush to make assumptions.

As for the dramatically faster, the ripgrep author doesn't claim this. What he claims (and supports with benchmarks) is the obverse, that there are no other tools dramatically faster than ripgrep.

Basically the stated goal is to be "fancy" like ack etc. and yet remain as fast as good ole' grep.

Re: Don’t underestimate grep-based code scanning

#25

Post by bruce_one »

arethuza wrote: post #19
jimmaswell wrote: post #11

Still never going to beat AST-integrated searching like VS has for C#. Which has a regex search too.

Are there any stand-alone AST based search tools?

I've found a few, eg https://www.graspjs.com/ is a Javascript one.

I do wonder if there is a multi-language aware one though.

Re: Don’t underestimate grep-based code scanning

#26

Post by _kbh_ »

K0nserv wrote: post #2

Just a small note that I would highgly recommend ripgrep[0] over standard grep. It's another modern tool that has been created by leveraging Rust and it's from BurntSushi[1] who is excellent. 0: https://github.com/BurntSushi/ripgrep 1. https://github.com/BurntSushi

rg + fzf really makes for a great toolset for some quick first pass code review.

Re: Don’t underestimate grep-based code scanning

#27

Post by adwn »

Annatar wrote: post #22
adrianN wrote: post #16

Earlier quoted context omitted.

fgrep -r takes about six times longer than rg on the repository that I'm currently working on.

Real fgrep does not implement -r because that would be implementing tools within tools, which is against the UNIX®️ philosophy. Try /usr/bin/find . -depth -type f -print | /usr/bin/xargs -i /usr/bin/fgrep string '{}' and run it several times so that the filesystem cache is primed.

> Real fgrep does not implement -r

That's BS. The fgrep on my system – GNU grep 3.1 – provides recursive search (-r). What now, are you claiming that's not "real fgrep"? [1]

> that would be implementing tools within tools, which is against the UNIX®️ philosophy

Even more BS. Or are you telling me that "rm -r" is also against the "UNIX philosophy"?

> /usr/bin/find . -depth -type f -print | /usr/bin/xargs -i /usr/bin/fgrep string '{}'

Terrible, terrible advice. Cumbersome, error-prone, and slow as molasses. A quick test: Searching for 'asdfadsgf' in the Linux kernel repository takes 0.25 s using rg, 12 s using GNU fgrep -f, and 228 s (!) using your command.

You know, when your ideology results in the worst results of all, you should really reconsider your ideology.

[1] https://en.wikipedia.org/wiki/No_true_Scotsman

Re: Don’t underestimate grep-based code scanning

#28

Post by pjc50 »

jpalomaki wrote: post #18

Random idea: maybe you could supercharge this by introducing to grep some constructs from programming languages. Now you have things like "word character", "whitespace", "start of line". In supercharged version you would have "function", "identifier"

Then it's not grep, it's something much slower which has to parse the syntax.

Re: Don’t underestimate grep-based code scanning

#29

Post by pjc50 »

Annatar wrote: post #22
adrianN wrote: post #16

Earlier quoted context omitted.

fgrep -r takes about six times longer than rg on the repository that I'm currently working on.

Real fgrep does not implement -r because that would be implementing tools within tools, which is against the UNIX®️ philosophy. Try /usr/bin/find . -depth -type f -print | /usr/bin/xargs -i /usr/bin/fgrep string '{}' and run it several times so that the filesystem cache is primed.

.. which falls over as soon as you have a file with a space in the name.

Edit: this highlights the big weakness in the "UNIX philosophy", in which the only record delimiter that's conventionally recognized in pipelines is the newline but the shell recognizes characters as filename delimiters that are also allowed in filenames. Causing a cascade of delimiter bugs. Sometimes you really do need a bit more structure to your data.

(The UNIX philosophy is best understood in contrast to what went before - the COBOL or JCL style where files had fixed records, in turn based on fixed-column punchcard layouts.)

Re: Don’t underestimate grep-based code scanning

#30

Post by LandR »

jpalomaki wrote: post #18

Random idea: maybe you could supercharge this by introducing to grep some constructs from programming languages. Now you have things like "word character", "whitespace", "start of line". In supercharged version you would have "function", "identifier"

Visual Studio with Resharper does this.

It's very very fast / almost instant even with hundreds of source code files and millions of lines of code.

I hit ctrl+T and then can search everything, this give me a drop down that filters out the more I type, select the item in the dropdown and it goes to that source file.

I can also type:

/t and search just types

/m members

/mm methods

/u unit tests

/f file

/fp project

/e event

/mp property

/mf field

/ff project folder

e.g.

/t Foo

will find all the Foos

/mm SavePhoto

will find any methods called SavePhoto

Same works in JetBrains Rider for C# stuff.

I couldn't dev without this now, and it's all built into my IDE.

Post reply on HN