Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We are stuck with egrep and fgrep (unless you like beating people)

utcc.utoronto.ca

Post reply on HN
Print view View original

291–300 of 354 posts

Re: We are stuck with egrep and fgrep (unless you like beating people)

#291

Post by teddyh »

mannykannot wrote: post #284
teddyh wrote: post #203

Earlier quoted context omitted.

You can’t prove a negative.

There is no largest prime.

Using that model, we can prove conclusively that, since a behavior has changed (a warning printed), it might cause problems. Therefore, we cannot prove that it cannot cause problems. What we would really like, though, is an actual problem shown to exist. Just like in mathematics; it’s one thing to prove that it’s impossible to prove something could not exist, but another thing entirely to show it existing.

Re: We are stuck with egrep and fgrep (unless you like beating people)

#292

Post by hnlmorg »

fnordpiglet wrote: post #280
hnlmorg wrote: post #146

There seems to be a lot of confusion in this discussion about what the change is and how egrep and fgrep work. These are not symlinks like some have suggested but rather shell scripts. You can see the exact commit diff here: https://git.savannah.gnu.org/gitweb/?p=grep.git;a=blobdiff;f... I remember around ~10 years ago being told "you should never use `egrep` because it is slower than `grep -E`." precisely because th…

I am not sure I agree with the logic. On an absolute instruction count scale grep will often dwarf the instructions necessary for the higher level logic captured in the shell script. Insert appeal to andahls law here. Writing in some more optimal language often requires considerably more work, and if the marginal benefit of reducing a few forks is minuscule compared to grep performance, why would you do this?

It depends on where your hot path is. If you’re writing the kind of performance critical code where the different between ‘egrep’ and ‘grep -E’ matters, then you shouldn’t be writing it in $SHELL to begin with. Not just because of the expense fork()ing, but because Bash is interpreted, because each exec requires loading new code into your L1 cache, and so on. It’s just never going to be as quick as something written in most other languages with a mature regex library.

I’m not saying this as a criticism against shell scripting (I’m an author of an alternative $SHELL and shell scripting language, so do have a love for shell scripting). I’m saying it as a criticism against people who try to optimise against the wrong things.

I guess in a way we are “violently agreeing” (as some say) because your point about the engineering effort mattering more than micro-optimisations is an example of the right kind of optimisations. Which is what I’m also, albeit badly, trying to describe.

Re: We are stuck with egrep and fgrep (unless you like beating people)

#293

Post by kccqzy »

Phelinofist wrote: post #290
kccqzy wrote: post #220

Earlier quoted context omitted.

If you have interacted with Paul Eggert in real life (I have; he's a professor at UCLA), you would've known that he is a man that values purity over practicality. Not surprising at all.

Is it the same guy that maintains tzdb? https://lwn.net/Articles/870478/

Yes he is. He's a maintainer for a lot of GNU software including Emacs, sort, etc.

Running

    fgrep -U -l -r "Paul Eggert" /usr/bin
should turn up quite a few results. (That's of course an undercount because he doesn't always put his name into everything he touches.)

Re: We are stuck with egrep and fgrep (unless you like beating people)

#295

Post by manv1 »

It's ironic that GNU is using POSIX as a justification for a decision.

From RMS:

Following a standard is important to the extent it serves users. We do not treat a standard as an authority, but rather as a guide that may be useful to follow. Thus, we talk about following standards rather than "complying" with them. See the section Non-GNU Standards in the GNU Coding Standards.

We strive to be compatible with standards on most issues because, on most issues, that serves users best. But there are occasional exceptions.

For instance, POSIX specifies that some utilities measure disk space in units of 512 bytes. I asked the committee to change this to 1K, but it refused, saying that a bureaucratic rule compelled the choice of 512. I don't recall much attempt to argue that users would be pleased with that decision.

Since GNU's second priority, after users' freedom, is users' convenience, we made GNU programs measure disk space in blocks of 1K by default.

https://opensource.com/article/19/7/what-posix-richard-stall...

Re: We are stuck with egrep and fgrep (unless you like beating people)

#296

Post by mirker »

richard_todd wrote: post #244
mirker wrote: post #228

Earlier quoted context omitted.

As someone who learned about regular expressions before extensively using grep, I found grep to be quite unintuitive, since extended grep is what I am conceptually thinking about from a “theory” point if view. One difference between grep and compression is that grep is bounded by regular languages, but compression is more free form. It’s conceivable to therefore view grep as mature enough of a technology to be finish…

Consider the popularity of ffmpeg and imagemagick. Putting the mainstream algorithms together under one interface seems to be an appealing model to a lot of people, even though new algorithms are constantly being developed in those areas. Personally, for audio encoding, I'm much happier just installing ffmpeg than I would be gathering and learning a bunch of flac/ogg/opus/mp3/etc encoders.

Yes, those are quite different. But you also can’t pipe lossy compression data willy-nilly without compounding data loss, so it’s hard to imagine an alternative with unix-style tools.

The difference may also stem from those being application-focused rather than command line, so they don’t have a burden of maintaining legacy applications forever.

Re: We are stuck with egrep and fgrep (unless you like beating people)

#297

Post by cestith »

TillE wrote: post #125
cestith wrote: post #108

Earlier quoted context omitted.

Here's one example. This is in code my team inherited a long time ago, and there are many more like it. databases=`find /var/lib/mysql -type d | sed 's/\/var\/lib\/mysql\///g' | egrep -v 'mysql|test|performance|schema'`

That doesn't do anything with stderr, so it doesn't break.

It does output to STDERR an extra warning.

Re: We are stuck with egrep and fgrep (unless you like beating people)

#298

Post by tgv »

rgoulter wrote: post #121
AshamedCaptain wrote: post #50

Earlier quoted context omitted.

Both of your suggestions are basically equivalent to "think before you upgrade" and/or "don't upgrade". This is the second elephant in the room with all these distros that "don't use globals" and/or statically link everything (or do an analogue to that, like nix). There's very little benefit for desktops. So the upgrade to dependency X breaks component Y, and you are forced not to update X, or at least, Y's copy of X…

> The situation is as unatenable long-term as it is on a regular distro. I may be misremembering, but isn't this pretty much the default user experience on rolling distributions like Arch Linux? -- You consult the wiki to see what changes you need to make are (or maybe see what breaks after an update), make the change, and get on with your desktop experience.

Idk, but Arch Linux is a bad model, in my experience. I had inherited a system that ran Arch Linux, but hadn't been updated in a while. Imagine my surprise when I needed to update something, and couldn't. A part of the upgrade path (or whatever you want to call it) had been removed. So now I have an Arch Linux system that hasn't been updated for way too long, and we need to consider abandoning the product that runs on that machine, or invest in porting it to some modern environment. It's hidden behind a firewall with access for only 2 IP addresses, so it's unlikely to get hacked, but it's most undesirable.

If you like that model, fine, but don't force it onto the whole world, unless you can commit more resources to it than to Arch Linux, and basically keep all upgrade paths alive forever.

Re: We are stuck with egrep and fgrep (unless you like beating people)

#299

Post by mike_hock »

hnlmorg wrote: post #146

There seems to be a lot of confusion in this discussion about what the change is and how egrep and fgrep work. These are not symlinks like some have suggested but rather shell scripts. You can see the exact commit diff here: https://git.savannah.gnu.org/gitweb/?p=grep.git;a=blobdiff;f... I remember around ~10 years ago being told "you should never use `egrep` because it is slower than `grep -E`." precisely because th…

What's the problem? Just make egrep and fgrep shell aliases in the default /etc/profile (if you're a distro maintainer).

Also, where's the extra fork?

    #!/bin/sh
    exec grep -E "$@"

Re: We are stuck with egrep and fgrep (unless you like beating people)

#300

Post by jraph »

hnlmorg wrote: post #211
chaxor wrote: post #184

Earlier quoted context omitted.

> "if concerned with performance why are you running code as a shell script" Rewrite grep in python before running - got it. :P

That's not what I suggested. I was saying if you're writing performance sensitive code with a hot loop calling `egrep` then a smarter approach might be to use a language better tuned for performance which supports regex libraries. Shell scripts have their place and can out perform most languages if you're writing simple logic that can run across large datasets in parallel, such as cat very-large-file | grep "do not w…

> please excuse the useless use of `cat`, it's there to illustrate the direction of data flow

Note that you can write
Instead of

    cat very-large-file |
To avoid a useless use of cat and keep the direction. Not that I care very much though. Using cat is less surprising to most people.
Post reply on HN