Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ripgrep – A new command line search tool

blog.burntsushi.net

Post reply on HN
Print view View original

181–190 of 219 posts

Re: Ripgrep – A new command line search tool

#181

Post by alphapapa »

libman wrote: post #36

Tragically the news that LLVM is switching to a non-Copyfree license (see copyfree.org/standard/rejected) has ruined everything... Nothing written in Rust can be called Free Software. :(

According to that site, Free Software (as defined by the FSF) is not Copyfree.

As far as I can tell, Copyfree is like the BSD license, except you can only restrict changes to the license text itself; the license itself may not be changed or added to, but everything else in the project may be used and changed in any way. So, effectively, it seems like a license where the license itself is copyrighted with all rights reserved, but the rest of the project is public domain.

So, just like BSD, anyone can take a Copyfree project proprietary, turning it non-free.

The GPL is as important today as ever.

Re: Ripgrep – A new command line search tool

#182

Post by burntsushi »

luckman212 wrote: post #175
skanga wrote: post #173

Earlier quoted context omitted.

Looks interesting. But I did not find binaries for it and do not want to setup a rust env to try it out.

The releases are right there on GitHub: https://github.com/BurntSushi/ripgrep/releases

I think the GP was asking about xsv, not ripgrep. There are binary releases for xsv though: https://github.com/BurntSushi/xsv/releases

Re: Ripgrep – A new command line search tool

#183

Post by prattmic »

brandmeyer wrote: post #180
prattmic wrote: post #162

Earlier quoted context omitted.

- The initial mapping of each file in any thread must halt all of the threads which are otherwise active. - Every page fault in any mapping must also halt all of the threads. These are certainly not the case on Linux, and I'd imagine not on other OSes, as it would be terrible for performance. Each mapping (i.e., mmap(2) call) is synchronized with other paths that read process memory maps, such as other mmap(2), munma…

So, I did some testing with 'perf'. This is on an older Intel processor, 2-cores with hyperthreading. These were all done on the same set of files, using the binary release of ripgrep v0.1.16 on Debian Jessie: At -j1, --mmap: 95 kdTLB load-misses, 2800 page faults At -j2: --mmap: 170 kdTLB load-misses, 2840 page faults At -j3: --mmap: 230 kdTLB load-misses, 2800 page faults At -j4, --mmap: 4180 context switches, 2900…

I did some experimentation last night as well. I suspected a lot of the cost came from the unmapping the files and the required invalidations and TLB shootdowns required to do so.

I made rg simply not munmap files when it was done with them (I made this drop do nothing: https://github.com/danburkert/memmap-rs/blob/master/src/unix...)

Searching for PM_RESUME in the Linux source gave me these results:

    --no-mmap: ~400ms
    --mmap (with munmap): ~750ms
    --mmap (without munmap): ~550ms
So eliding munmap made a big difference, but it was still not enough to beat out reading the files. perf shows that the mmap syscall itself is just too expensive (this is --mmap (with munmap)):

      Children      Self  Command  Shared Object       Symbol
    -   81.88%     0.00%  rg       rg                  [.] __rust_try
       - __rust_try
          - 50.57% std::panicking::try::call::ha112cda315d6c57d
             - 47.73% rg::Worker::search_mmap::h5179a76c63e344d0
                - 23.91% __GI___munmap
                     6.08% smp_call_function_many
                     3.14% rwsem_spin_on_owner
                     1.86% native_queued_spin_lock_slowpath
                     0.94% osq_lock
                     0.67% native_write_msr_safe
                     0.52% unmap_page_range
                + 21.41% _$LT$rg..search_buffer..BufferSearcher$LT$$u27$a$C$$u20$W$GT$$GT$::run::hd0f8b2830716be0c
                  0.80% memchr
          - 17.99% __mmap64
               5.20% rwsem_down_write_failed
               1.96% rwsem_spin_on_owner
               0.77% osq_lock
               0.59% native_queued_spin_lock_slowpath
          + 6.79% 0x1080d
            1.72% __GI___libc_close
            1.27% __memcpy_sse2_unaligned
            0.98% __fxstat64
            0.56% __GI___ioctl

Re: Ripgrep – A new command line search tool

#184

Post by pyroholic »

mbrubeck wrote: post #14
fsiefken wrote: post #2

nice, but does it compile and run on armhf? I don't see any binaries

I was able to build ripgrep from source for ARM, cross-compiling from my laptop running Debian, following the directions here: https://github.com/japaric/rust-cross#tldr-ubuntu-example (I haven't actually run it because I don't have an ARM linux device handy.)

Cross-compiled and ran a couple basic searches on an armv7l device. So at least the basic functionality works just fine.

Re: Ripgrep – A new command line search tool

#185

Post by jonstewart »

jonstewart wrote: post #17

Nice! Lightgrep[1] uses libicu et al to look up code points for a user-specified encoding and encode them as bytes, then just searches for the bytes. Since ripgrep is presumably looking just for bytes, too, and compiling UTF-8 multibyte code points to a sequence of bytes, perhaps you can do likewise with ICU and support other encodings. ICU is a bear to build against when cross-compiling, but it knows hundreds of enc…

Yep! It's a multipattern engine so we use an NFA--slower than a normal engine, but effective when you have thousands of patterns to search for in binary streams. Since we are often searching binary, we'll encounter text fragments in lots of different encodings, so they're all compiled into the NFA. We wrote a paper about its Unicode support: http://www.dfrws.org/sites/default/files/session-files/paper....

Re: Ripgrep – A new command line search tool

#186

Post by jonstewart »

burntsushi wrote: post #72
jonstewart wrote: post #17

Nice! Lightgrep[1] uses libicu et al to look up code points for a user-specified encoding and encode them as bytes, then just searches for the bytes. Since ripgrep is presumably looking just for bytes, too, and compiling UTF-8 multibyte code points to a sequence of bytes, perhaps you can do likewise with ICU and support other encodings. ICU is a bear to build against when cross-compiling, but it knows hundreds of enc…

I hadn't heard of liblightgrep, nice. It's on my short list for looking more closely. I doubt I'd ever be comfortable with Rust's regex engine growing a dependency on libicu, but it's still worth understanding your implementation. Some questions, if you don't mind. The big one is: does your regex engine use finite automata, and does it put the text decoding into the automaton itself? For example, when you compile the…

Reply fail, see above.

Re: Ripgrep – A new command line search tool

#187

Post by chx »

I am not sure how excited I am ... I readily accept this to be faster than ag -- but ag already scans 5M lines in a second for a string literal on my machine. Not having to switch tools when I need a recursive regexp is win enough to tolerate a potential .4s vs .32s second everyday search.

Re: Ripgrep – A new command line search tool

#189

Post by diimdeep »

wamatt wrote: post #102
diimdeep wrote: post #98

Earlier quoted context omitted.

AFAIK mdfind heavily depends on file system events in macOS, it would be painful or impossible to implement such system in cross platform way with support for file systems like FAT etc.

sure, FAT. OTOH some people think it's not a totally intractable problem A cross-platform file change monitor with multiple backends: Apple OS X File System Events, BSD kqueue, Solaris/Illumos File Events Notification, Linux inotify, Microsoft Windows and a stat()-based backend. http://emcrisostomo.github.io/fswatch/ https://github.com/emcrisostomo/fswatch

nice.
Post reply on HN