Live data from Hacker News

look(1): lines beginning with given string

linux.die.net

31–40 of 47 posts

Re: look(1): lines beginning with given string

#31
post #23

If you don't have this basic linux utility, you can use grep '^string' file.txt but there's a greater chance of having look installed than grep.

The interesting thing about look is that it uses binary search, so it is much faster.

Binary search is useless unless the file is sorted. Most often my files are not sorted, so egrep '^line' works just fine.

Re: look(1): lines beginning with given string

#32

Earlier quoted context omitted.

The interesting thing about look is that it uses binary search, so it is much faster.

Binary search is useless unless the file is sorted. Most often my files are not sorted, so egrep '^line' works just fine.

The interesting use case for me is that you can make a sorted index and look up records efficiently, i.e. within a script. It works with ctags indexes, for instance.

Re: look(1): lines beginning with given string

#33
post #5

Earlier quoted context omitted.

A better one-line description would be "look(1): binary search for lines with a given prefix in a sorted file" Which tells you exactly what the pros/cons are compared to plain grep.

I created a 1.1G file with dictionary words and timed... % du -h a 1.1G a % time look 'dog' a | wc -l 53856 real 0m0.021s user 0m0.020s sys 0m0.003s % time grep '^dog' a | wc -l 53856 real 0m28.593s user 0m0.977s sys 0m2.223s Ok grep performed worse than what I expected (sort took a long time though).

That seem abysmally slow for grep. On my machine, with a 1.1G file made of 1200 copies of /usr/share/dict/words, GNU grep 2.16 takes about 1.5 seconds. Try with LC_ALL=C?

Re: look(1): lines beginning with given string

#34
post #23

If you don't have this basic linux utility, you can use grep '^string' file.txt but there's a greater chance of having look installed than grep.

As long as I have been using *nixes, I have never met a machine that didn't have grep. It was written by Ken Thompson himself in 1974, or 41 years ago, and first came out in Unix 4th edition.

grep is also specified by POSIX, whereas look isn't.

Re: look(1): lines beginning with given string

#35
post #23

If you don't have this basic linux utility, you can use grep '^string' file.txt but there's a greater chance of having look installed than grep.

The interesting thing about look is that it uses binary search, so it is much faster.

Binary search is only faster if access to the elements is constant-time, which is not true for a file with varying-length records (lines). Line 1000 (counting from 0) could be at offset 2000 if each line is a single character + newline, or it could be at offset 2000000 if each line is 999 characters and a newline, or somewhere in between.

Re: look(1): lines beginning with given string

#36

Earlier quoted context omitted.

The interesting thing about look is that it uses binary search, so it is much faster.

Binary search is only faster if access to the elements is constant-time, which is not true for a file with varying-length records (lines). Line 1000 (counting from 0) could be at offset 2000 if each line is a single character + newline, or it could be at offset 2000000 if each line is 999 characters and a newline, or somewhere in between.

It's probably much faster in most cases of text files though.

I also think your constant time claim sounds too strong. You can eat a lot of end of line search time after the binary search and still beat a linear/regex search.

As someone else pointed out, look can also exit sooner.

Re: look(1): lines beginning with given string

#37

Earlier quoted context omitted.

The interesting thing about look is that it uses binary search, so it is much faster.

Binary search is useless unless the file is sorted. Most often my files are not sorted, so egrep '^line' works just fine.

It obvious that look was written as an optimization for this specific case though. The default file is the user dictionary which is interesting. Do you think they didn't know about grep in 1979?

Re: look(1): lines beginning with given string

#38

Earlier quoted context omitted.

The interesting thing about look is that it uses binary search, so it is much faster.

Binary search is only faster if access to the elements is constant-time, which is not true for a file with varying-length records (lines). Line 1000 (counting from 0) could be at offset 2000 if each line is a single character + newline, or it could be at offset 2000000 if each line is 999 characters and a newline, or somewhere in between.

You don't need to know the number of the line you're looking at. You jump to the middle of the file, and you compare the nearest line to the search pattern. Then you recurse on the top or bottom half.

Re: look(1): lines beginning with given string

#39
post #5

Earlier quoted context omitted.

A better one-line description would be "look(1): binary search for lines with a given prefix in a sorted file" Which tells you exactly what the pros/cons are compared to plain grep.

I created a 1.1G file with dictionary words and timed... % du -h a 1.1G a % time look 'dog' a | wc -l 53856 real 0m0.021s user 0m0.020s sys 0m0.003s % time grep '^dog' a | wc -l 53856 real 0m28.593s user 0m0.977s sys 0m2.223s Ok grep performed worse than what I expected (sort took a long time though).

Seems that most time for grep was spent waiting on disk. This underlines the reason look may be significantly faster: actually reading the bad lines may take time.

Re: look(1): lines beginning with given string

#40
look a |wc -l => 5985

look -b a |wc -l => 1228

look -bf a |wc -l => 1228

You must sort the /usr/share/dict/words file according to ignore case for the option bf to work properly.

Edit: I was expecting

sort -f /usr/share/dict/words | look -bf a | wc -l to be 5985 but the result is 1228, don't know why.

Post reply on HN