Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Pythonpy – the swiss army knife of the command line

github.com

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Show HN: Pythonpy – the swiss army knife of the command line

#31

Post by vram22 »

ctrijueque wrote: post #3

Nice. Similar to PyP from Sony Imageworks: http://opensource.imageworks.com/?p=pyp The Pyed Piper Tutorial: http://www.youtube.com/watch?v=eWtVWF0JSJA

Some time ago, I had blogged a series of posts about various ways of doing pipe-like operations in Python, including one experimental project of mine (which is not like real Unix pipes, which are IPC, but is intra-process), and some others, including PyP. Below are links to some of those posts, which may be of interest. Note: some of the posts link to each other, so I've posted them below in reverse chronological order.

Swapping pipe components at runtime with pipe_controller:

http://jugad2.blogspot.in/2012/10/swapping-pipe-components-a...

Using PipeController to run a pipe incrementally:

http://jugad2.blogspot.in/2012/09/using-pipecontroller-to-ru...

PipeController v.01 released - simulating UNIX-style pipes in Python:

http://jugad2.blogspot.in/2012/08/pipecontroller-v01-release...

Some ways of doing UNIX-style pipes in Python:

http://jugad2.blogspot.in/2011/09/some-ways-of-doing-unix-st...

Re: Show HN: Pythonpy – the swiss army knife of the command line

#32

Post by vram22 »

mitchty wrote: post #23
S4M wrote: post #17

I like it! Now I can count the number of files I have in a directory: ls |py -l 'sum(1 for x in l)' And also I can count the number of files whose names matche a certain extension: ls |grep .py |py -l 'sum(1 for x in l)' I am not saying it was not possible before, but I don't know how to do it in bash. Looking forward to use that for some basic stats on file. The following should sum up the first column of a csv. cat…

You'll probably want to learn the regular unix way as well. Tools like this are nice, but you're already 99% there as it is for most things. ls | wc -l ls | grep -c .py (note the cat and pipe are not needed) awk -F',' '{sum+=$1}END{print sum}' data.csv

> ls | grep -c .py

That won't work if there are files with names like foo.py.txt.

These will:

ls | grep -c '.py$'

or better, because less names to filter out:

ls -l *.py | wc -l

Re: Show HN: Pythonpy – the swiss army knife of the command line

#33

Post by gfxmonk »

ctrijueque wrote: post #3

Nice. Similar to PyP from Sony Imageworks: http://opensource.imageworks.com/?p=pyp The Pyed Piper Tutorial: http://www.youtube.com/watch?v=eWtVWF0JSJA

I wrote a similar tool a while back: http://gfxmonk.net/dist/doc/piep/

Mostly out of frustration for PyP not being lazy (on large inputs it reads the entire file up-front, or at least used to).

But it was quite interesting to implement all the standard python idioms (like slicing) in a lazy way, and it's not that complicated a tool. I still use it a lot whenever I have a nontrivial pipeline to write.

Re: Show HN: Pythonpy – the swiss army knife of the command line

#34

Post by Russell91 »

Qantourisc wrote: post #4

ls | py -x '"mv %s %s.txt" % (x,x)' | sh has me worried, I can't find the escape code directly. Also you might wish to offer the following api ls | py -X '["mv","%s","%s.txt"] and run those using processes or sub-process. This would ensure the right argument in the right place. Just an idea.

Hmm, I just added this example yesterday, and it seems to be highly contested, so I'll exchange it for something less controversial. That being said, if you're concerned about spaces, the best way to do this is either using shutil.move, or by generating output like mv 'file name' 'file name.txt' to pipe into sh. Of course, since you need to wrap your commands with single quotes to avoid bash quirks, it may seem difficult to use single quotes in a pythonpy expression. Fear not though - just use sharp quotes (`), which pythonpy will swap out for single quotes:

  ls | py -x '"mv `%s` `%s`" % (x,x)' | sh

Re: Show HN: Pythonpy – the swiss army knife of the command line

#35

Post by Russell91 »

mitchty wrote: post #23
S4M wrote: post #17

I like it! Now I can count the number of files I have in a directory: ls |py -l 'sum(1 for x in l)' And also I can count the number of files whose names matche a certain extension: ls |grep .py |py -l 'sum(1 for x in l)' I am not saying it was not possible before, but I don't know how to do it in bash. Looking forward to use that for some basic stats on file. The following should sum up the first column of a csv. cat…

You'll probably want to learn the regular unix way as well. Tools like this are nice, but you're already 99% there as it is for most things. ls | wc -l ls | grep -c .py (note the cat and pipe are not needed) awk -F',' '{sum+=$1}END{print sum}' data.csv

Yea, IMO the best way to do this with pythonpy is:

  py -l 'len(l)'
That being said, I still use wc -l every time. In fact, I always prefer regular unix commands (e.g. grep, xargs, head) to pythonpy when possible. But there are some things, such as:

  py -l 'l[::2]'
which while possible with tools like sed, are just much better expressed with pythonpy.

Re: Show HN: Pythonpy – the swiss army knife of the command line

#36

Post by [deleted] »

gfxmonk wrote: post #33
ctrijueque wrote: post #3

Nice. Similar to PyP from Sony Imageworks: http://opensource.imageworks.com/?p=pyp The Pyed Piper Tutorial: http://www.youtube.com/watch?v=eWtVWF0JSJA

I wrote a similar tool a while back: http://gfxmonk.net/dist/doc/piep/ Mostly out of frustration for PyP not being lazy (on large inputs it reads the entire file up-front, or at least used to). But it was quite interesting to implement all the standard python idioms (like slicing) in a lazy way, and it's not that complicated a tool. I still use it a lot whenever I have a nontrivial pipeline to write.

[deleted]

Re: Show HN: Pythonpy – the swiss army knife of the command line

#37

Post by Russell91 »

gfxmonk wrote: post #33
ctrijueque wrote: post #3

Nice. Similar to PyP from Sony Imageworks: http://opensource.imageworks.com/?p=pyp The Pyed Piper Tutorial: http://www.youtube.com/watch?v=eWtVWF0JSJA

I wrote a similar tool a while back: http://gfxmonk.net/dist/doc/piep/ Mostly out of frustration for PyP not being lazy (on large inputs it reads the entire file up-front, or at least used to). But it was quite interesting to implement all the standard python idioms (like slicing) in a lazy way, and it's not that complicated a tool. I still use it a lot whenever I have a nontrivial pipeline to write.

Yea, fortunately pythonpy supports lazy iteration over sys.stdin when you really need it. Just like in python, the syntax won't be as nice as using a list. But it works:

  py 'itertools.count(1)' | py 'itertools.islice(stdin, 0, 10, 2)'
However, the number of times that you need this are surprisingly rare. Most lazy operations don't require that each row be aware of the surrounding row context, and using the much simpler:

  py -x 'new_row_from_old_row(x)'
will get the job done in a lazy fashion. Usually, when you need rows to be context aware, as in:

  py -l 'sorted(l)'
or

  py -l 'set(l)'
it's just not possible to accomplish your task without reading in all of stdin.

Re: Show HN: Pythonpy – the swiss army knife of the command line

#38

Post by gfxmonk »

Russell91 wrote: post #37
gfxmonk wrote: post #33

Earlier quoted context omitted.

I wrote a similar tool a while back: http://gfxmonk.net/dist/doc/piep/ Mostly out of frustration for PyP not being lazy (on large inputs it reads the entire file up-front, or at least used to). But it was quite interesting to implement all the standard python idioms (like slicing) in a lazy way, and it's not that complicated a tool. I still use it a lot whenever I have a nontrivial pipeline to write.

Yea, fortunately pythonpy supports lazy iteration over sys.stdin when you really need it. Just like in python, the syntax won't be as nice as using a list. But it works: py 'itertools.count(1)' | py 'itertools.islice(stdin, 0, 10, 2)' However, the number of times that you need this are surprisingly rare. Most lazy operations don't require that each row be aware of the surrounding row context, and using the much simpl…

Cool :), glad it's supported, at least for the simple case of line-wise transforms.

Some things can't be done without reading everything. But there are still a number of operations on "all of stdin" that can safely be done lazily. I'm particularly fond of "divide stdin into chunks of lines separated by " [0]. Which does need context, but only enough to determine where the current chunk ends (typically a few lines).

`py` seems to be aimed at a single expression per invocation (nice and simple), while `piep` recreates pipelines internally (more complex but also means pipelines can produce arbitrary objects rather than single-line strings). So I'm not really sure how you'd do the above in `py` anyway.

[0] http://gfxmonk.net/dist/doc/piep/#piep.list.BaseList.divide

Re: Show HN: Pythonpy – the swiss army knife of the command line

#39

Post by whit537 »

canistr wrote: post #24
thejosh wrote: post #11

Earlier quoted context omitted.

This is what gittip is for. And Kickstarter/Indiegogo/Crowdfunding has shown genuine interest to help fund useful tools and libraries lately which is great.

I think if Git Tip had a button on a GitHub repo (a la Heroku button), this would be amazing.

How about the Gittip badges from http://shields.io/?

Re: Show HN: Pythonpy – the swiss army knife of the command line

#40

Post by cohomologo »

I attempted to install on Windows - it doesn't work due to the lack of os.geteuid(), which is called in the setup.py file to check for root. With a little bit of finagling I could get around that and make it install, but not work due to undefined SIGPIPE in the signal module. It makes sense that this could be UNIX only, but that wasn't stated anywhere on the webpage. Perhaps that should be mentioned somewhere.
Post reply on HN