Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scripts I wrote that I use all the time

evanhahn.com

Post reply on HN
Print view View original

391–400 of 408 posts

Re: Scripts I wrote that I use all the time

#391

Post by aktau »

1vuio0pswjnm7 wrote: post #374
aktau wrote: post #250

I have a bunch, but one that I rarely see mentioned but use all the time is memo(1) ( https://github.com/aktau/dotfiles/blob/master/bin/memo ). It memoizes the command passed to it. $ memo curl https://some-expensive.com/api/call | jq . | awk '...' Manually clearing it (for example if I know the underlying data has changed: $ memo -c curl https://some-expensive.com/api/call In-pipeline memoization (includes the input…

. #!/usr/bin/env bash # # memo(1), memoizes the output of your command-line, so you can do: # # $ memo | ... # # Instead of # # $ > tmpfile # $ cat tmpfile | ... # $ rm tmpfile to save output, sed can be used in the pipeline instead of tee for example, x=$(mktemp -u); test -p $x||mkfifo $x; zstd -19 tmpfile.zst & |sed w$x| ; # You can even use it in the middle of a pipe if you know that the input is not # extremely l…

Those commands are a (1) harder to grok and (2) do not actually use the memoized result (tmpfile.zst) to speed up a subsequent run.

Can you give a more complete example of how you would use this to speed up developing a pipeline?

Re: Scripts I wrote that I use all the time

#392

Post by aktau »

news_hacker wrote: post #382
aktau wrote: post #250

I have a bunch, but one that I rarely see mentioned but use all the time is memo(1) ( https://github.com/aktau/dotfiles/blob/master/bin/memo ). It memoizes the command passed to it. $ memo curl https://some-expensive.com/api/call | jq . | awk '...' Manually clearing it (for example if I know the underlying data has changed: $ memo -c curl https://some-expensive.com/api/call In-pipeline memoization (includes the input…

I've been using bkt ( https://github.com/dimo414/bkt ) for subprocess caching. It has some nice features, like providing a ttl for cache expiration. In-pipeline memoization looks nice, I'm not sure it supports that

I was not aware of bkt. Thanks for the link. It seems very similar to memo, and has more features:

  - Explicit TTL
  - Ability to include working directory et al. as context for the cache key.
There do appear to be downsides (from my PoV) as well:

  - It's a rust program, so it needs to be compiled (memo is a bash/zsh script and runs as-is).
  - There's no mention of transparent compression, either in the README or through simple source code search. I did find https://github.com/dimo414/bkt/issues/62 which mentions swappable backends. The fact that it uses some type of database instead of just the filesystem is not a positive for me, I prefer the state to be easy to introspect with common tools. I will often memo commands that output gigabytes of data, which is usually highly compressible. Transparent compression fixes that up. One could argue this could be avoided with a filesystem-level feature, like ZFS transparent compression. But I don't know how to detect that in a cross-FS fashion.
I opened https://github.com/dimo414/bkt/discussions/63 so the author of bkt can perhaps also participate.

Re: Scripts I wrote that I use all the time

#393

Post by 1vuio0pswjnm7 »

1vuio0pswjnm7 wrote: post #374
aktau wrote: post #250

I have a bunch, but one that I rarely see mentioned but use all the time is memo(1) ( https://github.com/aktau/dotfiles/blob/master/bin/memo ). It memoizes the command passed to it. $ memo curl https://some-expensive.com/api/call | jq . | awk '...' Manually clearing it (for example if I know the underlying data has changed: $ memo -c curl https://some-expensive.com/api/call In-pipeline memoization (includes the input…

. #!/usr/bin/env bash # # memo(1), memoizes the output of your command-line, so you can do: # # $ memo | ... # # Instead of # # $ > tmpfile # $ cat tmpfile | ... # $ rm tmpfile to save output, sed can be used in the pipeline instead of tee for example, x=$(mktemp -u); test -p $x||mkfifo $x; zstd -19 tmpfile.zst & |sed w$x| ; # You can even use it in the middle of a pipe if you know that the input is not # extremely l…

If provide sample showing (a) input format of text and (b) desired output format of text, then perhaps can provide an example of how to do the text processing

Re: Scripts I wrote that I use all the time

#394

Post by bawis »

klaussilveira wrote: post #288
soiltype wrote: post #6

This is exactly the kind of stuff I'm most interested in finding on HN. How do other developers work, and how can I get better at my work from it? What's always interesting to me is how many of these I'll see and initially think, "I don't really need that." Because I'm well aware of the effect (which I'm sure has a name - I suppose it's similar to induced demand) of "make $uncommon_task much cheaper" -> "$uncommon_ta…

This is one of the things I miss the most about hacker conferences. The sharing of tools, scripts, tips and tricks. It was, and still is, just as fun as trading cards.

What's a hacker conference?

Re: Scripts I wrote that I use all the time

#395

Post by virgoerns »

o11c wrote: post #53

I keep meaning to generalize this (directory target, multiple sources, flags), but I get quite a bit of mileage out of this `unmv` script even as it is: #!/bin/sh if test "$#" != 2 then echo 'Error: unmv must have exactly 2 arguments' exit 1 fi exec mv "$2" "$1"

How do you use it so it's more ergonimic than typing arguments manually reversed? `unmv !$` ?

Re: Scripts I wrote that I use all the time

#396

Post by virgoerns »

hackeraccount wrote: post #341
some_guy_nobel wrote: post #80

These are great, and I have a few matching myself. Here are some super simple ones I didn't see that I use almost every day: cl="clear" g="git" h="history" ll="ls -al" path='echo -e ${PATH//:/\\n}' lv="live-server" And for common navigation: dl="cd ~/Downloads" dt="cd ~/Desktop"

I'm terrible about remembering shortcuts (edit a bash line in an editor? Can never remember it) but clear (CTRL-l) is one that really stuck. That and exit (CTRL-d). A guy I used to work with just mentioned it casually and someone it just seared itself into my brain.

FYI, ctrl-d isn't a shortcut to exit terminal. It sends EOF (end of file) character which, when reaches shell, closes stdinput file of shell. It generally closes any active interactive input, like all repls, interactive input to sed etc. When interactive shell loses possibility to get more input, it closes as soon as possible and then its parent, the terminal window, also closes. More-less :)

Re: Scripts I wrote that I use all the time

#397

Post by yellowapple »

bawis wrote: post #394
klaussilveira wrote: post #288

Earlier quoted context omitted.

This is one of the things I miss the most about hacker conferences. The sharing of tools, scripts, tips and tricks. It was, and still is, just as fun as trading cards.

What's a hacker conference?

A conference in which hackers congregate.

My favorite recent one was Handmade Seattle, but that one's kaput as of this year, and it seems everything else along similar lines is overseas and/or invite-only.

Re: Scripts I wrote that I use all the time

#398

Post by yellowapple »

latexr wrote: post #295
tom_ wrote: post #287

Earlier quoted context omitted.

If you write these sorts of things in Python, argparse is worth investigating: https://docs.python.org/3/library/argparse.html - it's pretty easy to use, makes it easy to separate the command line handling from the rest of the code, and, importantly, will generate a --help page for you. And if you want something it can't do, you can still always write the code yourself!

I don’t like Python in general, but even so I’ll say that argparse is indeed very nice. When I was writing ruby, I always felt that OptionParser¹ wasn’t as good. Swift has Argument Parser², officially from Apple, which is quite featureful. For shell, I have a a couple of bespoke patterns I have been reusing in every script for many years. ¹ https://github.com/ruby/optparse ² https://github.com/apple/swift-argument-pa…

Many years ago I wrote a library I called “Ruby on Bales”¹ specifically due to my frustrations with the state of command-line argument parsing for Ruby scripts. I haven't touched it in a long while; maybe I should revisit it.

----

¹ https://github.com/YellowApple/bales

Re: Scripts I wrote that I use all the time

#399

Post by yellowapple »

chipsrafferty wrote: post #380
duskdozer wrote: post #168

Earlier quoted context omitted.

Maybe, but 1. even if it costs more time, it could also save more annoyance which could be a benefit 2. by publishing the scripts, anyone else who comes across them can use them and save time without the initial cost. similarly, making and sharing these can encourage others to share their own scripts, some of which the author could save time with

In my experience, it's not "maybe" but "almost certainly" which is why I stopped doing this. Every time I get a new system I would have to set everything up again, it's not cross platform, doesn't work when using someone else's computer, suddenly breaks for some reason or another, or you forget it exists... The annoyance of all these factors for outweighs the benefits, in my experience. It's just that the scripts fee…

> Every time I get a new system I would have to set everything up again

Sounds like something you could automate with a script :)

Re: Scripts I wrote that I use all the time

#400

Post by yellowapple »

janalsncm wrote: post #361
normie3000 wrote: post #231

Earlier quoted context omitted.

> I disagree with that xkcd The xkcd doesn't seem to be pushing an agenda, just providing a lookup table. Time spent vs time saved is factual.

The title of the comic is “ Is It Worth the Time?”. To take a concrete example, if I spend 30 minutes on a task every six months, over 5 years that’s 5 hours of “work” hours. So the implication is that it’s not worth automating if it takes more than 5 hours to automate. But if those are 5 hours of application downtime, it’s pretty clearly worth it even if I have to spend way more than 5 hours to reduce downtime.

Time saved also ain't the only factor here. I'll often automate something not because it actually saves a lot of time, but rather because it codifies an error-prone process and having it scripted out reduces the risk of human error by enough of a degree to be worth spending more time on it than I'd save.
Post reply on HN