How to Make Money Using Grep, Sed and Awk • Hacker News Board

Re: How to Make Money Using Grep, Sed and Awk

#31

Post by xrisk » Wed, Oct 11, 2017, 1:07 PM UTC

iDemonix wrote: post #28

yomansat wrote: post #25

I looked up the freelance job in question, it pays $36 for a 215 page publication from the 80s, including checking where automation has failed. A question to those who do freelance work: sites like Upwork often have low budgets, where do you guys find better projects with better compensation?

I gave up with all of them. A friend of mine is using Upwork and Freelancer (awful) to find gigs, and he tried to get me involved so I could help him out with overspill. It seemed like you had to put ludicrously low rates to get terrible work, and despite paying peanuts, everyone expects amazing results. I ditched it, joined some communities online, did a tiny bit of networking, and announced I was looking for work r…

I'm from India and I'm horrified that so many people are willing to do work for nothing or next to nothing...

Re: How to Make Money Using Grep, Sed and Awk

#32

Post by KGIII » Wed, Oct 11, 2017, 1:42 PM UTC

zwischenzug wrote: post #22

tinix wrote: post #16

Earlier quoted context omitted.

There's no reason you can't orchestrate awk and other tools from python though. (I doubt I'm the only one that's done that before).

I wrote a tool to do that: https://github.com/ianmiell/shutit Among many other things I use it to test Kubernetes/OpenShift clusters in Vagrant for Chef recipes: https://github.com/ianmiell/shutit-openshift-cluster/blob/ma... and here's some others in a more 'native' python format: https://github.com/ianmiell/shutit-scripts/ https://github.com/ianmiell/shutit-scripts/blob/master/logme... https://github.com/ianmiell/s…

> crazy talk about pushing awk through python

> oh! I wrote a tool to do that!

sniffles

That might just be my favorite HN exchange ever. Y'all remind me of emacs users. Between this and the unusual number of Perl mentions on HN today, my cockles are suitably heated.

That is a compliment. Nary a day passes where HN doesn't give me a reason to keep clicking links and learning.

I started off giggling and now I'm moved from my tablet to a laptop so that I can investigate your shutit - as it looks like a handy tool for learning. The scales look the most interesting.

I have been here for a while, but only recently (past couple of months) decided to comment. I lurked for like 12 months, just to see if I'd fit in. Why? HN continually has commentary about things I haven't yet learned.

In short, this is my awkward attempt to thank you. I'll be spending the afternoon trying to enjoy your shutit scales.

Re: How to Make Money Using Grep, Sed and Awk

#33

Post by xelxebar » Wed, Oct 11, 2017, 2:11 PM UTC

jaymzcampbell wrote: post #7

saagarjha wrote: post #3

Earlier quoted context omitted.

In my experience it tends to be at around 50 to 100 lines. At this point you usually encounter at least one "gotcha" or difficultly that bash can't handle in an elegant way and realize that it would probably be a lot faster if you just used another language instead.

As soon as I start adding ifs and battling to remember the exact syntax I give up and move to Python and argparse. Recently I've started using the Begins[1] library to make useful command line tools and scripts even more quickly. I appreciate it's less and less portable at this stage but so much more productive. Most of the time these scripts are very specific anyway so it's not a concern. The other thing I've notice…

There is a lot of historical arcana baked into the command line and bash, but I've found it worth learning like any other language.

Anyway, just for future reference you can use bash's `help` command to get documentation about builtins. In the case of conditionals, we want to loo at `test`:

    $ help test

I find this a lot easier than wading through the bash man page.

Re: How to Make Money Using Grep, Sed and Awk

#34

Post by zwischenzug » Wed, Oct 11, 2017, 2:14 PM UTC

KGIII wrote: post #32

zwischenzug wrote: post #22

Earlier quoted context omitted.

I wrote a tool to do that: https://github.com/ianmiell/shutit Among many other things I use it to test Kubernetes/OpenShift clusters in Vagrant for Chef recipes: https://github.com/ianmiell/shutit-openshift-cluster/blob/ma... and here's some others in a more 'native' python format: https://github.com/ianmiell/shutit-scripts/ https://github.com/ianmiell/shutit-scripts/blob/master/logme... https://github.com/ianmiell/s…

> crazy talk about pushing awk through python > oh! I wrote a tool to do that! sniffles That might just be my favorite HN exchange ever. Y'all remind me of emacs users. Between this and the unusual number of Perl mentions on HN today, my cockles are suitably heated. That is a compliment. Nary a day passes where HN doesn't give me a reason to keep clicking links and learning. I started off giggling and now I'm moved f…

Great, thanks!

You can find more info here on my blog:

https://zwischenzugs.wordpress.com/

do ping me with your experience.

Re: How to Make Money Using Grep, Sed and Awk

#35

Post by tonyedgecombe » Wed, Oct 11, 2017, 2:25 PM UTC

angarg12 wrote: post #2

At what point it pays off to switch to python/node instead of keep growing a bash script?

My rule is one line, I don't want to write any bash script to a file, one-liners are as far as it goes for me.

Re: How to Make Money Using Grep, Sed and Awk

#36

Post by bsdpunk » Wed, Oct 11, 2017, 2:37 PM UTC

ggm wrote: post #21

The best way to make money with grep/sed/awk is to become sufficiently l33t in them, you can be gainfully employed by others to "do things" on "the goddam unixes" hey.. we got more unixes here. can somebody grep them out the way?

Best Comment is Best

Re: How to Make Money Using Grep, Sed and Awk

#37

Post by herbst » Wed, Oct 11, 2017, 3:13 PM UTC

yomansat wrote: post #25

I looked up the freelance job in question, it pays $36 for a 215 page publication from the 80s, including checking where automation has failed. A question to those who do freelance work: sites like Upwork often have low budgets, where do you guys find better projects with better compensation?

Haven't worked below 60$/ for fun things and 80$/h for boring things on Upwork without much reputation. Sure I could find a better paid job, but if you can sell yourself as skilled you don't have to be afraid of the averagely low numbers on there

Re: How to Make Money Using Grep, Sed and Awk

#38

Post by nunez » Wed, Oct 11, 2017, 4:33 PM UTC

If author used better variable names, then the code would not have been as hard to look at

Re: How to Make Money Using Grep, Sed and Awk

#39

Post by tslug » Wed, Oct 11, 2017, 5:53 PM UTC

CJefferson wrote: post #14

mar77i wrote: post #9

Dude! You need help with using your tools' features to reduce your pipeline lengths. - grep derp | sed whatever is the same as sed '/derp/ whatever' (you might be interested in sed -r for that matter) - sed whatever | sed whatever\ else is the same as sed -e whatever;whatever\ else This list is by no means complete but saves a lot of external processes already, and seriously: think about writing the whole thing in pu…

I disagree, unless it's slow enough to effect performance, I prefer to combine simple pipes. The only time I have to use more complicated constructions is to get around the stupid problem that passing filenames with pipes is almost impossible to do safely.

I learned when writing bash scripts for cygwin under Windows that Windows doesn't support copy-on-write fork()'s. This means that any new process is extremely expensive to fork, because it copies the memory from the process it was forking from, even if it doesn't need it.

As a result, I did as much as I could using only bash internals, and very rarely did I use pipes, because they always fork()'d a bash subshell in addition to whatever the process was.

In about 7k lines of bash script written to completely automate the iterative development of two games with a shared game engine featuring dockerized server containers, I was able to avoid using sed in almost all of it. When I did use it (pretty sure only two places), it was basically for mass substitution of variables inside of text files, and the multi-line syntax worked very nicely for this, as I could form the sed line with a loop over all the variables I wanted to replace.

It turns out for most bash scripts, you can do the most common sed substitutions using bash's rich variable substitution expressions. For instance, you see in a lot of bash scripts calling commands like "dirname" and "basename" to get the directory and filename of paths. There's a much faster way to do this in bash:

  path=/tmp/my/path/to/stuff.txt
  dir="${path%/*}"
  file="${path##*/}"
  other="${path/stuff/other}"

That dir line means "delete the shortest string that matches /* from the end of the string". That file line means "delete the longest string matching */ from the start of the string." That other line means "replace the word stuff with the word other in this string".

Re: How to Make Money Using Grep, Sed and Awk

#40

Post by tslug » Wed, Oct 11, 2017, 6:00 PM UTC

xelxebar wrote: post #33

jaymzcampbell wrote: post #7

Earlier quoted context omitted.

As soon as I start adding ifs and battling to remember the exact syntax I give up and move to Python and argparse. Recently I've started using the Begins[1] library to make useful command line tools and scripts even more quickly. I appreciate it's less and less portable at this stage but so much more productive. Most of the time these scripts are very specific anyway so it's not a concern. The other thing I've notice…

There is a lot of historical arcana baked into the command line and bash, but I've found it worth learning like any other language. Anyway, just for future reference you can use bash's `help` command to get documentation about builtins. In the case of conditionals, we want to loo at `test`: $ help test I find this a lot easier than wading through the bash man page.

Holy crap, xelxebar. Thank you! This is a game-changer. I've been wading through the bash man pages like a fool.