Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Awk in 20 Minutes (2015)

ferd.ca

Post reply on HN
Print view View original

71–80 of 128 posts

Re: Awk in 20 Minutes (2015)

#71

Post by eternauta3k »

mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

I'm tending towards Python for anything more than a single line shell script, because I find it easier to keep Python scripts tidy and well-structured. Not saying this is impossible, but I find the bash syntax for nontrivial operations hard to learn. This probably also applies if you replace bash with awk and python with perl.

Re: Awk in 20 Minutes (2015)

#72

Post by mongol »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

Analysis of large log files. Granted, today Splunk and other tools are more used.

Re: Awk in 20 Minutes (2015)

#73

Post by xorcist »

ggm wrote: post #11
bcyn wrote: post #10

So, I don't use awk right now. This answers "how to awk" more than "why use awk" for me (understandable given the 20 min claim). Does anyone have concrete, practical examples of use cases where awk made your life much easier?

Awk '{ print $2 }' does LWSP gobbling.. cut -d' ' -f2 doesn't and this difference alone makes awk useful to me on a daily basis. Awk has a hash like perl which is very efficient. I use an awk expression to print uniq as they come in counted through the hash insert on new instead of uniq which prints at end. Awk count unique over 300,000,000 ips was as fast as perl and python and smaller memory footprint

For the simplest uses, cut is more readable.

If argument expansion is required, consider using read. That way the argument is given a name. Use "read a b c Just remember to pipe to read with care, as the right hand part of a pipe is a subshell in which variables are local. So "read a b c Another way to expand arguments is to simply define a shell function and use $2. Something like "process_line() { echo $2 }".

When you have to reach for something like awk, your script would probably improve by being mostly awk. In which case most people are probably looking at perl or python anyway. Even if awk is a nice language, the arrival of perl mostly killed it. It is not wrong to say perl was the next version of awk.

Re: Awk in 20 Minutes (2015)

#74

Post by xorcist »

j_z_reeves wrote: post #46

nice, I finally took the time to read the man pages for awk. And whipped out a script to count the number of errors occurred for a particular day for a postgres log file. cat logfile | awk '/ERROR:/ {counts[$1] = counts[$1] + 1}; END { for (day in counts) print day " : " counts[day]}' | sort I just needed to know how awk programs are structured, the rest is just simple programming! EDIT: I'm not sure if it's actually…

Apart from the useless use of cat, since sort does the work here something like the following would probably suffice:

grep ERROR logfile | cut -f 1 -d ' ' | sort | uniq -c

Re: Awk in 20 Minutes (2015)

#75

Post by Timpy »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

I'm sure use cases vary to some degree, but awk clicked for me when I had some data that wasn't formatted well. I didn't use it in a script, I just worked in the shell and kept iterating until it looked good, then redirected it into a text file. I use it for quick one-offs like this all the time now.

I've also used awk to get button presses from an input stream on a MIDI controller. For me, I found that the up front cost of learning a few awk commands quickly made it's return on investment.

Re: Awk in 20 Minutes (2015)

#76

Post by saberience »

eternauta3k wrote: post #70
saberience wrote: post #69

Earlier quoted context omitted.

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

> I've managed my whole career with ctrl-f, and highlight-all matches. Imagine you have to do this for 100 files.

Fair :)

Re: Awk in 20 Minutes (2015)

#77

Post by Wilem82 »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

Using any kind of management position as justification for anything tech-related won't have the effect you're probably aiming for. Dev going management is always downshifting.

Re: Awk in 20 Minutes (2015)

#78

Post by submeta »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

There are hundreds of use-cases. It is not important what the use-cases are. - There are two types of developers: Those who use hacking to solve all kinds of tasks that need hundreds of manual steps. And those who never think of automating smaller steps. The second kind of devs can be very good software engineers, but they prefer IDEs instead of tweaking Vim or Emacs. The first kind of dev will look for possiblities to automate repeated steps, opportunities for tweaking, transforming, hacking. For the fun of it.

Re: Awk in 20 Minutes (2015)

#79

Post by indymike »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

Well, you might have have got the job done faster with awk (and other tools like it)...

ETL and anything dealing with importing large data sets come to mind.

Re: Awk in 20 Minutes (2015)

#80

Post by scruple »

saberience wrote: post #69
mauvehaus wrote: post #21

I learned perl in college in the 5.6 days, and did a lot of text processing with it for a time. At some point, I bit the bullet and learned awk, and I've mostly abandoned perl as a result. Why? awk is small enough that it fits in my head, or at least the bits I need every couple of months do. And if I forget, it only takes 20 minutes to put them back in. perl, by contrast, is far too large to fit in my head and comes…

What are you using Perl/awk for that it's needed so often? I've been a software engineer since 2005 and worked my way up to being a VP of Engineering currently and never had to use either Perl or awk (or similar). I often read about these tools on Hackernews and I find it quite mystifying as I manage to have written Java, Scala, C#, SQL, and so on for 15 years and happily never needed them. Is this a certain kind of…

Are you working in a Linux development environment? Targeting Linux systems? You're not going to come across much of a use for awk if you're in the Windows world. I'm making an assumption based on C#.

I've used perl and awk extensively for all sorts of things. Log parsing embedded device logs to generate reports was a big one for me, that's where I really learned both languages. And they are superb tools for that task. We also used awk quite a bit for configuration parsing as a sort of intermediary between different processes and scripts on those devices.

As I moved into web development, I've found myself using both much less. I haven't touched perl more than once in the last 6 years -- I was given a script to maintain recently but it only needed an hour or two of my time to add a small feature to. That perl script is part of a legacy build system. I still use awk every couple of months but it's just part of some pipeline on the command line.

Post reply on HN