Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Learn Awk? (2016)

blog.jpalardy.com

Post reply on HN
Print view View original

171–180 of 246 posts

Re: Why Learn Awk? (2016)

#171

Post by macintux »

mekazu wrote: post #165
macintux wrote: post #156

Earlier quoted context omitted.

Perl was my second programming love, but awk is much shorter and easier to remember for the simple cases where I need it. Remembering which Perl command-line arguments simulate awk’s line-by-line processing is harder than just remembering awk.

I use Perl similarly to awk if I need to use regex rather than white space delimited fields. I think if you know Perl really well and can remember the command line arguments - particularly -E, -n, -I and -p - then it’s a good swap in substitute for grep, sed, awk, cut, bash, etc when whatever 5 min task you’re working on gets a tiny bit more complex. Similarly a decent version perl 5 seems to be installed everywhere…

I would say Perl’s native support for regular expressions makes it more useful on the CLI than Python, but Python is also very low on my preferred languages list.

I do, however, use it for JSON pretty printing in a pipeline: python -mjson.tool IIRC.

Re: Why Learn Awk? (2016)

#172

Post by chongli »

nextos wrote: post #146
bloopernova wrote: post #83

I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into. I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job. For instance, "mail…

I disagree, it's quite elegant if you think in terms of relational algebra operators: * Projection (Π): awk and cut for simple cases * Selection (σ): grep for simple cases, otherwise sed & awk * Rename (ρ): sed * Set operators: join, comm...

Bravo! This is one of the most insightful comments I've read in a long time! I have been using some of these tools for years but I never thought of describing them this way. Now I can think of writing a complex query in relational algebra and translating it into these commands in a very natural way.

Re: Why Learn Awk? (2016)

#173

Post by zoom6628 »

jerf wrote: post #73
Scarbutt wrote: post #18

Earlier quoted context omitted.

And awk doesn't offers cut's column range selection ;)

Absolutely. Everything comes with costs & benefits. But I'm not sure I've, in my entire 23-year professional programming career, ever encountered a fixed-width text format in the wild. I've used cut even so for places where by coincidence the first couple of columns happened to be the same size, but that's really a hack. Obviously, other people have different experiences which is why I quality it so. (I only narrowly…

Some topics where you will most definitely come across fixed-width formats: - processing older system printouts that are save to text file - banking industry formats for payment files and statements - industrial machine control

and my favourite....... - source code.

My first intro to awk was using it to process COBOL code to find redundant copy libs, consolidate code, and generally cleanup code (very very crude linting). And it was brilliant. Fast, logical, readable, reliable - was everything i needed.

It is also an eminently suitable language for teaching programming because it introduces the basic concept of SETUP - LOOP - END . which is exactly the same as one will find in most business systems, you find it in arduino sketches, hell you even find it in a browser which is basically just a whole universe of stuff sitting atop a very fast loop that looks for events.

AWK fan for sure - my heirachy of languages these days would be cmd line where there is specific command doing all i need, AWK for anything needing to slice and dice records that dont contain blobs, python for complete programs, and python+nuitka or lazarus or C# when need speed and/or cross platform.

Re: Why Learn Awk? (2016)

#174

Post by hnick »

F-0X wrote: post #114
jerf wrote: post #73

Earlier quoted context omitted.

Absolutely. Everything comes with costs & benefits. But I'm not sure I've, in my entire 23-year professional programming career, ever encountered a fixed-width text format in the wild. I've used cut even so for places where by coincidence the first couple of columns happened to be the same size, but that's really a hack. Obviously, other people have different experiences which is why I quality it so. (I only narrowly…

Within my first year of professional development, I encountered several fixed-width files I needed to read and write. I suppose exposure depends a lot on the specific industry.

Also big mainframe users (banks, insurance) often send fixed width data to us.

Re: Why Learn Awk? (2016)

#175

Post by nickpsecurity »

samatman wrote: post #25

> Imagine programming without regular expressions. I live in this future and it's beautiful. Steps to programming without regular expressions: 1) find a PEG library for your language of choice There is no step 2.

Quite a few alternatives to RE's I put in this answer:

https://softwareengineering.stackexchange.com/questions/1949...

Re: Why Learn Awk? (2016)

#176

Post by hnick »

umvi wrote: post #23
rb808 wrote: post #16

Earlier quoted context omitted.

scripts are the antithesis of modern software development. No Unit testing, No CI, no monitoring, no consistency, often no source control. Appeals to me as a hacker but only my scripts are intelligible - which is a bad sign. :)

I'm not actually convinced unit testing is all that valuable, unless the unit under test has a very clear input -> output transformation (like algorithms, string utilities, etc). If it doesn't (and most units don't), unit tests just encumber you.

Yes I worked in a print production industry and unit testing was almost impossible. To generate a string of text reliably and correctly is not difficult, but to position it on the page respecting other elements, flow, overlap, current branded font and size, and white space - unit testing is basically useless. And these are the errors we faced most.

Re: Why Learn Awk? (2016)

#177

Post by hnick »

berkes wrote: post #72

Last week I threw out AWK and replaced it with Ruby (Could've been Python, Perl or PHP even). Because AWK is not suited for CSV. Please prove me wrong! I had to parse 9million lines. Some of which contain "quoted records", others, same column, are unquoted. Some contain comma's, in the fields, most don't. CSV is like that: more like a guideline than actual sense. Two hours of googling and hacking later, I gave up and…

CSV is not really a standard, various programs have their own interpretation (even MS Excel allows a header line before the data that contain specifications for the file, such as delimiter, which you probably won't account for in a home-grown solution).

So use a dedicated tool or library or you will run into trouble one day.

For one off jobs, I open it in Excel or similar and save as tab or pipe delimited text. This usually plays much more nicely with command line utilities, assuming it didn't mangle any numbers.

Re: Why Learn Awk? (2016)

#178

Post by jolmg »

emmelaich wrote: post #153
emmelaich wrote: post #147

Earlier quoted context omitted.

To pile on :-) you often want -w (match word) flag to grep. In awk, I couldn't find how to do this. I tried /\bfoo\b/ and /\ / but neither worked. I don't know why and don't care enough which brings me to my major awk irritation ... It doesn't use extended or perl REs, which makes it quite different to ruby, perl, python, java. Now, according to the man page it does ; at least on OSX (man re_format) but as mentioned…

UGH! Found the problem; it simply doesn't work. Assuming the OSX awk is the same as the freebsd awk there is a very old open bug on this: awk(1) does not support word-boundary metacharacters https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=171725

\ work with GNU's awk:

  $ printf "fishstick\nfish\ngoldfish\n" | awk '/\/' 
  fish

Re: Why Learn Awk? (2016)

#179

Post by samstave »

darau1 wrote: post #92
bloopernova wrote: post #83

I use awk because there's an almost 100% chance that it's going to be installed on any unix system I can ssh into. I use awk because I like to visually refine my output incrementally. By combining awk with multiple other basic unix commands and pipes, I can get the data that I want out of the data I have. I'm not writing unit tests or perfect code, I'm using rough tools to do a quick one-off job. For instance, "mail…

> it's not elegant I completely disagree.

Eloquently put!

Re: Why Learn Awk? (2016)

#180

Post by LilBytes »

jiggawatts wrote: post #169
rectang wrote: post #10

Awk, like shell, is a fraught programming environment, full of silent failure and hidden gotchas. Even for one-liners. I use shell because I have to, not because I like it. I dread maintaining shell scripts which have a bunch of awk and sed in them. The Unix ideal of small single-purpose tools and text processing is separable from these old warhorses.

PowerShell is a lot better at this. It was designed for reading by humans, not for saving precious bytes over 300-baud terminal connections. So for example, selecting columns is easy, and can be done by name. Here's a useful little snippet demonstrating converting and processing the CSV output of the legacy "whoami" Windows command. It lists the groups a user is a member of without poking domain controllers using LDA…

PowerShell Core (PS5/open source) is crushing it in this space over Bash\AWK specifically.

I'm a pretty advocate for how it works in these use cases now.

Generally I still use Bash\Sed\AWK for the one off instances when ever I need something done there and then. But if I'm writing something that's going to be involved in my CI\CD with a high chance of needing to be maintained. I'll nearly always write it in PowerShell.

Post reply on HN