51–60 of 181 posts
I use an awk interpreter called mawk.[1] This is noticeably faster than gawk or other standard variants. [1] https://invisible-island.net/mawk/
Try multiple interpreters with timings.
Gawk's profiler can be invaluable.
My favorite thing has been `| ruby -e "puts STDIN.to_a. ..."`, allows to run any kind of code on the standard input, much easier than remembering awk/sed various options and much more powerful. edit: same thing can be done with Python/Perl
Yes, easier if you know ruby.. sed/awk is good for munching strings in small scripts and it is also always installed on a unix system (AFAIK).
Good enough programming is something you code that provides something people want -- and you never look at the code again. Find the problem, solve the problem, walk away from the problem. That's not sexy. It's not going to get you an article to write for a famous magazine, but it's good enough.
We have lost sight of "good enough" in programming, and without some kind of guardrails, we end up doing stuff we like or stuff that sounds good to other programmers. For instance, while I love cloud computing, I'm seeing "how-to" articles written about setting up a VPC for doing something like playing checkers. Yes, it was an oversimplified article, and you have to write that way, but without wisdom, how is the reader supposed to know that? What criteria do they use to determine whether it's a co-lo server, a lambda, or a world-wide distributed cloud?
We're going like gangbusters selling programmers and companies on all kinds of new and complex ways of doing things. They like it. We like it. But it is in anybody's best interest over the long run?
Recently I rewrote a pet project for the third time. First time it was C#, SQL Server, and an ORM. Then it was F#, MySQL, and linux. The last time it was pure FP in F# and microservices.
Some of you may know where this is going.
Just as I finished writing the app in a real microservices format, I realized. Holy cow! This whole thing was just a few Unix commands and some pipes.
My thinking went from all kinds of concerns about transactions and ORM-fun to just some nix stuff in a small script. The problem stayed the same.
Something else happened too. At each step, I did less and less maintenance. The last rewrite has had no maintenance required at all. In my spare time, I'm going to do the nix one using no servers at all on a static SPA. In a very meaningful way, there's no app, there's no server, and there's nothing to maintain. Yet I still get the functionality I need. And I never maintain it.
Of course that's not possible for every app, but the key thing I learned wasn't the magic of serverless static SPAs or the joys of unix. It was that I didn't know whether or not it was possible or not until I did it. By thinking in a pure FP fashion and deploying in true microservices, the rest just "fell out" of the work. At first I was actually thinking in a way that would have only led to more and more complexity and maintenance requirements.
My belief is that we get our thinking right first, use code budgets, and try for a simple unix solution. If it doesn't work, why? At least then we've made an effort to be good enough programmers. That beats most everybody else.
This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are…
80% of bioinformatics.
Case in point from my own recent work: I've been analysing characteristics of Google+ Communities, mostly looking for plausibly active good-faith instances. There are 8.1 million communities in total, and thanks to some friendly assistance, I'd identified slihtly more than 100,000 with both 100 or more members, and visible activity within the preceeding 31 days, as of early 2019. The task of Web scraping those 100k c…
Earlier quoted context omitted.
If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.
One time I met a company who insisted they were sending tens of TB of data per day and would need multi-PB per year storage compressed. Took one look at the data: All json, all GUIDS and bools. If we just pre-parse it, the entire dataset for a year fits in a few 100s GB uncompressed -- literally could fit on a macbook air for most of the year. The funny thing about "big data" in my experience, is just how small it ac…
Add in images and the rest of the Web payload (800 KiB per page), and that swells to Petabyte range. But the actual scale of the user-entered text is stunningly small.
I agree with the sentiment that many solutions are over-engineered, but when you need to process billions of records a day, you do need more complex systems. Bottom line: when facing an engineering problem, start with the simplest, fastest to implement solution, and build complexity as necessary. The simple solution suffices most of the time.
A good GPU can do >1bn calculations per frame.
I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…
Things fade and shine in succession. Good bits will always come back. It's a bit like the saying about mathematics truthiness nature: doesn't matter who or when you look, they will re-emerge as is. composing tiny bits is always good, whether it's unix commands, lisp functions, or forth words.. Societies are large and full of random fluxes and waves.. right now it might be the time for Wirth 17 pages long solutions ..…
It's not so much about "tiny" but about no fuss, low complexity, no-nonsense, etc. For example, Forth may be tiny, but it's unusable for most applications. Assembler languages are tiny, too.
For my next book, I'm working through a concept I call "good enough programming. Good enough programming is something you code that provides something people want -- and you never look at the code again. Find the problem, solve the problem, walk away from the problem. That's not sexy. It's not going to get you an article to write for a famous magazine, but it's good enough. We have lost sight of "good enough" in prog…
Edit: Can I can sign up somewhere to get a heads up when your book is available? Would be appreciated!
Earlier quoted context omitted.
Medium data. Substantial data. Just-enough-data.
My favorite term is "annoying sized" data, not enough to warrant clusters and HPC, but enough to make a decent laptop crawl to a halt. It's that uncomfortable in-between that makes up the bulk of the data I usually encounter.