141–150 of 181 posts
Earlier quoted context omitted.
You know, "billions a day" is only on the order of 10K per second. A single machine can handle that.
That would be amazing, wouldn't it? It's not true though, the problem with dealing with billion of operations a day is the spikes, most of the time you don't get a nice homogene rate for 24 hours straight.
I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…
Also, I'd like to point out that in both this blog post and the Taco Bell one, the "UNIX way" is being compared against a straw man example, not a real example of over-engineering. Neither post actually provides any evidence of any real inefficiency. Both authors are just trying to explain and improve their own thinking about programming, not trying to cast judgement on a generation.
I totally agree that most of the *nix tools are most of the time the best ones, but things get trickier when there is a complex dynamic pipeline e.g. the input conditions the kind of processes and these are also dynamic based on other inputs.
I've been a pipeline junkie for a long time, but i've only recently started to get into awk. The thing i can do with awk but not other tools is to write stateful filters, which accumulate information in associative arrays as they go. For example, if you want to do uniq without sorting the input, that's: awk '{ if (!($0 in seen)) print $0; seen[$0] = 1; }' This works best if the number of unique lines is small, either…
awk '{ if (!($2 in seen)) print $0; seen[$2] = 1; }' You can even shorten this a bit! "awk '!seen[$2]++'" does the same thing -- awk will print the whole line when it's provided a truthy value. It's definitely more code-golfy than being explicit about what's actually going on though
Earlier quoted context omitted.
Literally just a Bash while-read loop over community IDs. It's embarrassingly trivial. I'm planning on posting the data, probably to https://social.antefriguserat.de/ and will include procssing scripts. This is the fetch-script, which saves both the HTML and HEAD responses: #!/bin/bash sample_file=$1 comm_path='community-pages' base_url='https://plus.google.com/communities' i=0 time sed -e 's,^.*/,,' $sample_file | w…
This seems to be a perfect use case for GNU Parallel[0] to download and process, say 10 ids, in parallel. If you have already downloaded/processed and have no need to do it again, then probably doesn't matter now. [0]: https://www.gnu.org/software/parallel/
That's why the script echoed the curl commant rather than run it directly. It fed xargs.
The other problem was failed or errored (non 3xx/4xx, or incomplete HTML -- no "" tag found) responses. There was no runtime detection of these. Instead, I checked for those on completion of the first run and re-pulled those in a few minutes, a few thousand from the whole run, most of which ended up being 4xx/3xx ultimately.
Earlier quoted context omitted.
If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.
I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework
Earlier quoted context omitted.
Python is fantastic for little (or large!) bits of logic, but its handling of input is clunky enough to put me off for tiny things. AFAIK the boilerplate you need to get to working on the fields on each line is: import sys for line in sys.stdin: fields = line.split() # now you can do your logic If you want to use regular expressions, that's another import. Python also doesn't play well with others in a pipeline. You…
This is exactly where perl (namely, perl -ne) is so very, very useful.
https://github.com/thisredone/rb
Probably not as fast as many of the individual unix tools it could replace, but does look like a great way to leverage one's knowledge of ruby.
I really agree with aspects of this, and I think CLIs and Unix pipes are way more powerful than we treat them, but be forewarned that there are problems with doing everything with pipes. You need to code more defensively with them. For example, it is rare, but every so often a newline will be fail to be emitted. kinda\n likethis\n \n example\n There are many other gotchas, but that one is a doozy because if you're us…
This is one of the reasons I prefer PowerShell, it requires a lot fewer text parsing shenanigans. UNIX tools simply failed to evolve. Single io stream pipelining on raw ASCII was perfectly reasonable in the 1970s but it isn't the 1970s anymore. We should be composing tools with multiple typed io stream paths in GUIs (or TUIs I suppose), leveraging two or even three dimensional layouts. All our interfaces should be co…
Earlier quoted context omitted.
My favorite term is "annoying sized" data, not enough to warrant clusters and HPC, but enough to make a decent laptop crawl to a halt. It's that uncomfortable in-between that makes up the bulk of the data I usually encounter.
The one nice thing about AWS and its ilk is the ability to spin up big chunky VM for few hours/days for ad-hoc data processing for pretty small amount of money. That can in many cases shift the boundary of annoying sized enough, with the added bonus that your own workstation is not fully occupied by the processing. Of course such approach is not applicable to all scenarios, but it is a useful trick to keep in your sl…
Does anyone have suggestions on books to grow my scripting fu? (End of chapter exercises tend to be useful for me) I know bash, and know a lot of basic commands, but I'm not familiar with some more advanced things. I don't know awk or sed for example.
So I end up writing a lot of bash scripts. A lot of small python scripts. Stupid scripts. Over time, you build up a library of tricks. You read threads like this where you pick up new tricks. It isn't something you'll learn once. Some of the tools take years to really get the feel of.