31–40 of 156 posts
It's good to note that `uniq -u` does remove duplicates, but it doesn't output any instances of a line which has been duplicated. This is probably not clear to a lot of people reading this.
Hate to sound like Steve-Jobs here, but: "You're using it wrong."
Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how your language works. Both things that will prevent you from being hindered next time around when you have to do a similar big-data job.
Perhaps you are more comfortable using pre-defined lego-blocks to build your logic. Perhaps you understand the unix commands better than you do your chosen language. But understand that programming is the same, just in a different conceptual/knowledge space. And remember, always use the right tool for the job!
(I use Unix commands daily as they're quick/dirty in a jiffy, but for complex tasks I am more productive solving the problem in a language I am comfortable in instead of searching through man pages for obscure flags/functionality)
Neat selling points: csvfix eval and csvfix exec
also: the last commit to csvfix was 6 days ago; it's active, mature, and the developer is very responsive. If you can think of a capability that he hasn't yet, tell him and you'll have it in no time:)
For last line, I always did tac [file] | head -n 1 Mainly because I can never remember basic sed commands (Strange, OS X doesn't seem to have tac, but Cygwin does...)
What a great start. Unless he's a biologist, the author means generic, not genetic.
The author goes on to show that he can use command-line utilities to accomplish what database clients do much more easily.
uniq also doesn't deal well with duplicate records that aren't adjacent. You may need to do a sort before using it. sort | uniq But that can screw with your header lines, so be careful there two.
sort -u sort and uniq in one step.
"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how yo…
For performing the tasks outlined by his examples, Unix utilities are easier for the user as well as executing faster than writing your own code in a general purpose programming language, unless one puts in the time to tune the implementation.
One could rebuild AWK in C and get similar performance, but why not just use some extremely simple AWK? And is anybody going to replicate the amazing speed of grep without a huge time investment? [1]
There is a right tool for the job, and having seen dozens of programmers be exposed to new big data sets, I can tell you that the ones who become productive quickly and stay more productive are the ones who adopt whatever tool is best for the job, not the ones that stick to their one favorite programming language. In fact, a good sign of somebody who will quickly fail is someone who says "forget those Unix tools, I'm just going to write this in X".
[1] http://ridiculousfish.com/blog/posts/old-age-and-treachery.h...
"While dealing with big genetic data sets I often got stuck with limitation of programming languages in terms of reading big files." Hate to sound like Steve-Jobs here, but: "You're using it wrong." Let me elaborate. If you're coming across limitations of "too-big" or "too-long" in your language of choice: Then you're just a few searches away from both being enlightened on how to solve your task at hand and on how yo…
For example, working with file lines naively in Ruby means reading the whole lot into a giant array and doing transformations an array at a time, rather than in a streaming fashion.
The shell gives you fairly safe concurrency and streaming for free.
Personally, if it's a complex task, I generally write a tool such that it can be put into a shell pipeline.
Knowing the command line well - so that you don't often have to look up man pages for obscure flags / functionality - has its own rewards, as these commands turn into something you use all the time in the terminal. Rather than spending a few minutes developing a script in an editor, you can incrementally build a pipeline over a few seconds. Doing your script in a REPL is a better approximation, but it's a bit less immediate.
For last line, I always did tac [file] | head -n 1 Mainly because I can never remember basic sed commands (Strange, OS X doesn't seem to have tac, but Cygwin does...)