Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What you need may be “pipeline +Unix commands” only

nanxiao.me

Post reply on HN
Print view View original

11–20 of 181 posts

Re: What you need may be “pipeline +Unix commands” only

#11

Post by Aeolun »

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

You forget that using more developers means more headcount, and more headcount means I have more responsibility as a manager.

These crazy complex solutions also look a lot more difficult on the slides than the simple 3 layer architecture that’s often shown to me.

With something that simple, and requiring so few people, how am I ever going to convince my clients to pay me multiple millions of dollars for it.

Re: What you need may be “pipeline +Unix commands” only

#12

Post by gizmo686 »

gsich wrote: post #3
TicklishTiger wrote: post #2

if your data set can be disposed by an awk script, it should not be called “big data”. Why not? I don't see how awk is limited to a certain amount of data.

If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.

I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework

Re: What you need may be “pipeline +Unix commands” only

#13

Post by nik1aa5 »

I have been using the command line for all type of work for years now. The most satisfying is to realize that there is always more to learn. And once you grasped the basics, they fit together like LEGO bricks.

While I think it's important to make that argument, the posted article and the one it refers to lack some guidance on how to reach "command line mastery". I recently came across this great resource here on HN:

https://github.com/jlevy/the-art-of-command-line

It gives great overview of the toolbox you have on the command line. Equipped with `man` you're ready to optimize your everyday work. And always remember to write everything down and ask yourself WHY something works the way it works. The interface of the standard tools is thought out very well. Getting comfortable with this mindset pays off.

Re: What you need may be “pipeline +Unix commands” only

#14

Post by Aeolun »

gsich wrote: post #3
TicklishTiger wrote: post #2

if your data set can be disposed by an awk script, it should not be called “big data”. Why not? I don't see how awk is limited to a certain amount of data.

If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.

You can still connect a fairly big NAS to a beefy server and do the processing, unless thoughput rate becomes an issue. Saturating a 10Gbit link means you can probably process up to 100TB a day.

Re: What you need may be “pipeline +Unix commands” only

#15

Post by Aeolun »

gizmo686 wrote: post #12
gsich wrote: post #3

Earlier quoted context omitted.

If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.

I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework

Most stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.

Re: What you need may be “pipeline +Unix commands” only

#17

Post by codetrotter »

kurczynski wrote: post #10

I hate to be that guy, but they're NOT "Unix" tools, as the name GNU literally states. The post makes a good point that I fully agree with, just doesn't explain it well enough.

I and many others take “Unix” to mean any UNIX or UNIX-like operating system, whereas “UNIX” with all capital letters means only the ones that are certified UNIX.

Re: What you need may be “pipeline +Unix commands” only

#19

Post by groestl »

Aeolun wrote: post #15
gizmo686 wrote: post #12

Earlier quoted context omitted.

I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework

Most stream processing doesn’t rely too much on RAM, unless you literally need all the data in memory at the same time.

If it _needs_ to be in RAM, then either you got a big enough machine (then by definition it's not Big Data) or it's impossible. If you manage to come by with RAM using smart algorithms, although the full dataset would never fit in RAM, then it's Big Data. So I'd argue, stream processing is Big Data, exactly because it doesn't rely too much on RAM.
Post reply on HN