What you need may be “pipeline +Unix commands” only • Hacker News Board

Re: What you need may be “pipeline +Unix commands” only

#21

Post by gsich » Sun, Mar 03, 2019, 10:21 AM UTC

gizmo686 wrote: post #12

gsich wrote: post #3

Earlier quoted context omitted.

If your data fits on a single harddrive it's not big data. So I would set the current limit to at least 14 TB.

I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework

That is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.

Re: What you need may be “pipeline +Unix commands” only

#22

Post by johncolanduoni » Sun, Mar 03, 2019, 10:22 AM UTC

ambrop7 wrote: post #16

Unless your data contains spaces, tabs, or, god forbid, newlines. Unix pipeline tools lack any sort of useful data structuring capabilities, making them appropriate for one-off tasks at most.

Yeah I feel like there’s a pretty broad medium between Hadoop and awk. I generally find a python script to be a much clearer solution than an dense line of awk; the latter tends to turn into clever code golf pretty quickly.

Re: What you need may be “pipeline +Unix commands” only

#23

Post by atoav » Sun, Mar 03, 2019, 10:23 AM UTC

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

As a young programmer I certainly found some love for all these unix tools and learn new things every day.

I think the problem is scale. Back in the day (before I was born), very few people were programmers and the resources they could use were limited. This means they didn’t need insanely complex solutions because they already needed complex solutions just to make it work on the limited hardware. People were trying to solve problems with computers. Nowaday you take a problem that could be solved by an microcontroller with three buttons and make it a cloud app with web server, web interface and all kind of other things like containers.

We donlt really tend to ask the question what a good solution would look like. Often it is the case that you just use the technology the developer wants to learn

Re: What you need may be “pipeline +Unix commands” only

#24

Post by aboutruby » Sun, Mar 03, 2019, 10:29 AM UTC

My favorite thing has been `| ruby -e "puts STDIN.to_a. ..."`, allows to run any kind of code on the standard input, much easier than remembering awk/sed various options and much more powerful.

edit: same thing can be done with Python/Perl

Re: What you need may be “pipeline +Unix commands” only

#25

Post by craigsmansion » Sun, Mar 03, 2019, 10:29 AM UTC

rofo1 wrote: post #6

I feel like the art of UNIX is slowly fading into oblivion, especially with the new generation of programmers/developers. Eventually, they'll become the ones that decide the fate of software engineers (by being hiring managers, etc.) and we'll see more and more monstrosity like the article portraits, instead of cleverly using UNIX tools where applicable. There's so many things that the software world is doing wrong t…

> I feel like the art of UNIX is slowly fading into oblivion

I think that by itself isn't a problem, but fading right along with it is the capacity to decompose and structure the problem domain.

Even if one ends up writing a solution in a different language for whatever reasons, starting out by mapping the problem with UNIX command line tools will result in a better understanding of the problem; an understanding that is language agnostic and can be transferred to any preferred method of implementation.

Re: What you need may be “pipeline +Unix commands” only

#26

Post by mrighele » Sun, Mar 03, 2019, 10:38 AM UTC

kurczynski wrote: post #10

I hate to be that guy, but they're NOT "Unix" tools, as the name GNU literally states. The post makes a good point that I fully agree with, just doesn't explain it well enough.

Many of the GNU tools are reimplementation of already existing tools.

For example the initial implementation of AWK was in 1977 [1], a few years before GNU even existed [2], so it _is_ a Unix tool.

[1] https://en.wikipedia.org/wiki/AWK#History [2] https://en.wikipedia.org/wiki/GNU#History

Re: What you need may be “pipeline +Unix commands” only

#27

Post by taejavu » Sun, Mar 03, 2019, 10:38 AM UTC

kurczynski wrote: post #10

I hate to be that guy, but they're NOT "Unix" tools, as the name GNU literally states. The post makes a good point that I fully agree with, just doesn't explain it well enough.

awk predates GNU by at least 7 years. It is most certainly a UNIX tool.

Re: What you need may be “pipeline +Unix commands” only

#28

Post by groestl » Sun, Mar 03, 2019, 10:38 AM UTC

gsich wrote: post #21

gizmo686 wrote: post #12

Earlier quoted context omitted.

I thought the boundary point was RAM. It is relativly simple to work with data across multiple drives. When you pass the boundary of being able to work in a single systems RAM, you genneally need a more significant rework

That is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.

I think the limit is more in the double digits TB range right now.

Re: What you need may be “pipeline +Unix commands” only

#29

Post by gsich » Sun, Mar 03, 2019, 10:42 AM UTC

groestl wrote: post #28

gsich wrote: post #21

Earlier quoted context omitted.

That is harder to define. Server mainboards can hold more RAM then consumer mainboards. So with 32GB per slot and 4 slots I would set a limit to 128GB? Also this would make so much more tasks "big data". Games with 50+GB are not big data, neither is e.g video conversion.

I think the limit is more in the double digits TB range right now.

See? As I said this not easily definable. A single HDD is though. What you could do is a single RAM module.

Re: What you need may be “pipeline +Unix commands” only

#30

Post by speedplane » Sun, Mar 03, 2019, 10:43 AM UTC

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs.

Unix commands are great up to a few GBs of data, Excel is even better if you're dealing with less than a few tens of MBs. But to deal with Terabytes of data quickly and efficiently, these tools totally break down.