Live data from Hacker News

Dt: Duck tape for your Unix pipes

dt.plumbing

81–90 of 162 posts

Re: Dt: Duck tape for your Unix pipes

#81

Trivia: "DT" is Japanese slang for "dōtei" ("童貞", means "virgin"). Names are hard.

There's a fairly successful version control system whose name is an insult in British English.

And wait until you hear what the most popular open source alternative to Photoshop is called...

Re: Dt: Duck tape for your Unix pipes

#82
post #69

Earlier quoted context omitted.

That's different. Apple as a trademark for a computer (or for a record company) is fine. Apple as a trademark for apples... not so.

Just that the "difference" stops when they now sue any grocery related company having anything like an apple in their logos..

Anyone can sue anyone else over anything. If those grocery companies actually fought them in court instead of caving in, Apple would get their ass handed to them.

Re: Dt: Duck tape for your Unix pipes

#83
post #16

Nushell has filled this void for me. It elevates pipes by structuring them into lists or tables. It has all sorts of goodies for manipulating those data structures like inserting/updating rows and cells, better string interpolation, useful utility functions like case transformation, and more.

That's why I use powershell

Re: Dt: Duck tape for your Unix pipes

#84

In my profressional life I have cause to work with Powershell a great deal, and I love the object oriented pipelines that it has. Being able to filter, sort, or transform properties of the objects passed over makes it much more straightforward to work with than bash et al.

I’m with you here. Once you get the hang of it you even miss the most awkward pipelines once they’re gone and you’re just passing around single dimensional strings. The memory overhead with sets of objects and their properties vs vanilla strings is significant though and easy to bump up against when working with large data sets. Best to try to keep all of the processing for that dataset in a single pipeline if you ca…

Personally I only use shell scripting for simple tasks, not anything that would run into a memory limit. If it requires more heavy lifting, I'll write it in a programming language like C#. Nothing we do in production relies on shell scripts, they exist only as shortcuts for our workflow

Re: Dt: Duck tape for your Unix pipes

#85
post #47

Earlier quoted context omitted.

Null separators between fields is structured data. It might just be the simplest that can possibly work (because the null character is outside the normal data range for text, unlike the newline, which demonstrably doesn't work in that sense) but it is structure. A list structure, specifically. The world would be a much happier place if -print0 (or whatever) was universal, but no such luck.

Sure, but the reason it works with any tool is because it's generic and simple. If each tool had to implement anything more sophisticated like a JSON parser and serializer, it would be a nightmare to maintain. Projects like Nushell essentially need to handle every type of output from any command, and every type of input to any command, which is an absurd amount of work, and just not scalable. Subtle changes in this s…

Murex (https://GitHub.com/lmorg/murex) doesn’t replace coreutils with builtins but manages interop with commands just fine.

Most output is relatively easy to parse, sometimes you need to annotate the pipe with what format to expect but that’s easy enough to do. And Murex does come with builtins that cover some of the more common coreutils use cases for instances when you want greater assurances of the quality of the data - but those are named differently to their coreutil counterparts to avoid confusion.

Re: Dt: Duck tape for your Unix pipes

#86

Earlier quoted context omitted.

I’m with you here. Once you get the hang of it you even miss the most awkward pipelines once they’re gone and you’re just passing around single dimensional strings. The memory overhead with sets of objects and their properties vs vanilla strings is significant though and easy to bump up against when working with large data sets. Best to try to keep all of the processing for that dataset in a single pipeline if you ca…

Personally I only use shell scripting for simple tasks, not anything that would run into a memory limit. If it requires more heavy lifting, I'll write it in a programming language like C#. Nothing we do in production relies on shell scripts, they exist only as shortcuts for our workflow

I think that's a good rule of thumb but not universally applicable. Powershell is used a lot with Microsoft sysadmin/identity/mail software and other vendor software used to manage or integrate with said MS software. If you're processing tens of thousands of AD objects you can easily hit memory issues. You could have your staff learn a few best practices and continue using the wealth of existing tooling/knowledge that exists for Powershell or write a bunch of custom .net code for every simple integration or job they want to run to process the objects in their directory.

Re: Dt: Duck tape for your Unix pipes

#87

Earlier quoted context omitted.

I’m with you here. Once you get the hang of it you even miss the most awkward pipelines once they’re gone and you’re just passing around single dimensional strings. The memory overhead with sets of objects and their properties vs vanilla strings is significant though and easy to bump up against when working with large data sets. Best to try to keep all of the processing for that dataset in a single pipeline if you ca…

Yeah the memory usage of the pipeline is real. Often using foreach instead to save memory is needed. Or good old fashioned array chunking.

Exactly. If you really have to you can take advantage of its ability to use .net libraries directly and force immediate cleanup/gc of variables but it's super hacky and always found pipelining to work better once you get the hang of processing data in flight.

Re: Dt: Duck tape for your Unix pipes

#88
post #85
post #47

Earlier quoted context omitted.

Sure, but the reason it works with any tool is because it's generic and simple. If each tool had to implement anything more sophisticated like a JSON parser and serializer, it would be a nightmare to maintain. Projects like Nushell essentially need to handle every type of output from any command, and every type of input to any command, which is an absurd amount of work, and just not scalable. Subtle changes in this s…

Murex ( https://GitHub.com/lmorg/murex ) doesn’t replace coreutils with builtins but manages interop with commands just fine. Most output is relatively easy to parse, sometimes you need to annotate the pipe with what format to expect but that’s easy enough to do. And Murex does come with builtins that cover some of the more common coreutils use cases for instances when you want greater assurances of the quality of th…

That's the first time I'm hearing about this project (which I take you are the creator of?). Very interesting!

How would you say Murex compares to Nushell? The syntax seems vaguely similar. Are there any fundamental differences?

Re: Dt: Duck tape for your Unix pipes

#89
post #38

Earlier quoted context omitted.

What an epic failure by the trademark office. That's as if a company had tried to trademark "Hammer" as a brand name for hammers, and the trademark office just said "Yeah, sure!"

Or like "Apple" as a brand name for... nevermind

Not to forget "Mail" for an email program. Made searching so much easier ... Not.

And never mind the dog named dog (a long time ago, some IEEE journal had a great cartoon where someone painted "house" on their house, "car" on their car, ... you get it. The cartoon illustrated an article about good and bad names for systems.

Re: Dt: Duck tape for your Unix pipes

#90
post #82

Earlier quoted context omitted.

Just that the "difference" stops when they now sue any grocery related company having anything like an apple in their logos..

Anyone can sue anyone else over anything. If those grocery companies actually fought them in court instead of caving in, Apple would get their ass handed to them.

I welcome you to try sueing your neighbor for murdering you.

Courts do not accept all lawsuits.

Post reply on HN