Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Dt: Duck tape for your Unix pipes

dt.plumbing

Post reply on HN
Print view View original

101–110 of 162 posts

Re: Dt: Duck tape for your Unix pipes

#101

Post by shortrounddev2 »

brazzledazzle wrote: post #86
shortrounddev2 wrote: post #84

Earlier quoted context omitted.

Personally I only use shell scripting for simple tasks, not anything that would run into a memory limit. If it requires more heavy lifting, I'll write it in a programming language like C#. Nothing we do in production relies on shell scripts, they exist only as shortcuts for our workflow

I think that's a good rule of thumb but not universally applicable. Powershell is used a lot with Microsoft sysadmin/identity/mail software and other vendor software used to manage or integrate with said MS software. If you're processing tens of thousands of AD objects you can easily hit memory issues. You could have your staff learn a few best practices and continue using the wealth of existing tooling/knowledge tha…

Personally we use Linux boxes for everything remote, and I just develop on Windows locally

Re: Dt: Duck tape for your Unix pipes

#102

Post by p-e-w »

imiric wrote: post #99
p-e-w wrote: post #77

Earlier quoted context omitted.

> Projects like Nushell essentially need to handle every type of output from any command, and every type of input to any command, which is an absurd amount of work, and just not scalable. I think you're misunderstanding how Nushell works. They don't parse outputs, or generate inputs, from/for standard Unix commands. Instead, they implement their own commands with the same names as standard commands, and generate/cons…

You're right, I misunderstood the way it worked. But I'm not sure that approach is better. They either need to maintain full compatibility with existing tools, or users need to learn the idiosyncrasies of Nushell tools. And commands not reimplemented by Nushell wouldn't work, or they would need some kind of generic wrapper, which would have the drawbacks I mentioned. But, hey, this obviously has users who prefer it,…

> They either need to maintain full compatibility with existing tools, or users need to learn the idiosyncrasies of Nushell tools.

The existing tools aren't fully compatible with each other either. There are significant differences between GNU and BSD tools, for example, and yet more differences with BusyBox and others. The idea of "standard" tools is unfortunately an illusion, so not much is lost there.

But more importantly, most of the many options the traditional tools offer are related to output selection and formatting. In Nushell, those problems are solved in a unified way by piping to builtin commands that work with structured data. So instead of learning twenty different cryptic flags for `ls`, you just learn three or four postprocessing commands, and use them for `ls` and everything else.

Re: Dt: Duck tape for your Unix pipes

#103

Post by MisterTea »

p-e-w wrote: post #36
imiric wrote: post #30

Earlier quoted context omitted.

The Unix philosophy doesn't dictate any specific data format to interoperate between commands, besides this being data streams, and typically text. Commands are free to interpret these streams however they want, which is why this seems like "hodgepodge", but in no way were tools breaking any sort of promise. FWIW I've been using *nix machines for decades, and very rarely have I had the need to use structured data bet…

> but in no way were tools breaking any sort of promise The Unix philosophy as usually stated includes the rule "write programs that work together". Most Unix commands don't work together in any meaningful sense. For example, `ps | kill` will not terminate all processes, as one might naively expect. I'd say this is a clear violation of both the letter and the spirit of the Unix philosophy.

> For example, `ps | kill` will not terminate all processes, as one might naively expect.

Who expects it to behave like that? Quite the assumption.

Expecting automagic inference based on typing is building a system based on assumption. Have fun making breaking changes to a foundational component that every command will rely on. Everyone will assume you want typed data and if you don't accept/generate typed data then what? Will every shell program have to be burdened with this nonsense and require a rewrite?

In the end ps does exactly what it was meant to do: print information on processes. Your assumption is missing the step where you extract the necessary information you want, e.g. the pid. That is where unix philosophy comes in. Your assumption also seems to make foot guns easier as ps|kill should not be that easy.

Re: Dt: Duck tape for your Unix pipes

#104

Post by imiric »

hnlmorg wrote: post #85
imiric wrote: post #47

Earlier quoted context omitted.

Sure, but the reason it works with any tool is because it's generic and simple. If each tool had to implement anything more sophisticated like a JSON parser and serializer, it would be a nightmare to maintain. Projects like Nushell essentially need to handle every type of output from any command, and every type of input to any command, which is an absurd amount of work, and just not scalable. Subtle changes in this s…

Murex ( https://GitHub.com/lmorg/murex ) doesn’t replace coreutils with builtins but manages interop with commands just fine. Most output is relatively easy to parse, sometimes you need to annotate the pipe with what format to expect but that’s easy enough to do. And Murex does come with builtins that cover some of the more common coreutils use cases for instances when you want greater assurances of the quality of th…

Murex is pretty neat, thanks for sharing.

Still, you must have issues parsing all variations of output, depending on the flags passed to the source command and its version. How do you parse the output of ls or ps without knowing the column headers, delimiters, or which version of the command was ran (GNU, BSD, BusyBox, etc.)? Piping data into commands also must require a wrapper of some sort.

Not knocking on the project, it does look interesting, especially the saner scripting language. But the usefulness seems limited to the commands and workflows it supports.

Re: Dt: Duck tape for your Unix pipes

#106

Post by [deleted] »

seemaze wrote: post #40
dynisor wrote: post #26

Earlier quoted context omitted.

That is not what the thing you quoted says, or even correct. All duct tape is made of fabric and it is… well duct tape, so ducts were the intended use case when the name was created. Duct tape has kinda become a general term for all cloth-based tapes though, so you can indeed find duct tape not intended for ducts, but the fact the name was originally “duck tape” has nothing to do with that.

ASHRAE/UMC actually specify tape 'for ducts' to be UL 181A-P/181B-FX [0] which is commonly made of metal foil and not cloth. [0] https://tapeuniversity.com/products/foil-tapes/ul181ap-ul181...

[deleted]

Re: Dt: Duck tape for your Unix pipes

#107

Post by jareklupinski »

Cthulhu_ wrote: post #64
p-e-w wrote: post #38

Earlier quoted context omitted.

What an epic failure by the trademark office. That's as if a company had tried to trademark "Hammer" as a brand name for hammers, and the trademark office just said "Yeah, sure!"

Or like "Apple" as a brand name for... nevermind

ever since i read the story about the "Max Headroom" guys about how they picked their name (because it was written at the entrance of every parking garage)

i think the best way of naming a company is to make it something that is said hundreds of times by people every day

"an apple a day" etc... wonder if I can name my comapny "the The company"

or "the corporation of Is" haha

Re: Dt: Duck tape for your Unix pipes

#108

Post by imiric »

p-e-w wrote: post #102
imiric wrote: post #99

Earlier quoted context omitted.

You're right, I misunderstood the way it worked. But I'm not sure that approach is better. They either need to maintain full compatibility with existing tools, or users need to learn the idiosyncrasies of Nushell tools. And commands not reimplemented by Nushell wouldn't work, or they would need some kind of generic wrapper, which would have the drawbacks I mentioned. But, hey, this obviously has users who prefer it,…

> They either need to maintain full compatibility with existing tools, or users need to learn the idiosyncrasies of Nushell tools. The existing tools aren't fully compatible with each other either. There are significant differences between GNU and BSD tools, for example, and yet more differences with BusyBox and others. The idea of "standard" tools is unfortunately an illusion, so not much is lost there. But more imp…

> The existing tools aren't fully compatible with each other either.

Right, but those incompatibilities, as well as the way commands interoperate, are left to the user to resolve. No monolithic tool could realistically make that easier, unless they reimplement everything from scratch, as Nushell has done. But then you have to work with an entirely different and isolated ecosystem, and you depend on a single project to maintain all your workflows for you. Again, the ability for loosely coupled tools to work together is one of the strengths of Unix.

We clearly have a difference of opinion here, so let's agree to disagree. :)

Re: Dt: Duck tape for your Unix pipes

#109

Post by nmz »

lelanthran wrote: post #74
regularfry wrote: post #34

Earlier quoted context omitted.

Null separators between fields is structured data. It might just be the simplest that can possibly work (because the null character is outside the normal data range for text, unlike the newline, which demonstrably doesn't work in that sense) but it is structure. A list structure, specifically. The world would be a much happier place if -print0 (or whatever) was universal, but no such luck.

The painful part of all this is that there were and still are actual record separator and field separator characters in ASCII. ASCII was perfect for table output. All tools had to do when outputting is use the standard existing characters when no TTY is detected, else use tabs/spaces and newlines. I always wondered why no one uses the ASCII rs and FS codes.

How would you deal with a tree?

Re: Dt: Duck tape for your Unix pipes

#110

Post by hnlmorg »

imiric wrote: post #104
hnlmorg wrote: post #85

Earlier quoted context omitted.

Murex ( https://GitHub.com/lmorg/murex ) doesn’t replace coreutils with builtins but manages interop with commands just fine. Most output is relatively easy to parse, sometimes you need to annotate the pipe with what format to expect but that’s easy enough to do. And Murex does come with builtins that cover some of the more common coreutils use cases for instances when you want greater assurances of the quality of th…

Murex is pretty neat, thanks for sharing. Still, you must have issues parsing all variations of output, depending on the flags passed to the source command and its version. How do you parse the output of ls or ps without knowing the column headers, delimiters, or which version of the command was ran (GNU, BSD, BusyBox, etc.)? Piping data into commands also must require a wrapper of some sort. Not knocking on the proj…

Basically the same way you’d parse a CSV except white space delimited. You assume the headings are the first row. You can use named headings or numbered heading (like AWK) so you have options depending on the input and whether it contains headings.

The current implementation does break a little if records contain a space as part of its input (eg ‘command parameter parameter’ in ps) but I’m working on some code that would look at column alignment as well as separators etc — basically reading the output like a human might but without going to the extreme of machine learning. (I’m already doing this to parse man pages and --help output as part of automatic autocompletions so I know the theory works, I just haven’t yet applied that to more generalised command output).

Post reply on HN