Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Collapse of the Unix Philosophy

kukuruku.co

Post reply on HN
Print view View original

541–550 of 616 posts

Re: The Collapse of the Unix Philosophy

#541

Post by jstimpfle »

pdimitar wrote: post #533
jstimpfle wrote: post #515

Earlier quoted context omitted.

I clearly said I'm not defending shell. Even when the author is responsible himself for wanting to put a fixed string where a pattern is expected. But this is about text formats. Text is simple. It's only the overengineering farts who think they have to wrap everything in three levels of parens. It doesn't make a difference. > Their fault? Absolutely and definitely. But it's the job of the tech to slap you through th…

> Text is simple. Sigh. I am not here to argue with your out-of-context sweeping generalizations. So I won't. BTW, do you have a particular gripe with S-expressions / LISP? You ranted twice about parens in your comment towards me. And no -- me, the OP, and several others in this thread will definitely not stop with this "vague FUD", "bs", "trolling" -- all your quotes from other comments -- simply because it's someth…

Again, you have ignored that this discussion is not about shell (which I know, including its few flaws, and can easily deal with, but am in no way trying to describe as easy to learn given that there are so many broken scripts and tutorials out there. It's hard to just learn the quoting rules for once, and browse through "bash pitfalls" once, simply because people don't know where to look for good resources. And I have freely admitted it was hard for me as well. Nevertheless I seriously recommend learning it rigorously because it has tremendous practical benefits).

This discussion is about text representations. Why do you keep claiming that text formats are broken when you can't give a single example?

> BTW, do you have a particular gripe with S-expressions / LISP? You ranted twice about parens in your comment towards me.

I will rant again until people stop making stupid claims.

I actually like LISP as a programming language. There is just zero benefit from writing record- or even word-oriented data in a (random) free-form syntax that is meant for representing trees. If I wanted I could parse /etc/passwd format like this:

  struct_passwd = namedtuple("passwd", "pw_name pw_passwd pw_uid pw_gid pw_gecos pw_dir pw_shell")
  passwd = [struct_passwd(* line.rstrip('\n').split(':')) for line in open('/etc/passwd')]
That's it. It works. There, I even made a nice datatype for you. And there's already more integrity checking in these two lines compared to a json.parse() or similar.

It works so nicely that I'm even making a text format for such databases with strong schema support that can still quite easily be used with generic text tools (git, grep, sed, awk, diff...). http://jstimpfle.de/projects/wsl/main.html

> So forgive all of us working folk who don't keep Star Trek-like exact logs on every problem we ever bump into. /s

Never asked for that. Give a single reasonable example why text file practice is bad, to get any credibility. It can't be that hard.

> ... And that's a fact of the daily life of many devs. You can call that a vague FUD if you wish.

Well, it's a bit less vague now that you have actually described a little better. But there is no connection to text representations. Sorry, you replied to the wrong thread.

Re: The Collapse of the Unix Philosophy

#542

Post by _acme »

pjmlp wrote: post #350
em3rgent0rdr wrote: post #239

Earlier quoted context omitted.

Suggesting that Unix has been continuing just by inertia doesn't explain why Apple adopted Unix for macOS or why most newer computers nowadays run a Unix OS.

Apple did not adopt UNIX per se. Apple failed miserably to create a new OS, decided to buy a company instead and they happened to get the one where Steve Jobs was. NextSTEP was partly based on UNIX, because it was competing against Solaris and needed some compatibility for easing the port of applications. It used a micro-kernel like architecture, drivers where written in Objective-C, the whole userspace used the Foun…

Apple took NeXTSTEP and with minimal changes had macOS certified as UNIX, so it is actually UNIX, unlike many UNIX-like operating systems.

Re: The Collapse of the Unix Philosophy

#543

Post by arpa »

ghettoimp wrote: post #531
arpa wrote: post #432

Earlier quoted context omitted.

Bash is not bad at all; it's unrestrained and arcane at times, agreed, but it does have the nature of a true programming language, just like powershell does. Now .bat scripts, that's truly inelegant and unsafe.

While it's certainly not the worst, and certainly can be used for many things, bash is definitely pretty damn bad. You've got horrible string escaping nonsense ($IFS, ...), an utter lack of any sensible types, constant incompatibilities due to command-line tools supporting different arguments on different platforms, ... While Perl or similar are not without many faults, at least they solve a lot of this. Think of eve…

You could use something like jq for parsing JSON. And while I agree with you regarding some of your points, they could be easily refuted by pointing fingers at other things: yes, string escaping can give a grown man nightmares, but hey, it's not working with strings in C; and once you get the hang of it, it is pretty simple. Lack of typing can hurt, sure - but it's not BASIC; With a little imagination you can have strict types even.

And cli tools having different args on different platforms? come on, it's not even bash's fault...

Basically what i'm saying, everything you can do on $LANG, you can do in bash. And it's not that hard. It just has a loooong learning curve and seems to provide with just enough things to shoot yourself in the foot. But then again, you can be stupid in any language...

Re: The Collapse of the Unix Philosophy

#544

Post by swhipple »

jstimpfle wrote: post #454
TeMPOraL wrote: post #448

Earlier quoted context omitted.

> Name a well-thought out text file format that can't correctly be parsed e.g. by a Python one-liner with basic string operations. And please don't include: JSON, XML, YAML, sexps, because it's not possible, at least not without a library. Well, because this library should be a part of the OS API. A set of conrete cases where existing practice is bad is Unix itself (and its descendants). Think of every time a script…

> because every program has to contain its own, half-assed parser for textual data. As I said, name me a format that I can't parse correctly as a Python one-liner. I work as a systems administrator and my scripts (mostly shell, python) don't break. I'm not kidding you. Of course when writing shell scripts (which I think you imply) I need to know how to write non-broken shell scripts (in a clean and straightforward wa…

> As I said, name me a format that I can't parse correctly as a Python one-liner.

mboxo? [1] It is a popular text format that cannot be unambiguously parsed.

More generally, most Unix tools' output is also not able to be unambiguously parsed. For example, use gcc to compile a file, then collect the warnings? The regex "^.+:\d+:\d+: warning.*" will be right most of the time, but there's no 'correct' way to parse gcc output (there is not a surjective mapping of output to input).

There are various ways to work around the problem: mboxrd format uses an escape sequence to work around the earlier problem mentioned with mboxo. `ls -l --dired' (GNU) will allow you to parse ls by appending filename byte offsets to the output. `wc --libxo xml` (FreeBSD) will give the output in XML, which is unambiguous as well. multipart/form-data (RFC2388) is used to embed binary data in a text format, by using a byte sequence which doesn't appear in the data.

Binary formats present their own set of issues, but "accidentally unparseable" is more common in text-based formats (or ad-hoc text output).

[1] https://jdebp.eu/FGA/mail-mbox-formats.html

Re: The Collapse of the Unix Philosophy

#545

Post by Too »

Arizhel wrote: post #113
pjmlp wrote: post #98

Earlier quoted context omitted.

The thing is UNIX, by definition, is never going to move beyond its original design, meaning POSIX + C. Whereas Mac OS, Windows, iOS, Android, ChromeOS have moved into more productive language runtimes, with rich frameworks, improving safety across OS layers, even if they have a few bumps along the way.

That's not true. Windows in particular is greatly limited by past design choices, such as filename limitations, the security model, the fact that many programs have to be run as administrator because that's how they were written years ago, etc. There's nothing preventing you from running different language runtimes and such on Linux/Unix systems; people do it all the time. Have you not noticed Mono? It's been around…

Old, badly written, programs requiring admin was a problem the first year after Vista was released. Today, not anymore.

All programs of interest have migrated and there is also a compatibility layer redirecting older programs to write to a fake system directory.

Re: The Collapse of the Unix Philosophy

#546

Post by ChuckMcM »

tnone wrote: post #401
ChuckMcM wrote: post #156

That was a sad thing to read, the author is so clueless they don't even know when the reasons they imagine something might have been broken are wrong. Back when UNIX was born the character was a first class citizen in every computer on the planet, and many languages used it as part of their syntax. Static binaries were invented when Sun and Berkeley co-developed shared libraries and there needed to be binaries that y…

You know that thing about how old ideas in science can only really be replaced when their supporters die? Unix greybeards, this is about you.

I think there is some truth to that. And I also think there is a tendency to look backward through time and come away with the wrong impression. It is starting with people new to web design wondering how anyone thought HTML was a good idea. So perhaps it's a 30 year window for it to start.

In my experience it gets worse and worse over time.

Re: The Collapse of the Unix Philosophy

#547

Post by jstimpfle »

swhipple wrote: post #544
jstimpfle wrote: post #454

Earlier quoted context omitted.

> because every program has to contain its own, half-assed parser for textual data. As I said, name me a format that I can't parse correctly as a Python one-liner. I work as a systems administrator and my scripts (mostly shell, python) don't break. I'm not kidding you. Of course when writing shell scripts (which I think you imply) I need to know how to write non-broken shell scripts (in a clean and straightforward wa…

> As I said, name me a format that I can't parse correctly as a Python one-liner. mboxo? [1] It is a popular text format that cannot be unambiguously parsed. More generally, most Unix tools' output is also not able to be unambiguously parsed. For example, use gcc to compile a file, then collect the warnings? The regex "^.+:\d+:\d+: warning.*" will be right most of the time, but there's no 'correct' way to parse gcc o…

Thanks!

It's true that filenames with whitespace or newlines are bad for interoperability ("make" is another example). There are three simple options: escaping filenames, making filenames NUL-terminated or declare such filenames as invalid. The latter way seems to have won for practical reasons, and it's a pity that "safe filenames" were never standardized (but C-identifier plus extension should be safe everywhere).

Mbox is definitely broken (for example body lines that start with "From" are changed to "> From"). I don't think it is ambiguous today (all software I know interprets "From " at the beginning of a line as a new mail), but it clearly was not much designed at all. It still has some precious properties which is why it's still in use today. For example, appending a new email (Mail server) is very fast. Crude interactive text search works also very well in practice, although automation can't really be done without a library.

Email is complex data (not line- or record-oriented), so various storage formats achieving various tradeoffs are absolutely justified.

> Binary formats present their own set of issues, but "accidentally unparseable" is more common in text-based formats.

It's true, especially with formats from the 70s where the maxime was "be liberal in what you accept", and where some file formats weren't really designed at all.

On the other hand, "accidentally unextendable" (for example, fixed-width integers) and "accidental data loss" is much more common in binary formats.

Re: The Collapse of the Unix Philosophy

#548

Post by Too »

jstimpfle wrote: post #438
TeMPOraL wrote: post #416

Earlier quoted context omitted.

Piping isn't the culprit. It's what you pipe that is, and Unix Philosophy says "pipe whatever the hell you want, and let the users upstream sort it out". It's not about encoding everything in exactly the same way. It's about providing the basic, shared protocol for representing structure. With typical Unix tools, you don't have "simpler encodings", you have no encoding at all . Each tool outputs whatever its particul…

There exists a shared protocol. It's called "explain it". But that's typically not even needed, the user can just look at the data and figure it out. > With typical Unix tools, you don't have "simpler encodings", you have no encoding at all. You don't even believe this yourself, do you? That's so obviously wrong. Name a well-thought out text file format that can't correctly be parsed e.g. by a Python one-liner with b…

> There exists a shared protocol. It's called "explain it". But that's typically not even needed, the user can just look at the data and figure it out.

This is the root cause of 99% of all parse errors and security holes in the world.

If you just "look" on the output of ls in some arbitrary directory there is nothing there telling you that a file name can contain a newline that will mess up the output. Write your parser with this assumption and it's broken. (See OP)

If i had a penny for every csv-"parser" I've seen that is just data=input.split(','); i would be a rich man now. Because the developer when looking at their data had no comma in any cell. Doesn't mean the customer don't have it.

Re: The Collapse of the Unix Philosophy

#549

Post by the_why_of_y »

braveo wrote: post #518
moosingin3space wrote: post #229

Earlier quoted context omitted.

What in particular are you referring to? I ask honestly, because this "my way or the highway" attitude has never been the Rust community's way.

Their approach to overflow, for one.

IIRC the reason not to detect overflow by default in release builds isn't any variant of "You Just Don't Get It", it's rather "yes we'd like to do that but unfortunately there is a trade-off with a significant runtime performance hit, so we don't, for now".

Re: The Collapse of the Unix Philosophy

#550

Post by jstimpfle »

Too wrote: post #548
jstimpfle wrote: post #438

Earlier quoted context omitted.

There exists a shared protocol. It's called "explain it". But that's typically not even needed, the user can just look at the data and figure it out. > With typical Unix tools, you don't have "simpler encodings", you have no encoding at all. You don't even believe this yourself, do you? That's so obviously wrong. Name a well-thought out text file format that can't correctly be parsed e.g. by a Python one-liner with b…

> There exists a shared protocol. It's called "explain it". But that's typically not even needed, the user can just look at the data and figure it out. This is the root cause of 99% of all parse errors and security holes in the world. If you just "look" on the output of ls in some arbitrary directory there is nothing there telling you that a file name can contain a newline that will mess up the output. Write your par…

I'm pretty sure most security errors come from implementations of complex binary formats. (Okay, there is the web world and I hear people still haven't learnt to escape their SQL queries).

ls is only for human consumption. I said this elsewhere in this thread.

CSV is utterly broken (at least was RFC'ed at some point, but the escaping rules are still shit. We have known for decades how to do it better).

Post reply on HN