The solution I often contemplate is to address the process by which the data is created. (Let's assume we're talking about data you would find on the web.) That is, why don't we impose rules on that process? Why can't we "mandate" that the output be structured data from the get go? Instead we allow vast quantities of unstructered data to be created and then we try to normalize it. I know this is a radical view, but it could make sense in some circumstances.
I disagree with your comment about CSV. But it's impossible to have a meaningful argument unless you provide an example: Give me a job to do, a CSV file and let me have a go at it. I'm serious. Post a link to a CSV file, define a task and let's see what we can do just using plain ole UNIX. Could be a fun exercise.
As for your hypothetical command, I do not understand what is so difficult about this. The stat command is what you want, not ls. No self-respecting UNIX user would parse ls when he can use stat (I recommend the BSD one over GNU.).
But here's what I would do:
1. If your UNIX filenames have spaces in them, rename them. There is no sensible reason to leave spaces in filenames in UNIX. Fix this first before it becomes a problem.
2. Write a one-liner and save it as a function, perhaps in your .profile, or maybe in RCS, or save it as a script. There's so many ways to manipulate output as a stream. Pick one that suits your tastes. That's the beauty of UNIX. Make your own solutions as you go. There is no right or wrong answer. It is a form of customization. My choice will no doubt make some people cringe. Assuming there's no user named "[0-9]M":
whatever(){ ls -lhS |tr '\011' '\040'|sed '/ [0-9]M /!d' ;}
or save what's between the brackets as a file named "whatever". Maybe you save it in a directory called "x" and add that to your PATH. Then you do
. whatever
Of course how long the list of files is going to be makes a difference. I might take a different approach if the output was going to be an enormous list.
There are so many ways to get what you're after. The point is that you should be able to tap out a one-liner that does the job. Maybe it takes a few iterations to get the right output. Tweak it until the output is what you want. Viewing command-line history is perfect for seeing the process of creating a one-liner to manipuate output. You can see the line grow incrementally as you build it, until you finally have the output you want. This sort of history allows you to go back to any stage in the process. If you're a vi fan, you can use vi-mode on the command line to move around the line quickly as you edit. Eventually you can hit "v" and edit the thing visually in your EDITOR, then save it. I've built over 700 useful functions this way and the number keeps growing.
I do understand there should be a way to "extract" the file size column the way Pike decribes in the article. To do this I think you have to free yourself from "line-oriented" thinking and imagine another type of structure. And I think using another language you can do it. But for something as simple as this -- manipulating ls command output (cf. manipulating large datasets) -- an "ugly" one-liner suits me fine. The more you use the boring old utilities the more you can get them to do.
Regular Expressions are indeed "crude". But, to me, that is just fine in a lot of cases.