I agree with you that visual tools are probably not finished as a means for non-programmers to automate things. You point about regular expressions and SQL is very good. Some business people have been doing amazing stuff with Excel or even Access and I know people who use visual regex builders to reorganise the tags and titles of their mp3 files.
What's important, though, is that these people don't build UIs. First and foremost they manipulate or analyse data. The data is at the center. The important thing is a simple data model that follows the closure principle (filter tables and you get tables, filter text and you get text). The UI is there to support this simple data model and it has to be similarly reductionist I think. Placing arbitrary UI widgets on forms is not the way to do this.
Yes, more fundamental research please :-) Maybe machine learning could be used to let users correct the system's guesses rather than initiating some action. That would go further in the direction that graphical UIs have begun, which is to let users select from what is offered to them and react instead of making them give commands to the system.
Or maybe some of the thinking that has gone into computer games would be useful, I don't know.