Layer-free shell syntax
Here's a question I've been pondering for a while, to which I don't have any good answer.
I often find myself answering questions from Unix users who are having a little trouble working out how to get some complicated piece of shell syntax to work. The impression I'm left with is that a major source of confusion is the fact that the POSIX shell syntax as a whole - defined as the entire sequence of steps that get from a single input string (typed on the command line or read from a script) to the decisions of individual programs about what to do - is composed of a great many layers, and it doesn't take all that unusual a situation before it becomes important to understand what happens in which layer in order to debug your problem.
Examples of gotchas arising from this, many of which you need to develop quite a deep understanding to sort out, include:
- searching for a string starting with a minus sign using a command along the lines of ‘grep \-10 file.txt’. (‘The backslash causes other characters to be treated literally rather than specially, so why not this one?’)
- expecting ‘~’ and ‘~user’ to still be expanded after a colon (‘PATH=/here:~/there:$PATH’).
- getting subtly wrong results when adding a prefix to an existing command line, if the latter had redirections. (E.g. prefixing ‘sudo’ or ‘gdb --args’, and finding that in the former case the file is not opened with root privilege and in the latter case all of gdb's interactive output gets lost.)
- almost anything involving multiple layers of escaping. (The typical example being regular expressions, where you have to first escape special characters to stop them being special to the shell, and then escape them a second time to stop them being special to the application. And don't forget to shell-escape the extra characters you used for the second job, of course.)
- when a one-character option takes an argument either with or without a separating space, and you want to pass an empty string as argument, omitting the space before the quoted empty string. (‘If ‘-s foo’ and ‘-sfoo’ and ‘-s ""’ all do what I expect, why on earth would ‘-s""’ not?’)
Some of these even catch me out on occasion, and I know perfectly well why they don't work. Typically I realise my mistake as soon as I see the error message or odd behaviour - but it's telling that I nonetheless still get it wrong some of the time, because the real rules of how this stuff works are just too cumbersome to think through from first principles for every command I type, and so I use a simpler intuitive model most of the time and only resort to applying my full understanding when that can't cope.
And that's the point, really. The system is just too complicated to manage sensibly. So: if we were designing a shell syntax from scratch, would it be possible to arrange it in such a way that you don't ever have to reason about layers? The ground rule would be: the system is permitted to divide labour between multiple processes or pieces of code, as long as that division is its problem rather than the user's, and (barring the diagnosis of actual bugs) a user never has to keep in mind the divisions between those components just to figure out what a given command will do or what command they need to do a given task.
But the catch is that you aren't allowed to solve the problem by reducing the expressive power of the syntax. If you do, application authors will take up the slack by implementing all the things you didn't, and they'll all do it differently from each other, or not do it at all, and nobody will be better off. (See, for example, the haphazard state of wildcard support in Windows command-line programs.)
Here's a simple example of the sort of thing I'm thinking of. In a summer job when I was a student, I had occasion to write a tool that presented an internal command line, and I got to make up the syntax used on that command line. Basically on a whim, I set it up as follows: spaces were special (they separated words), double quotes were special (they stopped spaces from separating words), and the only other special things were braces, which were used to enclose special tag names. Initially, when starting to scan a command line and looking for the actual command word, the only things you were allowed to do with braces was to use them to escape the existing special characters, so that {{} and {}} and {"} translated into the literal characters {, } and ". But the main command processor only parsed the input line as far as was necessary to find the command name - and parsing the rest of the line was deferred until that command had had a chance to add extra sequences to the list of braced escapes. So, for instance, if a command had decided to treat the minus sign specially, that command could also have added support for an escaped version {-} which was not treated specially; so a user would not have had to draw the mental distinction between characters treated specially by the ‘shell’ and by one particular subcommand, and would have been able to work on the much simpler principle that all special characters were defusable by bracing them. (Or, perhaps better still, the subcommands could all have adopted the principle that everything not braced was treated literally, and invented braced tags to do everything special they might need.)
Of course it was easy for me to do something like that in a small language supporting only a few commands, particularly when there was no division into subprocesses (each of my commands was just a procedure within the same source file). And even so, my design wasn't perfect (e.g. it would still have suffered from the ‘-s""’ flaw in my above list). And it had very simple functionality by comparison to the full scope of real shell syntax. And, of course, I wouldn't even dare to imagine that any replacement shell syntax design would be realistically adoptable at this stage, what with a huge established source base.
But it illustrates the sort of idea I'm thinking of. So, just as an exercise for curiosity's sake … could the entire edifice of shell design be reworked, in principle, by means such as the above or by totally different means, so as to avoid the need to have users keep track of the multiple layers of code involved?
Also at Dreamwidth. (
comments | Leave a comment )