@david_chisnall i agree in general but specifically with glob expansion we have the other option on display with windows and it is *definitely worse*

@whitequark I am not convinced by this. The problems on Windows are mostly due to the way that arguments are passed to and from a process, not due to how they’re processed afterwards. VMS managed to get both parts right and was a lot more consistent. Windows also suffers from two other issues:

A lot of things that take command-line arguments have DOS heritage and DOS didn’t have shared libraries either.
PowerShell changed how argument parsing should work in a bunch of subtle ways and so you end up with big differences between things that are native PowerShell scriptlets, things that try to mimic PowerShell conventions, and things that don’t even try.

Oh, and the fact that Microsoft spent the ‘90s trying to convince everyone that no one should ever touch a command line, so intentionally didn’t do anything to improve the command-line experience (and happily let it degrade).

@whitequark

I think you’re conflating a couple of things here. The big problem with CreateProcess is that they take a single string for all command-lime arguments, whereas execve takes a vector of arguments. This is the root cause of a load of quoting errors because there is no way of passing multiple arguments to a child process on Windows that doesn’t involve quoting them. And, because this API is used directly, everyone rolls their own quoting code to go from their internal vector-of-arguments representation to the child process’s one.

This has nothing to do with glob expansion, which happens later after you’ve split arguments. And on UNIX-like systems, doing it on the shell causes all sorts of weird behaviour. For example, on FreeBSD, I often do pkg info foo* to print info about packages that start with some string. If I forget to quote the last argument, this behaves differently depending on whether the current directory contains one or more files that have the prefix that I used. If they do, the shell expands them and pkg info returns nothing because I don’t have any installed packages that match those files. If they don’t, the shell passes the star to the program, which does glob expansion but against a namespace that is not the filesystem namespace. The pkg tool knows that this argument is a set of names of installed packages, not files in the current directory, but it can’t communicate that to the shell and so the shell does the wrong thing.

Similarly, on DOS the rename command took a load of source files and a destination file or pattern. You could do rename *.c *.txt and it would expand the first pattern, then do the replacement based on the two patterns. UNIX’s mv can’t do that and I deleted a bunch of files by accident when I started using Linux because it’s not obvious to a user what actually happens when you write mv *.c *.txt. There is a GNU (I think?) rename command and its syntax is far more baroque than the DOS one because it is fighting against the shell doing expansion without any knowledge of the argument structure.

@david_chisnall I count glob expansion and quoting together because you can't (reasonably) do glob expansion without escaping, and having different rules for escaping glob metacharacters than for escaping other command line characters is as bad as having glob rules differ between applications.

@david_chisnall @whitequark but I think the answer for globbing and adjacent matters would lie in programs declaring the "types" of their arguments and shells understanding them? This would also help with tab completion, etc.

But in the end, I guess lowest common denominator tends to win, because innovation is always pushing against limits, and inertia is a big force...

@coder @david_chisnall I was about to point this out as a natural evolution of my other suggestion earlier, yes.

@whitequark @coder

It is not possible to move all processing outside of the tool unless your model is to allow the tool to provide complex arbitrary code to the shell. For example, consider gcc or clang. Some of their flags depend on the target, so you need to parse some of the, and then do lookups against complex data structures that depend on targets. It’s sufficiently complex that writing it in declarative code is hard.

Some folks on the .NET team had a very nice solution to this for autocompletion, where a .EXE had a special section of .NET IL that PowerShell could load. If you used their declarative framework for argument parsing then it would generate this for you and it would make autocomplete work beautifully in PowrShell (not sure if this was ever released, I played with a prototype). I can imagine a lightweight WAsm interpreter being an interesting approach for doing this on *NIX.

The best argument for doing expansion in the shell is one that is sadly not realised in UNIX. If you do file expansion and opening in the shell, you can start processes with file descriptors instead of (or as well as) paths. Processes can then be started with access only to files that are either listed in a manifest or passed on the command line. Build a system like this, and you have a nice way of respecting the principle of least privilege.

@david_chisnall @coder since at the beginning you stated that shared libraries are available: I am completely fine with applications being loadable shared libraries that the shell interoperates with via a function call based interface; at a point where you have structured data on the shell/application interface you already have your application be conceptually a function call, so you might as well implement it using a function call interface (or multiple functions, as you suggest here)

@david_chisnall @coder I do also think that it is viable, if restrictive enough that it goes against unix sensibilities, to have a declarative interface that limits how shell/filesystem-specific argument types (filename, glob, hole, etc) are used; I think it is an interesting direction for design because it will make the overly complex and difficult to use interfaces like that of `gcc` more difficult to build and so will make the CLI more pleasant to use overall

@whitequark @coder

That is a lot closer to how MULTICS worked. MULTICS shared libraries were a lot richer than the ones that UNIX ever added and were security boundaries much like a UNIX process.

You might have noticed that I’ve copied MULTICS a lot in CHERIoT RTOS.

@david_chisnall @coder I am completely unfamiliar with MULTICS. do you have any good introduction to it?

@whitequark @coder Not really. They published a load of things, but hanging out with Peter G. Neumann and listening to everything he said was the most useful for me.

@whitequark

I don’t think I understand this. The escaping problems happen as a result of expanding globs early. If you pass *.txt as an entry in argv, you don’t need to escape it. The receiving tool then passes it to glob, which returns an array of paths. If you’re matching something that isn’t a file path, you can use `fnmatch‘ and get the same behaviour, but again you’re dealing with structured data.

Quoting is a requirement when you are serialising structured data. If you expand globs and pass them as a string, you have introduced a requirement for quoting. If you pass all of the arguments to a program as a single string, you have similarly introduced a requirement for quoting. The shell performing early expansion adds a bunch of quoting requirements (POSIX shells not having array types and writing arguments as a string, even though they are passed as a vector of strings, also introduces quoting requirements). These are not intrinsic to the problem.

Windows requires a load of quoting because you flatten a structured vector of arguments to a single string.

UNIX requires quoting in the shell because the shell expands globs to a single string.

VMS, as I recall, had neither of these problems because it passed arguments as a vector and provided a shared library routine for expanding file patterns.

@david_chisnall in this system, how do you move a file called literally `*.txt`? or do you propose to ban glob metacharacters from filenames?

@whitequark Banning glob meta characters from files is how most platforms that followed this model worked. It’s probably a good idea, I’ve never created a file that contained one deliberately (I have created quite a few by accident).

Again, too late to put that particular genie back in the bottle for UNIX, but that doesn’t mean it was a good idea.

@david_chisnall OK, that does avoid the quoting issue and results in a consistent system, provided you can coordinate development between your shell and your filesystem interface.

I wonder though whether I'd prefer a system where a prefix (sigil character?) denotes the type of the token that follows. it's more verbose, but also unambiguous in a way that's compatible with existing systems and might be easier to read in a script as well

@whitequark The little language Jon Anderson built for the Fabrique build system had an explicit type for files, as distinct from strings. He did this so that you can add sandboxing things (the build system knows all of the files a tool should access and so can set policies around it) but I’ve found it something I miss in other environments.

I think there are two dimensions here:

What namespace are you talking about? UNIX defaults to the filesystem namespace because it was the only namespace originally (no network, no IPC except pipes) but these days you have different global (e.g. network) namespaces and different context-specific namespaces (e.g. the installed package namespace, or even the namespace of options this tool supports). Plan 9 addressed this by putting everything in the same namespace, but it often feels a bit shoehorned in.
What processing is done on this string to resolve it against things in that namespace.

Sigils feel like a good fit for the second because that should be a constrained set. Encouraging every application to do their own regex-like thing is probably bad, for example. The first set should be extensible, so sigils are maybe not the right thing here, but explicit namespaces that can be omitted when they are unambiguous might be nice.

@david_chisnall I agree! also, since namespaces are inherently system-wide, this structure enables a middle ground where each namespace comes its own little parser as a loadable shell component, but applications are purely declarative (but can depend on particular namespaces being available).

this parallels how the system operator would learn and think about commands: new namespaces come with higher cognitive effort than new commands, so the primitive is more heavyweight