Sven Mascheck documents Unix shell history and behavior, including shebang handling, the Bourne shell, echo and printf differences, and argument limits.
HN links it as a detailed reference on shell quirks, such as set -e, echo and printf behavior, and the Bourne shell's goto.
http://www.in-ulm.de/~mascheck/various/echo+printf/ is the raw data version.
POSIX defines echo as only taking string parameters and no options, but notes that behaviour facing `-n` is implementation-defined.
BSD and GNU echo implement `echo -n` as not printing a trailing newline, but `echo` commonly calls to a shell builtin which may or may not follow that behaviour (and may switch behaviour depending on whether the shell is in "sh mode" or not), so `echo -n` could print `-n<newline>` or nothing whatsoever (empty string and suppressed newline) depending on the utils set, the shell, and the shell's runmode.
GNU echo also supports -e, -E, --version and --help options, and much like -n shells and other utils set may or may not support these.
For instance on my machine (OSX 10.11)
* zsh (builtin) interprets -e, -E and -n as options (but not version or help)
* bash (builtin) also does, except when invoked as sh in which case it does not and all parameters are literal (this may also apply to zsh)
* dash (builtin) interprets -n, but none of the others. bash note may also apply to it.
* BSD echo interprets -n but will print -e, -E, version and help literally
* GNU echo interprets all of the above
So if you use echo with any non-literal parameter, or with one of the parameters listed above, in a script you distribute to un-controlled third-parties as an sh script (rather than e.g. a bash or zsh script specifically) you will suffer from portability issues.
And that's just…
> 100% reliable file iteration. I want to do a "for each file in this directory" in a manner that doesn't ever run into trouble with spaces, newlines or unusual characters.
In UNIX file names, there is exactly one unusual character: ASCII NULL (or '\0'). Every other character is usual, including spaces, newlines, tabs and other historical ASCII control characters.
------
> No length limits. If I'm processing 10K files, I don't want to run into the problem that the command line is too long.
Shell does not impose length limits, the UNIX kernel does by defining the limit of how much can be passed into the «execve» syscall which UNIX shells use to create new processes. You can find the length limit on your system by running «getconf ARG_MAX» in the shell as it varies across different systems (it varies even across different versions of the same system; in Linux, it is now reportedly 1/4th of ulimit -s). The command line limit is calculated using following constituents:
command line limit = «environment size (env | wc -c)» *plus* «(ARG_MAX *minus* environment size» *minus* «POSIX recommended 2048»).
How the length limit works is explained at length (please pardon the pun) here: https://www.in-ulm.de/~mascheck/various/argmax/
------
> Excellent path parsing. […] finding the file extension […]
File names in UNIX do not have extensions, they simply have names. Is «.bashrc»: 1) a full and complete file name or 2) an empty file name with the «bashrc» extension? It is (1).…
dash is an Almquist shell. At this point there are several major ones (the Debian one, the FreeBSD one, the NetBSD one, and the BusyBox one) and a whole raft of minor ones (Minix, Cygwin, Android, et al.).
* https://www.in-ulm.de/~mascheck/various/ash/
And Wikipedia does not say that. It says that the Minix Almquist shell, specifically, had a non-conformant test utility. The Almquist shell did not, after all, have a built-in test command at all to start with, so the standard conformance of that utility wasn't a matter for the Almquist shell.
Once again Wikipedia is wrong, because Thomas Dickey's original page that purportedly supports this claim points out that the Minix test command was an external command and not part of the Almquist shell. M. Dickey even pointed to the source code for the external command on GitHub.
* https://invisible-island.net/autoconf/portability-test.html#portability
* https://github.com/Stichting-MINIX-Research-Foundation/minix/commits/master/bin/test/test.c
* https://commons.wikimedia.org/wiki/File:Don't%20abbreviate%20as%20Wiki%20(English%20version).png
https://news.ycombinator.com/item?id=35768615
Perhaps dash now has these features but does NetBSD sh or FreeBSD sh have them.
What's the point of "pure sh" if it's restricted to specific versions of shells. Opinions may differ but I'd rather learn the "lowest common denominator". I use the same scripts on both Linux and BSD so I focus on portability. I do not use bash for scripting because it's larger and slower.
One source I consult is https://www.in-ulm.de/~mascheck/bourne/
It is sometimes submitted to HN, but not IMO enough
https://news.ycombinator.com/from?site=in-ulm.de
I'm using the latest version of dash but I still don't think it has these ternary operators. Even if it has them, I'm not going to use them in scripts unless I know they'll work on older dash versions, NetBSD sh, FreeBSD sh, etc. Needless to say the process of updating these shells is generally very slow. It takes years.
"Pure sh" implies there is some advantage over "bashisms". I thought maybe it was portability but who knows.
The thing that surprised me was that you can't write an interpreter in an interpreted language, at least not in obsd. It is possible if you jump through a few hoops but you can't directly call it.
An example: if you made a language in python /bin/my_lang: #does nothing but pretend it does
#!/usr/local/bin/python3 import sys print('my_lang args', sys.argv) for line in sys.stdin: print('invalid_line:', line) my_script:
#!/bin/my_lang line of stuff another line of stuff chmod u+x my_script ./my_script Probably for the best, but I was a bit sad that my recursive interpreter scheme was not going to work.
Update: looks like linux does allow nested interpreters, good for them.
https://www.in-ulm.de/~mascheck/various/shebang/#interpreter-script
really that whole document is a delightful read.
Everyone talks about the macro abuse, nobody talks about the memory management abuse:
https://www.in-ulm.de/~mascheck/bourne/segv.html
> In comp.arch, 05/97, <5m2mu4$guf$1@murrow.corp.sgi.com>, John Mashey writes:
> For speed, Steve B had used a clever trick of using a memory arena without checking for the end, but placing it so that running off the end would cause a memory fault, which the shell then trapped, allocated more memory, then returned to the instruction that caused the trap and continued. The MC68000 (in order to go fast) had an exception model that broke this (among other things) and caused some grief to a whole generation of people porting UNIX to 68Ks in the early 1980s.