Aaron Maxwell

It usually doesn't matter much, but there are some situations where it can matter a lot. For one thing, you can't use seek() on a pipe, so e.g. `cat bigfile | tail` has to read through the entire file to find the end, but `tail bigfile` will read the file backward from the end, completely skipping the irrelevant beginning and middle. With `pv bigfile | whatever`, pv (which is basically a pipeline progress indicator) can tell how big file is and tell you how for through you are as a percentage; with `cat bigfile | pv | whatever`, it has no idea (unless you add a flag to tell it). Also, `cat bigfile | head` will end up killing cat with a SIGPIPE signal after head exits; if you're using something like "Unofficial bash strict mode" [1], this will cause your script to exit prematurely.

Another sometimes-important difference is that if there are multiple input files, `somecommand file1 file2 file3` can tell what data is coming from which file; with `cat file1 file2 file3 | somecommand` they're all mashed together, and the program has no idea what's coming from where.

In general, though, I think it's mostly a matter of people's expertise level in using the shell. If you're a beginner, it makes sense to learn one very general way to do things (`cat |`), and use it everywhere. But as you gain expertise, you learn other ways of doing it, and will choose the best method for each specific situation. While `cat |` is usually an ok method to read from a file, it's almost never the best…

The thing I wished I had learned earlier is "quick and dirty assertions". If you write lots of functions in Bash, you quickly end up getting tripped up by cases where an argument is omitted and the function does something totally batshit given the missing (empty string) argument. Now, the canonical way to handle this is to put validators on your input, (and make sure those validators don't crash with cryptic errors if someone calling your function is using "set -u") like so:

function() myfunc {
local foo="${1:-}"
if [ -z "$foo" ]; then
echo "Invalid first parameter!" >&2
return 127
fi
...
}

...but man, that's time consuming when you have lots of parameters.

Instead, the quick and dirty way is to just "assert" via [parameter expansion](https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html):

function myfunc() {
local foo="${1:?First parameter must be provided}"
...
}

Much quicker, especially when throwing things together in a hurry. It has a gotcha, though: ":?" assertion doesn't cause a function to return early, it shuts down the whole interpreter after outputting the error. So it's more like a true assert() statement than an input validator. If you'd only ever call your function in a subshell, this won't matter (because the subshell will exit early with a nonzero code, big deal), but otherwise it can be a nasty surprise to users when an argument-validation issue inside a function shuts the program…

It's cool, nice work! And thank you for sharing it.

This is just spitballing, you've already solved problem you set out to and get full marks.

There are always many different ways to do things with bash/shell. Here, you could probably do the entire thing in AWK, which you seem to already be aware of. This might be more readable or it might not.

If this were a script someone wrote for more general usage, I'd recommend to use the unofficial "bash strict mode"[0], and make sure to check that the awk/sed calls succeed. Shell has a nasty habit of limping along as far as it can if something goes wrong along the way and it's very easy to just end up with bad output and not notice until it's too late.

Another thing I only recently discovered is envsubst, a tool that's commonly available and tailor-made for substituting variables. You could pipe a template string or pile into that instead of the repetitive calls to echo. What you have here is fine and instantly readable, but if it were three times as long it might not be so nice. Here you could use awk to extract the variables and envsubst to put them into your template.

Honestly I don't think shell is a good language for writing perfect, robust things anyway. But it's good for exactly what you're doing, though, writing something for yourself that solves a problem without spending too much time on it (although it's fine to spend a lot of time on it if you're learning and/or just enjoying yourself).

[0]…

On one hand, if you're managing a few servers with shell scripts, you'll invariably end up with a framework of sorts. On the other hand, after a few years playing with bash and posix shell -- I'm fairly certain I'd prefer bash, to posix shell -- it seems rather unlikely you can easily use the same set of scripts for BSD, open solaris and various linux distributions -- if your target is linux, pretty much all distributions ship with bash in base install (just remember to use proper "bang" of #!/usr/bin/env bash or equivalent).

And shell script is a fickle language -- I'd at least consider using "trap" and/or "set -e", "set -u" -- maybe "set -p" rather than "echo 'what I am about to do'; what I am about to do". See eg: http://redsymbol.net/articles/unofficial-bash-strict-mode/

I agree that ansible is far from perfect - but I'm not convinced posixcube is a great alternative right now.

From a brief look at the code, it does look nicely structured and commented, though!

I've played a bit with the idea of doing something similar (mostly because I'm not convinced all the complexity present in ansible is necessary for my use case -- which appear to be similar to your use-case -- a few small simple server configs).

I've seen people here on HN recommending some makefiles for deployment (along with simple shell scripts) -- and I've played a bit with using m4 for templates rather than here-documents -- and it looks like that might work fine. Does mean your…

I agree with some of your frustrations, but it's honestly not that bad once you setup your environment properly and get used to some of the quirks.

> I feel like I spend 45 minutes every time I write a shell script to determine simple things like "is this environment variable set".

Adopt some best practices[1] and have a template to start new scripts with. ShellCheck also helps with avoiding the most common pitfalls.

> I find myself reaching for purpose-built tools rather than ad-hoc pipelines

It helps to distinguish day-to-day interactive work in the shell from writing portable shell scripts that will be robust and require little maintenance for years to come. For the former you want user friendly tools that help you the most with a particular task. For the latter you want to use battle-tested tools with a stable interface that are available on many systems or are preferably built-in to the shell.

> I am getting more and more frustrated at basic shell mechanics like history handling

That's mostly a solved problem in many shells. Zsh supports it natively[2].

I wouldn't want to discourage you from writing your own shell, good luck with that, but there are plenty of good POSIX compatible and alternative shells out there. We can agree that "real" programming languages are much more powerful, safer and friendlier, but plain old shell scripts are the right tool for the job in many situations and shouldn't be ignored at all cost.

[1]:…

> I once deleted a /lib directory due to a misspelled variable in a script!

Ah yes, the old "rm -rf $TEMPBASEEDIR/lib" bug. It is so easy to have that happen.

Or like I did, write a script that assumes a certain environment variable is set outside the script... and then forget to set the variable before running it!

To prevent these problems, use "set -u" at the top of your bash scripts. This will halt the script instead of silently inserting an empty string if you use an undefined variable.

"set -e" (halt on nonzero return code) and "set -o pipefail" (propagate error returns through a pipe) are also useful options, or combine them with:

set -euo pipefail

Here's a great article that describes these options and some tips for using them:

http://redsymbol.net/articles/unofficial-bash-strict-mode/