Ilya Sher

> I'm rambling a bit, I'm not sure my thoughts on the matter are that coherent. [...] Do shells languages have to be as capable as general purpose programming languages, and if so, why?

I think your thoughts here are definitely coherent, and in fact you're getting to the heart of the issue. This exact question is pretty central in motivating many next-generation shells. I'll refer to the documentation of a few of them below. Feel free to follow the associated link to tug on any of the threads and see if the overall argument holds in your opinion.

> Python and Ruby aren't good shell replacements in general. Shell is a domain-specific language for dealing with concurrent processes and the file system. Python and Ruby have too much abstraction over these concepts, sometimes in the name of portability (e.g. to Windows). They hide what's really going on.

— Oil Shell: Why create a new shell?: Shouldn't scripts over 100 lines be rewritten in Ruby or Python? ( https://www.oilshell.org/blog/2021/01/why-a-new-shell.html#shouldnt-scripts-over-100-lines-be-rewritten-in-python-or-ruby )

> Traditional shells use strings for all kinds of data. They can be stored in variables, used as function arguments, written to output and read from input. Strings are very simple to use, but they fall short if your data has an inherent structure. A common solution is using pseudo-structures of “each line representing a record, and each (whitespace-separated) field represents a property”, which is…

I know the text stream interface has lots of limitations and formats are often unspecified or underspecified.

I would just add that if you happen to have numbered fields with consistent delimiters (and no quoting or escaping of delimiters), you can do all of these things with cut, sort, grep, and head, to say nothing of awk, which can typically do them in a single line.

ls -l foo/ | awk '$NF == "bar.baz" {print $NF, $5}' | head -10

The numbered fields with consistent delimiters part is a big if, and of course the fields are numbered and not named, so you don't get to do the "name,size" thing (you have to know explicitly that for this particular command, the filename is the last field and the size is the fifth field).

One direction that some people have gone is to treat JSON as a universal form of structured data (where text is a universal form of unstructured data). You could rebuild many Unix tools or near-equivalents to work well with JSON records instead of, say, lines. Your pipe example would be pretty straightforward in jq if you had a version of ls that gave you JSON outputs.

Here's a crummy, primitive version of ls that gives JSON outputs. :-)

#!/usr/bin/env python

import json, os, sys

ret = {}
for arg in sys.argv[1:]:
here = {}
for entry in os.listdir(arg):
path = os.path.join(arg, entry)
stat = os.stat(path)
here[entry] = {
"size": stat.st_size,
"ctime": stat.st_ctime,
"atime":…

(author here) Hm I've tried fish, and it seems very nice for interactive use.

However I don't see it being used AT ALL for the cloud/linux use case? Those are the cases where you tend to get 1000+ lines of shell scripts.

For example, I mention Kubernetes/Docker/Chef, and I've never seen fish in that space.

I also don't know of any Linux distro that uses fish as their foundation -- they all appear to use a POSIX shell like bash/dash/busybox ash. fish is "on top".

See Success with Aboriginal, Alpine, and Debian http://www.oilshell.org/blog/2018/01/15.html -- these distros are built with thousands of lines of shell scripts.

Either 1) I don't know about such usage, 2) people don't know that fish can be used this way, or 3) there is some problem with using fish this way.

I link to this post in the FAQ, which I think is a lot closer to Oil:

https://ilya-sher.org/2017/07/07/why-next-generation-shell/

It's basically the "devops" use case. (And as I mention the main difference between Oil and NGS is that Oil is compatible / has an upgrade path from bash.)

If you rephrase that to tools which deals with what the endpoints emits there is a boatload. Most emits JSON but XML can be found as well.

Personally I use jaq. Gojq is nice as well. Both. Inspired by the original ja. You can find many lists[1].

I would especially point out jid[2] which often comes in handy to explore apis.

Or if we stay at the http calling level and the command line: wget or fetch.

If you include scripting languages then most http libraries will do. With Python a simple approach would be "python -m json.tool".

[1] https://ilya-sher.org/2018/04/10/list-of-json-tools-for-command-line/

[2] https://github.com/simeji/jid