Linux command line and shell scripting bible

What Hacker News says

1. tristorSep 2021
   I don't have any one complete book that I can recommend, and I don't even really have a great reading list for this. But I'll make an attempt to share what I think is useful as a starting point.

   1. Systems Operations is first and foremost about understanding systems, in all of their complexity, which means understanding the internals of your OS primarily.

   2. Performance and networking, in particular, are super important areas to focus on understanding when it comes to learning the topic to help with software development.

   3. A lot of it is about understanding concepts in abstract and being able to extrapolate to other situations and apply these concepts, so there's actually quite a lot of useful information that can be learned on one OS and still applied to another OS (or on one game engine and applied to another, et al).

   Here's a few books I think are worth reading, not in any particular order of prevalence, but loosely categorized

   Databases:

   High Performance MySQL: https://www.amazon.com/gp/product/1449314287/

   SQL Queries for Mere Mortals: https://www.amazon.com/gp/product/0321992474/

   The Art of SQL: https://www.amazon.com/gp/product/0596008945/

   Networking:

   TCP/IP Illustrated: https://www.amazon.com/exec/obidos/ISBN=0201633469/wrichards... (updates on author's site at http://www.kohala.com/start/tcpipiv1.html)

   The TCP/IP Guide: https://www.amazon.com/TCP-Guide-Comprehensive-Illustrated-P...

   UNIX Network Programming: https://www.amazon.com/dp/0131411551

   Beej's Guide to Network Programming: http://beej.us/guide/bgnet/

   Operating Systems:

   Operating Systems Concepts: https://www.amazon.com/Operating-System-Concepts-Abraham-Sil... (various editions, I have the 7th edition... I recommend you find the latest)

   Modern Operating Systems: https://www.amazon.com/Modern-Operating-Systems-Andrew-Tanen... (the "Tanenbaum Book")

   Operating Systems Design and Implementation: https://www.amazon.com/Operating-Systems-Design-Implementat-... (the other one, the "MINIX Book")

   Windows Internals:

   Part 1: https://www.amazon.com/Windows-Internals-Part-architecture-m...

   Part 2: https://www.amazon.com/Windows-Internals-Part-2-7th/dp/01354... (I had the pleasure of being taught from this book by Mark Russinovich and David Solomon at a previous employer, was an amazing class and these books are incredible resources even applied outside of Windows, we used 5th edition, I linked 7th, which has the 2nd part pending publication).

   MacOS Internals:

   Part 1: https://www.amazon.com/MacOS-iOS-Internals-User-Mode/dp/0991...

   Part 2: https://www.amazon.com/MacOS-iOS-Internals-II-Kernel/dp/0991...

   Part 3: https://www.amazon.com/MacOS-iOS-Internals-III-Insecurity/dp...

   Linux Kernel Programming:

   Part 1: https://www.amazon.com/Linux-Kernel-Development-Cookbook-pro...

   Part 2: https://www.amazon.com/Linux-Kernel-Programming-Part-Synchro...

   The Linux Programming Interface: https://www.amazon.com/Linux-Programming-Interface-System-Ha...

   General Systems Administration:

   Essential Systems Administration: https://www.amazon.com/gp/product/0596003439/

   UNIX and Linux Systems Administration Handbook: https://www.amazon.com/UNIX-Linux-System-Administration-Hand...

   The Linux Command Line and Shell Scripting Bible: https://www.amazon.com/Linux-Command-Shell-Scripting-Bible/d...

   UNIX Shell Programming: https://www.amazon.com/Unix-Shell-Programming-Stephen-Kochan...

   BASH Hackers Wiki: https://wiki.bash-hackers.org/

   TLDP Advanced BASH Scripting Guide: https://tldp.org/LDP/abs/html/

   The Debian Administrator's Handbook: https://debian-handbook.info/browse/stable/

   TLDP Linux System Administrator's Guide: https://tldp.org/LDP/sag/html/index.html

   Performance & Benchmarking:

   Systems Performance: https://www.amazon.com/Systems-Performance-Brendan-Gregg-dp-... (this is Brendan Gregg's book where you learn about the magic of dtrace)

   BPF Performance Tools: https://www.amazon.com/Performance-Tools-Addison-Wesley-Prof... (the newer Brendan Gregg book about BPF, stellar)

   The Art of Computer Systems Performance Analysis: https://www.cse.wustl.edu/~jain/books/perfbook.htm (no longer available from Amazon, but is available direct from publisher. This is basically the one book you should read about creating and structuring benchmarks or performance tests)

   I guess that's a "reading list", but this is just a small part of what you need to know to excel in systems operations.

   I would say for the typical software developer writing web applications, the most important thing to know is how databases work and how networking works, since these are going to be the primary items affecting your application performance. But there's obviously topics not included in this list that are also worth understanding, such as browser/DOM internals, how caching and CDNs work, and web-specific optimizations that can be achievable with HTTP/2 or QUIC.

   For the average software developer writing desktop applications, I'd say make sure you /really/ understand OS internals... at the base everything you do on a computer system is based on what the OS provides to you. Even though you are abstracted (possibly many layers) away from this, being able to peel back the layers and understand what's /really/ happening is essential to writing high-quality application code that is performant and secure, as well as making you a champ at debugging issues.

   If you're trying to get into systems operations as a field, this is just a brush over the top surface and there's a lot deeper diving required.

   Show morethread
2. nikcubApr 2011
   The best way to learn the UNIX tools is in the context of shell scripting and using the combination of tools to achieve a purpose (such as parsing log files, setting up shortcuts as part of your dev toolchain)

   1. Understand the philosophy behind UNIX. ie. everything is a file and it is small apps that are very good at a single task that can be combined through pipes etc. to process larger tasks (this has lost its way somewhat in Linux, but still holds true)

   2. Look at your computer usage through development, sysadmin, etc. and find parts that you want to automate, and then set out to write scripts to complete these tasks. for eg. in each one of my projects I have a script called start.sh which has a bunch of tasks implemented (eg. backup, serve, dns (to update dyndns for callbacks), push, diff, deploy, stage (run remote commands) etc.). Remember that git is itself a set of shell scripts, as are most server management commands. Take a peak inside each of these to learn how they are implemented.

   3. Once you have a good understanding of what you want to implement, just go for it. Learn along the way by using Bash scripting, regular expressions, an understanding of UNIX/Linux, sed, awk, yacc/bison (to analyze code and extract info). You will end up building your own collection of shell scripts, environment variables, .vimrc, ssh, curl (you can do almost any API request and auth with curl and a shell script - automate tweets, RSS feeds) etc. (I have been meaning to publish my own, the most recent one I wrote greps my code for TODO's and pushes them to a simple webapp where I can view them, sort them, etc. along side my 'personal' todo list)

   4. You need two sets of resources, one is for learning, the other is for reference which you keep handy. Here are my own recommendations:

   Learning:

   The learning style book comes down a lot to personal preference. You can go wrong, so get a feel for each topic by reading online tutorials and then scan the TOC and sample chapters of books that look good. A subscription to O'Reilly Safari comes in handy. There are now also a lot of screencasts online, try searching YouTube for the topic with 'screencast' or 'tutorial', once you find a good screencast publisher look at the rest of his/her videos.

   You can also get the O'Reilly books on special in bundles sometimes, check their website

   There is no book that really introduces UNIX and then covers most of these topics as I described. ie. 'UNIX for Developers'.

   * UNIX:

   - FreeBSD handbook: http://www.freebsd.org/doc/handbook/

   - Linux Documentation Project: http://tldp.org/LDP/

   - Linux Command Line and Shell Scripting Bible: http://www.amazon.com/Linux-Command-Shell-Scripting-Bible/dp...

   - UNIX In a Nutshell (I haven't read this in a while but it is on my shelf): http://oreilly.com/catalog/9781565924277

   - Learning UNIX (also O'Reilly): http://www.amazon.com/Learning-UNIX-Operating-System-Fifth/d...

   * Bash:

   - http://linuxcommand.org/

   - http://bash.cyberciti.biz/guide/Main_Page (better, and v good)

   - Learning the Bash shell: http://oreilly.com/catalog/9780596009656/

   - Advanced Bash Scripting (online): http://tldp.org/LDP/abs/html/

   - YouTube playlist: http://www.youtube.com/view_play_list?p=2284887FAE36E6D8

   * Regular Expressions:

   - Mastering Regular Expressions (this is such a great book. I didn't like regexp until I picked this up eons ago): http://oreilly.com/catalog/9781565922570

   * sed/awk/grep etc.

   - sed & awk (O'Reilly): http://oreilly.com/catalog/9781565922259

   * Vim/Vi:

   - vimcasts (screencasts): http://vimcasts.org

   - more screencasts (these are great): http://vimeo.com/user1690209/videos

   - learning vi: http://www.amazon.com/dp/1565924266

   Reference:

   On my browser toolbar, I have a folder called '_ref', it has around 70+ bookmarks (online documentation, cheatsheats) in it each titled what the link is a reference for (ie. 'vi', 'bash' etc.). I will publish it online, and will post the links here. I have collected the links over years.

   Show morethread
3. JdeBPSep 2020
   The -- mechanism in general, not specific to the cp command albeit mostly limited to programs that use getopt/popt for their argument parsing, is described all over the place. I first encountered it in a book, written by Eric Foxley, entitled UNIX for Super-Users and published in 1985.

   * It's perennially asked about on Unix and Linux Stack Exchange, ranging from https://unix.stackexchange.com/q/11376/5132 almost a decade ago to https://unix.stackexchange.com/q/570729/5132 last year.

   * It's in a whole bunch of duplicate questions there, including (but not limited to) https://unix.stackexchange.com/questions/linked/1519?lq=1 .

   * It's in the Single Unix Specification, and has been for a long time. (https://pubs.opengroup.org/onlinepubs/7990989775/xbd/utilcon...)

   * It's in the getopt(3) manual page from the Linux man-pages project. It's in the getopt(1) manual page from the util-linux project.

   * It's in getopt(1) and getopt(3) in the NetBSD Manual. The same goes for the OpenBSD and FreeBSD manuals.

   * It's in Blum's and Bresnehan's 2015 Linux Command Line and Shell Scripting Bible, on page 377.

   * It's in Marcel Gagné's 2002 Linux System Administration: A User's Guide, on page 48.

   * It's in Kirk Waingrow's 1999 UNIX Hints & Hacks.

   * It was Q&A number 32 in part 2 of the Usenet comp.unix.shell FAQ document. (http://cfajohnson.com/shell/cus-faq-2.html#Q32)

   Specific to the cp command, or rather to the many cp commands that you could end up using, as this is not a "bash command":

   * No, the OpenBSD, FreeBSD, and NetBSD cp(1) manuals do not mention it.

   * Neither does the GNU CoreUtils cp(1) manual page, as you noted. It's instead in the GNU CoreUtils doco section on "common options". The actual CoreUtils cp(1) manual doesn't mention that there even are "common options". (https://manpages.debian.org/sid/coreutils/cp.1.en.html) That's only mentioned, as an aside, in the info page. (https://gnu.org/software/coreutils/manual/html_node/cp-invoc...)

   * For the cp command that is part of Rob Landley's toybox, this is covered in the introduction to the toybox(1) manual page, and there isn't really a distinct cp(1) manual page. (http://landley.net/toybox/help.html)

   * No, BusyBox doesn't similarly document this for its cp command, or anywhere on busybox(1). (https://busybox.net/downloads/BusyBox.html)

   * It's tangentially in several GNU CoreUtils FAQ Q&As, inasmuch as CoreUtils cp is the same as CoreUtils rm in this regard. (https://gnu.org/software/coreutils/faq/coreutils-faq.html#Ho...)

   * It's tangentially in the Linux Documentation Project's Advanced Bash-Scripting Guide, again because cp is the same as rm here. (https://tldp.org/LDP/abs/html/special-chars.html#DOUBLEDASHR...)

   * For an operating system where this is actually in the cp(1) manual itself, look to Solaris, and its successor Illumos, where you'll find it in the NOTES section. (https://illumos.org/man/1/cp)

   * Another operating system where this is actually in the cp(1) manual itself is AIX, actually in the FLAGS table alongside all of the other flags. (https://ibm.com/support/knowledgecenter/ssw_aix_72/c_command...)

   Show morethread
4. KoshkinMay 2020
   > a single source of truth

   Naturally, these have been called "bibles": Unix Bible, Linux Bible, Linux Command Line and Shell Scripting Bible, etc.

   thread