Adam Drake

> syslog-ng supports the systemd journal natively so you'd never see a difference from before if you don't want to.

We use rsyslog with systemd and it works too, however it's just another level of abstraction so, I'm not fond of piling stuff over and over. Binary logs' and journals' usefulness can be also debated. It of course brings some advantages to the table but, it doesn't enable anything groundbreaking for me.

> The principle is dogma that has held back Unix imo. It's useful for simple CLIs but that is about it. By your definition the kernel itself goes against the principle.

Actually I don't consider it dogma, because it has some very important results and corollaries. This principle reduces secondary complexity (glue logic, complexity required to do many things with one code base, etc.) greatly hence the software can be relentlessly optimized. grep, find, sed, awk, cut, head, tail, less all are extremely performant tools. I develop high performance scientific applications and making a tool accomplish more than a couple different functions either reduce performance (generalized algorithms) or increase size a lot (specialized functions for everything). Its refactoring, verification, maintenance gets complicated and expensive fast.

UNIX pipes and small optimized tools are much more useful beyond simple CLI commands and small scripts. Since the tools are fast, low memory footprint performance behemoths, you can build very fast and reliable machinery with them. You…

Ha, I was explaining this just yesterday to a few people at $WORK (I'm not at TB, though we do use a bunch of Zig in prod).

"Multiprocessing" (multiple CPU cores independently executing and only able to coordinate via some message-passing system -- a definition which encompasses both multi-core CPUs and horizontally scaled distributed systems, contrasting slightly with the normal definition) is challenging for a few reasons.

Firstly, the details are an open math question, but I'll blindly state that some problems aren't amenable to parallelization. I.e., no algorithm can meaningfully improve performance via parallelization no matter the implementation. Think through how you would more quickly compute hash(hash(hash(hash(...)))) for example. The serial dependency makes things challenging. That isn't too dissimilar to the problem TB faces.

Secondly, message passing is expensive. If the only way two CPU cores can coordinate is through a multi-level cache, at best you're incurring ~tens of nanoseconds of latency per message. Contrast that with a base rate of 512 bytes processed per nanosecond with enough attention to detail on typical modern server hardware (4 pipelined AVX512 instructions at 2GHz). Messages are several orders of magnitude worse than your normal work, so if you need very many of them then you're hosed from a performance perspective (worse with longer delays, like networked computers). Even very parallelizable problems at an abstract level can suffer…

I missed the edit window on this but: I don't think you're going to get computers to not be messy and full of huge time sinks. That's just the nature of humanity being reflected in a silicon mirror.

I like the idea of focusing first and foremost on debuggability, on manipulating program execution histories as first-class objects. There are a huge number of interesting tradeoffs there with performance.

What I mean by "power" is that Unix makes it easy enough to do things like "interactive text chat in three lines of code" (untested, and quality at best "usable prototype" rather than "production ready")

#!/bin/bash
ch="$
{2?Usage: $0 /tmp/channelfile}"
touch "$ch"; chmod 666 "$ch"; tail -f "$ch" & # chat of the beast
while read -e line; do echo "<$USER> $line"; done >> "$ch"

or "which other English words might be Greek neuters, plural and all, like automaton/automata"

grep "on$" /usr/share/dict/words | sed 's/..$/a/' | sort |
join - <(sort /usr/share/dict/words)

or "which directory is growing in my home directory and filling up the fucking disk"

du -k ~ > tmp.du.1
du -k ~ > tmp.du.2
diff -u tmp.du.[12]

or "build the thumbnails for any images I haven't thumbnailed yet", at least as long as none of them have spaces in the names, using GNU make (untested, but adapted from real code):

thumbnails: $(patsubst %.jpg,thumbs/%.jpg,$(wildcard *.jpg))
thumbs/%.jpg: %.jpg
convert -geometry 80x160 $< $@

or…

>I mean if you're doing data science the data is not always organized and of course you would want multi-processing

Not necessarily - I might not want it or need it. It's a few TB, it can be on a fast HD, on an even faster SSD, or even in memory. I can crunch them quite fast even with basic linear scripts/tools.

And organized could just mean some massaging or just having them in csv format.

This is already the same rushed notions about "needing this" and "must have that" that the OP describes people jumping to, that leads them to suggest huge setups, distributed processing, multi-machine infrastructure, for use cases and data sizes that could fit on a single server with redundancy and be done it.

DHH has often written about this for their Basecamp needs (scalling vertically where others scale horizontally having worked for them for most of their operation), there's also this classic post: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

>1 TB of memory is like 5 grand from a quick Google search then you probably need specialized motherboards.

Not that specialized, I've work with server deployments (HP) with 1, 1.5 and 2TB RAM (and > 100 cores), it's trivial to get.

And 5 or even 30 grand would still be cheaper (and more effective and simpler) than the "big data" setups some of those candidates have in mind.

All the new shiny. From experience I can tell you

a) most companies deal with small amounts of data. Small can mean dozens of megabytes to dozens or hundreds of gigabytes. A single well provisioned server will typically be able to handle that very well. Also an SQL database can do a great deal if you know what you're doing.

b) inappropriately used big data frameworks are expensive performance killers. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html for example.

c) Good quality programming, as in understanding the machine, memory layout and why it matters, and a good understanding of algorithms (and a hefty dose of common sense), will often yield you more speedup than buying almost any number of new machines.

c) Hiring is often driven by fads and companies often don't like being told 'you don't need this roomful of servers', they like to waste money, so maybe do learn them (the profligacy with money is likely to be coming to an end with the economic damage of covid).

Takeaway: brainpower will get you much further than horsepower

I wouldn't underestimate how much a modern machine with a bunch of RAM and SSDs can do vs HDFS. This post[1] is now 10 years old and has find + awk running an analysis in 12 seconds (at speed roughly equal to his hard drive) vs Hadoop taking 26 minutes. I've had similar experiences with much bigger datasets at work (think years of per-second manufacturing data across 10ks of sensors).

I get that that post is only on 3.5GB, but, consumer SSDs are now much faster at 7.5GB/s vs 270MB/s HDD back when the article was written. Even with only mildly optimised solutions, people are churning through the 1 billion rows (±12GB) challenge in seconds as well. And, if you have the data in memory (not impossible) your bottlenecks won't even be reading speed.

[1]: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html