Nelson Elhage

> It is recommended to learn if subtransactions are used in your systems. If they are, it does not immediately mean that they need to be eliminated – it all depends on the risks of your particular case. Evaluating such risks is a complex task that requires deep Postgres expertise, so if you need such an analysis, consider hiring a PostgreSQL expert.

It is quite disheartening that this essentially echoes this quote from the related blog post https://buttondown.email/nelhage/archive/22ab771c-25b4-4cd9-b316-31a86f737acc:

> It is entirely possible to operate Postgres safely and with high performance and throughput at scale … but essentially the only way to do it is to have ready access to deep PostgreSQL experience and expertise on your team; people who Just Know where the landmines are because they’ve seen them before.

And https://blog.nelhage.com/post/some-opinionated-sql-takes/:

> My personal choice: MySQL

> The devil is in the details, but for me, as a default, and despite all these pet peeves, I would start with MySQL.

> As for Postgres, I have enormous respect for it and its engineering and capabilities, but, for me, it’s just too damn operationally scary. In my experience it’s much worse than MySQL for operational footguns and performance cliffs, where using it slightly wrong can utterly tank your performance or availability. In addition, because MySQL is, in my experience, more widely deployed, it’s easier to find and hire engineers with experience deploying and…

why does the quality of your code matter? it matters if you plan on changing it often. visiting code just to improve it's quality will likely cause bugs and cost time/money to no end. focus your energy on code that's changing most frequently or growing the fastest.

0) human angle

meet to determine what of these prevent sustainable operations or growth. if something becomes quadratically more painful in relation to team growth, deal with it quickly

determine what things are annoyances that would save real time (getting angry at the codebase costs time)

buy time from leadership; if leadership is not on-board with this, then it will always suffer

1) code review

reviewing code when a company is growing is a good way for original authors to impart knowledge or suggest architectural improvements (often ones they wish they made). it also creates accountability and a discussion forum for the items that follow.

2) testing

testing is like a ratchet and it can be used to build a punch-list of the most important things to fix over time.

coming up with a way to test that gets you a lot for a little is key at first. here's an example for an API: (1) snapshot your database, (2) make a bunch of api calls, (3) turn those api calls into test cases (there are creative ways to do this quickly), (4) run them all the time, (5) reduce the number of failing tests over time

find ways to turn your system into a function (use property based testing or fuzzing and you get a _lot_ for a…

> Unless there is clear evidences upfront that the project will be a piece of software where local performance is highly critical, it makes sense to favor code readability and maintainability over optimality.

Sure, but let's also not confuse "optimal" with "reasonable". One of the major challenge of modern programs is how slow they are at every level. Very often, this bad performance can be attributed to a style of programming that tries to completely ignore that the program will run on a real machine with real hardware. A little bit of mechanical sympathy (e.g., operations that make good use of the CPU caches or don't confuse the branch predictor) can yield a program that is 10x faster than a naive implementation, with little to no loss of readability or maintainability. (In fact, as noted by Nelson Elhage [1], faster programs enable simpler architectures, which helps make them more readable and maintainable.)

In FP languages, programmers face an extra difficulty: the distance between the code they write and the machine code that will be executed is greater than in languages like Rust or Go. They will need to be knowledgeable about the language, its libraries, and its compiler to avoid the pitfalls that could make their programs slower than would be reasonable (e.g., avoiding unnecessary thunking in Haskell).

[1] https://blog.nelhage.com/post/reflections-on-performance/#performant-foundations-simplify-architecture

Another recent Postgres-complaint post from one of the best engineers I've worked with: https://blog.nelhage.com/post/some-opinionated-sql-takes/

Quoting his conclusion:

> As for Postgres, I have enormous respect for it and its engineering and capabilities, but, for me, it’s just too damn operationally scary. In my experience it’s much worse than MySQL for operational footguns and performance cliffs, where using it slightly wrong can utterly tank your performance or availability. … Postgres is a fine choice, especially if you already have expertise using it on your team, but I’ve personally been burned too many times.

He wrote that shortly after chasing down a gnarly bug caused by an obscure Django/Postgres crossover: https://buttondown.email/nelhage/archive/22ab771c-25b4-4cd9-b316-31a86f737acc

Personally, I'd still opt for Postgres every time – the featureset is incredible, and while it may have scary footguns, it's better to have footguns than bugs – at least you can do something about them.

Still, I absolutely wish the official Postgres docs did a better job outlining How Things Can Go Wrong, both in general and on the docs page for each given feature.

This article helped me a bit when porting Shake to x86_64 (the first in the series is linked from the article):

http://blogs.msdn.com/b/oldnewthing/archive/2004/01/14/58579.aspx

The x86_64 calling convention can be a bit hairy. C.f. http://blog.nelhage.com/2010/10/amd64-and-va_arg/

The best thing about really large C++ applications with assembly glue deep in their core is that nothing works until you rewrite the glue for the new platform.

You can't debug the thing like a normal app when its hand-rolled threading and mutex library from 1997 doesn't work. So you tread forward; only suckers use debuggers. You can tell what it's supposed to do by reading the assembly and function names from the other platforms... (ppc, alpha (SGI), and i386 in this case).

Naturally, none of the assembly parts had any documentation. Honestly, it's probably good that they didn't (j/k). My guess as to what they were thinking: "if they don't understand it then they shouldn't be writing the asm for the next platform". I guess they were right.

Finally, you get the glue right and it's an "ah-ha" moment. Everything suddenly starts working.

I put a bit further thought into this. The claim is that because the pty has user permissions its possible to hijack it, but that really hasn't been true for years which is why the two PoC do things like use reptyr (involves replacing fd's with ptrace [1], needing either a parent relationship or capabilities that aren't default) or running a command like netcat directly from the pty you want to control [2] (essentially ttyjack [3]).

If you look at the implementation of TIOCSTI [4], 99% sure this is what the new PoC is doing, you can see why you can't do this across ttys. This goes back ages, even into some 2.6 kernels it seems before I got bored going back kernel versions.

I went and tested all this on a Ubuntu 22.04 box, not possible unless there is a new trick to hijack ttys. I tried the TIOCLINUX across ttys physically at the machine and that didn't work either.

[1] https://blog.nelhage.com/2014/08/new-reptyr-feature-tty-stealing/ [2] https://twitter.com/hackerfantastic/status/1786080968958161229 [3] https://github.com/jwilk/ttyjack [4] https://elixir.bootlin.com/linux/latest/source/drivers/tty/tty_io.c#L2281