Curt Clifton, a software developer, hosts papers and posts on his site, including the "Out of the Tar Pit" PDF.
HN links it for the "Out of the Tar Pit" paper, which commenters call the best they've read on software complexity.
> Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)
That's a huge part of the problem. Was looking for a monograph to pass on my successor that wasn't so familiar with linux - modern linux - as in systemd, docker, nssswitch, pam_homed, network-manager is penetrable but more often than not I'm just looking at the c source in github and I am in my 30ies, starting in my teens running linux nonstop and I'm still throwing up my hands every other day...
Add time constraints and pressure to deliver (don't waste your time understanding this, just do xyz) and here we are - on the other hand it's not okay that you have to devote years of trail and error to get a comprehensive undestanding of the system.
I guess a nuclear power plant has at least a training plan and complete reference book - that still needs to be read and grokked and trained upon but modern linux is often just undestandable by reading the source if you hit a problem - for some projects like things in the freedesktop ecosystem and partly systemd even that is kind of difficult because i.e. this bug https://github.com/systemd/systemd/issues/19118 explains the problem - there is no overview documentation, no clear way to look up what's happening, not even a good way to introspect and it's several components that interact with each other fail subtly. I've chosen this one because I've also hit it, not because I want to blame systemd, which is not so bad there are much worse…
I always find myself sitting down to read Out of the Tar Pit[0] at least a couple times per year. It has been—and continues to be—one of the most seminal texts in my career. I still remember the first time I read the following passage on complexity, and how it just turned on all the mental light bulbs:
>> Essential Complexity is inherent in, and the essence of, the problem (as seen by the users).
>> Accidental Complexity is all the rest — complexity with which the development team would not have to deal in the ideal world (e.g. complexity arising from performance issues and from suboptimal language and infrastructure).
>> Note that the definition of essential is deliberately more strict than common usage. Specifically when we use the term essential we will mean strictly essential to the users’ problem (as opposed to — perhaps — essential to some specific, implemented, system, or even — essential to software in general).
The best skill I've learned, and continued to practice and improve, is the ability to strip how we talk about problems we want to solve with software down to what's truly essential to the problem. Making a habit of doing so helps clarify the contours of the problem itself, and improves discussions around solving because the boundaries of what's truly essential become clear—and then everyone involved knows that every choice we make from that point is additional, accidental complexity we are adding to the problem ourselves.
Far too often I have seen even…
You're being condescending, telling me to "learn the theory". We've clearly had different experiences and work with different kinds of systems. You're automatically discounting what I say by assuming I must be a newbie because I disagree with you. No. We both have things to learn from each other and you telling me "go RTFM, newbie" is shutting down that opportunity.
> Virtually all of the systems I write have at least two back-ends to maintain that flexibility.
It sounds like that's a worthwhile tradeoff for you, but it is a tradeoff, giving up some of the unique power of each individual database in order to support both. Realize that most people don't do this.
> A good object model is NOT a relational model, and a good relational model is NOT an object model.
Maybe this is our biggest disagreement. I believe there is a model that can get the biggest benefits of both. I'm not alone in this; papers in Software Engineering, "Out of the Tar Pit", prescribes this as a possible solution to a lot of woes in software engineering:
"The classical ways to approach the difficulty of state include object oriented programming which tightly couples state together with related behaviour, and functional programming which — in its pure form — eschews state and side-effects all together ... We argue that it is possible to take useful ideas from both and that — when combined with some ideas from the relational database world - this approach offers significant potential for simplifying…
I find similar arguments around SQL.
So much time & frustration expended simply to avoid typing out the magic database commands... And the constant ego trips attempting to outperform 30+ year old query planner codebases on 7-way+ joins by using baby's first ORM.
> the tar pit
If you find yourself stuck in one of these, I strongly recommend giving this a shot: https://curtclifton.net/papers/MoseleyMarks06a.pdf
"but it won't scale"
We are in the era of hyperscale SQL engines. Database engines that are spread out across multiple servers, racks and buildings. Engines so vast & complex the compute & storage responsibilities have to be separated into different stacks. But, they (the good ones) still work just like the old school approach from an application perspective. The workload necessary to actually saturate one of these databases would be incredible. I some days wonder if Twitter could be rewritten on top of one without much suffering.
And, if you aren't trying to go big and bold or spend a bunch of money, there's always SQLite. It also supports basically all the same damn things. It can run entirely in memory. It has FTS indexing. Your CTE-enabled queries will work just fine on it. If you find SQLite doesn't scale with you, swapping to a different engine really isn't that big of a deal either. You will have some dialect conflicts but it's generally very workable, especially if you use some thin layer like Dapper between your code and the actual connection instances.
ClojureScript is awesome.
There are several tools available for live-coding (REPL-driven development) such as more recent shadowcljs or the older Figwheel.
Rum is a powerful wrapper over React.js and you can live code and see instantaneous results in the browser (using shadowcljs or figwheel).
ClojureScript's strengths largely stem from Clojure -- collections as fundamental types, REPL, macros, immutable datastructures* and lovely syntax.
As far as cons go, it takes some time to start up a REPL and making release builds is an extra step, but should work just fine unless you're a newbie like me and you leave strange unnecessary lines in your code. Javascript and js/CSS interop is there, but it's not super obvious and takes some getting used to what it looks like -- luckily there are enough online writeups to make that process easier.
The addage "why clojurescript? Because clojure rocks and javascript reaches" sums up nicely the motivation.
You can have largely isomorphic code on the front-end and back-end, which lends nicely to creating server-side rendered versions of SPAs for search engine indexing and fast load times. However, the server-side copy and the client rendered copy must match exactly for this to work, which can be a bit of a "thread the needle" kind of problem.
* immutable datastructures are awesome, contrast with mutable datastructures and "location oriented programming" -- any time you "edit" a value you're actually creating a new datum. This is…
The main problem with ORM is the "object" part. I feel object-oriented programming had its chance and it failed.
Both event sourcing and relational models (="table modelling") are better models for programming systems.
So by using an ORM, you are (at least in the name, and as traditionally done) taking a better programming model (relational models) and mapping it to an inferior one (OOP) -- also getting a ton of leaky abstractions involved too.
I wish there were more "ORMs" that were made with the assumption that Object-Oriented Programming is BAD, and started out with assumptions of wanting to program either in an event sourcing model and/or a relational model, and see where that would take things.
Related:
Fred Brooks, The Mythical Man Month (1975): "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious."
Out of the tar pit paper: https://curtclifton.net/papers/MoseleyMarks06a.pdf
--
My take on the software industry today is that OOP failed to produce good and maintainable code, but instead of fixing that, one is adding the idea of micro-services to try to contain the damage done by OOP.
If more people moved to event sourcing, functional code and focusing on data over code, there would be much less need for micro-services etc (or, they just become less relevant).