Jonathan Blow

Developers end up at local maxima because a whole bunch of practices add up to a working system. It's a mistake to champion any single practice without understanding how it fits into an ecosystem: time budget, testing, language features available, efficiency, tolerance for errors, imposed deadlines, third-party technical ecosystem, etc. It's all about the system where they all work together.

To truly suggest something as a best practice, you need to understand what it requires from the technical and social environment around it, and when it applies and doesn't apply. We're nowhere near that level of discipline as a field. This lack of discipline is why we wince when we hear that a startup is implementing something like voting machines: we know the financial incentives that lead to startups focusing on speed, and we can imagine the discipline that voting systems require. We know there's a mismatch between the social environment and the likely project management approach. But we're blind to the social and technical requirements that allow other practices to work well. When is it appropriate to use inheritance? Someone will inevitably say, "You shouldn't use it?" Why does it work so well in Rails?

IMO it's how developers like John Carmack[0] and Martin Fowler[1] can make opposite arguments about things like the benefits of having code inline or in functions. These two opinions work in conjunction with their environments; maybe video game and aerospace coding requires budgeted…

John Carmack has written about it and agrees with you with the caveat that functional programming alleviates or even eliminates this kind of issue. AKA if your extracted procedures are just functions, then it can improve your code. And since this is Elixir we are looking at presumably mostly functional code.

http://number-none.com/blow/blog/programming/2014/09/26/carmack-on-inlined-code.html

I want to add a third condition that makes this decision easier IMO, because I think it is insufficient to say "if it is a pure function you can extract it". It can often be the case, but it can also lead to what you describe regardless.

It is insufficient if the extracted function is just a one-off helper. The remaining downstream code should read as an _abstracted_ form. Meaning the naming of the functions is clear, uniform and in the best cases they represent a domain layer: A mini-language that communicates clearly what is going on, but draws away from the details. And most importantly it should follow John Ousterhout's proposed rule about interface surface to implementation depth ratio, which should be kept as small as possible.

To the example: I don't think the amount of cases handled should have any implication on extracting functions here (as mentioned in the example). It would still be clearer to just return the data directly. Nothing is gained until the body of the functions themselves have enough _depth_ to provide a useful abstraction, the function names would naturally…

> But as we get closer to 100 lines I doubt that it's more understandable and readable to have it all in one big blob of code.

Well, but that's exactly what I'm pushing back against. I think the rule of 30 is often a mistake. I think if you're going out of your way to avoid long functions, then you are probably over-abstracting your code.

I don't necessarily know that I would inline a quicksort function, because that's genuinely something that I might want to use in multiple places. It's an already-existing, well-understood abstraction. But I would inline a dedicated custom sorting method that's only being used in one place. I would inline something like collision detection, nobody else should be calling that outside of a single update loop. In general, it's a code smell to me if I see a lot of helper functions that only exist to be called once. Those are prime candidates for inlining.

This is kind of a subtle argument. I would recommend http://number-none.com/blow/john_carmack_on_inlined_code.html as a starting point for why inlined code makes sense in some situations, although I no longer agree with literally everything in this article, and I think the underlying idea I'm getting at is a bit more general and foundational.

> Do you think that inlining the partition function would make this easier or harder to read?

Undoubtedly easier, although you should label that section with a comment and use a different variable name than `i`. Your secondary function is just a…

This is an old debate and yet one that is difficult to think through. I share your experience of having trouble convincing someone else not to combine things too hastily. Partly it's a "my gut feel is different to yours" situation, but I often don't have the confidence that I could articulate my reasoning without indulging in a wordy lecture on minutiae.

Lately I have been fixating on the following line of thinking: the unit of deduplication--usually a function, but sometimes even bigger--is the same thing as the unit of abstraction. When you dedupe, you've also given birth to a new abstraction, and those don't come for free. Now there's a new thing in the world that you had to give a name to, and that somebody else might come along and re-use as well, perhaps not in a context where you originally intended. The new thing is now bearing the load of different concerns, and without anyone intending it, it now connects those concerns. The cost of deduplication isn't just the work of the initial refactor; it's the risk that those future connections will break something or make your system harder to understand.

This reminds me of another famous Carmack pronouncement about the value of longer functions [1], which I think has some parallels here. In the same way we're taught to DRY up our code, we're taught to break up long functions. I sort of think of these two things as the same problem, because I view their costs as essentially the same: they risk proliferating new and…

So this article concludes that starting with a certain project size, over-engineered code is easier to work with than under-engineered code, and scales well when the projects grows (even "linear with the size").

But what this article is actually about is about entanglement of individual parts. Under-engineered implies spaghetti code with lots of entanglement. Over-engineered code does have the minimum needed amount of entanglement.

Of course this is right on the topic of entanglement.

But this is not so much my understanding of the terms over-engineered vs under-engineered.

I understand over-engineered as having too much abstractions. And those can turn out to be the wrong abstractions when they are actually used, which then requires more work to refactor this.

I understand under-engineered as having too little abstractions, and having build in some assumptions which might hold for one use case but not anymore for others. This can also require some effort to fix this.

In practice, this is also a continuum, and you might even have both things mixed.

And when actually writing some new code, it is often hard to know the right amount of complexity.

This is why the overall quality greatly improves when you do a couple of iterations of rewrites from scratch. Because you keep improving on just the right needed abstractions.

Related:

http://number-none.com/blow/blog/programming/2014/09/26/carmack-on-inlined-code.html

https://github.com/Droogans/unmaintainable-code

I think mostly because having scads of clever conditionals all over the place greatly increases the risk of subtle and hard-to-reproduce bugs. From a 2007 Carmack email [1] :

> The way we have traditionally measured performance and optimized our games encouraged a lot of conditional operations -- recognizing that a particular operation doesn't need to be done in some subset of the operating states, and skipping it. This gives better demo timing numbers, but a huge amount of bugs are generated because skipping the expensive operation also usually skips some other state updating that turns out to be needed elsewhere.

> We definitely still have tasks that are performance intensive enough to need optimization, but the style gets applied as a matter of course in many cases where a performance benefit is negligible, but we still eat the bugs. Now that we are firmly decided on a 60hz game, worst case performance is more important than average case performance, so highly variable performance should be looked down on even more.

There was fascinating breakdown [2] of a single GTA V frame about five years back. It was a real eye-opener for me - whole full-screen rendering passes for things like water reflection and refraction when there isn't even any water visible in the scene.

[1] http://number-none.com/blow/john_carmack_on_inlined_code.html

[2] http://www.adriancourreges.com/blog/2015/11/02/gta-v-graphics-study/