Brave New Geek

This is probably the most detailed analysis of retry techniques that I've seen. I really appreciated the circuit breaker and deadline propagation sections.

But this is why I've pretty much abandoned all connection-oriented logic in favor of declarative programming:

https://en.wikipedia.org/wiki/Declarative_programming

Loosely, that means that instead of thinking of communication as client-server or peer-to-peer remote procedure calls (RPC), I think of it as state transfer. Specifically, I've moved aware from REST towards things like Firebase that encapsulate retry logic. Under this model, failure is never indicated, apps just hang until communication is reestablished.

I actually think that apps can never really achieve 100% reliability, because there's no way to ever guarantee communication:

https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

https://en.wikipedia.org/wiki/Byzantine_fault

https://en.wikipedia.org/wiki/Two_Generals%27_Problem

Although deadline propagation neatly models the human experience of feeling connected to the internet or not.

Also this is why I think that microservices without declarative programming are an evolutionary dead end. So I recommend against starting any new work with them in this era of imperative programming where so many developer hours are lost to managing mutable state. A better way is to use append-only databases like CouchDB which work similarly to Redux to provide the last-known state as the reduction of all…

> Do you have a citation for that?

I do indeed have a citation for that by Rob Pike - and no matter how downvoted I am, it stays a fact:

> The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

(https://bravenewgeek.com/go-is-unapologetically-flawed-heres-why-we-use-it/)

and also:

> Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical.

(https://go.dev/talks/2012/splash.article)

> As far as I remember that’s as most a second order effect -- they wanted a language that felt as clean and straightforward as C, to benefit everybody including experienced developers, not primarily inexperienced ones.

No, it's not a second order effect. Or rather: both what you say (a language "as clean and straightforward as C") as well as a language for devs early in their career (i.e. inexperienced) are both second order effects from the underlying reason that the language is targeted at developers at google.

I'm totally in agreement with this. Both processes should also be idempotent (you should be able to pull multiple times without side effects, and push and pull should be able to happen at the same time without side effects).
When everything is working well, the 'push' does all the work, and though the 'pull' runs every few minutes/seconds/whatever it never has anything to do.

This same thing applies to time-based events: your system should not assume that the process is always running, so if something needs to happen at exactly 9:00am (and it's not okay to just skip it if missed), it should be able to run anytime later with the same outcome as if it ran at 9:00am.

When I was first getting into this, it helped me to understand that events must follow one of the semantics: at-most-once, at-least-once and exactly-once -- with the trick that exactly-once is not strictly possible [1].

There are only two hard problems in distributed systems:
2. Exactly-once delivery
1. Guaranteed order of messages
2. Exactly-once delivery

-Mathias Verraes [2]

[1] http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

[2] https://twitter.com/mathiasverraes/status/632260618599403520

(project author)

I appreciate all the work that Tyler has put into his series of blog posts on messaging systems (and NSQ bug reports!) but I think this particular article is one that misses the mark [1].

While raw performance is important, I think comparing the guarantees and related operational and development semantics are far more interesting and useful.

These systems vary a great deal in the interfaces and functionality they expose to users building on top of them and operators dealing with them when they're deployed in production (and inevitably break!). This is ultimately what matters most.

P.S. single node (localhost) performance is essentially useless. Most of these tools are designed to be deployed as the backbone of large distributed systems (NSQ in particular), so scalability and performance in that context is a better indication of real-world performance.

[1] He's since posted this follow-up that outlines many of the same things http://www.bravenewgeek.com/benchmark-responsibly/

You almost never want to use average as your metric when dealing with time. For time, the 95 or 99 percentile latency is going to be measure of how many requests out of a hundred a less than your metric. Esp when the thing you are measuring is a piece of larger system, which is the crux, because of how a resource in turn requests k-more resources. Each request has an equal chance of being over the latency percentile. Even at 10 requests, only 90% of sessions will be under the 99% latency.

See also, Gil Tene https://www.youtube.com/watch?v=lJ8ydIuPFeU

A nice overview, http://bravenewgeek.com/everything-you-know-about-latency-is-wrong/

See also, http://highscalability.com/blog/2015/10/5/your-load-generator-is-probably-lying-to-you-take-the-red-pi.html

> Whenever the plumbing does not guarantees EOIO but the business demands it, it gets pushed into the app layer where TCP (retries and acks) is reimplemented, to various success levels.

Interesting! You're saying the exact opposite of what Tyler Treat is saying:

> Even with smart middleware, problems still leak out and you have to handle them at the edge—you’re now being taxed twice. This is essentially the end-to-end argument. Push responsibility to the edges, smart endpoints, dumb pipes, etc. It’s the idea that if you need business-level guarantees, build them into the business layer because the infrastructure doesn’t care about them.

(source: http://bravenewgeek.com/smart-endpoints-dumb-pipes/)

What do you think of his argument?