HAProxy Blog

When reading along: Keep in mind that I'm not a core developer and thus are not directly involved in development, design decisions, or roadmap. I have some understanding of the internals and the associated challenges based on my contributions and discussions on the mailing list, but the following might not be entirely correct.

> I wonder if a disk-writing process could be spun out before dropping privileges?

I mean … it sure can and that appears the plan based on the last comment in that issue. However the “no disk access” policy is also useful for security. HAProxy can chroot itself to an empty directory to reduce the blast radius and that is done in the default configuration on at least Debian.

> but I don't see why disk accesses need to block requests

My understanding is that historically Linux disk IO was inherently blocking. A non-blocking interface (io_uring) only became available fairly recently: https://stackoverflow.com/a/57451551/782822. And even then it's a operating system specific interface. For the BSD's you need a different solution.

If your process is blocked for even one millisecond when handling two million of requests per second (https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/) then you drop 2k requests or increase latency.

> or why they have to occur in the same thread as requests?

“have” is a strong word, of course nothing “has” to be. One thing to keep in mind is that HAProxy…

> Hard disagree. Markdown is the way to go.

As other child comments have stated, this is entirely subjective, and many would disagree with you (myself included). I live primarily in the terminal, and the ability to seamlessly look up syntax, flags, features, etc. without ever leaving the keyboard is wonderful. Additionally, I agree with milesrout that I do not want emoji, animations, etc. in my manuals (nor my terminal). I want searchable text. I also agree with lupusreal that having a single large page, at least as an option, is ideal.

A perfect example of the two approaches is HAProxy. Here [0] is their commercial site. It's very flashy, modern web, etc. I don't necessarily hate it, but it's definitely designed to get the attention who may not necessarily know what they want, and can be sold to. In comparison, here [1] is their community site. It is extremely Web 1.0, and I love it. Everything you need, nothing you don't. Similarly, their docs [2] are [3] logically [4] split, and manage to be readable, navigable, yet information-dense. I don't need a tutorial with code snippets every few paragraphs in my docs.

> Additionally, developers are very much used to markdown, and most other docs are written in markdown (maybe some RST, but probably mostly markdown) so this is not great for the person writing the docs either.

Web devs, perhaps, though I imagine there's more variety than you think. Kernel devs likely do not have the same opinion as you (though I do not wish to…

A few notes on Power-Of-Two-Choices (Po2C) which is part of the solution described in the paper.

NGINX appears to have Po2C available as an option on the "random" balancing algorithm. [1] It only considers load (requests in flight), unlike the paper which also considers recent latency (and does many other things).

However NGINX "Plus" (the paid version) appears to have a couple additional settings that use actual traffic as latency probes (not asynchronous synthetic probes as described in the paper). [also 1] It's not fully clear to me but I think if you use these options it's going to only use latency, not load. The paper described in this post actually uses both load (RIF) and latency. Eager to try these out.

Also, I noted that HAProxy author and kernel maintainer Willy Tarreau wrote an article in 2019 about adding Power-of-two-choices to HAProxy. [2] He builds a test bed with a few micro services and compares several balancing algorithms to Po2C, considering response time, throughput, and max load.

One conclusion is that "At the very least, Power of Two is always better than Random. So, we decided to change the default number of draws for the Random algorithm from one to two, matching Power of Two." However, in the response time data, the test appears to consider only average response time. I would love to see the p90 and p99 tail latency of these tests. The main focus of the paper in this post is minimizing p90/99, not average latency.

Tarreau also helpfully…

Postman supports dynamic variables for tests [1]. Of course nothing requires you to use postman, you could just as easily script this with just ruby and curb/httparty.

The reason for this is that a lot of api endpoints support a large number of parameters, and we want to be sure that we test a decent sample of real world usage instead of only a few examples that we can think of to implement in controller/request tests.

We can take a month’s worth of request logs from production, deduplicate them, run the requests against prod and record responses, then run same requests against stage and verify responses match even after you rewrite half the backing queries. This helps a lot with peace of mind when we deploy :)

For another, more advanced approach to this idea, see traffic mirroring/shadowing [2].

[1]: https://learning.postman.com/docs/writing-scripts/test-scripts/

[2]: https://www.haproxy.com/blog/haproxy-traffic-mirroring-for-real-world-testing/

Thanks for the feedback. I've provided a few replies below but I think you have made some great points and I will share them internally.

> - the site does not do a good job of describing the products and what they do.

The page does lead off by telling you it is a software load balancer, but I agree, we can do a better job of giving more information on the product itself. Most people who pursue HAProxy Enterprise are already familiar with HAProxy. With that said, we're actually working on a complete overhaul to this page and I believe the new version will address this.

> - it is hard to find requirements from the first page.

This is just a reference point and typically organizations are working closely with a sales engineer.

> - is there a simpler HA setup than the 3 layer one here:

A simpler version is found under active/standby here [1]

[1] https://www.haproxy.com/documentation/hapee/2-2r1/high-availability/active-standby/

In our maps blog post we have a section where we talk about applying different rate limits for different paths (https://www.haproxy.com/blog/introduction-to-haproxy-maps/#rate-limiting-by-url-path). For this I'd change the path fetch to a req.hdr(authorization) (or however you send API keys) to limit per API key. You could also add a 20/second with another stick table using the methods in this DDoS post (just with a string key to store API key rather than ip/src to store the IP address).

You can put HAProxy directly on a box running another service to provide protection/reporting if you don't want to replace or add HAProxy to your ELB's (though sometimes that just adds too much complexity), we also talk about using HAProxy and ELB in our AWS blog post (https://www.haproxy.com/blog/haproxy-amazon-aws-best-practices-part-1/).